Reinforcement Learning

Transcription

1 Reinforcement Learning March May, 2013

2 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha ed.24 RL in finite MDPs 03/25/2015 (10:15-12:15) PT1 RL in finite MDPs 03/27/2015 (10:15-12:15) PT1 RL in continuous MDPs 04/01/2015 (10:15-12:15) Sala conferenze RL in continuous MDPs 04/08/2015 (10:15-12:15) Sala conferenze RL in POMDPs 04/10/2015 (10:15-12:15) Sala conferenze Multi agent RL 04/15/2015 ( 10:15-12:15) Sala conferenze Applications 04/17/2015 ( 10:15-12:15) Sala conferenze

3 Modelling the Environment Deterministic vs Stochastic Finite vs Continuous States Finite vs Continuous Actions Discrete vs Continuous Time Fully vs Partially Observable Stationary vs Non Stationary

4 Introduction to MDPs formally describe an environment for reinforcement learning The environment is fully observable the current state completely characterizes the process Almost all RL problems can be formalized as MDPs Optimal control primarily deals with continuous MDPs Partially observable problems can be converted into MDPs Bandits are MDPs with one state

5 Stochastic A stochastic process is an indexed collection of random variables {X t} e.g., time series of weekly demands for a product Discrete case: At a particular time t, labeled by integers, system is found in exactly one of a finite number of mutually exclusive and exhaustive categories or states, labeled by integers, too Process could be embedded in that time points correspond to occurrence of specific events (or time may be evenly spaced) Random variables may depend on others, e.g., { max{(3 D t+1 ), 0} if X t 0 X t+1 = max{(x t D t+1 ), 0} if X t 0 or K X t+1 = α k X t k + ξ t with ξ t N (µ, σ 2 ) k=0

6 Examples of Stochastic Exchange rates Photon emission Epidemic models Earthquakes Budding yeast

7 Assumption The future is independent of the past given the present Definition A stochastic process X t is said to be ian if and only if P(X t+1 = j X t = i, X t 1 = k t 1,..., X 1 = k 1, X 0 = k 0 ) = P(X t+1 = j X t = i) The state captures all the information from history Once the state is known, the history may be thrown away The state is a sufficient statistic for the future The conditional probabilities are transition probabilities If the probabilities are stationary (time invariant), we can write: p ij = P(X t+1 = j X t = i) = P(X 1 = j X 0 = i)

8 A process (or chain) is a memoryless stochastic process, i.e., a sequence of random states s 1, s 2,... with the property Definition A Process is a tuple S, P, µ S is a (finite) set of states P is a state transition probability matrix, P ss = P(s s) a set of initial probabilities µ 0 i = P(X 0 = i) for all i Looking forward in time, n step transition probabilities p (n) i,j = P(X t+n = j X t = i) = P(X n = j X 0 = i)

9 Process Example 1 Student process

10 Process Example 1 Student process Sample paths C1 C2 C3 Pass Sleep C1 FB FB C1 C2 Sleep C1 C2 C3 Pub C2 C3 Pass Sleep C1 FB FB C1 C2 C3 Pub C1 FB FB FB C1 C2 Sleep

11 Process Example 1 Student process FB C1 C2 C3 Pub Pass Sleep P =

12 Process Example 2 Land of Oz Rain Nice Snow P = Rain Nice Snow

13 Chapman Kolmogorov n step transition probabilities can be obtained from 1 step transition probabilities recursively p (n) ij = M k=0 p (v) ik p(n v) kj, i, j, n; 0 v n We can get this via the matrix too P (n) = P } {{ P } = P n = PP n 1 = P n 1 P n times Given the transition matrix P and the starting distribution represented by the probability vector µ, the probability distribution over the state space after n steps is: µ (n) = µp n

14 Chapman Kolmogorov for Student Process FB C C P = C Pub Pass Sleep FB C C P 2 = C Pub Pass Sleep FB C C P 10 = C Pub Pass Sleep FB C C P 100 = C Pub Pass Sleep

15 Chapman Kolmogorov for Land of Oz P = P 2 = P 3 = P 4 = P 5 = P 6 =

16 First Passage Times Number of transitions to go from i to j for the first time if i = j, this is the recurrence time First Passage Times are random variables n step recursive relationship for first passage probability f (1) ij = p (1) ij = p ij f (2) ij = p (2) ij f (1) ij. f (n) ij = p (n) ij f (1) ij p jj p (n 1) jj f (2) ij p (n 2) jj f (n 1) ij p jj For fixed i and j, these f (n) ij so that n=1 f (n) ij 1 are non negative numbers If n=1 f (n) ii = 1 that state is a recurrent state

17 Classification of States State j is accessible from i if p (n) ij > 0 (for some n 0) If state j is accessible from i and vice versa, the two states are said to communicate As a result of communication, one may partition the general process into states in disjoint classes Positive recurrent: a state that is recurrent and has a finite expected return time If the process can only visit the state at integer multiples of t, we call it periodic Positive recurrent states that are aperiodic are called ergodic states A state j is said to be an absorbing state if p jj = 1 A state which is not absorbing is called transient

18 Classification of Absorbing A process is called absorbing if it has at least one absorbing state and if that state can be reached from every other state (not necessarily in one step) Ergodic A process is called ergodic (or irreducible) if it is possible to go from every state to every state (not necessarily in one move) Regular A process is called regular if some power of the transition matrix has only positive elements

19 Examples Drunkard s Walk P =

20 Examples Switching Process P = 1 [ ]

21 Examples Land of Oz Rain Nice Snow P =

22 Stationary Distribution Theorem Consider process with transition probability matrix P. A probability vector p is called a stationary distribution of the process if: pp = p Let P be the transition matrix for a regular chain. Then, as n, the powers P n approach a limiting matrix W with all rows the same vector w. The vector w is a strictly positive probability vector (i.e., the components are all positive and they sum to one).

23 Stationary Distribution Theorem Let P be a regular transition matrix, w be the common row of W, and let c be the column vector all of whose components are 1. Then wp = w, and any row vector v such that vp = v is a constant multiple of w Pc = c, and any column vector x such that Px = x is a multiple of c. Theorem Let P be the transition matrix for a regular chain and v an arbitrary probability vector. Then lim n vpn = w, where w is the stationary distribution of the chain.

24 Fundamental Matrix Absorbing Canonical form: renumber the states so that transient states come first: [ ] Q R P = 0 I Q n n 0 Theorem For an absorbing process the matrix I Q has an inverse N (called fundamental matrix) and N = i=0 Qi. The ij entry n ij of the matrix N is the expected number of times the chain is in state s j, given that it starts in state s i. The initial state is counted if i = j.

25 Fundamental Matrix Drunkard s Walk Example 1 P = [ Q R 0 I ] = 0.5 N = (I Q) 1 = Canonical form

26 Fundamental Matrix Ergodic Proposition Let P be the transition matrix of an ergodic chain, and let W be the matrix all of whose rows are the fixed probability row vector for P. Then the matrix I P + W has an inverse Z that is called fundamental matrix. Theorem The mean first passage matrix M for an ergodic chain is determined from the fundamental matrix Z and the fixed row probability vector w by m ij = z jj z ij w j.

27 Fundamental Matrix Land of Oz Example I P + W = Z = w = [ ] m 12 = z 22 z 12 w 2 = = 4

28 Mixing Rate Definition The total variation distance between two probability distributions µ and ν on Ω is Definition Given ɛ, the mixing time is τ(ɛ) = min t µ ν TV = 1 µ(x) ν(x) 2 x Ω { max x Ω Pt (x, ) w TV < ɛ, } t t Definition A chain is rapidly mixing if τ(ɛ) is O(poly(log( S ɛ )))

29 Spectral gap Definition A process is time reversible if w i P ij = w j P ji. The eigenvalues of P are 1 = λ 1 > λ 2 λ 3 λ S 1 Definition The spectral gap β of a process defined by transition matrix P is 1 max( λ 2, λ S ) Theorem (Alon, Sinclair) ( ) 1 β 1 β log τ(ɛ) 1β ( ) 2ɛ log 1 ɛ min i w i

30 A reward process is a process with values. Definition A Process is a tuple S, P, R, γ, µ S is a (finite) set of states P is a state transition probability matrix, P ss = P(s s) R is a reward function, R s = E[r s] γ is a discount factor, γ [0, 1] a set of initial probabilities µ 0 i = P(X 0 = i) for all i

31 Example Student MRP

32 Return Time horizon finite: finite and fixed number of steps indefinite: until some stopping criteria is met (absorbing states) infinite: forever Cumulative reward total reward: V = average reward: i=1 r r n V = lim n n discounted reward: V = γ i 1 r i mean variance reward i=1 r i

33 Infinite horizon Discounted Return Definition The return v t is the total discounted reward from time step t. v t = r t+1 + γr t+2 + = γ k r t+k+1 k=0 The discount γ [0, 1) is the present value of future rewards The value of receiving reward r after k + 1 time steps is γ k r Immediate reward vs delayed reward γ close to 0 leads to myopic evaluation γ close to 1 leads to far sighted evaluation γ can be also interpreted as the probability that the process will go on

34 Why discount? Most reward (and decision) processes are discounted, why? Mathematically convenient to discount rewards Avoids infinite returns in cyclic processes Uncertainty about the future may not be fully represented If the reward is financial, immediate rewards may earn more interest than delayed rewards Animal/human behavior shows preference for immediate reward It is sometimes possible to use undiscounted reward processes (i.e. γ = 1), e.g. if all sequences terminate

35 Value Function The value function V (s) gives the long term value of state s Definition The state value function V (s) of an MRP is the expected return starting from state s V (s) = E[v t s t = s]

36 Example

37 Example

38 Example

39 Bellman Equation for MRPs The value function can be decomposed into two parts: immediate reward r discounted value of successor state γv (s ) V (s) = E[v t s t = s] = E[r t+1 + γr t+2 + γ 2 r t s t = s] = E[r t+1 + γ(r t+2 + γr t ) s t = s] = E[r t+1 + γv t+1 s t = s] = E[r t+1 + γv (s t+1 ) s t = s] Bellman Equation V (s) = E[r + γv (s ) s] = R s + γ s S P ss V (s )

40 Bellman Equation in Matrix Form The Bellman equation can be expressed concisely using matrices V = R + γpv where V and R are column vectors with one entry per state and P is the state transition matrix. V (1) R(1) P 11 P 1n V (1). =. + γ V (n) R(n) P n1 P nn V (n)

41 Solving the Bellman Equation The Bellman equation is a linear equation It can be solved directly V = R + γpv (I γp)v = R V = (I γp) 1 R Computational complexity is O(n 3 ) for n states Direct solution only possible for small MRPs There are many iterative methods for large MRPs, e.g., Dynamic programming Monte Carlo evaluation Temporal Difference learning

42 Discrete time Finite A decision process (MDP) is reward process with decisions. It models an environment in which all states are and time is divided into stages. Definition A Process is a tuple S, A, P, R, γ, µ S is a (finite) set of states A is a (finite) set of actions P is a state transition probability matrix, P(s s, a) R is a reward function, R(s, a) = E[r s, a] γ is a discount factor, γ [0, 1] a set of initial probabilities µ 0 i = P(X 0 = i) for all i

43 Example: Student MDP

44 Goals and s Is a scalar reward an adequate notion of a goal? Sutton hypothesis: That all of what we mean by goals and purposes can be well thought of as the maximization of the cumulative sum of a received scalar signal (reward) Probably ultimately wrong, but so simple and flexible we have to disprove it before considering anything more complicated A goal should specify what we want to achieve, not how we want to achieve it The same goal can be specified by (infinite) different reward functions A goal must be outside the agent s direct control thus outside the agent The agent must be able to measure success: explicitly frequently during her lifespan

45 Policies A policy, at any given point in time, decides which action the agent selects A policy fully defines the behavior of an agent Policies can be: ian History dependent Deterministic Stochastic Stationary Non stationary

46 Stationary Stochastic ian Policies Definition A policy π is a distribution over actions given the state: π(a s) = P[a s] MDP policies depend on the current state (not the history) i.e., Policies are stationary (time independent) Given an MDP M = S, A, P, R, γ, µ and a policy π The state sequence s 1, s 2,... is a process S, P π, µ The state and reward sequence s 1, r 2, s 2,... is a reward process S, P π, R π, γ, µ, where P π = π(a s)p(s s, a) R π = π(a s)r(s, a) a A a A

47 Value Functions Given a policy π, it is possible to define the utility of each state: Policy Evaluation Definition The state value function V π (s) of an MDP is the expected return starting from state s, and then following policy π V π (s) = E π[v t s t = s] For control purposes, rather than the value of each state, it is easier to consider the value of each action in each state Definition The action value function Q π (s, a) is the expected return starting from state s, taking action a, and then following policy π Q π (s, a) = E π[v t s t = s, a t = a]

48 Example: Value Function of Student MDP

49 Bellman Expectation Equation The state value function can again be decomposed into immediate reward plus discounted value of successor state, V π (s) = E π [r t+1 + γv π (s t+1 ) s t = s] ( = a A π(a s) R(s, a) + γ s S P(s s, a)v π (s ) The action-value function can similarly be decomposed ) Q π (s, a) = E π [r t+1 + γq π (s t+1, a t+1 ) s t = s, a t = a] = R(s, a) + γ P(s s, a)v π (s ) s S = R(s, a) + γ P(s s, a) π(a s )Q π (s, a ) s S a A

50 Bellman Expectation Equation (Matrix Form) The Bellman expectation equation can be expressed concisely using the induced MRP with direct solution V π = R π + γp π V π V π = (I γp π ) 1 R π

51 Bellman operators for V π Definition The Bellman operator for V π is defined as T π : R S R S (maps value functions to value functions): ( ) (T π V π )(s) = a A π(a s) R(s, a) + γ s S P(s s, a)v π (s ) Using Bellman operator, Bellman expectation equation can be compactly written as: T π V π = V π V π is a fixed point of the Bellman operator T π This is a linear equation in V π and T π If 0 < γ < 1 then T π is a contraction w.r.t. the maximum norm

52 Bellman operators for Q π Definition The Bellman operator for Q π is defined as T π : R S A R S A (maps action value functions to action value functions): (T π Q π )(s, a) = R(s, a) + γ s S P(s s, a) a A π(a s ) (Q π (s, a )) Using Bellman operator, Bellman expectation equation can be compactly written as: T π Q π = Q π Q π is a fixed point of the Bellman operator T π This is a linear equation in Q π and T π If 0 < γ < 1 then T π is a contraction w.r.t. the maximum norm

53 Optimal Value Function Definition The optimal state value function V (s) is the maximum value function over all policies V (s) = max V π (s) π The optimal action value function Q (s, a) is the maximum action value function over all policies Q (s, a) = max Q π (s, a) π The optimal value function specifies the best possible performance in the MDP An MDP is solved when we know the optimal value function

54 Example: Optimal Value Function of Student MDP

55 Optimal Policy Value functions define a partial ordering over policies Theorem π π if V π (s) V π (s), For any Process s S There exists an optimal policy π that is better than or equal to all other policies π π, π All optimal policies achieve the optimal value function, V π (s) = V (s) All optimal policies achieve the optimal action value function, Q π (s) = Q (s) There is always a deterministic optimal policy for any MDP A deterministic optimal policy can be found by maximizing over Q (s, a) { π 1 if a = arg max a A Q (s, a) (a s) = 0 otherwise

56 Example: Optimal Policy for Student MDP

57 Bellman Optimality Equation Bellman Optimality Equation for V V (s) = max a = max a Q (s, a) { R(s, a) + γ s S P(s s, a)v (s ) } Bellman Optimality Equation for Q Q (s) = R(s, a) + γ s S P(s s, a)v (s ) = R(s, a) + γ s S P(s s, a) max a Q (s, a )

58 Example: Bellman Optimality Equation in Student MDP

59 Bellman Optimality Operator Definition The Bellman optimality operator for V is defined as T : R S R S (maps value functions to value functions): ( ) (T V )(s) = max a A R(s, a) + γ s S P(s s, a)v (s ) Definition The Bellman optimality operator for Q is defined as T : R S A R S A (maps action value functions to action value functions): (T Q )(s, a) = R(s, a) + γ s S P(s s, a) max a Q (s, a )

60 Properties of Bellman Operators Monotonicity: if f 1 f 2 component wise T π f 1 T π f 2, T f 1 T f 2 Max Norm Contraction: for two vectors f 1 anf f 2 T π f 1 T π f 2 γ f 1 f 2 T f 1 T f 2 γ f 1 f 2 V π is the unique fixed point of T π V is the unique fixed point of T For any vector f R S and any policy π, we have lim k (T π ) k f = V π, lim k (T ) k f = V

61 Solving the Bellman Optimality Equation Bellman optimality equation is non linear No closed form solution for the general case Many iterative solution methods Dynamic Programming Value Iteration Policy Iteration Linear Programming Reinforcement Learning Q learning SARSA

62 Extensions to MDP Undiscounted, average reward MDPs Infinite and continuous MDPs Partially observable MDPs Semi-MDPs Non stationary MDPs, games

63 Exercises Model the following problems But who s counting Blackjack