Reinforcement Learning. Spring 2018 Defining MDPs, Planning

Size: px

Start display at page:

Download "Reinforcement Learning. Spring 2018 Defining MDPs, Planning"

Anissa Welch
5 years ago
Views:

1 Reinforcement Learning Spring 2018 Defining MDPs, Planning

2 understandability 0 Slide 10 time You are here

3 Markov Process Where you will go depends only on where you are

4 Markov Process: Information state This spider doesn t like to turn back The information state of a Markov process may be different from its physical state

5 Markov Reward Process Random wandering through states will occasionally win you a reward

6 The Fly Markov Reward Process 1.0 s /3 s 2 s 1 1/3 s 0 1/3 2/3 R=-1 R=-1 R=-1 R=0 There are, in fact, only four states, not eight Manhattan distance between fly and spider = 0 (s 0 ) Distance between fly and spider = 1 (s 1 ) Distance between fly and spider = 2 (s 2 ) Distance between fly and spider = 3 (s 3 ) Can, in fact, redefine the MRP entirely in terms of these 4 states

7 The discounted return G t = r t+1 + γr t+2 + γ 2 r t+3 + = γ k r t+k k=0 Total future reward all the way to the end

8 c 1 c 2 Markov Decision Process c 3 c 8 c 4 c 7 c 5 c 6 Markov Reward Process with following change: Agent has real agency Agent s actions modify environment s behavior

9 The Fly Markov Decision Process s 0 Process ends a + s 1 a s 2 s 1 1 s 1 s 2 a + a - 2/3 s 3 a- 2/3 s 3 s 2 1 s 0 s 2 1/3 s 1 s 3 1/3

10 Policy c 1 c 2 c 3 c 8 c 4 c 7 c 5 c 6 The policy is the agent s choice of action in each state May be stochastic

11 The Bellman Expectation Equations The Bellman expectation equation for state value function v π s = π a s R s a + γ P a s,s v π s a A s The Bellman expectation equation for action value function q π s, a = R s a + γ s P a s,s a A π a s q π s, a

12 Optimal Policies The optimal policy is the policy that will maximize the expected total discounted reward at every state: E G t S t = s = E γ k r t+k S t = s k=0 Optimal Policy Theorem: For any MDP there exist optimal policies π that is better than or equal to every other policy: π π π v s v π s q s, a q π s, a s s, a do we consider the discounted return, rather than the actual return σ k=0 r t+k?

13 The optimal value function π a s = 1 for argmax q s, a a 0 otherwise v s = max a q s, a

14 Bellman Optimality Equations Optimal value function equation v s = max a R s a + γ s P a s,s v s Optimal action value equation q s, a = R s a + γ s P a s,s max a q s, a

15 Planning with an MDP Problem: Given: an MDP S, P, A, R, γ Find: Optimal policy π Can either Value-based Solution: Find optimal value (or action value) function, and derive policy from it OR Policy-based Solution: Find optimal policy directly

16 Value-based Planning Value -based solution Breakdown: Prediction: Given any policy π find value function v π s Control: Find the optimal policy

17 Prediction DP Iterate v π (k+1) s = a A π a s R s a + γ s P a (k) s,s v π s

18 Policy Iteration Start with any policy π (0) Iterate (k = 0 convergence): Use prediction DP to find the value function v π (k) s Find the greedy policy π k+1 s = greedy v π (k) s

19 Value iteration v (k) s = max a R s a + γ s P a (k 1) s,s v s Each state simply inherits the cost of its best neighbour state Cost of neighbor is the value of the neighbour plus cost of getting there

20 Problem so far Given all details of the MDP Compute optimal value function Compute optimal action value function Compute optimal policy This is the problem of planning Problem: In real life, nobody gives you the MDP How do we plan???

21 Model-Free Methods AKA model-free reinforcement learning How do you find the value of a policy, without knowing the underlying MDP? Model-free prediction How do you find the optimal policy, without knowing the underlying MDP? Model-free control

22 Model-Free Methods AKA model-free reinforcement learning How do you find the value of a policy, without knowing the underlying MDP? Model-free prediction How do you find the optimal policy, without knowing the underlying MDP? Model-free control Assumption: We can identify the states, know the actions, and measure rewards, but have no knowledge of the system dynamics The key knowledge required to solve for the best policy A reasonable assumption in many discrete-state scenarios Can be generalized to other scenarios with infinite or unknowable state

23 Model-Free Assumption Can see the fly Know the distance to the fly Know possible actions (get closer/farther) But have no idea of how the fly will respond Will it move, and if so, to what corner

24 Model-Free Methods AKA model-free reinforcement learning How do you find the value of a policy, without knowing the underlying MDP? Model-free prediction How do you find the optimal policy, without knowing the underlying MDP? Model-free control

25 Model-Free Assumption Can see the fly and distance to the fly But have no idea of how the fly will respond to actions Will it move, and if so, to what corner But will always try to reduce distance to fly (have a known, fixed, policy) What is the value of being a distance D from the fly?

26 Methods Monte-Carlo Learning Temporal-Difference Learning TD(1) TD(K) TD(λ)

27 Monte-Carlo learning to learn the value of a policy π Just let the system run while following the policy π and learn the value of different states Procedure: Record several episodes of the following Take actions according to policy π Note states visited and rewards obtained as a result Record entire sequence: S 1, A 1, R 2, S 2, A 2, R 3,, S T Assumption: Each episode ends at some time Estimate value functions based on observations by counting

28 Monte-Carlo Value Estimation Objective: Estimate value function v π (s) for every state s, given recordings of the kind: S 1, A 1, R 2, S 2, A 2, R 3,, S T Recall, the value function is the expected return: v π s = E G t S t = s = E R t+1 + γr t γ T t 1 R T S t = s To estimate this, we replace the statistical expectation E G t S t = s by the empirical average avg G t S t = s

29 A bit of notation We actually record many episodes episode 1 = S 11, A 11, R 12, S 12, A 12, R 13,, S 1T1 episode 2 = S 21, A 21, R 22, S 22, A 22, R 23,, S 2T2 Different episodes may be different lengths

30 Counting Returns For each episode, we count the returns at all times: S 11, A 11, R 12, S 12, A 12, R 13, S 13, A 13, R 14,, S 1T1 G 1,1 Return at time t G 1,1 = R 12 + γr γ T 1 2 R 1T1 G 1,2 = R 13 + γr γ T 1 3 R 1T1 G t = R 1,t+1 + γr 1,t γ T 1 t 2 R 1T1

31 Counting Returns For each episode, we count the returns at all times: S 11, A 11, R 12, S 12, A 12, R 13, S 13, A 13, R 14,, S 1T1 Return at time t G 1,1 = R 12 + γr γ T 1 2 R 1T1 G 1,2 = R 13 + γr γ T 1 3 R 1T1 G 1,2 G t = R 1,t+1 + γr 1,t γ T 1 t 2 R 1T1

32 Counting Returns For each episode, we count the returns at all times: S 11, A 11, R 12, S 12, A 12, R 13, S 13, A 13, R 14,, S 1T1 Return at time t G 1,1 = R 12 + γr γ T1 2 R 1T1 G 1,2 = R 13 + γr γ T1 3 R 1T1 G 1,t = R 1,t+1 + γr 1,t γ T1 t 1 R 1T1

33 Estimating the Value of a State To estimate the value of any state, identify the instances of that state in the episodes: S 11, A 11, R 12, S 12, A 12, R 13, S 13, A 13, R 14,, S 1T1 s a s b s a Compute the average return from those instances v π s a = avg G 1,1, G 1,3,

34 Estimating the Value of a State For every state s Initialize: Count N s = 0, Total return v π s = 0 For every episode e For every time t = 1 T e Compute G t If (S t == s)» N s = N s + 1» v π s = v π s + G t v π s = v π s /N(s) Can be done more efficiently..

35 Online Version For all s Initialize: Count N s = 0, Total return totv π s = 0 For every episode e For every time t = 1 T e Compute G t N S t = N S t + 1 totv π S t = totv π S t + G t For every state s : v π s = totv π s /N(s) Updating values at the end of each episode Can be done more efficiently..

36 Monte Carlo estimation Learning from experience explicitly After a sufficiently large number of episodes, in which all states have been visited a sufficiently large number of times, we will obtain good estimates of the value functions of all states Easily extended to evaluating action value functions

37 Estimating the Action Value function To estimate the value of any state-action pair, identify the instances of that state-action pair in the episodes: S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T s a a x s b a y s a a y Compute the average return from those instances q π s a, a x = avg G 1,1,

38 Online Version For all s, a Initialize: Count N s, a = 0, Total value totq π s, a = 0 For every episode e For every time t = 1 T e Compute G t N S t, A t = N S t, A t + 1 totq π S t, A t = totq π S t, A t + G t For every s, a : q s, a = totq π s, a /N(s, a) Updating values at the end of each episode

39 Monte Carlo: Good and Bad Good: Will eventually get to the right answer Unbiased estimate Bad: Cannot update anything until the end of an episode Which may last for ever High variance! Each return adds many random values Slow to converge

40 Online methods for estimating the value of a policy: Temporal Difference Leaning (TD) Idea: Update your value estimates after every observation S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Update for S 1 Update for S 2 Update for S 3 Do not actually wait until the end of the episode

41 Incremental Update of Averages Given a sequence x 1, x 2, x 3, a running estimate of their average can be computed as k μ k = 1 k i=1 x i This can be rewritten as: μ k = (k 1)μ k 1 + x k k And further refined to μ k = μ k k x k μ k 1

42 Incremental Update of Averages Given a sequence x 1, x 2, x 3, a running estimate of their average can be computed as μ k = μ k k x k μ k 1 Or more generally as μ k = μ k 1 + α x k μ k 1 The latter is particularly useful for non-stationary environments

43 Incremental Updates μ k = μ k k x k μ k 1 α = 0.1 α = 0.05 μ k = μ k 1 + α x k μ k 1 α = 0.03 Example of running average of a uniform random variable

44 Incremental Updates μ k = μ k k x k μ k 1 α = 0.1 μ k = μ k 1 + α x k μ k 1 α = 0.05 α = 0.03 Correct equation is unbiased and converges to true value Equation with α is biased (early estimates can be expected to be wrong) but converges to true value

45 Actual update Updating Value Function Incrementally v π s = 1 N(s) N(s) i=1 G t(i) N(s) is the total number of visits to state s across all episodes G t(i) is the discounted return at the time instant of the i-th visit to state s

46 Online update Given any episode S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Update the value of each state visited N S t = N S t + 1 v π S t = v π S t + 1 N(S t ) G t v π S t Incremental version Still an unrealistic rule v π S t = v π S t + α G t v π S t Requires the entire track until the end of the episode to compute Gt

47 Online update Given any episode S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Update the value of each state visited N S t = N S t + 1 v π S t = v π S t + 1 N(S t ) G t v π S t Incremental version Still an unrealistic rule Problem v π S t = v π S t + α G t v π S t Requires the entire track until the end of the episode to compute Gt

48 TD solution v π S t = v π S t + α G t v π S t Problem But G t = R t+1 + γg t+1 We can approximate G t+1 by the expected return at the next state S t+1

49 Counting Returns For each episode, we count the returns at all times: S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Return at time t G 1 = R 2 + γr γ T 2 R T G 2 = R 3 + γr γ T 3 R T G t = R t+1 + γr t γ T t 2 R T Can rewrite as G 1 = R 2 + γg 2 Or G 1 = R 2 + γr 3 + γ 2 G 3 G t = R t+1 + σ N i=1 γ i R t+1+i + γ N+1 G t+1+n

50 TD solution But v π S t = v π S t + α G t v π S t Problem G t = R t+1 + γg t+1 We can approximate G t+1 by the expected return at the next state S t+1 v π S t+1 G t R t+1 + γv π S t+1 We don t know the real value of v π S t+1 bootstrap it by its current estimate but we can

51 TD(1) true online update v π S t = v π S t + α G t v π S t Where G t R t+1 + γv π S t+1 Giving us v π S t = v π S t + α R t+1 + γv π S t+1 v π S t

52 TD(1) true online update Where v π S t = v π S t + αδ t δ t = R t+1 + γv π S t+1 v π S t δ t is the TD error The error between an (estimated) observation of G t and the current estimate v π S t

53 TD(1) true online update For all s Initialize: v π s = 0 For every episode e For every time t = 1 T e v π S t = v π S t + α R t+1 + γv π S t+1 v π S t There s a lookahead of one state, to know which state the process arrives at at the next time But is otherwise online, with continuous updates

54 TD(1) Updates continuously improve estimates as soon as you observe a state (and its successor) Can work even with infinitely long processes that never terminate Guaranteed to converge to the true values eventually Although initial values will be biased as seen before Is actually lower variance than MC!! Only incorporates one RV at any time TD can give correct answers when MC goes wrong Particularly when TD is allowed to loop over all learning episodes

55 TD vs MC What are v(a) and v(b) Using MC Using TD(1), where you are allowed to repeatedly go over the data

56 TD look ahead further? TD(1) has a look ahead of 1 time step G t R t+1 + γv π S t+1 But we can look ahead further out G t (2) = R t+1 + γr t+2 + γ 2 v π S t+2 G t (N) = R t+1 σ N i=1 γ i R t+1+i + γ N+1 v π S t+n

57 TD(N) with lookahead v π S t = v π S t + αδ t (N) Where δ t (N) = R t+1 + N i=1 γ i R t+1+i + γ N+1 v π S t+n v π S t δ t (N) is the TD error with N step lookahead

58 Lookahead is good Good: The further you look ahead, the better your estimates get Problems: But you also get more variance At infinite lookahead, you re back at MC Also, you have to wait to update your estimates A lag between observation and estimate So how much lookahead must you use

59 Looking Into The Future How much various TDs look into the future Which do we use?

60 Solution: Why choose? Each lookahead provides an estimate of G t Why not just combine the lot with discounting?

61 TD(l) G t λ = (1 λ) n=1 λ n 1 G t (n) Combine the predictions from all lookaheads with an exponentially falling weight Weights sum to 1.0 V S t V S t + α G t λ V S t

62 Something magical just happened TD(l) looks into the infinite future I.e. we must have all the rewards of the future to compute our updates How does that help?

63 The contribution of future rewards to S t the present update R t+1 1 λ R t+2 (1 λ)λ R t+3 (1 λ)λ 2 R t+4 (1 λ)λ 3 R t+5 (1 λ)λ 4 (1 λ)λ 5 (1 λ)λ 6 TIME All future rewards contribute to the update of the value of the current state R t+6 Rt+7

64 The contribution of current reward to (1 λ)λ 6 (1 λ)λ 5 S t+7 (1 λ)λ 4 S t+6 S t+5 past states (1 λ)λ 3 (1 λ)λ 2 S t+4 S t+3 (1 λ)λ S t+2 1 λ S t+1 R t TIME All current reward contributes to the update of the value of all past states!

65 TD(l) backward view R t Add these weights to compute contribution to red state.. (1 λ)λ 6 (1 λ)λ 5 (1 λ)λ 4 (1 λ)λ 3 (1 λ)λ 2 (1 λ)λ 1 λ The Eligibility trace: Keeps track of total weight for any state TIME Which may have occurred at multiple times in the past

66 TD(l) Maintain an eligibility trace for every state E 0 s = 0 E t s = γe t 1 s + 1 S t = s Computes total weight for the state until the present time

67 TD(l) At every time, update the value of every state according to its eligibility trace δ t = R t+1 + γv S t+1 V S t V s V s + αδ t E t S t Any state that was visited will be updated Those that were not will not be, though

68 The magic of TD(l) Managed to get the effect of inifinite lookahead, by performing infinite lookbehind Or at least look behind to the beginning Every reward updates the value of all states leading to the reward! E.g., in a chess game, if we win, we want to increase the value of all game states we visited, not just the final move But early states/moves must gain much less than later moves When λ = 1 this is exactly equivalent to MC

69 Story so far Want to compute the values of all states, given a policy, but no knowledge of dynamics Have seen monte-carlo and temporal difference solutions TD is quicker to update, and in many situations the better solution TD(l) actually emulates an infinite lookahead But we must choose good values of a and l

70 Optimal Policy: Control We learned how to estimate the state value functions for an MDP whose transition probabilities are unknown for a given policy How do we find the optimal policy?

71 Value vs. Action Value The solution we saw so far only computes the value functions of states Not sufficient to compute the optimal policy from value functions alone, we will need extra information, namely transition probabilities Which we do not have Instead, we can use the same method to compute action value functions Optimal policy in any state : Choose the action that has the largest optimal action value

72 Value vs. Action value Given only value functions, the optimal policy must be estimated as: π s = argmax R a s + P a ss V s a A Needs knowledge of transition probabilities Given action value functions, we can find it as: π s = argmax Q s, a a A This is model free (no need for knowledge of model parameters)

73 Problem of optimal control From a series of episodes of the kind: S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Find the optimal action value function q s, a The optimal policy can be found from it Ideally do this online So that we can continuously improve our policy from ongoing experience

74 Exploration vs. Exploitation Optimal policy search happens while gathering experience while following a policy For fastest learning, we will follow an estimate of the optimal policy Risk: We run the risk of positive feedback Only learn to evaluate our current policy Will never learn about alternate policies that may turn out to be better Solution: We will follow our current optimal policy 1 ε of the time But choose a random action ε of the time The epsilon-greedy policy

75 GLIE Monte Carlo Greedy in the limit with infinite exploration Start with some random initial policy π Start the process at the initial state, and follow an action according to initial policy π Produce the episode S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Process the episode using the following online update rules: Compute the ε-greedy policy for each state Repeat 1 ε for a = argmax Q(s, a ) a π a s = ε N a 1 otherwise

76 GLIE Monte Carlo Greedy in the limit with infinite exploration Start with some random initial policy π Start the process at the initial state, and follow an action according to initial policy π Produce the episode S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T Process the episode using the following online update rules: Compute the ε-greedy policy for each state Repeat 1 ε for a = argmax Q(s, a ) a π a s = ε N a 1 otherwise

77 On-line version of GLIE: SARSA Replace G t with an estimate TD(1) or TD(l) Just as in the prediction problem TD(1) SARSA Q S, A Q S, A + α R + γq S, A Q(S, A)

78 Initialize Q(s, a) for all s, a Start at initial state S 1 Select an initial action A 1 For t = 1.. Terminate Get reward R t SARSA Let system transition to new state S t+1 Draw A t+1 according to ε -greedy policy Update 1 ε for a = argmax Q(s, a ) a π a s = ε N a 1 otherwise Q S t, A t = Q S t, A t + α R t + γq S t+1, A t+1 Q S t, A t

79 SARSA(l) Again, the TD(1) estimate can be replaced by a TD(l) estimate Maintain an eligibility trace for every state-action pair: E 0 s, a = 0 E t s, a = γe t 1 s, a + 1 S t = s, A t = a Update every state-action pair visited so far δ t = R t+1 + γq S t+1, A t+1 Q S t, A t Q s, a Q s, a + αδ t E t s, a

80 SARSA(l) For all s, a initialize Q(s, a) For each episode e For all s, a initialize E s, a = 0 Initialize S 1, A 1 For t = 1 Termination Observe R t+1, S t+1 Choose action A t+1 using policy obtained from Q δ = R t+1 + γq S t+1, A t+1 Q(S t, A t ) E S t, A t += 1 For all s, a Q s, a = Q s, a + αδe(s, a) E s, a = γλe(s, a)

81 On-policy vs. Off-policy SARSA assumes you re following the same policy that you re learning Its possible to follow one policy, while learning from others E.g. learning by observation The policy for learning is the whatif policy S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T መA 2 መA 3 Modifies learning rule to hypothetical Q S t, A t = Q S t, A t + α R t+1 + γq S t+1, A t+1 Q S t, A t Q S t, A t = Q S t, A t + α R t+1 + γq S t+1, መA t+1 Q S t, A t Q will actually represent the action value function of the hypothetical policy

82 SARSA: Suboptimality SARSA: From any state-action S, A, accept reward R, transition to next state S, choose next action A Use TD rules to update: δ = R + γq S, A Q S, A Problem: which policy do we use to choose A

83 SARSA: Suboptimality SARSA: From any state-action S, A, accept reward R, transition to next state S, choose next action A Problem: which policy do we use to choose A If we choose the current judgment of the best action at S we will become too greedy Never explore If we choose a sub-optimal policy to follow, we will never find the best policy

84 Solution: Off-policy learning The policy for learning is the whatif policy S 1, A 1, R 2, S 2, A 2, R 3, S 3, A 3, R 4,, S T መA 2 መA hypothetical 3 Use the best action for S t+1 as your hypothetical off-policy action But actually follow an epsilon-greedy action The hypothetical action is guaranteed to be better than the one you actually took But you still explore (non-greedy)

85 Q-Learning From any state-action pair S, A Accept reward R Transition to S Find the best action A for S Use it to update Q(S, A) But then actually perform an epsilon-greedy action A " from S

86 Q-Learning (TD(1) version) For all s, a initialize Q(s, a) For each episode e Initialize S 1, A 1 For t = 1 Termination Observe R t+1, S t+1 Choose action A t+1 at S t+1 using epsilon-greedy policy obtained from Q Choose action መA t+1 at S t+1 as መA t+1 = argmax a δ = R t+1 + γq S t+1, መA t+1 Q(S t, A t ) Q S t, A t = Q S t, A t + αδ Q(S t+1, a)

87 Q-Learning (TD(l) version) For all s, a initialize Q(s, a) For each episode e For all s, a initialize E s, a = 0 Initialize S 1, A 1 For t = 1 Termination Observe R t+1, S t+1 Choose action A t+1 at S t+1 using epsilon-greedy policy obtained from Q Choose action መA t+1 at S t+1 as መA t+1 = argmax a δ = R t+1 + γq S t+1, መA t+1 Q(S t, A t ) E S t, A t += 1 For all s, a Q s, a = Q s, a + αδe(s, a) E s, a = γλe(s, a) Q(S t+1, a)

88 What about the actual policy? Optimal greedy policy: 1 for a = argmax Q(s, a ) π a s = a 0 otherwise Exploration policy 1 ε for a = argmax Q(s, a ) a π a s = ε N a 1 otherwise Ideally ε should decrease with time

89 Q-Learning Currently most-popular RL algorithm Topics not covered: Value function approximation Continuous state spaces Deep-Q learning Action replay Application to real problem..

Reinforcement Learning. Summer 2017 Defining MDPs, Planning

Reinforcement Learning. Summer 2017 Defining MDPs, Planning Reinforcement Learning Summer 2017 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state