Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1
Today Eligibility traces n-step TD returns Forward and backward view Function approximation stochastic gradient descent (SGD) on-policy prediction and control 2
Scaling-up reinforcement learning Sparse rewards e.g. gridworld Large state spaces: Go: 10 170 states Camera input, e.g. 1280 720 24 2 10 7 (3 color channels, 8 bits each) Continuous spaces: e.g. inverted pendulum, mobile robot, etc. 3
Recall: n-step return n-step returns for n = 1, 2,... : n-step return: n = 1 TD R t+1 + γv (S t+1 ) n = 2 R t+1 + γr t+2 + γ 2 V (S t+2 ) n = 3 R t+1 + γr t+2 + γ 2 R t+3 + γ 3 V (S t+3 ).. n = MC R t+1 + γr t+2 + γ 2 R t+3 +... + γ T R T G t:t+n = R t+1 + γr t+2 + γ 2 R t+3 +... + γ n 1 R t+n + γ n V (S t+n ) n-step temporal difference update: V (S t ) V (S t ) + α (G t:t+n V (S t )) 4
If + n<t,theng G + Q(S +n,a +n ) (G ) Example: n-step Q(SSarsa,A ) Q(S,A )+ [G Q(S,A )] If is being learned, then ensure that ( S )is"-greedy wrt Q Until = T 1 Path taken Action values increased by one-step Sarsa Action values increased by Sarsa(!) by 10-step with Sarsa!=0.9 G G G Figure How 7.4: to choose Gridworld n? example of the speedup of policy learning due to the use of n-step methods. The first panel shows the path taken by an agent in a single episode, ending at a location of high reward, marked by the G. In this example the values were all initially 0, and all rewards were zero except for a positive reward at G. The arrows in the other two panels show which action values were strengthened as a result of this path by one-step and n-step Sarsa methods. The one-step method strengthens only the last action of the sequence of actions that led to the high reward, whereas the n-step method strengthens the last n 5
n=1 λ-returnfigure 12.2 further illustrates the weighting on the sequence of n-step returns in the -return. The one-step return is given the largest weight, 1 ; the two-step return The is given λ-return the Gnext t λ combines largest weight, all n-step (1 returns: ) ; the three-step return is given the weight (1 ) 2 ; and so on. The weight fades by with each additional step. After a terminal state has been reached, Gt λ = all(1 subsequent λ) λn-step n 1 G t:t+n returns are equal to G t.ifwe n=1 TD("), "-return 1!" (1!") " (1!") " 2 # = 1 Figure 12.1: The backup digram for TD( ). If = 0, then the overall backup reduces to its 6 " T-t-1
λ-return weighting function 254 Gt λ = (1 λ) CHAPTER λ n 1 G t:t+n12. ELIGIBILITY TRACES n=1 weight given to the 3-step return is (1 ) 2 total area = 1 Weight 1!" decay by " weight given to actual, final return is T t 1 t Time T Figure 12.2: Weighting given in the -return to each of the n-step returns. 7
to all the future rewards and decide how best to combine them. We might imagine Forwardourselves view riding the stream of states, looking forward from each state to determine its update, as suggested by Figure 12.4. After looking forward from and updating one state, we move on to the next and never have to work with the preceding state again. Future states, on the other hand, are viewed and processed repeatedly, once from each vantage point preceding them. Rr T Rr t+1 t+1 s t S t+1 S t Rr t+3 Ss t+3 Rr t+2 S t+2 s t+1 s t+2 Time Update Figure values 12.4: The by looking forward view. forward Weto decide future howrewards to update andeach states state by looking forward to future rewards and states. Update values towards λ-return Can only be computed for terminated sequences 8
Backward view This is why that algorithm was called TD(0). In terms of Figure 12.5, TD(0) is the case in which only the one state preceding the current one is changed by the TD error. For larger values of, butstill < 1, more of the preceding states e t Ss t-3 e t Ss t-2 e t Ss t-1!t t e t Ss t Ss t+1 Time Figure 12.5: The backward or mechanistic view. Each update depends on the current TD Forward error combined view provides with eligibility theory traces of past events. Backward view provides a mechanism how to perform updates Update every step; works for incomplete sequences 9
Eligibility traces Credit assignment problem: Frequency: assign credit to most frequent states Recency: assign credit to recent states s : e(s) γλe(s) e(s t ) e(s t ) + 1 10
the case in which only the one state preceding the current one is changed by the BackwardTD view error. For larger values of, butstill < 1, more of the preceding states e t Ss t-3 e t Ss t-2 e t Ss t-1!t t e t Ss t Ss t+1 Time Figure 12.5: The backward or mechanistic view. Each update depends on the current TD Keep errorancombined eligibility with trace eligibility for every traces state of past s events. Update value V (s) for every state s: δ t = R t+1 + γv (S t+1 ) V (S t ) s : V (s) V (s) + αδ t e t (s) 11
TD(λ) and TD(0) When λ = 0: { 1 for s = St e(s) = 0 else δ t = R t+1 + γv (S t+1 ) V (S t ) s : V (s) V (s) + αδ t e t (s) Same as TD(0): V (S t ) V (S t ) + α [R t+1 + γv (S t+1 ) V (S t )] 12
Large state spaces Backgammon: 10 20 states Go: 10 170 states Continuous spaces: inverted pendulum mountain car helicopter... 13
Small domains: tabular representation So far V and Q were just lookup tables: k = 50 k = 50 0.59 0.67 0.77 3 0.645 0.744 0.848 1.000 3 0.57 0.64 0.60 0.74 0.66 0.85 0.53 0.67 0.57 1.00 2 0.566 0.572-1.000 0.57 2 0.51 0.51 0.46 0.57 0.53-0.60 0.30-1.00 1 0.491 0.431 0.475 0.277 1 2 3 4 0.49 0.40 0.48-0.65 1 0.45 0.41 0.43 0.42 0.40 0.29 0.28 0.13 0.44 0.40 0.41 0.27 1 2 3 4 Problems with large MDPs: too many states to store in memory too slow to learn How can we scale the tabular solution methods to arbitrarily large state/action spaces? generalize from previous encounters of similar states (or state action pairs) function approximation (supervised learning) 14
Example: mountain car 15
State aggregation in mountain car 10 0 10 8 50 8 Velocity 6 4 100 150 200 Velocity 6 4 2 250 2 0 0 2 4 6 8 10 Position 300 0 0 2 4 6 8 10 Position 16
State aggregation in mountain car 10 8 Velocity 6 4 2 0 0 2 4 6 8 10 Position 17
Idea of value function approximation Parameterized functional form, with weights w R d : ˆv π (s, w) v π (s) Generally, much less weights than states d S obvious for continuous state spaces changing single weight, changes value estimate of many states when one state is update, change generalizes to many states Update w with MC or TD learning 18
Function approximators Linear combinations of features state aggregation tile coding polynomials radial basis functions (RBFs) Fourier bases... Neural networks Decision trees Non-parametric approaches 19
s 7! v is trained on, then whatever the prior estimate, ˆv(s, Tile coding t ), the new estimate will be ˆv(s, t+1 )=v. Usually one wishes to change more slowly than this, to allow for generalization and stochastic variation in target outputs. For example, one might choose = 1 10m, in which case the estimate for the trained state would move one- Continuous 2D state space Tiling 1 Tiling 2 Tiling 3 Tiling 4 Point in state space to be represented Four active tiles/features overlap the point and are used to represent it Figure 9.9: Multiple, overlapping grid-tilings on a limited two-dimensional space. These tilings are o set from one another by a uniform amount in each dimension. 20
Function approximators for RL Differentiable function approximators, e.g.: Linear combination of features Neural networks RL specific problems: non-stationary non-iid data bootstrapping delayed targets 21
Stochastic gradient descent (SGD) Approximate value function ˆv(s, w) differentiable for all s S Weight vector w = (w 1, w 2,..., w d ) T w t weight vector at time t = 0, 1, 2,... Gradient of f (w): ( δf (w) f (w) =, δw 1 δf (w),..., δw 1 ) T δf (w) δw d Adjust w in the direction that reduces the error on sample S t v π (S t ): alpha is a step size parameter w t+1 = w t 1 2 α [v π(s t ) ˆv(S t, w t )] 2 }{{} squared sample error = w t + α[v π (S t ) ˆv(S t, w t )] ˆv(S t, w) why not use α = 1, thus eliminating full error on sample? 22
Linear methods Special case where ˆv(, w) is linear in the weights Feature vector x(s) represents state s: Each component of x is a feature Examples: x(s) = (x 1 (s), x 2 (s),..., x d (s)) T distance of robot to landmarks piece on a specific location on a chess board Value function is simply the inner product between w and x(s): ˆv(s, w) = w T x(s) = d w i x i (s) i=1 Value function is represented as a linear combination of features Gradient is simply ˆv(s, w) = x(s) 23
Prediction with function approximation We assumed the true value function v π (S t ) is known Substitute target U t for v π (s) w t+1 = w t + α[u t ˆv(S t, w t )] ˆv(S t, w) U t might be a noisy or bootstrapped approximation of the true value Monte Carlo: U t = G t TD(0): U t = R t+1 + γˆv(s t+1, w) TD(λ): U t = Gt λ 24
Monte Carlo with function approximation Target is unbiased by definition: E[U t S t = s] = E[G t S t = s] = v π (S t ) Training data: D = { S 1, G 1, S 2, G 2,..., S T 1, G T 1, S T, 0 } Using SGD, w is guaranteed to converge to a local optimum MC prediction exhibits local convergence with linear and non-linear function approximation SGD update for sample S t G t : w w + α[g t ˆv(S t, w)] ˆv(S t, w) 25
TD with function approximation TD-target U t = R t+1 + γˆv(s t+1, w) is biased sample of the true value v π (S t ) Training data: D = { S 1, R 2 + γˆv(s 2, w), S 2, R 3 + γˆv(s 3, w),... S T 1, R T } SGD update for sample S t G t : w w + α[r t+1 + γˆv(s t+1, w) ˆv(S t, w)] ˆv(S t, w) Linear TD(0): w w + α[r t+1 + γw T x(s t+1 ) w T x(s t )]x(s t ) 26
Example: TD-Gammon by Gerald Tesauro (1992) Multi-layer perceptron (neural network) represents value function Only reward given at the end of game (1, 0.5, 0) Self-play: use the current policy to sample moves on both sides! Same ideas used in recent work on Go, Chess, Shogi: Alpha Go Zero (2015) Alpha Zero (2017) 27
Control with function approximation Control via generalized policy iteration (GPI): policy evaluation: approximate policy evaluation: ˆq(,, w) q π policy improvement: ɛ-greedy policy improvement 28
Types of action value function approximation Action as input: ˆq(s, a, w) q π (s, a) Multiple action value outputs: ˆq a (s, w) q π (s, a) 29
Action-value function approximation Approximate action-value function ˆq(s, a, w) q π (s, a) Linear case: d ˆq(s, a, w) = w T x(s, a) = w i x i (s, a) i=1 ˆq(s, a, w) = x(s, a) Minimize squared error on samples S t, A t q π : [ q π ˆq(S t, A t, w) ] 2 Use SGD to find local minimum: w t+1 = w t 1 2 α [q π(s t, A t ) ˆq(S t, A t, w t )] 2 = w t + α[q π (S t, A t ) ˆq(S t, A t, w t )] ˆq(S t, A t, w) 30
Control with function approximation Again, we must substitute target U t for true action-value q π (s, a): w t+1 = w t + α[u t ˆq(S t, A t, w t )] ˆq(S t, A t, w) Monte Carlo: U t = G t One-step Sarsa: U t = R t+1 + γˆq(s t+1, A t+1, w) 31
Batch reinforcement learning Gradient descent is simple But it is not sample efficient Batch methods seek to find best fitting value function Given the agent s experience ( training data ) 32
Least squares prediction Approximate value function ˆv(s, w) v π (s) Given experience D sampled using π: Which parameters w give the best fit for ˆv? D = { s 1, v π (s 1 ), s 2, v π (s 2 ),..., s T, v π (s T ) } Least squares algorithms find parameter vectors w minimising the sum-squared error between ˆv(s i ) and true values v π (s i ): T (v π (s i ) ˆv(s i, w)) 2 [ = E D (vπ (s) ˆv(s, w) 2] i=1 Sampling s, v π (s) D from experience and applying SGD, converges to the least squares solution: T w π = arg min (v π (s i ) ˆv(s i, w)) 2 w i=1 33
Linear least squares prediction (LSPI) Recall the linear TD update w w + α[r t+1 + γw T x(s t+1 ) w T x(s t )]x(s t ) At steady state w π the expected update must be zero: ] E D [α[u w T x(s)]x(s) = 0 T α [u i w T x(s i )]x(s i ) = 0 t=1 T T x(s i )u i = x(s i )x(s i ) T w t=1 w = Direct solution is O(d 3 ) t=1 ( T ) 1 x(s i )x(s i ) T T t=1 t=1 Incremental solution using Shermann-Morrison equality is O(d 2 ) x(s i )u i 34
Recent successes of deep reinforcement learning images from Human-level control through deep reinforcement learning (Google Deepmind / Nature) 35
Deep (supervised) learning Deep representation is a composition of many functions x w1 h 1 w2 h 2 w3... wn h n wn+1 y Linear transformation and non-linear activation functions h k Weight sharing Recurrent neural networks: across time steps Convolutional neural networks: across spatial (or temporal) regions Weights w optimized by stochastic gradient descent (SGD) Powerful function approximation and representation learning finds compact low-dimensional representation (features) State-of-the-art for image, text and audio 36
Naive deep Q-learning Q-learning update rule: ( Q(s, a) Q(s, a) + α r + γ max a ) Q(s, a) Q(s, a) Q is represented by a neural network with weights w: ˆq(s, a, w) Loss is the mean-squared TD-error: Minimize sample errors with SGD [ ( ) ] 2 L(w) = E r + γ max ˆq(s, a, w) ˆq(s, a, w]) a 37
Stability Naive Q-learning with neural networks oscillates or diverges: 1. Data is non i.i.d! trajectories samples are correlated (generated by interaction) 2. Policy changes rapidly with slight changes to Q-values policy may oscillate 3. Reward range is unknown gradients can be large instabilities during back-propagation 4. Maximization bias 38
Deep Q-networks (DQN) Deep Q-networks (DQN) address instabilities through: Experience replay store transitions S t, A t, R t, S t+1 sample random mini-batches removes correlation, restores i.i.d. property Target network second q network (second set of parameters) fixed parameters in target network periodically update target network parameters Reward clipping/normalization clip rewards to r [ 1, 1] batch normalization 39
DQN in Atari End-to-end learning: state: stack of 4 frames, raw pixels action: joystick commands (18 discrete actions) RESEARCH LETTER reward: change in score Convolution Convolution Fully connected Fully connected No input Figure 1 Schematic illustration of the convolutional neural network. The symbolizes sliding of each filter across input image) and two fully connected details of the architecture are explained in the Methods. The input to the neural layers with a single output for each valid action. Each hidden layer is followed image network fromconsists Human-level of ancontrol 84 3 84 through 3 4 image deep produced reinforcement by learning the preprocessing (Google Deepmind by / Nature) a rectifier nonlinearity (that is, maxð0,xþ). 40
DQN in Atari image from Human-level control through deep reinforcement learning (Google Deepmind / Nature) 41