Lecture 8: Policy Gradient

Transcription

1 Lecture 8: Policy Gradient Hado van Hasselt

2 Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient

3 Introduction Vapnik s rule Never solve a more general problem as an intermediate step. (Vladimir Vapnik, 1998) If we care about optimal behaviour: why not learn a policy directly?

4 Introduction General overview Model-based RL: + Easy to learn a model (supervised learning) - Wrong objective - Non-trivial going from model to policy (planning) Value-based RL: + Closer to true objective o Harder than learning models, but perhaps easier than policies - Still not exactly right objective Policy-based RL: + Right objective! - Learning & evaluating policies can be inefficient (high variance) - Ignores other learnable knowledge, which may be important

5 Introduction Policy-Based Reinforcement Learning In the last lecture we approximated the value or action-value function using parameters θ, v θ (s) v π (s) q θ (s, a) q π (s, a) A policy was generated directly from the value function e.g. using ɛ-greedy In this lecture we will directly parametrize the policy π θ (a s) = P [a s, θ] We focus on model-free reinforcement learning

6 Introduction Value-Based and Policy-Based RL Value Based Learnt Value Function Implicit policy (e.g. ɛ-greedy) Policy Based No Value Function Learnt Policy Actor-Critic Learnt Value Function Learnt Policy Value Function Value-Based Actor Critic Policy Policy-Based

7 Introduction Advantages of Policy-Based RL Advantages: Better convergence properties Effective in high-dimensional or continuous action spaces Can learn stochastic policies Sometimes policies are simple while values and models are complex Disadvantages: Susceptible to local optima Learning a policy can be slower than learning values

8 Introduction Rock-Paper-Scissors Example Example: Rock-Paper-Scissors Two-player game of rock-paper-scissors Scissors beats paper Rock beats scissors Paper beats rock Consider policies for iterated rock-paper-scissors A deterministic policy is easily exploited A uniform random policy is optimal (i.e. Nash equilibrium)

9 Introduction Aliased Gridworld Example Example: Aliased Gridworld (1) The agent cannot differentiate the grey states Consider features of the following form (for all N, E, S, W) walls actions {}}{{}}{ φ(s, a) = (}{{} 1 }{{} 0 }{{} 1 }{{} 0 }{{} 0 }{{} 1 }{{} 0 }{{} 0 ) N E S W N E S W Compare deterministic and stochastic policies

10 Introduction Aliased Gridworld Example Example: Aliased Gridworld (2) Under aliasing, an optimal deterministic policy will either move W in both grey states (shown by red arrows) move E in both grey states Either way, it can get stuck and never reach the money So it will traverse the corridor for a long time

11 Introduction Aliased Gridworld Example Example: Aliased Gridworld (3) An optimal stochastic policy moves randomly E or W in grey states π θ (wall to N and S, move E) = 0.5 π θ (wall to N and S, move W) = 0.5 Will reach the goal state in a few steps with high probability Policy-based RL can learn the optimal stochastic policy

12 Introduction Policy Search Policy Objective Functions Goal: given policy π θ (s, a) with parameters θ, find best θ But how do we measure the quality of a policy π θ? In episodic environments we can use the start value J 1 (θ) = v πθ (s 1 ) In continuing environments we can use the average value J avv (θ) = s d πθ (s)v πθ (s) Or the average reward per time-step J avr (θ) = s d πθ (s) a π θ (s, a)r a s where d πθ (s) is stationary distribution of Markov chain for π θ

13 Introduction Policy Search Policy Optimisation Policy based reinforcement learning is an optimization problem Find θ that maximises J(θ) Some approaches do not use gradient Hill climbing Genetic algorithms Greater efficiency often possible using gradient Gradient descent Conjugate gradient Quasi-newton We will focus on stochastic gradient descent

14 Finite Difference Policy Gradient Policy Gradient!"#$%"&'(%")*+,#'-'!"#$%%" (%")*+,#'.+/0+,# Let J(θ) be any policy objective function Policy gradient algorithms search for a!"#$%&'('%$&#()&*+$,*$#&&-&$$$$."'%"$ local maximum in J(θ) by ascending the '*$%-/0'*,('-*$.'("$("#$1)*%('-*$ gradient of the policy, w.r.t. parameters θ,22&-3'/,(-& %,*$0#$)+#4$(-$ %&#,(#$,*$#&&-&$1)*%('-*$$$$$$$$$$$$$ θ = α θ J(θ) Where θ J(θ) is the policy gradient!"#$2,&(',5$4'11#&#*(',5$-1$("'+$#&&-&$ J(θ) 1)*%('-*$$$$$$$$$$$$$$$$6$("#$7&,4'#*($ θ 1 %,*$*-.$0#$)+#4$(-$)24,(#$("#$ θ J(θ) =. '*(#&*,5$8,&',05#+$'*$("#$1)*%('-*$ J(θ) θ n,22&-3'/,(-& 9,*4$%&'('%:;$$$$$$ and α is a step-size parameter

15 Monte-Carlo Policy Gradient Likelihood Ratios Gradients on parameterized policies We need to compute an estimate of the policy gradient Assume policy π θ is differentiable almost everywhere E.g., π θ is a linear function, or a neural network Or perhaps we may have a parameterized class of controllers Goal is to compute θ J(θ) = θ E d [v πθ (S)]. We will use Monte Carlo samples to compute this gradient So, how does E d [v πθ (S)] depend on θ?

16 Monte-Carlo Policy Gradient Likelihood Ratios Monte Carlo Policy Gradient Consider a one-step case such that J(θ) = E[R(S, A)], where expectation is over d (states) and π (actions) We cannot sample R t+1 and then take a gradient: R t+1 is just a number that does not depend on θ Instead, we use the identity: θ E[R(S, A)] = E[ θ log π(a S)R(S, A)]. (Proof on next slide) The right-hand side gives an expected gradient that can be sampled

17 Monte-Carlo Policy Gradient Likelihood Ratios Monte Carlo Policy Gradient θ E[R(S, A)] = θ d(s) π θ (a s)r(s, a) s a = d(s) θ π θ (a s)r(s, a) s a = s = s d(s) a d(s) a π θ (a s) θπ θ (a s) R(s, a) π θ (a s) π θ (a s) θ log π θ (a s)r(s, a) = E[ θ log π θ (A S)R(S, A)]

18 Monte-Carlo Policy Gradient Likelihood Ratios Monte Carlo Policy Gradient θ E[R(S, A)] = E[ θ log π θ (A S)R(S, A)] (see previous slide) This is something we can sample Our stochastic policy-gradient update is then θ t+1 = θ t + αr t+1 θ log π θt (A t S t ). In expectation, this is the actual policy gradient So this is a stochastic gradient algorithm

19 Monte-Carlo Policy Gradient Likelihood Ratios Softmax Policy Consider a softmax policy on linear values as an example Weight actions using linear combination of features φ(s, a) θ Probability of action is proportional to exponentiated weight π θ (s, a) e φ(s,a) θ b eφ(s,b) θ The gradient of the log probability is θ log π θ (s, a) = φ(s, a) E πθ [φ(s, )]

20 Monte-Carlo Policy Gradient Likelihood Ratios Gaussian Policy In continuous action spaces, a Gaussian policy is common E.g., mean is a linear combination of state features µ(s) = φ(s) θ Lets assume variance may be fixed at σ 2 (can be parametrized as well, instead) Policy is Gaussian, a N (µ(s), σ 2 ) The gradient of the log of the policy is then θ log π θ (s, a) = (a µ(s))φ(s) σ 2

21 Monte-Carlo Policy Gradient Policy Gradient Theorem Policy Gradient Theorem The policy gradient theorem generalizes this approach to multi-step MDPs Replaces instantaneous reward R with long-term value q π (s, a) Policy gradient theorem applies to start state objective, average reward and average value objective Theorem For any differentiable policy π θ (s, a), 1 for any of the policy objective functions J = J 1, J avr, or 1 γ J avv, the policy gradient is θ J(θ) = E πθ [ θ log π θ (A S) q πθ (S, A)]

22 Monte-Carlo Policy Gradient Policy Gradient Theorem Monte-Carlo Policy Gradient (REINFORCE) Using return G t = R t+1 + γr t+2 + γ 2 R t as an unbiased sample of q πθ (S t, A t ) Update parameters by stochastic gradient ascent θ t = α θ log π θ (A t S t )G t function REINFORCE Initialise θ arbitrarily for each episode {S 1, A 1, R 2,..., S T 1, A T 1, R T } π θ do for t = 1 to T 1 do θ θ + α θ log π θ (A t S t )G t end for end for return θ end function

23 Monte-Carlo Policy Gradient Policy Gradient Theorem Labyrinth

24 Actor-Critic Policy Gradient Reducing Variance Using a Critic Monte-Carlo policy gradient can have high variance, because we use full return We can use a critic to estimate the action-value function, q w (s, a) q πθ (s, a) Actor-critic algorithms maintain two sets of parameters Critic Updates action-value function parameters w Actor Updates policy parameters θ, in direction suggested by critic One Actor-critic algorithm: follow an approximate policy gradient θ J(θ) E πθ [ θ log π θ (s, a) q w (s, a)] θ = α θ log π θ (s, a) q w (s, a)

25 Actor-Critic Policy Gradient Estimating the Action-Value Function The critic is solving a familiar problem: policy evaluation What is the value of policy π θ for current parameters θ? This problem was explored in previous lectures, e.g. Monte-Carlo policy evaluation Temporal-Difference learning TD(λ)

26 Actor-Critic Policy Gradient Action-Value Actor-Critic Simple actor-critic algorithm based on action-value critic Using linear value fn approx. q w (s, a) = φ(s, a) w Critic Updates w by linear Sarsa(0) Actor Updates θ by policy gradient function QAC Initialise s, θ Sample a π θ for each step do Sample reward R = R a s ; sample transition S PS, A Sample action A π θ (S ) w w + β(r + γq w (S, A ) q w (S, A))φ(S, A) θ = θ + α θ log π θ (s, a)q w (s, a) a a, s s end for end function [Sarsa(0)] [Policy gradient update]

27 Actor-Critic Policy Gradient Compatible Function Approximation Bias in Actor-Critic Algorithms Approximating the policy gradient introduces bias A biased policy gradient may not find the right solution e.g. if q w (s, a) uses aliased features, can we solve gridworld example? Luckily, if we choose value function approximation carefully: we can avoid introducing any bias i.e. we can still follow the exact policy gradient

28 Actor-Critic Policy Gradient Compatible Function Approximation Compatible Function Approximation Theorem (Compatible Function Approximation Theorem) If the following two conditions are satisfied: 1 Value function approximator is compatible to the policy w q w (s, a) = θ log π θ (s, a) 2 Value function parameters w minimise the mean-squared error [ ε = E πθ (qπθ (S, A) q w (S, A)) 2] Then the policy gradient is exact, θ J(θ) = E πθ [ θ log π θ (A S) q w (S, A)]

29 Actor-Critic Policy Gradient Compatible Function Approximation Proof of Compatible Function Approximation Theorem If w is chosen to minimise mean-squared error, gradient of ε w.r.t. w must be zero, w ε = 0 E πθ [(q πθ (S, A) q w (S, A)) w q w (S, A)] = 0 E πθ [(q πθ (S, A) q w (S, A)) θ log π θ (A S)] = 0 E πθ [q πθ (S, A) θ log π θ (A S)] = E πθ [q w (S, A) θ log π θ (A S)] So q w (s, a) can be substituted directly into the policy gradient, θ J(θ) = E πθ [ θ log π θ (s, a)q w (s, a)]

30 Actor-Critic Policy Gradient Compatible Function Approximation Compatible Function Approximation 1 Value function approximator is compatible to the policy w q w (s, a) = θ log π θ (s, a) 2 Value function parameters w minimise the mean-squared error [ ε = E πθ (qπθ (S, A) q w (S, A)) 2] Unfortunately, 1) only holds for certain function classes Unfortunately, 2) is never quite true These are typical limitations of theory; be aware of the assumptions

31 Actor-Critic Policy Gradient Advantage Function Critic Reducing Variance Using a Baseline We subtract a baseline function b(s) from the policy gradient This can reduce variance, without changing expectation E πθ [ θ log π θ (s, a)b(s)] = d πθ (s) θ π θ (s, a)b(s) s S a = d πθ b(s) θ π θ (s, a) s S a A = 0 So we can rewrite the policy gradient as θ J(θ) = E πθ [ θ log π θ (s, a) (q πθ (s, a) b(s))]

32 Actor-Critic Policy Gradient Advantage Function Critic Estimating the Advantage Function (1) The advantage function can significantly reduce variance of policy gradient So the critic should really estimate the advantage function For example, by estimating both v πθ (s) and q πθ (s, a) Using two function approximators and two parameter vectors, v ξ (s) v πθ (s) q w (s, a) q πθ (s, a) A(s, a) = q w (s, a) v ξ (s) And updating both value functions by e.g. TD learning

33 Actor-Critic Policy Gradient Advantage Function Critic Estimating the Advantage Function (2) For the true value function v πθ (s), the TD error δ π θ δ π θ = r + γv πθ (s ) v πθ (s) is an unbiased estimate of the advantage function E πθ [δ π θ s, a] = E πθ [R t+1 + γv πθ (S t+1 ) s, a] v πθ (s) = q πθ (s, a) v πθ (s) = A πθ (s, a) So we can use the TD error to compute the policy gradient θ J(θ) = E πθ [ θ log π θ (s, a) δ π θ ] In practice we can use an approximate TD error δ v = r + γv ξ (s ) v ξ (s) This approach only requires one set of critic parameters ξ

34 Actor-Critic Policy Gradient Eligibility Traces Critics at Different Time-Scales Critic can estimate value function v ξ (s) from many targets at different time-scales From last lecture... For MC, the target is the return G t θ = α(g t v ξ (s))φ(s) For TD(0), the target is the TD target r + γv ξ (s ) θ = α(r t+1 + γv ξ (S t+1 ) v ξ (s))φ(s) For forward-view TD(λ), the target is the λ-return G λ t θ = α(g λ t v ξ (s))φ(s) For backward-view TD(λ), we use eligibility traces δ t = R t+1 + γv ξ (s t+1 ) v ξ (s t ) e t = γλe t 1 + φ(s t ) θ = αδ t e t

35 Actor-Critic Policy Gradient Eligibility Traces Actors at Different Time-Scales The policy gradient can also be estimated at many time-scales θ J(θ) = E πθ [ θ log π θ (s, a) A πθ (s, a)] Monte-Carlo policy gradient uses error from complete return θ = α(g t v ξ (s t )) θ log π θ (s t, a t ) Actor-critic policy gradient uses the one-step TD error θ = α(r t+1 + γv ξ (s t+1 ) v ξ (s t )) θ log π θ (s t, a t )

36 Actor-Critic Policy Gradient Eligibility Traces Policy Gradient with Eligibility Traces Just like forward-view TD(λ), we can mix over time-scales θ = α(v λ t v ξ (s t )) θ log π θ (s t, a t ) Like backward-view TD(λ), we can also use eligibility traces By equivalence with TD(λ), substituting φ(s) = θ log π θ (s, a) δ t = R t+1 + γv ξ (S t+1 ) v ξ (S t ) e t+1 = γλe t + θ log π θ (A t S t ) θ = αδ t e t This update can be applied online, to incomplete sequences

37 Actor-Critic Policy Gradient Natural Policy Gradient Alternative Policy Gradient Directions Gradient ascent algorithms can follow any ascent direction A good ascent direction can significantly speed convergence Also, a policy can often be reparametrised without changing action probabilities For example, increasing score of all actions in a softmax policy The vanilla gradient is sensitive to these reparametrisations

38 Actor-Critic Policy Gradient Natural Policy Gradient Natural Policy Gradient The natural policy gradient is parametrisation independent It finds direction that maximally ascends objective function, when changing policy by a small, fixed amount nat θ π θ (s, a) = G 1 θ θ π θ (s, a) where G θ is the Fisher information matrix G θ = E πθ [ θ log π θ (s, a) θ log π θ (s, a) T ]

39 Actor-Critic Policy Gradient Natural Policy Gradient Natural Actor-Critic Using compatible linear function approximation, φ(s, a) = θ log π θ (s, a) So the natural policy gradient simplifies, θ J(θ) = E πθ [ θ log π θ (s, a)a πθ (s, a)] ] = E πθ [ θ log π θ (s, a)φ(s, a) T w = G θ w nat θ J(θ)= w = E πθ [ θ log π θ (s, a) θ log π θ (s, a) T ] w i.e. update actor parameters in direction of critic parameters

40 robot controlled by this CPG goes straight during ω is ω 0,butturnswhenanimpulsive Lecture 8: Policy Gradient input is added at a certain instance; the turning angle depends on both the CPG s internal Actor-Critic Policy Gradient state Λ and the intensity of the impulse. An action is to select an input out of these two Snake example candidates, and the controller represents the action selection probability which is adjusted by RL. Natural To carry Actor out RLCritic with the in CPG, Snake we employdomain the CPG-actor-critic model [24], in which the physical system (robot) and the CPG are regarded as a single dynamical system, called the CPG-coupled system, and the control signal for the CPG-coupled system is optimized by RL. (a) Crank course (b) Sensor setting Figure 2: Crank course and sensor setting

41 Actor-Critic Policy Gradient Snake example Natural Actor Critic in Snake Domain (2) θi b,therobotsoonhitthewallandgotstuck(fig.3(a)). Incontrast,thepolicyparameter after learning, θi a,allowedtherobottopassthroughthecrankcourserepeatedly(fig.3(b)). This result shows a good controller was acquired by our off-nac Pisition y (m) Pisition y (m) Position x (m) (a) Before learning Position x (m) (b) After learning Figure 3: Behaviors of snake-like robot

42 Actor-Critic Policy Gradient Snake example Summary of Policy Gradient Algorithms The policy gradient has many equivalent forms θ J(θ) = E πθ [ θ log π θ (A S) G t ] REINFORCE = E πθ [ θ log π θ (A S) q w (S, A)] Q Actor-Critic = E πθ [ θ log π θ (A S) A w (S, A)] Advantage Actor-Critic = E πθ [ θ log π θ (A S) δ t ] TD Actor-Critic = E πθ [δ t e t ] TD(λ) Actor-Critic G 1 θ θ J(θ) = w (linear) Natural Actor-Critic Each leads a stochastic gradient ascent algorithm Critic uses policy evaluation (e.g. MC or TD learning) to estimate q π (s, a), A π (s, a) or v π (s)