Olivier Sigaud. September 21, 2012

Transcription

1 Supervised and Reinforcement Learning Tools for Motor Learning Models Olivier Sigaud Université Pierre et Marie Curie - Paris 6 September 21, / 64

2 Introduction Who is speaking? 2 / 64

3 Introduction Two main loops Interaction between cortico-basal loop and cortico-cerebellar loop 3 / 64

4 Introduction Messages of the tutorial A tutorial for a life sciences audience (Not so) dierent learning mechanisms Global view of Reinforcement Learning approaches Understanding recent Actor-Critic advances As few equations as possible......at least in the beginning! If you are interested, you should read : Reinforcement Learning : an Introduction, Sutton and Barto (1998) MIT Press. 4 / 64

5 Introduction Plan Introduction (Not so) dierent learning mechanisms Dynamic programming Reinforcement learning Direct (Model-free) methods: TD learning Action-Value Function Approaches Model-based reinforcement learning Actor-Critic approaches Naive gradient method Modern Actor-Critic Natural gradient NAC versus INAC 5 / 64

6 (Not so) dierent learning mechanisms Associative learning Finding correlations between data The agent builds a model based on data Independent from performance Typical mechanism : Hebb's rule 6 / 64

7 (Not so) dierent learning mechanisms Supervised learning The supervisor indicates to the agent the expected answer The agent corrects a model based on the answer Typical mechanism : gradient backpropagation, RLS Applications : classication, regression, function approximation 7 / 64

8 (Not so) dierent learning mechanisms Self-supervised learning When an agent learns to predict, it proposes its prediction The environment provides the correct answer : next state Supervised learning without supervisor Dicult to distinguish from associative learning 8 / 64

9 (Not so) dierent learning mechanisms Equivalence associative / self-supervised learning Example : LS, RLS Hotly debated issue : does the cerebellum perform supervised learning? Can the Inferior Olive convey an error signal? From outside, both associative and self-supervised learning can explain the same facts With population coding, both mechanisms could be undistinguishable... 9 / 64

10 (Not so) dierent learning mechanisms Reinforcement learning The environment provides the value of action (reward, penalty) Application : behaviour optimisation 10 / 64

11 (Not so) dierent learning mechanisms Reinforcement learning In practice, the value signal is given as a scalar How good is ? Necessity of exploration 11 / 64

12 (Not so) dierent learning mechanisms The exploration/exploitation trade-o Exploring can be (very) harmful Shall I exploit what I know or look for a better policy? Am I optimal? Shall I keep exploring or stop? Decrease the rate of exploration along time ɛ-greedy : take the best action most of the time, and a random action from time to time 12 / 64

13 (Not so) dierent learning mechanisms Temporal Dierence learning in dopamine neurons Udapte of ring rate depending on received reward If the reward is bigger than expected, dopamine neurons re more, If the reward is well predicted, standard level If the reward is smaller than expected, dopamine neurons re less That's not so dierent from a self-supervised learning correction!?!? Not so easy to distinguish learning processes Not so easy to understand the roles of Cerebellum/BG Is there any general underlying (associative) mechanism? 13 / 64

14 Dynamic programming Markov Decision Processes S : states space A : action space T : S A Π(S) : transition function r : S A IR : reward function An MDP denes s t+1 and r t+1 as f (s t, a t) It describes a problem, not a solution 14 / 64

15 Dynamic programming Partial observability/perceptual aliazing Markov property : probability of next state does not depend on history Seldom veried in practice But algorithms tend to be robust 15 / 64

16 Dynamic programming Policy and value functions Goal of planning (and learning) : nd a policy π : S A maximising the agregation of reward on the long run The value function V π : S IR records the agregation of reward on the long run for each state (following policy π) The action value function Q π : S A IR records the agregation of reward on the long run for doing each action in each state (and then following policy π) The computation of value functions assumes the choice of an agregation criterion (discounted, average, etc.) 16 / 64

17 Dynamic programming Bellman equation Given the discounted reward agregation criterion : V (s 0) = r 0 + γv (s 1) Deterministic π : V π (s) = r(s, π(s)) + γ s p(s s, π(s))v π (s ) (1) 17 / 64

18 Dynamic programming Dynamic Programming Looking for an optimal policy boils down to nding an optimal (action-)value function Done through Dynamic Programming (DP) methods : Value Iteration and Policy Iteration They consist in propagating values over the state space So as to determine an optimal policy The optimal (highest) value function corresponds to the optimal policy 18 / 64

19 Dynamic programming Value Iteration in practice R 19 / 64

20 Dynamic programming Value Iteration in practice R 19 / 64

21 Dynamic programming Value Iteration in practice R 19 / 64

22 Dynamic programming Value Iteration in practice R 19 / 64

23 Dynamic programming Value Iteration in practice 19 / 64

24 Dynamic programming Policy Iteration in practice 20 / 64

25 Dynamic programming Policy Iteration in practice 20 / 64

26 Dynamic programming Policy Iteration in practice 20 / 64

27 Dynamic programming Policy Iteration in practice 20 / 64

28 Dynamic programming Policy Iteration in practice 20 / 64

29 Dynamic programming Policy Iteration in practice 20 / 64

30 Dynamic programming Policy Iteration in practice 20 / 64

31 Dynamic programming Policy Iteration in practice 20 / 64

32 Reinforcement learning Reinforcement learning In DP (planning), T and r are given Reinforcement learning goal : build π without knowing T and r Model-free approach : build π without estimating T nor r Actor-critic approach : special case of model-free Model-based approach : build a model of T and r and apply DP 21 / 64

33 Reinforcement learning Direct (Model-free) methods : TD learning Incremental estimation Estimating the average immediate (stochastic) reward in a state s E k (s) = (r 1 + r r k )/k E k+1 (s) = (r 1 + r r k + r k+1 )/(k + 1) Thus E k+1 (s) = k/(k + 1)E k (s) + r k+1 /(k + 1) Or E k+1 (s) = (k + 1)/(k + 1)E k (s) E k (s)/(k + 1) + r k+1 /(k + 1) Or E k+1 (s) = E k (s) + 1/(k + 1)[r k+1 E k (s)] Still needs to store k. Can be approximated as E k+1 (s) = E k (s) + α[r k+1 E k (s)] Will converge to the true average (slower or faster depending on α) without storing anything The formula is everywhere in reinforcement learning 22 / 64

34 Reinforcement learning Direct (Model-free) methods : TD learning Monte Carlo methods Much used in games (Go...) to evaluate a state Generate a lot of trajectories : s 0, s 1,..., s N with observed rewards r 0, r 1,..., r N Update state values V (s k ), k = 0,..., N 1 with : V (s k ) V (s k ) + α(s k )(r k + r k r N V (s k )) It uses the average estimation method from previous slide 23 / 64

35 Reinforcement learning Direct (Model-free) methods : TD learning Temporal Dierence error The goal of TD methods is to estimate the value function V (s) If estimations V (s t) and V (s t+1) were exact, we would get : V (s t) = r t + γr t+1 + γ 2 r t+2 + γ 3 r t V (s t+1) = r t+1 + γ(r t+2 + γ 2 r t Thus V (s t) = r t + γv (s t+1) δ k = r k + γv (s k+1 ) V (s k ) : measures the error between current values of V and the values they should have 24 / 64

36 Reinforcement learning Direct (Model-free) methods : TD learning Temporal Dierence learning in dopamine neurons Udapte rule of the V depending on received reward If r k is bigger than expected, δ > 0 increase V If r k is well predicted, δ = 0 If r k is smaller than expected, δ < 0 decrease V Temporal Dierence method : correct the error by tuning V (s t) with a Widrow-Ho equation : V k (s) V k (s) + α k δ k Bootstrap method : a better estimate from a previous one 25 / 64

37 Reinforcement learning Direct (Model-free) methods : TD learning Temporal Dierence (TD) Methods Temporal Dierence (TD) methods combine the properties of DP methods and Monte Carlo methods : in Monte Carlo, T and r are unknown, but the value update is global, a trajectory is needed in DP, T and r are known, but the value update is local TD : as in DP, V (s t) is updated locally given an estimate of V (s t+1) and T and r are unknown. Note : Monte Carlo can be reformulated incrementally using the temporal dierence δ k udpate 26 / 64

38 Reinforcement learning Direct (Model-free) methods : TD learning TD(0) V (s t) V (s t) + α[r t+1 + γv (s t+1) V (s t)] Combines the temporal dierence udpate (propagation from V (s t+1) to V (s t)) from DP and the incremental estimation method from Monte Carlo. Updates are local from s t, s t+1 and r t+1. Proof of convergence : [?] 27 / 64

39 Reinforcement learning Direct (Model-free) methods : TD learning TD(0) : limitation TD(0) evaluates V (s) One cannot infer π(s) from V (s) without knowing T : one must know which a leads to the best V (s ). Three solutions : Work with Q(s, a) rather than V (s). Learn a model of T : model-based (or indirect) reinforcement learning Actor-critic methods (simultaneously learn V and update π) 28 / 64

40 Reinforcement learning Action-Value Function Approaches Sarsa Action-Value Function Q π : S A IR so that, for each (s, a) S A, Q π (s, a) = r(s, a) + γ s p(s s, a)v π (s ) We have V (s) = max a A Q (s, a) Perform exploration (e.g. ɛ-greedy) For each observed (s t, a t, r t+1, s t+1, a t+1) : Q(s t, a t) Q(s t, a t) + α[r t+1 + γq(s t+1, a t+1) Q(s t, a t)] One must know the action a t+1, thus constrains exploration On-Policy method : more complex convergence proof [?] 29 / 64

41 Reinforcement learning Action-Value Function Approaches Q-Learning For each observed (s t, a t, r t+1, s t+1) : Q(s t, a t) Q(s t, a t) + α[r t+1 + γ max Q(s t+1, a) Q(s t, a t)] a A max a A Q(s t+1, a) instead of Q(s t+1, a t+1) O-Policy method : no more need to know a t+1 Convergence proved provided innite exploration [?] 30 / 64

42 Reinforcement learning Action-Value Function Approaches Q-Learning in practice (Q-learning video) Build a states actions table (Q-Table, eventually incremental) Initialise it (randomly or with 0 is not a good choice) Apply update equation after each action Problem : it is (very) slow [?] : action-specic reward values in the striatum (Science) 31 / 64

43 Reinforcement learning Model-based reinforcement learning Eligibility traces To improve over Q-learning Naive approach : store all (s, a) pair and back-propagate values Limited to nite horizon trajectories Speed/memory trade-o TD(λ), sarsa (λ) and Q(λ) : more sophisticated approach to deal with innite horizon trajectories A variable e(s) is decayed with a factor λ after s was visited and reinitialized each time s is visited again TD(λ) : V (s) V (s) + αδe(s), (similar for sarsa (λ) and Q(λ)), If λ = 0, e(s) goes to 0 immediately, thus we get TD(0), sarsa or Q-learning TD(1) = Monte-Carlo / 64

44 Reinforcement learning Model-based reinforcement learning Dyna architecture : ideas (Dyna-like video) General idea : planning with a learnt model is performing back-ups in the agent's head ([?,?]) Learn a model of T and r Use Policy Iteration (Dyna-PI) or Q-learning (Dyna-Q) to get V or Q There is also Dyna-AC 33 / 64

45 Reinforcement learning Model-based reinforcement learning Dyna architecture : biological background Learning T and r is self-supervised learning In the continuous case, need for function approximator LWPR, gaussian processes, RBF... [?] : combining model-free and model-based 34 / 64

46 Actor-Critic approaches Continuous action problems Until (very) recently, there was very few work on continuous action problems, though reinforcement learning (partially) comes from optimal control Problem : in standard TD methods, one must perform max over actions (several times), (optimisation problem when action is continuous) Gradient methods and Actor-Critic methods reduce the problem, they look for a local optimum (Pontryagine method rather than Bellman method) They get very popular nowadays 35 / 64

47 Actor-Critic approaches Convergence problems Continuous states and actions cannot be enumerated function approximation Combining function approximation and greedy methods based on action-value can diverge easily ([?,?]...) [?] : algorithms based on the advantage function A π (s, a) = Q π (s, a) V π (s) converge more robustly Since V π (s) = max aq π (s, a), A π (s, a) is a negative term equivalent to a regret 36 / 64

48 Actor-Critic approaches Biological background Basal ganglia might be organised into an Actor-Critic architecture 37 / 64

49 Actor-Critic approaches From Q(s, a) to Actor-Critic (1) state / action a 0 a 1 a 2 a 3 e e e e e e In Q learning, given a Q Table, one must determine the max at each step This becomes expensive if there are numerous actions 38 / 64

50 Actor-Critic approaches From Q(s, a) to Actor-Critic (2) state / action a 0 a 1 a 2 a 3 e * e * 0.43 e * 0.73 e * 0.81 e * e * One can store the best value for each state Then one can update the max by just comparing the changed value and the max No more maximum over actions (only in one case) 39 / 64

51 Actor-Critic approaches From Q(s, a) to Actor-Critic (3) state / action a 0 a 1 a 2 a 3 e * e * 0.43 e * 0.73 e * 0.81 e * e * state chosen action e 0 a 1 e 1 a 2 e 2 a 2 e 3 a 2 e 4 a 1 e 5 a 1 Store the max is equivalent to store the policy by the side of the Q Table Update the policy as a function of value updates Basic actor-critic scheme 40 / 64

52 Actor-Critic approaches Dynamic Programming and Actor-Critic (1) In both PI and AC, the architecture contains a representation of the value function (the critic) and the policy (the actor) In PI, the environment (T and r) is known PI alternates two stages : 1. Policy evaluation : update (V (s)) or (Q(s, a)) given the current policy 2. Policy improvement : follow the value gradient 41 / 64

53 Actor-Critic approaches Dynamic Programming and Actor-Critic (2) In AC, T and r are unknown, but not represented (model-free) Information from the environment generates update in the critic, then in the actor 42 / 64

54 Actor-Critic approaches Successive contributions 43 / 64

55 Actor-Critic approaches Naive design Discrete states and actions, stochastic policy An update in the critic generates a local update in the actor Critic : compute δ and update V (s) with V k (s) V k (s) + α k δ k Actor : P π (a s) = P π (a s) + α k δ k NB : no need for a max over actions NB2 : one must then know how to draw an action from a probabilistic policy (not obvious for continuous actions) 44 / 64

56 Actor-Critic approaches Naive gradient method Gradient methods : Parametrized policies We get stochastic policies parametrized by a vector θ, noted π θ (a s) Look for the gradient θ of the long term performance J(π θ ) J is written dierently depending on the criterion : average reward, discounted, etc. The average reward criterion gives rise to clearest prooves nowadays We want to maximize J(π θ ) with a gradient ascent The gradient is estimated with samples collected along trajectories We note θ = J(π θ(a s)) θ 45 / 64

57 Actor-Critic approaches Naive gradient method Naive gradient computation The long term performance J θ in any state is unkown Naive approach : in a state, perform trajectories from that state with dierent θ, estimate performance, then its gradient wrt θ (REINFORCE) Very expensive and slow Given that V π θ (or Q π θ ) summarizes performance of π θ in all states Can we use estimates of V π θ or Q π θ to compute the gradient? [?] shows that yes, we can : value-based methods can be as good as REINFORCE with much fewer samples 46 / 64

58 Actor-Critic approaches Modern Actor-Critic An estimation problem Estimating : approximating a function thanks to a model, some samples, and a method Variance : variability of estimation as a function of samples Bias : error on estimate after convergence due to method (sampling, etc.) due to the model (structure or parameters) 47 / 64

59 Actor-Critic approaches Modern Actor-Critic Bias/variance trade-o Poor model : low variance, large bias Rich model : low bias, large variance Bias/variance trade-o : tune parameters : more parameters = less bias, more variance If we get few samples, better estimation accepting more bias to prevent variance To prove convergence : ensure null bias, then minimise variance 48 / 64

60 Actor-Critic approaches Modern Actor-Critic Gradient computation with value function (1) For a stochastic policy in the discrete case, one shows ([?], theorem 1) that the average reward gradient is written : θ J(π θ ) = s S d π θ (s) a A θ π θ (a s)q π θ (s, a) Where d π θ (s) represents the probability density of being in s following policy π θ in stationary mode d π θ (s) corresponds to sampling peformed by the policy Similar formula for discounted reward case 49 / 64

61 Actor-Critic approaches Modern Actor-Critic Gradient computation with value function (2) We have θ J(π θ ) = s S d π θ (s) a A θπ θ (a s)q π θ (s, a) Thus the gradient can be computed as a function of Q π θ (s, a) Problem : the exact Q π θ (s, a) is unknown, it must be estimated with a parametric representation We set f ω(s, a) = ˆQπ θ ω (s, a) Question : under which condition on f ω(s, a) as a function of π θ (s, a) do we get an unbiased estimation of Q π θ (s, a)? 50 / 64

62 Actor-Critic approaches Modern Actor-Critic Compatible functions It is shown that ([?], theorem 2) such a condition is fω(s,a) = 1 ω π θ (a s) θπ θ (a s) = θ logπ θ (a s) The gradient is given by : θ J(π θ ) = s S d π (s) a A fω(s, a) θ π θ (a s) ω 51 / 64

63 Actor-Critic approaches Modern Actor-Critic Compatibility condition For a policy π θ (s, a) The gradient is given by : θ J(π θ ) = s S d π θ (s) a A fω(s, a) θ π θ (a s) ω is unbiased if fω(s,a) ω = 1 π θ (a s) θπ θ (a s) = θ logπ θ (a s) Thus if f ω(s, a) = ω T θ logπ θ (a s) Proof in [?] The relation must be ensured between the basis functions used in the actor and the critic 52 / 64

64 Actor-Critic approaches Modern Actor-Critic Baseline We have θ J(π θ ) = d π (s) s S a A θπ θ (a s)q π θ (s, a) Introduce a baseline b(s) in the computation of θ J(π θ ) It is proven that, for any b(s), θ J(π θ ) = d π θ (s) θ π θ (a s)(q π θ (s, a) b(s)) s S a A Look for b(s) that most reduces the variance 53 / 64

65 Actor-Critic approaches Modern Actor-Critic Optimal baseline The baseline with the least variance is V π θ (s) ([?], lemma 2) Thus one should compute θ J(π θ ) = d π θ (s) θ π θ (a s)(q π θ (s, a) V π θ (s)) s S a A Thus the best critic (Q compatible approximator) estimates the advantage function from [?] : A π (s, a) = Q π (s, a) V π (s) 54 / 64

66 Actor-Critic approaches Modern Actor-Critic Reminder : Successive contributions 55 / 64

67 Actor-Critic approaches Natural gradient Natural gradient denition A particular parametrisation induces a particular way to explore a sub-space of stochastic policies A parametrisation induces a variety in the space of stochastic policies If the variety is not euclidean, the gradient may not follow the steepest descent direction (another parametrisation would induce another direction) The natural gradient ensures that the steepest descent direction is followed whatever the parametrisation 56 / 64

68 Actor-Critic approaches Natural gradient Natural gradient computation ˆ θ J θ (π θ ) = F 1 (θ) θ J θ (π θ ) F is the Fischer Information matrix Besides, θ J θ (π θ ) = E[ θ logπ θ logπ T ]ω = F (θ)ω Thus ˆ θ J θ (π θ ) = F 1 (θ)f (θ)ω = ω Thus, to update the policy gradient, one can use θ t+1 = θ t + α tω t where 57 / 64

69 Actor-Critic approaches NAC versus INAC NAC based on (LSTD-Q(λ)) ω t is an estimator of A π [?] : one cannot estimate A π thanks to a bootstrap method Use V π (s) = v T f (s) where f (s) are state features Estimate jointly ω t and v t with a least square method Add features f (s) on state s 58 / 64

70 Actor-Critic approaches NAC versus INAC enac (simplication) NAC needs n trajectories to solve a n linear equations system Use of a forgetting factor not to reinitialize everything at each step You don't need to dene value function features Problem : not incremental, not biologically plausible 59 / 64

71 Actor-Critic approaches NAC versus INAC INAC [?] found a way to estimate ω t and v t with a bootstrap method As in NAC, we use V π (s) = v T f (s) where f (s) are state features We use δ t = r t+1 Ĵ t+1 + ˆVt+1 ˆVt Estimated with δ t = r t+1 Ĵ t+1 + v T t f (s t+1) v T t f (s t) v t is updated with temporal dierence : v t+1 = v t + α tδ tf (s t) 60 / 64

72 Actor-Critic approaches NAC versus INAC Natural gradient INAC There are four INAC update equations One is based on the natural gradient, requires less computation Critic update : 1. v t+1 = v t + α tδ t f (s t) 2. ω t+1 = [I α tψ(s t, a t)ψ(s t, a t) T ]ω t + α tδ tψ(s t, a t) Actor update : θ t+1 = θ t + β tω t+1 NB : β must converge to 0 faster than α Diculties : 1. Finding basis functions ensuring compatibility condition for continuous action problems 2. Tuning the parameters is horrible Late breaking news : enac is much more easy to tune 61 / 64

73 Actor-Critic approaches NAC versus INAC Model-based Incremental Natural Actor-Critic 62 / 64

74 Actor-Critic approaches NAC versus INAC Messages Distinguishing learning mechanisms may be misleading Actor-critic methods are ecient for motor control modelling They seem to be biologically plausible They are not model-based Much progress in maths since naive early proposal (NAC and INAC) That progress did not propagate to biological modelling There is room for model-based actor-critic architectures 63 / 64

75 Actor-Critic approaches NAC versus INAC Future directions Look for neurophysiological counterparts of recent actor-critic mechanisms See how this actor-critic architecture can be embedded into a wider motor learning model (attend my CMCW talk on thursday) Towards a Kalman lter based actor-critic architecture (Geist et al. 2009) 64 / 64