Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associate

Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associated to the value Teaching signal : difference between what you get and what you think you should get

Exploration-Exploitation Dilemma Possible choices (actions) of a bee: land on a blue or yellow flower

Bee searching for nectar Possible choices (actions) of the bee: land on a blue or yellow flower rewards (in drops of nectar) r b =8 r y =

What does the bee know? actual reward internal estimate r b m b r y m y

How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i last reward on a blue flower last reward on a yellow flower

How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward r b,i 5 1 15 trial number 8 reward r y,i r y,i 5 1 i 15 trial number

How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: 5 1 15 trial number 8 reward m y r y,i 5 1 15 trial number i

How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: 5 1 15 trial number reward 8 5 1 15 trial number i

How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i batch update: N N m b = 1 N r b,i m y = 1 N r y,i i=1 i=1 average reward on last N visits to a blue flower average reward on last N visits to a yellow flower

How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i batch update: N N m b = 1 N i=1 r b,i m y = 1 N i=1 r y,i online update: m b m b + ɛ(r b,i m b ) delta - rule m b m b + ɛδ } learning rate with δ = r b,i m b } prediction error

How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: 5 1 15 trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward 5 1 15 trial number i

How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward Idea: the bee could use its estimates about the flower rewards to change its policy! 5 1 15 trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward 5 1 15 trial number i

Changing the policy online p(b) = p(y) = exp(βm b ) exp(βm b ) + exp(βm y ) exp(βm y ) exp(βm b + exp(βm y ) actual reward internal estimate r b m b r y m y

Changing the policy online p(b) = p(y) = exp(βm b ) exp(βm b ) + exp(βm y ) exp(βm y ) exp(βm b + exp(βm y ) probability 1.8.6.4 if m b >m y p(b). p(y) 4 6 8 beta exploration exploitation

Learning the nectar reward: exploration-exploitation trade-off 1 softmax Gibbs-policy with β = (5/5) policy actual average reward estimated average reward 8 6 4 1 3 4 5 1 8 6 4 1 3 4 5 time steps r y r b m y m b

Learning the nectar reward: exploration-exploitation trade-off 1 actual average reward 8 6 4 r y r b softmax Gibbs-policy with β =.5 estimated average reward 1 3 4 5 1 8 6 4 1 3 4 5 time steps m y m b

Learning the nectar reward: exploration-exploitation trade-off 1 actual average reward 8 6 4 r y r b softmax Gibbs-policy with β = estimated average reward 1 3 4 5 1 8 6 4 1 3 4 5 time steps m y m b

Exploration-exploitation tradeoff average reward 55 5 45 softmax Gibbs-policy 4.5 1 1.5.5 3 beta

Real, 1991, Science 53:98-986 What do real bees do? bumblebees (N=5) solid line: blue=constantly rewarded yellow=randomly rewarded switch at trial 17! blue=randomly rewarded yellow=constantly rewarded switch dashed line: less variability on randomly rewarded flowers

Reinforcement Learning Model-free approaches Cogmaster CO6 / Boris Gutkin

A Brief History Psychology of Animal Learning Edward Thorndike (1874-1949)

A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike (1874-1949) Richard Bellman (19-1984)

A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike (1874-1949) Richard Bellman (19-1984) Artificial Intelligence Marvin Minsky (197-???) Harry Klopf (197-???)

A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike (1874-1949) Richard Bellman (19-1984) Reinforcement Learning Artificial Intelligence Richard Sutton (1956-???) Andrew Barto (1948-???) Marvin Minsky (197-???) Harry Klopf (197-???)

Conditioning stimulus reward Trial 1: Trial : Trial 3: Trial 4:...

Bee foraging stimulus action reward Trial 1: Trial : Trial 3: Trial 4:...

More complex tasks? stimulus action stimulus action reward Trial 1: Trial : Trial 3: zzz... Trial 4:...

Example: Rat in a maze D E F G 5 Rewards B C A

The credit assignment problem D E F G 5 Rewards good choice B C good choice A

The credit assignment problem D E F G 5 Rewards bad choice B C good choice A even though there was no reward, the first choice was a good one!

Formalizing the problem D E F G Possible choices (actions a) of 5 the rat: move left or right B C States: s = {A,B,C,D,E,F,G} A Rewards: Actions: r(s) a = {left, right} Policy: p(a s) depends on the state!

The value function D E F G Define: 5 B C Value of a state V(s) = sum of all future rewards A Note! value of a state depends on future actions!!!

If the maze is known... D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A

... and the policy is given... D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E)

... then so are the values of the states D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G)

Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)

Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)

Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =1 V (A) =p(a=left A)V (B) +p(a=right A)V (C)

Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =1 V (A) =1.75

If the value function is known... p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 V (A) =1.75

... the policy can be improved p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 Policy improvement: make policy more greedy p(a = left A) p(a = right A)

... the policy can be improved D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4

... the policy can be improved... D E F G Value of a state = 5 sum of all future rewards B C V (E) =5 V (F )= V (G) = A V (D) = Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4 p(a = left B) =.4 p(a = right B) =.6

... which in turn... D E F G Value of a state = 5 sum of all future rewards B C V (E) =5 V (F )= V (G) = A V (D) = Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4 p(a = left B) =.4 p(a = right B) =.6

... changes the value function New policy: D E F G p(a = left A) =.6 5 p(a = right A) =.4 B C p(a = left B) =.4 p(a = right B) =.6 A p(a = left C) =.6 p(a = right C) =.4 New value function: V (B) =3 V (C) =1. V (A) =.8

Iteration yields optimal policy Optimal policy: D E F G p(a = left A) =.6 1 5 p(a = right A) =.4 B C p(a = left B) =.4 p(a = right B) =.6 1 A p(a = left C) = 1.6 p(a = right C) =.4 New value function: V (B) =5 V (C) = V (A) = 5

Conclusions so far if we know the environment (we know where to find rewards r as a function of the states s), and have a policy, then we can compute the value function (policy evaluation) with the value function, we can improve the policy (policy improvement) iteration of procedure yields the optimal policy

What if the environment is unknown? state H: rat taken out Assume p(a s) = 5/5 policy D E F G 5 B C A

Sampling the environment... state H: rat taken out Assume p(a s) = 5/5 policy D E F G Generate episodes (trials) B 5 A C ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH -

Sampling the environment... state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH -

... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(A) = 1.9 note difference to the exact value V(A)=1.75 that we computed earlier!

... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - and so on for D,E,F, and G Monte Carlo Policy Evaluation (similar to batch update for bees)

Online learning idea: update value function after each action The animal goes through states a a a s s s s up to final state H ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H

Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H up to final state H

Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H up to final state H

Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H

Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) Rat s prediction error δ = V (s) ˆV (s) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H

Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) Rat s prediction error δ = V (s) ˆV (s) } ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H Problem: can t compute prediction error without V (s)!!!

Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without Solution (trick): use V (s) V (s) =r(s)+v (s )

Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s )

Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s)

Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s) } reward in state s } - (expected reward in state s)

Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s) Temporal-difference (TD)-learning: V (s) V (s)+ɛδ with δ = r(s)+ ˆV (s ) ˆV (s)

Rat in maze: learning V(s) V (A) V (B) V (C) TD-learning: V (s) V (s)+ɛδ δ = r(s )+V (s ) V (s)

Actor-critic learning Critic Policy evaluation learning V(s) Actor Policy improvement learning p(a s)

Actor-critic learning Critic Policy evaluation learning V(s) Actor Policy improvement learning p(a s) e.g. Monte-Carlo-update: TD(1) Temporal-difference-learning: TD() Generalization: TD(λ) e.g. (havent discussed it yet) but it s the same as on the left

Rat in maze: learning p(a s)

Discounting rewards The further a reward lies in the future, the less important it is mathematically convenient good approximation of human and animal behavior

Exponential Discounting The further a reward lies in the future, the less important it is Discounted sum of future rewards ( γ 1 ) V (s) =r(s)+γr(s )+γ r(s )+... Temporal difference rule with discounting: V (s) V (s)+ɛδ δ = r(s)+γ ˆV (s ) ˆV (s)

Secondary conditioning training 1. sound food salivation. clapping sound salivation testing 3. sound salivation 4. clapping salivation

Example : Secondary conditioning Phase 1 training Phase sound food salivation clapping sound salivation Phase 1 C Phase 1 trial start A Phase B Phase D trial end Note: no actions are required in this task!

Example : Secondary conditioning Phase 1 training Phase sound food salivation clapping sound salivation Phase 1 C Phase 1 trial start A Phase B Phase D trial end Initial values: V (A) = V (B) = V (C) =1 V (D) = Assume: the higher the value of a state, the more the dog will salivate when in that state. Therefore: state C has value 1

Example : Secondary conditioning 1 Value V(C).8 value.6.4. 4 6 8 trial no. Phase 1 (4 trials) B C D

Example : Secondary conditioning 1.8 Value V(C) Value V(B) value.6.4. 4 6 8 trial no. Phase 1 (4 trials) B C D

Example : Secondary conditioning value 1.8.6.4. Value V(C) Value V(B) This is conditioning - exactly 4 6 8 trial no. as in the Rescorla-Wagner model! Phase 1 (4 trials) B C D

Example : Secondary conditioning 1.8 Value V(C) Value V(B) value.6.4. Value V(D) 4 6 8 trial no. Phase (4 trials) A B D

Example : Secondary conditioning 1 Value V(C) value.8.6.4 Value V(B). Value V(D) 4 6 8 trial no. Phase (4 trials) A B D

Example : Secondary conditioning 1 Value V(C) value.8.6.4. Value V(C ) Value V(B) This is extinction - exactly 4 6 8 trial no. as in the Rescorla-Wagner model! Phase (4 trials) A B D

Example : Secondary conditioning 1.8 value.6.4 Value V(A) Value V(B) Stimulus A has become valuable. 4 6 8 trial no. Phase (4 trials) A B D

Define a set of states K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E F G H I B C A

Define possible actions in each state K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E F G H I Actions: a = {left, right A} B C a = {left, center, right B} a = {left, center, right C} A

Define rewards in each state K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E B F 6 G 4 H 4 C I 4 Actions: a = {left, right A} a = {left, center, right B} a = {left, center, right C} A Rewards: r(s)

Problem summary D E F G H I 6 4 4 4 B K A C States: s = {A,B,C,D,E,F,G,H,I,K} Actions: a = {left, right A} a = {left, center, right B} a = {left, center, right C} Rewards: r(s)

Define a policy 1 K Policy: (see arrow thickness) p(a s) D E F G H I 6 4 4 4.33 B C.5 A.5 1) all actions equally likely

Define a policy K Policy: (see arrow thickness) p(a s) D E F G H I 6 4 4 4 B C. A.8 1) all actions equally likely ) rightward bias

Define a policy K Policy: (see arrow thickness) p(a s) D E F G H I 6 4 4 4 B C A 1) all actions equally likely ) rightward bias 3) optimal (ε-greedy) policy (but! dont know this beforehand!)

How can we improve a policy? 1 K Policy: (see arrow thickness) p(a s) D E F G H I 6 4 4 4.33 B C.5 A.5 1) all actions equally likely ) rightward bias 3) optimal (ε-greedy) policy (but! dont know this beforehand!)

Solution: define a value-function K D E F G H I 6 4 4 4 Policy: p(a s) Value (sum of all future rewards) V (s) B C A

Solution: define a value-function K D E F G H I 6 4 4 4 Policy: p(a s) Value (sum of all future rewards) V (s) B A C Sequence of actions a s s a s a s Values: V (s) =r(s)+r(s )+... = r(s)+v (s ) but: this is a bit sloppy...

Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N f(x i ) i=1

Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) Caution!! Expected value is a math-term which has nothing to do with the value-function!

Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) important examples: Mean of a distribution: E[x] = µ = x A xp(x) Sample mean: ˆµ = 1 N N i=1 x i

Solution: define a value-function K D E F G H I 6 4 4 4 Policy: p(a s) Value (sum of all future rewards) V (s) B A C Sequence of actions a s s a s a s Value of state s V (s) =r(s)+e[r(s )+...] V (s) =r(s)+e[v (s )] Expected value with respect to the likelihood of all next states dictated by current policy!

Policy evaluation and improvement if we know the environment (we know where to find rewards r as a function of the states s), and have a policy, then we can compute the value function (policy evaluation) with the value function, we can improve the policy (policy improvement) iteration of procedure yields the optimal policy

Policy evaluation K V (K) = Policy: all actions equally likely D E F G H I 6 4 4 4 Evaluation: V (s) =r(s)+e[v (s )] B C start at the end (top) A

Policy evaluation K V (K) = V (D) = V (I) =4 D E F G H I 6 4 4 4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] B C start at the end (top) A

Policy evaluation D E F G H I 6 4 4 4 B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] start at the end (top) A

Policy evaluation D E F G H I 6 4 4 4 B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] start at the end (top) V (A) = 3 A

{ Policy improvement D E F G H I 6 4 4 4 B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Update policy: p(a s) p(a s)+ɛδ Delta-error: δ = q(a s) p(a s) 1: for optimal choice given current value function : otherwise V (A) = 3 A

Policy evaluation K V (K) = V (D) = V (I) =4 D E F G H I 6 4 4 4 B C 4 V (C) V (B) A 1.5 p(c A) p(b A)

Policy improvement + evaluation K V (K) = V (D) = V (I) =4 D E F G H I 6 4 4 4 4 V (C) B C V (B) A 1.5 p(c A) p(b A)

Policy improvement + evaluation D E F G H I 6 4 4 4 B K V (K) = V (D) = V (I) =4 C 4 V (B) V (C) A 1 p(b A).5 p(c A)

Online policy evaluation: TD-learning K D E F G H I 6 4 4 4 Policy given Exact evaluation: V (s) =r(s)+e[v (s )] B C Online evaluation ˆV (s) ˆV (s)+ɛδ A prediction error: δ = r(s)+e[v (s )] ˆV (s) r(s)+ ˆV (s ) ˆV (s) TD-Learning

Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another)

Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s)) What if we don t know the current state (noisy observations)? Solution: Partially-observable Markov-decision-processes