Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associate

Transcription

1

2

3

4

5 Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associated to the value Teaching signal : difference between what you get and what you think you should get

6 Exploration-Exploitation Dilemma Possible choices (actions) of a bee: land on a blue or yellow flower

7 Bee searching for nectar Possible choices (actions) of the bee: land on a blue or yellow flower rewards (in drops of nectar) r b =8 r y =

8 What does the bee know? actual reward internal estimate r b m b r y m y

9 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i last reward on a blue flower last reward on a yellow flower

10 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward r b,i trial number 8 reward r y,i r y,i 5 1 i 15 trial number

11 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i

12 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i

13 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i

14 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i

15 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i

16 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i

17 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i

18 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i

19 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i

20 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i batch update: N N m b = 1 N r b,i m y = 1 N r y,i i=1 i=1 average reward on last N visits to a blue flower average reward on last N visits to a yellow flower

21 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i batch update: N N m b = 1 N i=1 r b,i m y = 1 N i=1 r y,i online update: m b m b + ɛ(r b,i m b ) delta - rule m b m b + ɛδ } learning rate with δ = r b,i m b } prediction error

22 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i

23 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i

24 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i

25 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i

26 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i

27 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward Idea: the bee could use its estimates about the flower rewards to change its policy! trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i

28 Changing the policy online p(b) = p(y) = exp(βm b ) exp(βm b ) + exp(βm y ) exp(βm y ) exp(βm b + exp(βm y ) actual reward internal estimate r b m b r y m y

29 Changing the policy online p(b) = p(y) = exp(βm b ) exp(βm b ) + exp(βm y ) exp(βm y ) exp(βm b + exp(βm y ) probability if m b >m y p(b). p(y) beta exploration exploitation

30 Learning the nectar reward: exploration-exploitation trade-off 1 softmax Gibbs-policy with β = (5/5) policy actual average reward estimated average reward time steps r y r b m y m b

31 Learning the nectar reward: exploration-exploitation trade-off 1 actual average reward r y r b softmax Gibbs-policy with β =.5 estimated average reward time steps m y m b

32 Learning the nectar reward: exploration-exploitation trade-off 1 actual average reward r y r b softmax Gibbs-policy with β = estimated average reward time steps m y m b

33 Exploration-exploitation tradeoff average reward softmax Gibbs-policy beta

34 Real, 1991, Science 53: What do real bees do? bumblebees (N=5) solid line: blue=constantly rewarded yellow=randomly rewarded switch at trial 17! blue=randomly rewarded yellow=constantly rewarded switch dashed line: less variability on randomly rewarded flowers

35 Reinforcement Learning Model-free approaches Cogmaster CO6 / Boris Gutkin

36 A Brief History Psychology of Animal Learning Edward Thorndike ( )

37 A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike ( ) Richard Bellman ( )

38 A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike ( ) Richard Bellman ( ) Artificial Intelligence Marvin Minsky (197-???) Harry Klopf (197-???)

39 A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike ( ) Richard Bellman ( ) Reinforcement Learning Artificial Intelligence Richard Sutton (1956-???) Andrew Barto (1948-???) Marvin Minsky (197-???) Harry Klopf (197-???)

40 Conditioning stimulus reward Trial 1: Trial : Trial 3: Trial 4:...

41 Bee foraging stimulus action reward Trial 1: Trial : Trial 3: Trial 4:...

42 More complex tasks? stimulus action stimulus action reward Trial 1: Trial : Trial 3: zzz... Trial 4:...

43 Example: Rat in a maze D E F G 5 Rewards B C A

44 The credit assignment problem D E F G 5 Rewards good choice B C good choice A

45 The credit assignment problem D E F G 5 Rewards bad choice B C good choice A even though there was no reward, the first choice was a good one!

46 Formalizing the problem D E F G Possible choices (actions a) of 5 the rat: move left or right B C States: s = {A,B,C,D,E,F,G} A Rewards: Actions: r(s) a = {left, right} Policy: p(a s) depends on the state!

47 The value function D E F G Define: 5 B C Value of a state V(s) = sum of all future rewards A Note! value of a state depends on future actions!!!

48 If the maze is known... D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A

49 ... and the policy is given... D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E)

50 ... then so are the values of the states D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G)

51 ... then so are the values of the states D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)

52 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)

53 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)

54 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =1 V (A) =p(a=left A)V (B) +p(a=right A)V (C)

55 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =1 V (A) =1.75

56 If the value function is known... p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 V (A) =1.75

57 ... the policy can be improved p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 Policy improvement: make policy more greedy p(a = left A) p(a = right A)

58 ... the policy can be improved D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4

59 ... the policy can be improved... D E F G Value of a state = 5 sum of all future rewards B C V (E) =5 V (F )= V (G) = A V (D) = Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4 p(a = left B) =.4 p(a = right B) =.6

60 ... which in turn... D E F G Value of a state = 5 sum of all future rewards B C V (E) =5 V (F )= V (G) = A V (D) = Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4 p(a = left B) =.4 p(a = right B) =.6

61 ... changes the value function New policy: D E F G p(a = left A) =.6 5 p(a = right A) =.4 B C p(a = left B) =.4 p(a = right B) =.6 A p(a = left C) =.6 p(a = right C) =.4 New value function: V (B) =3 V (C) =1. V (A) =.8

62 Iteration yields optimal policy Optimal policy: D E F G p(a = left A) = p(a = right A) =.4 B C p(a = left B) =.4 p(a = right B) =.6 1 A p(a = left C) = 1.6 p(a = right C) =.4 New value function: V (B) =5 V (C) = V (A) = 5

63 Conclusions so far if we know the environment (we know where to find rewards r as a function of the states s), and have a policy, then we can compute the value function (policy evaluation) with the value function, we can improve the policy (policy improvement) iteration of procedure yields the optimal policy

64 What if the environment is unknown? state H: rat taken out Assume p(a s) = 5/5 policy D E F G 5 B C A

65 Sampling the environment... state H: rat taken out Assume p(a s) = 5/5 policy D E F G Generate episodes (trials) B 5 A C ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH -

66 Sampling the environment... state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH -

67 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(A) = 1.9 note difference to the exact value V(A)=1.75 that we computed earlier!

68 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(B) =.5

69 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(C) = 1

70 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - and so on for D,E,F, and G

71 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - and so on for D,E,F, and G Monte Carlo Policy Evaluation (similar to batch update for bees)

72 Online learning idea: update value function after each action The animal goes through states a a a s s s s up to final state H ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H

73 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H up to final state H

74 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H up to final state H

75 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H

76 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) Rat s prediction error δ = V (s) ˆV (s) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H

77 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) Rat s prediction error δ = V (s) ˆV (s) } ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H Problem: can t compute prediction error without V (s)!!!

78 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without Solution (trick): use V (s) V (s) =r(s)+v (s )

79 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s )

80 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s)

81 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s) } reward in state s } - (expected reward in state s)

82 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s) Temporal-difference (TD)-learning: V (s) V (s)+ɛδ with δ = r(s)+ ˆV (s ) ˆV (s)

83 Rat in maze: learning V(s) V (A) V (B) V (C) TD-learning: V (s) V (s)+ɛδ δ = r(s )+V (s ) V (s)

84 Actor-critic learning Critic Policy evaluation learning V(s) Actor Policy improvement learning p(a s)

85 Actor-critic learning Critic Policy evaluation learning V(s) Actor Policy improvement learning p(a s) e.g. Monte-Carlo-update: TD(1) Temporal-difference-learning: TD() Generalization: TD(λ) e.g. (havent discussed it yet) but it s the same as on the left

86 Rat in maze: learning p(a s)

87 Discounting rewards The further a reward lies in the future, the less important it is mathematically convenient good approximation of human and animal behavior

88 Exponential Discounting The further a reward lies in the future, the less important it is Discounted sum of future rewards ( γ 1 ) V (s) =r(s)+γr(s )+γ r(s )+... Temporal difference rule with discounting: V (s) V (s)+ɛδ δ = r(s)+γ ˆV (s ) ˆV (s)

89 Secondary conditioning training 1. sound food salivation. clapping sound salivation testing 3. sound salivation 4. clapping salivation

90 Example : Secondary conditioning Phase 1 training Phase sound food salivation clapping sound salivation Phase 1 C Phase 1 trial start A Phase B Phase D trial end Note: no actions are required in this task!

91 Example : Secondary conditioning Phase 1 training Phase sound food salivation clapping sound salivation Phase 1 C Phase 1 trial start A Phase B Phase D trial end Initial values: V (A) = V (B) = V (C) =1 V (D) = Assume: the higher the value of a state, the more the dog will salivate when in that state. Therefore: state C has value 1

92 Example : Secondary conditioning 1 Value V(C).8 value trial no. Phase 1 (4 trials) B C D

93 Example : Secondary conditioning 1.8 Value V(C) Value V(B) value trial no. Phase 1 (4 trials) B C D

94 Example : Secondary conditioning value Value V(C) Value V(B) This is conditioning - exactly trial no. as in the Rescorla-Wagner model! Phase 1 (4 trials) B C D

95 Example : Secondary conditioning 1.8 Value V(C) Value V(B) value.6.4. Value V(D) trial no. Phase (4 trials) A B D

96 Example : Secondary conditioning 1 Value V(C) value Value V(B). Value V(D) trial no. Phase (4 trials) A B D

97 Example : Secondary conditioning 1 Value V(C) value Value V(C ) Value V(B) This is extinction - exactly trial no. as in the Rescorla-Wagner model! Phase (4 trials) A B D

98 Example : Secondary conditioning 1.8 value.6.4 Value V(A) Value V(B) Stimulus A has become valuable trial no. Phase (4 trials) A B D

99 Define a set of states K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E F G H I B C A

100 Define possible actions in each state K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E F G H I Actions: a = {left, right A} B C a = {left, center, right B} a = {left, center, right C} A

101 Define rewards in each state K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E B F 6 G 4 H 4 C I 4 Actions: a = {left, right A} a = {left, center, right B} a = {left, center, right C} A Rewards: r(s)

102 Problem summary D E F G H I B K A C States: s = {A,B,C,D,E,F,G,H,I,K} Actions: a = {left, right A} a = {left, center, right B} a = {left, center, right C} Rewards: r(s)

103 Define a policy 1 K Policy: (see arrow thickness) p(a s) D E F G H I B C.5 A.5 1) all actions equally likely

104 Define a policy K Policy: (see arrow thickness) p(a s) D E F G H I B C. A.8 1) all actions equally likely ) rightward bias

105 Define a policy K Policy: (see arrow thickness) p(a s) D E F G H I B C A 1) all actions equally likely ) rightward bias 3) optimal (ε-greedy) policy (but! dont know this beforehand!)

106 How can we improve a policy? 1 K Policy: (see arrow thickness) p(a s) D E F G H I B C.5 A.5 1) all actions equally likely ) rightward bias 3) optimal (ε-greedy) policy (but! dont know this beforehand!)

107 Solution: define a value-function K D E F G H I Policy: p(a s) Value (sum of all future rewards) V (s) B C A

108 Solution: define a value-function K D E F G H I Policy: p(a s) Value (sum of all future rewards) V (s) B A C Sequence of actions a s s a s a s Values: V (s) =r(s)+r(s )+... = r(s)+v (s ) but: this is a bit sloppy...

109 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N f(x i ) i=1

110 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) Caution!! Expected value is a math-term which has nothing to do with the value-function!

111 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) important examples: Mean of a distribution: E[x] = µ = x A xp(x) Sample mean: ˆµ = 1 N N i=1 x i

112 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) important examples: Mean of a distribution: E[x] = µ = x A xp(x) Sample mean: ˆµ = 1 N N i=1 x i Variance of a distribution: Sample variance: E[(x µ) σ = x A(x ]= µ) p(x) ˆσ = 1 N N (x i µ) i=1

113 Solution: define a value-function K D E F G H I Policy: p(a s) Value (sum of all future rewards) V (s) B A C Sequence of actions a s s a s a s Value of state s V (s) =r(s)+e[r(s )+...] V (s) =r(s)+e[v (s )] Expected value with respect to the likelihood of all next states dictated by current policy!

114 Policy evaluation and improvement if we know the environment (we know where to find rewards r as a function of the states s), and have a policy, then we can compute the value function (policy evaluation) with the value function, we can improve the policy (policy improvement) iteration of procedure yields the optimal policy

115 Policy evaluation K V (K) = Policy: all actions equally likely D E F G H I Evaluation: V (s) =r(s)+e[v (s )] B C start at the end (top) A

116 Policy evaluation K V (K) = V (D) = V (I) =4 D E F G H I Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] B C start at the end (top) A

117 Policy evaluation D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] start at the end (top) A

118 Policy evaluation D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] start at the end (top) V (A) = 3 A

119 { Policy improvement D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Update policy: p(a s) p(a s)+ɛδ Delta-error: δ = q(a s) p(a s) 1: for optimal choice given current value function : otherwise V (A) = 3 A

120 { Policy improvement D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Update policy: p(a s) p(a s)+ɛδ Delta-error: δ = q(a s) p(a s) 1: for optimal choice given current value function : otherwise V (A) = 3 A 1.5 p(c A) p(b A)

121 Policy evaluation K V (K) = V (D) = V (I) =4 D E F G H I B C 4 V (C) V (B) A 1.5 p(c A) p(b A)

122 Policy improvement + evaluation K V (K) = V (D) = V (I) =4 D E F G H I V (C) B C V (B) A 1.5 p(c A) p(b A)

123 Policy improvement + evaluation K V (K) = V (D) = V (I) =4 D E F G H I V (C) B C V (B) A 1.5 p(c A) p(b A)

124 Policy improvement + evaluation D E F G H I B K V (K) = V (D) = V (I) =4 C 4 V (B) V (C) A 1 p(b A).5 p(c A)

125 Online policy evaluation: TD-learning K D E F G H I Policy given Exact evaluation: V (s) =r(s)+e[v (s )] B C Online evaluation ˆV (s) ˆV (s)+ɛδ A prediction error: δ = r(s)+e[v (s )] ˆV (s) r(s)+ ˆV (s ) ˆV (s) TD-Learning

126 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another)

127 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s))

128 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s)) What if we don t know the current state (noisy observations)? Solution: Partially-observable Markov-decision-processes

129 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s)) What if we don t know the current state (noisy observations)? Solution: Partially-observable Markov-decision-processes What if there are too many states?