Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associate
|
|
- Theresa Osborne
- 5 years ago
- Views:
Transcription
1
2
3
4
5 Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associated to the value Teaching signal : difference between what you get and what you think you should get
6 Exploration-Exploitation Dilemma Possible choices (actions) of a bee: land on a blue or yellow flower
7 Bee searching for nectar Possible choices (actions) of the bee: land on a blue or yellow flower rewards (in drops of nectar) r b =8 r y =
8 What does the bee know? actual reward internal estimate r b m b r y m y
9 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i last reward on a blue flower last reward on a yellow flower
10 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward r b,i trial number 8 reward r y,i r y,i 5 1 i 15 trial number
11 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i
12 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i
13 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i
14 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward m b r b,i choices: trial number 8 reward m y r y,i trial number i
15 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i
16 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i
17 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i
18 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i
19 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i 8 reward choices: trial number reward trial number i
20 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i batch update: N N m b = 1 N r b,i m y = 1 N r y,i i=1 i=1 average reward on last N visits to a blue flower average reward on last N visits to a yellow flower
21 How can the bee learn the rewards? greedy update: m b = r b,i m y = r y,i batch update: N N m b = 1 N i=1 r b,i m y = 1 N i=1 r y,i online update: m b m b + ɛ(r b,i m b ) delta - rule m b m b + ɛδ } learning rate with δ = r b,i m b } prediction error
22 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i
23 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i
24 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i
25 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i
26 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward choices: trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i
27 How can the bee learn the rewards? online update: m b m b + ɛδ b m y m y + ɛδ y 8 reward Idea: the bee could use its estimates about the flower rewards to change its policy! trial number prediction error: δ b = r b,i m b δ y = r y,i m y 8 reward trial number i
28 Changing the policy online p(b) = p(y) = exp(βm b ) exp(βm b ) + exp(βm y ) exp(βm y ) exp(βm b + exp(βm y ) actual reward internal estimate r b m b r y m y
29 Changing the policy online p(b) = p(y) = exp(βm b ) exp(βm b ) + exp(βm y ) exp(βm y ) exp(βm b + exp(βm y ) probability if m b >m y p(b). p(y) beta exploration exploitation
30 Learning the nectar reward: exploration-exploitation trade-off 1 softmax Gibbs-policy with β = (5/5) policy actual average reward estimated average reward time steps r y r b m y m b
31 Learning the nectar reward: exploration-exploitation trade-off 1 actual average reward r y r b softmax Gibbs-policy with β =.5 estimated average reward time steps m y m b
32 Learning the nectar reward: exploration-exploitation trade-off 1 actual average reward r y r b softmax Gibbs-policy with β = estimated average reward time steps m y m b
33 Exploration-exploitation tradeoff average reward softmax Gibbs-policy beta
34 Real, 1991, Science 53: What do real bees do? bumblebees (N=5) solid line: blue=constantly rewarded yellow=randomly rewarded switch at trial 17! blue=randomly rewarded yellow=constantly rewarded switch dashed line: less variability on randomly rewarded flowers
35 Reinforcement Learning Model-free approaches Cogmaster CO6 / Boris Gutkin
36 A Brief History Psychology of Animal Learning Edward Thorndike ( )
37 A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike ( ) Richard Bellman ( )
38 A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike ( ) Richard Bellman ( ) Artificial Intelligence Marvin Minsky (197-???) Harry Klopf (197-???)
39 A Brief History Psychology of Animal Learning Optimal Control Theory Edward Thorndike ( ) Richard Bellman ( ) Reinforcement Learning Artificial Intelligence Richard Sutton (1956-???) Andrew Barto (1948-???) Marvin Minsky (197-???) Harry Klopf (197-???)
40 Conditioning stimulus reward Trial 1: Trial : Trial 3: Trial 4:...
41 Bee foraging stimulus action reward Trial 1: Trial : Trial 3: Trial 4:...
42 More complex tasks? stimulus action stimulus action reward Trial 1: Trial : Trial 3: zzz... Trial 4:...
43 Example: Rat in a maze D E F G 5 Rewards B C A
44 The credit assignment problem D E F G 5 Rewards good choice B C good choice A
45 The credit assignment problem D E F G 5 Rewards bad choice B C good choice A even though there was no reward, the first choice was a good one!
46 Formalizing the problem D E F G Possible choices (actions a) of 5 the rat: move left or right B C States: s = {A,B,C,D,E,F,G} A Rewards: Actions: r(s) a = {left, right} Policy: p(a s) depends on the state!
47 The value function D E F G Define: 5 B C Value of a state V(s) = sum of all future rewards A Note! value of a state depends on future actions!!!
48 If the maze is known... D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A
49 ... and the policy is given... D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E)
50 ... then so are the values of the states D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G)
51 ... then so are the values of the states D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)
52 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =p(a=left B)V (D) +p(a=right B)V (E) V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)
53 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =p(a=left C)V (F ) +p(a=right C)V (G) V (A) =p(a=left A)V (B) +p(a=right A)V (C)
54 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =1 V (A) =p(a=left A)V (B) +p(a=right A)V (C)
55 Example: random policy (5/5) p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (B) =.5 V (C) =1 V (A) =1.75
56 If the value function is known... p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 V (A) =1.75
57 ... the policy can be improved p(a=left) =.5 p(a=right) =.5 D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 Policy improvement: make policy more greedy p(a = left A) p(a = right A)
58 ... the policy can be improved D E F G Value of a state = 5 sum of all future rewards B C V (D) = V (E) =5 V (F )= V (G) = A V (C) =1 V (B) =.5 Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4
59 ... the policy can be improved... D E F G Value of a state = 5 sum of all future rewards B C V (E) =5 V (F )= V (G) = A V (D) = Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4 p(a = left B) =.4 p(a = right B) =.6
60 ... which in turn... D E F G Value of a state = 5 sum of all future rewards B C V (E) =5 V (F )= V (G) = A V (D) = Policy improvement: make policy more greedy p(a = left A) =.6 p(a = right A) =.4 p(a = left B) =.4 p(a = right B) =.6
61 ... changes the value function New policy: D E F G p(a = left A) =.6 5 p(a = right A) =.4 B C p(a = left B) =.4 p(a = right B) =.6 A p(a = left C) =.6 p(a = right C) =.4 New value function: V (B) =3 V (C) =1. V (A) =.8
62 Iteration yields optimal policy Optimal policy: D E F G p(a = left A) = p(a = right A) =.4 B C p(a = left B) =.4 p(a = right B) =.6 1 A p(a = left C) = 1.6 p(a = right C) =.4 New value function: V (B) =5 V (C) = V (A) = 5
63 Conclusions so far if we know the environment (we know where to find rewards r as a function of the states s), and have a policy, then we can compute the value function (policy evaluation) with the value function, we can improve the policy (policy improvement) iteration of procedure yields the optimal policy
64 What if the environment is unknown? state H: rat taken out Assume p(a s) = 5/5 policy D E F G 5 B C A
65 Sampling the environment... state H: rat taken out Assume p(a s) = 5/5 policy D E F G Generate episodes (trials) B 5 A C ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH -
66 Sampling the environment... state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH -
67 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(A) = 1.9 note difference to the exact value V(A)=1.75 that we computed earlier!
68 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(B) =.5
69 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - V(C) = 1
70 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - and so on for D,E,F, and G
71 ... and estimating the value function state H: rat taken out Assume p(a s) = 5/5 policy D B E 5 A F C G Generate episodes (trials) Collect rewards. ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - and so on for D,E,F, and G Monte Carlo Policy Evaluation (similar to batch update for bees)
72 Online learning idea: update value function after each action The animal goes through states a a a s s s s up to final state H ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H
73 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H up to final state H
74 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H up to final state H
75 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H
76 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) Rat s prediction error δ = V (s) ˆV (s) ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H
77 Online learning idea: update value function after each action The animal goes through states a s s s s The value of a state is a V (s) =r(s)+r(s )+... a = r(s)+v (s ) Rat s internal estimate of value: ˆV (s) Rat s prediction error δ = V (s) ˆV (s) } ABDH - ABEH - 5 ACFH - ABEH - 5 ABEH - 5 ABDH - ACFH - ACGH - ABDH - ACGH - final state H Problem: can t compute prediction error without V (s)!!!
78 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without Solution (trick): use V (s) V (s) =r(s)+v (s )
79 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s )
80 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s)
81 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s) } reward in state s } - (expected reward in state s)
82 Online learning Rat s prediction error δ = V (s) ˆV (s) } Problem: can t compute prediction error without V (s) Solution (trick): use V (s) =r(s)+v (s ) r(s)+ ˆV (s ) δ = r(s)+ ˆV (s ) ˆV (s) Temporal-difference (TD)-learning: V (s) V (s)+ɛδ with δ = r(s)+ ˆV (s ) ˆV (s)
83 Rat in maze: learning V(s) V (A) V (B) V (C) TD-learning: V (s) V (s)+ɛδ δ = r(s )+V (s ) V (s)
84 Actor-critic learning Critic Policy evaluation learning V(s) Actor Policy improvement learning p(a s)
85 Actor-critic learning Critic Policy evaluation learning V(s) Actor Policy improvement learning p(a s) e.g. Monte-Carlo-update: TD(1) Temporal-difference-learning: TD() Generalization: TD(λ) e.g. (havent discussed it yet) but it s the same as on the left
86 Rat in maze: learning p(a s)
87 Discounting rewards The further a reward lies in the future, the less important it is mathematically convenient good approximation of human and animal behavior
88 Exponential Discounting The further a reward lies in the future, the less important it is Discounted sum of future rewards ( γ 1 ) V (s) =r(s)+γr(s )+γ r(s )+... Temporal difference rule with discounting: V (s) V (s)+ɛδ δ = r(s)+γ ˆV (s ) ˆV (s)
89 Secondary conditioning training 1. sound food salivation. clapping sound salivation testing 3. sound salivation 4. clapping salivation
90 Example : Secondary conditioning Phase 1 training Phase sound food salivation clapping sound salivation Phase 1 C Phase 1 trial start A Phase B Phase D trial end Note: no actions are required in this task!
91 Example : Secondary conditioning Phase 1 training Phase sound food salivation clapping sound salivation Phase 1 C Phase 1 trial start A Phase B Phase D trial end Initial values: V (A) = V (B) = V (C) =1 V (D) = Assume: the higher the value of a state, the more the dog will salivate when in that state. Therefore: state C has value 1
92 Example : Secondary conditioning 1 Value V(C).8 value trial no. Phase 1 (4 trials) B C D
93 Example : Secondary conditioning 1.8 Value V(C) Value V(B) value trial no. Phase 1 (4 trials) B C D
94 Example : Secondary conditioning value Value V(C) Value V(B) This is conditioning - exactly trial no. as in the Rescorla-Wagner model! Phase 1 (4 trials) B C D
95 Example : Secondary conditioning 1.8 Value V(C) Value V(B) value.6.4. Value V(D) trial no. Phase (4 trials) A B D
96 Example : Secondary conditioning 1 Value V(C) value Value V(B). Value V(D) trial no. Phase (4 trials) A B D
97 Example : Secondary conditioning 1 Value V(C) value Value V(C ) Value V(B) This is extinction - exactly trial no. as in the Rescorla-Wagner model! Phase (4 trials) A B D
98 Example : Secondary conditioning 1.8 value.6.4 Value V(A) Value V(B) Stimulus A has become valuable trial no. Phase (4 trials) A B D
99 Define a set of states K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E F G H I B C A
100 Define possible actions in each state K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E F G H I Actions: a = {left, right A} B C a = {left, center, right B} a = {left, center, right C} A
101 Define rewards in each state K: rat taken out States: s = {A,B,C,D,E,F,G,H,I,K} D E B F 6 G 4 H 4 C I 4 Actions: a = {left, right A} a = {left, center, right B} a = {left, center, right C} A Rewards: r(s)
102 Problem summary D E F G H I B K A C States: s = {A,B,C,D,E,F,G,H,I,K} Actions: a = {left, right A} a = {left, center, right B} a = {left, center, right C} Rewards: r(s)
103 Define a policy 1 K Policy: (see arrow thickness) p(a s) D E F G H I B C.5 A.5 1) all actions equally likely
104 Define a policy K Policy: (see arrow thickness) p(a s) D E F G H I B C. A.8 1) all actions equally likely ) rightward bias
105 Define a policy K Policy: (see arrow thickness) p(a s) D E F G H I B C A 1) all actions equally likely ) rightward bias 3) optimal (ε-greedy) policy (but! dont know this beforehand!)
106 How can we improve a policy? 1 K Policy: (see arrow thickness) p(a s) D E F G H I B C.5 A.5 1) all actions equally likely ) rightward bias 3) optimal (ε-greedy) policy (but! dont know this beforehand!)
107 Solution: define a value-function K D E F G H I Policy: p(a s) Value (sum of all future rewards) V (s) B C A
108 Solution: define a value-function K D E F G H I Policy: p(a s) Value (sum of all future rewards) V (s) B A C Sequence of actions a s s a s a s Values: V (s) =r(s)+r(s )+... = r(s)+v (s ) but: this is a bit sloppy...
109 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N f(x i ) i=1
110 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) Caution!! Expected value is a math-term which has nothing to do with the value-function!
111 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) important examples: Mean of a distribution: E[x] = µ = x A xp(x) Sample mean: ˆµ = 1 N N i=1 x i
112 Expected value Expected value of a function f(x) E [ f(x) ] = x A f(x)p(x) Sample estimate of E[f(x)]: take N values xi and average ˆf = 1 N N i=1 f(x i ) important examples: Mean of a distribution: E[x] = µ = x A xp(x) Sample mean: ˆµ = 1 N N i=1 x i Variance of a distribution: Sample variance: E[(x µ) σ = x A(x ]= µ) p(x) ˆσ = 1 N N (x i µ) i=1
113 Solution: define a value-function K D E F G H I Policy: p(a s) Value (sum of all future rewards) V (s) B A C Sequence of actions a s s a s a s Value of state s V (s) =r(s)+e[r(s )+...] V (s) =r(s)+e[v (s )] Expected value with respect to the likelihood of all next states dictated by current policy!
114 Policy evaluation and improvement if we know the environment (we know where to find rewards r as a function of the states s), and have a policy, then we can compute the value function (policy evaluation) with the value function, we can improve the policy (policy improvement) iteration of procedure yields the optimal policy
115 Policy evaluation K V (K) = Policy: all actions equally likely D E F G H I Evaluation: V (s) =r(s)+e[v (s )] B C start at the end (top) A
116 Policy evaluation K V (K) = V (D) = V (I) =4 D E F G H I Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] B C start at the end (top) A
117 Policy evaluation D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] start at the end (top) A
118 Policy evaluation D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Policy: all actions equally likely Evaluation: V (s) =r(s)+e[v (s )] start at the end (top) V (A) = 3 A
119 { Policy improvement D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Update policy: p(a s) p(a s)+ɛδ Delta-error: δ = q(a s) p(a s) 1: for optimal choice given current value function : otherwise V (A) = 3 A
120 { Policy improvement D E F G H I B K V (K) = V (D) = V (I) =4 C V (B) = V (C) =4 Update policy: p(a s) p(a s)+ɛδ Delta-error: δ = q(a s) p(a s) 1: for optimal choice given current value function : otherwise V (A) = 3 A 1.5 p(c A) p(b A)
121 Policy evaluation K V (K) = V (D) = V (I) =4 D E F G H I B C 4 V (C) V (B) A 1.5 p(c A) p(b A)
122 Policy improvement + evaluation K V (K) = V (D) = V (I) =4 D E F G H I V (C) B C V (B) A 1.5 p(c A) p(b A)
123 Policy improvement + evaluation K V (K) = V (D) = V (I) =4 D E F G H I V (C) B C V (B) A 1.5 p(c A) p(b A)
124 Policy improvement + evaluation D E F G H I B K V (K) = V (D) = V (I) =4 C 4 V (B) V (C) A 1 p(b A).5 p(c A)
125 Online policy evaluation: TD-learning K D E F G H I Policy given Exact evaluation: V (s) =r(s)+e[v (s )] B C Online evaluation ˆV (s) ˆV (s)+ɛδ A prediction error: δ = r(s)+e[v (s )] ˆV (s) r(s)+ ˆV (s ) ˆV (s) TD-Learning
126 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another)
127 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s))
128 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s)) What if we don t know the current state (noisy observations)? Solution: Partially-observable Markov-decision-processes
129 Known problems + solutions The value function always depends on the policy Solution: off-policy learning (follow one policy but learn values for another) What if we don t know the next state (unknown environment)? Solution: Q-learning (define a state-action value function Q(a,s)) What if we don t know the current state (noisy observations)? Solution: Partially-observable Markov-decision-processes What if there are too many states?
Animal learning theory
Animal learning theory Based on [Sutton and Barto, 1990, Dayan and Abbott, 2001] Bert Kappen [Sutton and Barto, 1990] Classical conditioning: - A conditioned stimulus (CS) and unconditioned stimulus (US)
More informationCS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
More informationRewards and Value Systems: Conditioning and Reinforcement Learning
Rewards and Value Systems: Conditioning and Reinforcement Learning Acknowledgements: Many thanks to P. Dayan and L. Abbott for making the figures of their book available online. Also thanks to R. Sutton
More information6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
More informationLecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free
More informationTemporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI
Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning
More informationPART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.
Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE
More informationCO6: Introduction to Computational Neuroscience
CO6: Introduction to Computational Neuroscience Lecturer: J Lussange Ecole Normale Supérieure 29 rue d Ulm e-mail: johann.lussange@ens.fr Solutions to the 2nd exercise sheet If you have any questions regarding
More informationReinforcement learning
einforcement learning How to learn to make decisions in sequential problems (like: chess, a maze) Why is this difficult? Temporal credit assignment Prediction can help Further reading For modeling: Chapter
More informationIntroduction to Reinforcement Learning
Introduction to Reinforcement Learning Rémi Munos SequeL project: Sequential Learning http://researchers.lille.inria.fr/ munos/ INRIA Lille - Nord Europe Machine Learning Summer School, September 2011,
More informationElements of Reinforcement Learning
Elements of Reinforcement Learning Policy: way learning algorithm behaves (mapping from state to action) Reward function: Mapping of state action pair to reward or cost Value function: long term reward,
More informationBasics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
More informationIntroduction to Reinforcement Learning. Part 5: Temporal-Difference Learning
Introduction to Reinforcement Learning Part 5: emporal-difference Learning What everybody should know about emporal-difference (D) learning Used to learn value functions without human input Learns a guess
More informationPART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.
Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE
More information, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
More informationReinforcement learning crash course
crash course Quentin Huys Translational Neuromodeling Unit, University of Zurich and ETH Zurich University Hospital of Psychiatry Zurich Computational Psychiatry Course Zurich, 1.9.2016 Overview : rough
More informationThis question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
More informationReinforcement learning
Reinforcement learning Based on [Kaelbling et al., 1996, Bertsekas, 2000] Bert Kappen Reinforcement learning Reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error
More informationReinforcement Learning II
Reinforcement Learning II Andrea Bonarini Artificial Intelligence and Robotics Lab Department of Electronics and Information Politecnico di Milano E-mail: bonarini@elet.polimi.it URL:http://www.dei.polimi.it/people/bonarini
More informationMonte Carlo is important in practice. CSE 190: Reinforcement Learning: An Introduction. Chapter 6: Temporal Difference Learning.
Monte Carlo is important in practice CSE 190: Reinforcement Learning: An Introduction Chapter 6: emporal Difference Learning When there are just a few possibilitieo value, out of a large state space, Monte
More informationLecture 23: Reinforcement Learning
Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:
More informationMARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
More informationReinforcement Learning. Spring 2018 Defining MDPs, Planning
Reinforcement Learning Spring 2018 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
More informationARTIFICIAL INTELLIGENCE. Reinforcement learning
INFOB2KI 2018-2019 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Reinforcement learning Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More informationTemporal Difference Learning & Policy Iteration
Temporal Difference Learning & Policy Iteration Advanced Topics in Reinforcement Learning Seminar WS 15/16 ±0 ±0 +1 by Tobias Joppen 03.11.2015 Fachbereich Informatik Knowledge Engineering Group Prof.
More informationChapter 7: Eligibility Traces. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1
Chapter 7: Eligibility Traces R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Midterm Mean = 77.33 Median = 82 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
More informationChristopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
More informationReinforcement Learning and NLP
1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value
More informationBalancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu
More informationReinforcement Learning
1 Reinforcement Learning Chris Watkins Department of Computer Science Royal Holloway, University of London July 27, 2015 2 Plan 1 Why reinforcement learning? Where does this theory come from? Markov decision
More informationIntroduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
More informationReinforcement Learning. George Konidaris
Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom
More informationReinforcement Learning
Reinforcement Learning Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ Task Grasp the green cup. Output: Sequence of controller actions Setup from Lenz et. al.
More informationReinforcement Learning
Reinforcement Learning Policy gradients Daniel Hennes 26.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Policy based reinforcement learning So far we approximated the action value
More informationCMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING
More informationReinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo MLR, University of Stuttgart
More informationToday s Outline. Recap: MDPs. Bellman Equations. Q-Value Iteration. Bellman Backup 5/7/2012. CSE 473: Artificial Intelligence Reinforcement Learning
CSE 473: Artificial Intelligence Reinforcement Learning Dan Weld Today s Outline Reinforcement Learning Q-value iteration Q-learning Exploration / exploitation Linear function approximation Many slides
More informationReinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
More informationMarks. bonus points. } Assignment 1: Should be out this weekend. } Mid-term: Before the last lecture. } Mid-term deferred exam:
Marks } Assignment 1: Should be out this weekend } All are marked, I m trying to tally them and perhaps add bonus points } Mid-term: Before the last lecture } Mid-term deferred exam: } This Saturday, 9am-10.30am,
More informationMachine Learning. Reinforcement learning. Hamid Beigy. Sharif University of Technology. Fall 1396
Machine Learning Reinforcement learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Machine Learning Fall 1396 1 / 32 Table of contents 1 Introduction
More informationChapter 6: Temporal Difference Learning
Chapter 6: emporal Difference Learning Objectives of this chapter: Introduce emporal Difference (D) learning Focus first on policy evaluation, or prediction, methods hen extend to control methods R. S.
More informationCourse 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016
Course 16:198:520: Introduction To Artificial Intelligence Lecture 13 Decision Making Abdeslam Boularias Wednesday, December 7, 2016 1 / 45 Overview We consider probabilistic temporal models where the
More information15-889e Policy Search: Gradient Methods Emma Brunskill. All slides from David Silver (with EB adding minor modificafons), unless otherwise noted
15-889e Policy Search: Gradient Methods Emma Brunskill All slides from David Silver (with EB adding minor modificafons), unless otherwise noted Outline 1 Introduction 2 Finite Difference Policy Gradient
More informationLecture 2: Learning from Evaluative Feedback. or Bandit Problems
Lecture 2: Learning from Evaluative Feedback or Bandit Problems 1 Edward L. Thorndike (1874-1949) Puzzle Box 2 Learning by Trial-and-Error Law of Effect: Of several responses to the same situation, those
More informationCSC242: Intro to AI. Lecture 23
CSC242: Intro to AI Lecture 23 Administrivia Posters! Tue Apr 24 and Thu Apr 26 Idea! Presentation! 2-wide x 4-high landscape pages Learning so far... Input Attributes Alt Bar Fri Hun Pat Price Rain Res
More informationCS Machine Learning Qualifying Exam
CS Machine Learning Qualifying Exam Georgia Institute of Technology March 30, 2017 The exam is divided into four areas: Core, Statistical Methods and Models, Learning Theory, and Decision Processes. There
More informationReinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of
More informationMachine Learning I Reinforcement Learning
Machine Learning I Reinforcement Learning Thomas Rückstieß Technische Universität München December 17/18, 2009 Literature Book: Reinforcement Learning: An Introduction Sutton & Barto (free online version:
More informationReinforcement Learning Part 2
Reinforcement Learning Part 2 Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ From previous tutorial Reinforcement Learning Exploration No supervision Agent-Reward-Environment
More informationReinforcement Learning
Reinforcement Learning 1 Reinforcement Learning Mainly based on Reinforcement Learning An Introduction by Richard Sutton and Andrew Barto Slides are mainly based on the course material provided by the
More informationLecture 8: Policy Gradient
Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve
More informationToday s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes
Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks
More informationCSC321 Lecture 22: Q-Learning
CSC321 Lecture 22: Q-Learning Roger Grosse Roger Grosse CSC321 Lecture 22: Q-Learning 1 / 21 Overview Second of 3 lectures on reinforcement learning Last time: policy gradient (e.g. REINFORCE) Optimize
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
More informationOn and Off-Policy Relational Reinforcement Learning
On and Off-Policy Relational Reinforcement Learning Christophe Rodrigues, Pierre Gérard, and Céline Rouveirol LIPN, UMR CNRS 73, Institut Galilée - Université Paris-Nord first.last@lipn.univ-paris13.fr
More informationReinforcement Learning: the basics
Reinforcement Learning: the basics Olivier Sigaud Université Pierre et Marie Curie, PARIS 6 http://people.isir.upmc.fr/sigaud August 6, 2012 1 / 46 Introduction Action selection/planning Learning by trial-and-error
More informationComputational Reinforcement Learning: An Introduction
Computational Reinforcement Learning: An Introduction Andrew Barto Autonomous Learning Laboratory School of Computer Science University of Massachusetts Amherst barto@cs.umass.edu 1 Artificial Intelligence
More informationAn Adaptive Clustering Method for Model-free Reinforcement Learning
An Adaptive Clustering Method for Model-free Reinforcement Learning Andreas Matt and Georg Regensburger Institute of Mathematics University of Innsbruck, Austria {andreas.matt, georg.regensburger}@uibk.ac.at
More informationProf. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be
REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while
More informationReinforcement Learning Active Learning
Reinforcement Learning Active Learning Alan Fern * Based in part on slides by Daniel Weld 1 Active Reinforcement Learning So far, we ve assumed agent has a policy We just learned how good it is Now, suppose
More informationUsing Gaussian Processes for Variance Reduction in Policy Gradient Algorithms *
Proceedings of the 8 th International Conference on Applied Informatics Eger, Hungary, January 27 30, 2010. Vol. 1. pp. 87 94. Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms
More informationActive Policy Iteration: Efficient Exploration through Active Learning for Value Function Approximation in Reinforcement Learning
Active Policy Iteration: fficient xploration through Active Learning for Value Function Approximation in Reinforcement Learning Takayuki Akiyama, Hirotaka Hachiya, and Masashi Sugiyama Department of Computer
More informationQ-Learning in Continuous State Action Spaces
Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental
More informationReinforcement Learning
Reinforcement Learning Ron Parr CompSci 7 Department of Computer Science Duke University With thanks to Kris Hauser for some content RL Highlights Everybody likes to learn from experience Use ML techniques
More informationChapter 6: Temporal Difference Learning
Chapter 6: emporal Difference Learning Objectives of this chapter: Introduce emporal Difference (D) learning Focus first on policy evaluation, or prediction, methods Compare efficiency of D learning with
More informationReinforcement Learning with Function Approximation. Joseph Christian G. Noel
Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is
More informationLecture 25: Learning 4. Victor R. Lesser. CMPSCI 683 Fall 2010
Lecture 25: Learning 4 Victor R. Lesser CMPSCI 683 Fall 2010 Final Exam Information Final EXAM on Th 12/16 at 4:00pm in Lederle Grad Res Ctr Rm A301 2 Hours but obviously you can leave early! Open Book
More informationThe convergence limit of the temporal difference learning
The convergence limit of the temporal difference learning Ryosuke Nomura the University of Tokyo September 3, 2013 1 Outline Reinforcement Learning Convergence limit Construction of the feature vector
More informationHuman-level control through deep reinforcement. Liia Butler
Humanlevel control through deep reinforcement Liia Butler But first... A quote "The question of whether machines can think... is about as relevant as the question of whether submarines can swim" Edsger
More informationCourse basics. CSE 190: Reinforcement Learning: An Introduction. Last Time. Course goals. The website for the class is linked off my homepage.
Course basics CSE 190: Reinforcement Learning: An Introduction The website for the class is linked off my homepage. Grades will be based on programming assignments, homeworks, and class participation.
More informationREINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari
More information15-780: ReinforcementLearning
15-780: ReinforcementLearning J. Zico Kolter March 2, 2016 1 Outline Challenge of RL Model-based methods Model-free methods Exploration and exploitation 2 Outline Challenge of RL Model-based methods Model-free
More informationLecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
More informationOff-Policy Actor-Critic
Off-Policy Actor-Critic Ludovic Trottier Laval University July 25 2012 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July 25 2012 1 / 34 Table of Contents 1 Reinforcement Learning Theory
More informationMachine Learning and Bayesian Inference. Unsupervised learning. Can we find regularity in data without the aid of labels?
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone extension 6372 Email: sbh11@cl.cam.ac.uk www.cl.cam.ac.uk/ sbh11/ Unsupervised learning Can we find regularity
More informationBe able to define the following terms and answer basic questions about them:
CS440/ECE448 Section Q Fall 2017 Final Review Be able to define the following terms and answer basic questions about them: Probability o Random variables, axioms of probability o Joint, marginal, conditional
More informationNotes on Reinforcement Learning
1 Introduction Notes on Reinforcement Learning Paulo Eduardo Rauber 2014 Reinforcement learning is the study of agents that act in an environment with the goal of maximizing cumulative reward signals.
More informationLecture 9: Policy Gradient II 1
Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John
More informationAn Introduction to Reinforcement Learning
An Introduction to Reinforcement Learning Shivaram Kalyanakrishnan shivaram@cse.iitb.ac.in Department of Computer Science and Engineering Indian Institute of Technology Bombay April 2018 What is Reinforcement
More informationPlanning in Markov Decision Processes
Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Planning in Markov Decision Processes Lecture 3, CMU 10703 Katerina Fragkiadaki Markov Decision Process (MDP) A Markov
More informationarxiv: v1 [cs.ai] 1 Jul 2015
arxiv:507.00353v [cs.ai] Jul 205 Harm van Seijen harm.vanseijen@ualberta.ca A. Rupam Mahmood ashique@ualberta.ca Patrick M. Pilarski patrick.pilarski@ualberta.ca Richard S. Sutton sutton@cs.ualberta.ca
More informationReinforcement Learning
CS7/CS7 Fall 005 Supervised Learning: Training examples: (x,y) Direct feedback y for each input x Sequence of decisions with eventual feedback No teacher that critiques individual actions Learn to act
More informationReinforcement Learning. Machine Learning, Fall 2010
Reinforcement Learning Machine Learning, Fall 2010 1 Administrativia This week: finish RL, most likely start graphical models LA2: due on Thursday LA3: comes out on Thursday TA Office hours: Today 1:30-2:30
More informationStochastic Gradient Descent
Stochastic Gradient Descent Machine Learning CSE546 Carlos Guestrin University of Washington October 9, 2013 1 Logistic Regression Logistic function (or Sigmoid): Learn P(Y X) directly Assume a particular
More informationCS 570: Machine Learning Seminar. Fall 2016
CS 570: Machine Learning Seminar Fall 2016 Class Information Class web page: http://web.cecs.pdx.edu/~mm/mlseminar2016-2017/fall2016/ Class mailing list: cs570@cs.pdx.edu My office hours: T,Th, 2-3pm or
More informationQ-learning. Tambet Matiisen
Q-learning Tambet Matiisen (based on chapter 11.3 of online book Artificial Intelligence, foundations of computational agents by David Poole and Alan Mackworth) Stochastic gradient descent Experience
More informationBayesian Networks BY: MOHAMAD ALSABBAGH
Bayesian Networks BY: MOHAMAD ALSABBAGH Outlines Introduction Bayes Rule Bayesian Networks (BN) Representation Size of a Bayesian Network Inference via BN BN Learning Dynamic BN Introduction Conditional
More informationECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control
ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control Lecturer: Nikolay Atanasov: natanasov@ucsd.edu Teaching Assistants: Tianyu Wang: tiw161@eng.ucsd.edu Yongxi Lu: yol070@eng.ucsd.edu
More information15-780: Graduate Artificial Intelligence. Reinforcement learning (RL)
15-780: Graduate Artificial Intelligence Reinforcement learning (RL) From MDPs to RL We still use the same Markov model with rewards and actions But there are a few differences: 1. We do not assume we
More informationThe Nature of Learning - A study of Reinforcement Learning Methodology
The Nature of Learning - A study of Reinforcement Learning Methodology Anders Christian Kølle 1 Department of Computer Science University of Copenhagen November 21, 23 1 andersk@diku.dk One needs to explore
More informationChapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationLecture 9: Policy Gradient II (Post lecture) 2
Lecture 9: Policy Gradient II (Post lecture) 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver
More informationReinforcement Learning
Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation
More informationSequential Decision Problems
Sequential Decision Problems Michael A. Goodrich November 10, 2006 If I make changes to these notes after they are posted and if these changes are important (beyond cosmetic), the changes will highlighted
More informationBias-Variance Error Bounds for Temporal Difference Updates
Bias-Variance Bounds for Temporal Difference Updates Michael Kearns AT&T Labs mkearns@research.att.com Satinder Singh AT&T Labs baveja@research.att.com Abstract We give the first rigorous upper bounds
More informationReinforcement Learning and Control
CS9 Lecture notes Andrew Ng Part XIII Reinforcement Learning and Control We now begin our study of reinforcement learning and adaptive control. In supervised learning, we saw algorithms that tried to make
More information1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5
Table of contents 1 Introduction 2 2 Markov Decision Processes 2 3 Future Cumulative Reward 3 4 Q-Learning 4 4.1 The Q-value.............................................. 4 4.2 The Temporal Difference.......................................
More informationTemporal difference learning
Temporal difference learning AI & Agents for IET Lecturer: S Luz http://www.scss.tcd.ie/~luzs/t/cs7032/ February 4, 2014 Recall background & assumptions Environment is a finite MDP (i.e. A and S are finite).
More informationREINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
More informationReinforcement learning an introduction
Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,
More information