CS 598 Statistical Reinforcement Learning. Nan Jiang

Transcription

1 CS 598 Statistical Reinforcement Learning Nan Jiang

2 Overview

3 What s this course about? A grad-level seminar course on theory of RL 3

4 What s this course about? A grad-level seminar course on theory of RL with focus on sample complexity analyses 3

5 What s this course about? A grad-level seminar course on theory of RL with focus on sample complexity analyses all about proofs, some perspectives, 0 implementation 3

6 What s this course about? A grad-level seminar course on theory of RL with focus on sample complexity analyses all about proofs, some perspectives, 0 implementation Seminar course can be anywhere between students presenting papers and a rigorous course 3

7 What s this course about? A grad-level seminar course on theory of RL with focus on sample complexity analyses all about proofs, some perspectives, 0 implementation Seminar course can be anywhere between students presenting papers and a rigorous course In this course I will deliver most (or all) of the lectures 3

8 What s this course about? A grad-level seminar course on theory of RL with focus on sample complexity analyses all about proofs, some perspectives, 0 implementation Seminar course can be anywhere between students presenting papers and a rigorous course In this course I will deliver most (or all) of the lectures No text book; material is created by myself (course notes) 3

9 What s this course about? A grad-level seminar course on theory of RL with focus on sample complexity analyses all about proofs, some perspectives, 0 implementation Seminar course can be anywhere between students presenting papers and a rigorous course In this course I will deliver most (or all) of the lectures No text book; material is created by myself (course notes) will share slides/notes on course website 3

10 Who should take this course? This course will be a good fit for you if you 4

11 Who should take this course? This course will be a good fit for you if you are interested in understanding RL mathematically 4

12 Who should take this course? This course will be a good fit for you if you are interested in understanding RL mathematically want to do research in theory of RL or related area 4

13 Who should take this course? This course will be a good fit for you if you are interested in understanding RL mathematically want to do research in theory of RL or related area want to work with me 4

14 Who should take this course? This course will be a good fit for you if you are interested in understanding RL mathematically want to do research in theory of RL or related area want to work with me are comfortable with maths 4

15 Who should take this course? This course will be a good fit for you if you are interested in understanding RL mathematically want to do research in theory of RL or related area want to work with me are comfortable with maths This course will not be a good fit if you 4

16 Who should take this course? This course will be a good fit for you if you are interested in understanding RL mathematically want to do research in theory of RL or related area want to work with me are comfortable with maths This course will not be a good fit if you are mostly interested in implementing RL algorithms and making things to work we won t cover any engineering tricks (which are essential to e.g., Deep RL); check other RL courses offered across the campus 4

17 Prerequisites Maths 5

18 Prerequisites Maths Linear algebra, probability & statistics, basic calculus 5

19 Prerequisites Maths Linear algebra, probability & statistics, basic calculus Markov chains 5

20 Prerequisites Maths Linear algebra, probability & statistics, basic calculus Markov chains Optional: stochastic processes, numerical analysis 5

21 Prerequisites Maths Linear algebra, probability & statistics, basic calculus Markov chains Optional: stochastic processes, numerical analysis Useful for research: TCS background, empirical processes and statistical learning theory, optimization, online learning 5

22 Prerequisites Maths Linear algebra, probability & statistics, basic calculus Markov chains Optional: stochastic processes, numerical analysis Useful for research: TCS background, empirical processes and statistical learning theory, optimization, online learning Exposure to ML 5

23 Prerequisites Maths Linear algebra, probability & statistics, basic calculus Markov chains Optional: stochastic processes, numerical analysis Useful for research: TCS background, empirical processes and statistical learning theory, optimization, online learning Exposure to ML e.g., CS 446 Machine Learning 5

24 Prerequisites Maths Linear algebra, probability & statistics, basic calculus Markov chains Optional: stochastic processes, numerical analysis Useful for research: TCS background, empirical processes and statistical learning theory, optimization, online learning Exposure to ML e.g., CS 446 Machine Learning Experience with RL 5

25 Coursework Some readings after/before class 6

26 Coursework Some readings after/before class Ad hoc homework to help digest particular material. Deadline will be lenient & TBA at the time of assignment 6

27 Coursework Some readings after/before class Ad hoc homework to help digest particular material. Deadline will be lenient & TBA at the time of assignment Main assignment: course project (work on your own) 6

28 Coursework Some readings after/before class Ad hoc homework to help digest particular material. Deadline will be lenient & TBA at the time of assignment Main assignment: course project (work on your own) Baseline: reproduce theoretical analysis in existing papers 6

29 Coursework Some readings after/before class Ad hoc homework to help digest particular material. Deadline will be lenient & TBA at the time of assignment Main assignment: course project (work on your own) Baseline: reproduce theoretical analysis in existing papers Advanced: identify an interesting/challenging extension to the paper and explore the novel research question yourself 6

30 Coursework Some readings after/before class Ad hoc homework to help digest particular material. Deadline will be lenient & TBA at the time of assignment Main assignment: course project (work on your own) Baseline: reproduce theoretical analysis in existing papers Advanced: identify an interesting/challenging extension to the paper and explore the novel research question yourself Or, just work on a novel research question (must have a significant theoretical component; need to discuss with me) 6

31 Course project (cont.) See list of references and potential topics on website 7

32 Course project (cont.) See list of references and potential topics on website You will need to submit: 7

33 Course project (cont.) See list of references and potential topics on website You will need to submit: A brief proposal (~/2 page). Tentative deadline: end of Feb 7

34 Course project (cont.) See list of references and potential topics on website You will need to submit: A brief proposal (~/2 page). Tentative deadline: end of Feb what s the topic and what papers you plan to work on 7

35 Course project (cont.) See list of references and potential topics on website You will need to submit: A brief proposal (~/2 page). Tentative deadline: end of Feb what s the topic and what papers you plan to work on why you choose the topic: what interest you? 7

36 Course project (cont.) See list of references and potential topics on website You will need to submit: A brief proposal (~/2 page). Tentative deadline: end of Feb what s the topic and what papers you plan to work on why you choose the topic: what interest you? which aspect(s) you will focus on? 7

37 Course project (cont.) See list of references and potential topics on website You will need to submit: A brief proposal (~/2 page). Tentative deadline: end of Feb what s the topic and what papers you plan to work on why you choose the topic: what interest you? which aspect(s) you will focus on? Final report: clarity, precision, and brevity are greatly valued. More details to come 7

38 Course project (cont.) See list of references and potential topics on website You will need to submit: A brief proposal (~/2 page). Tentative deadline: end of Feb what s the topic and what papers you plan to work on why you choose the topic: what interest you? which aspect(s) you will focus on? Final report: clarity, precision, and brevity are greatly valued. More details to come All docs should be in pdf. Final report should be prepared using LaTeX. 7

39 Course project (cont. 2) Rule of thumb 8

40 Course project (cont. 2) Rule of thumb. learn something that interests you 8

41 Course project (cont. 2) Rule of thumb. learn something that interests you 2. teach me something! (I wouldn t learn if I could not understand your report due to lack of clarity) 8

42 Course project (cont. 2) Rule of thumb. learn something that interests you 2. teach me something! (I wouldn t learn if I could not understand your report due to lack of clarity) 3. write a report similar to (or better than!) the notes I will share with you 8

43 Contents of the course many important topics in RL will not be covered in depth (e.g., TD). read more if you want to get a more comprehensive view of RL 9

44 Contents of the course many important topics in RL will not be covered in depth (e.g., TD). read more if you want to get a more comprehensive view of RL the other opportunity to learn what s not covered in lectures is the project, as potential topics for projects are much broader than what s covered in class. 9

45 My goals Encourage you to do research in this area 0

46 My goals Encourage you to do research in this area Introduce useful mathematical tools to you 0

47 My goals Encourage you to do research in this area Introduce useful mathematical tools to you Learn from you 0

48 Logistics Course website:

49 Logistics Course website: logistics, links to slides/notes (uploaded after lectures), and resources (e.g., textbooks to consult, related courses), deadlines & announcements

50 Logistics Course website: logistics, links to slides/notes (uploaded after lectures), and resources (e.g., textbooks to consult, related courses), deadlines & announcements Time & Location: Tue & Thu 2:30-:45pm, 304 Siebel.

51 Logistics Course website: logistics, links to slides/notes (uploaded after lectures), and resources (e.g., textbooks to consult, related courses), deadlines & announcements Time & Location: Tue & Thu 2:30-:45pm, 304 Siebel. TA: Jinglin Chen (jinglinc)

52 Logistics Course website: logistics, links to slides/notes (uploaded after lectures), and resources (e.g., textbooks to consult, related courses), deadlines & announcements Time & Location: Tue & Thu 2:30-:45pm, 304 Siebel. TA: Jinglin Chen (jinglinc) Office hours: By appointment Siebel.

53 Logistics Course website: logistics, links to slides/notes (uploaded after lectures), and resources (e.g., textbooks to consult, related courses), deadlines & announcements Time & Location: Tue & Thu 2:30-:45pm, 304 Siebel. TA: Jinglin Chen (jinglinc) Office hours: By appointment Siebel. Questions about material: ad hoc meetings after lectures subject to my availability

54 Logistics Course website: logistics, links to slides/notes (uploaded after lectures), and resources (e.g., textbooks to consult, related courses), deadlines & announcements Time & Location: Tue & Thu 2:30-:45pm, 304 Siebel. TA: Jinglin Chen (jinglinc) Office hours: By appointment Siebel. Questions about material: ad hoc meetings after lectures subject to my availability Other things about the class (e.g., project): by appointment

55 Introduction to MDPs and RL

56 Reinforcement Learning (RL) Applications 3

57 Reinforcement Learning (RL) Applications [Levine et al 6] [Ng et al 03] 3

58 Reinforcement Learning (RL) Applications [Levine et al 6] [Ng et al 03] 3 [Singh et al 02]

59 Reinforcement Learning (RL) Applications [Levine et al 6] [Ng et al 03] 3 [Singh et al 02] [Lei et al 2]

60 Reinforcement Learning (RL) Applications 3 [Levine et al 6] [Ng et al 03] [Mandel et al 6] [Singh et al 02] [Lei et al 2]

61 Reinforcement Learning (RL) Applications 3 [Levine et al 6] [Ng et al 03] [Singh et al 02] [Mandel et al 6] [Tesauro et al 07] [Lei et al 2]

62 Reinforcement Learning (RL) Applications 3 [Levine et al 6] [Ng et al 03] [Singh et al 02] [Lei et al 2] [Mandel et al 6] [Tesauro et al 07] [Mnih et al 5][Silver et al 6]

63 Shortest Path s 0 b 4 3 d 3 f g 2 c 2 e 4

64 Shortest Path state s 0 action 2 b c e d 3 f g 4

65 Shortest Path state s 0 action 2 b c e d 3 f g Greedy is suboptimal due to delayed effects Need long-term planning 4

66 Shortest Path V * (g) = 0 state s 0 action 2 b c e d 3 f V * (f) = g Greedy is suboptimal due to delayed effects Need long-term planning 4

67 Shortest Path V * (g) = 0 state s 0 action 2 b c e d 3 f V * (f) = g Bellman Equation V * (d) = min{3 + V * (g), 2 + V * (f)} Greedy is suboptimal due to delayed effects Need long-term planning 4

68 Shortest Path V * (d) = 2 V * (g) = 0 state s 0 action 2 b c e d 3 f V * (f) = g Bellman Equation V * (d) = min{3 + V * (g), 2 + V * (f)} Greedy is suboptimal due to delayed effects Need long-term planning 4

69 Shortest Path V * (b) = 5 V * (d) = 2 V * (g) = 0 V * (s 0 ) = 6 s 0 b 4 3 d 3 f g 2 c V * (c) = 4 2 e V * (e) = 2 V * (f) = Bellman Equation V * (d) = min{3 + V * (g), 2 + V * (f)} Greedy is suboptimal due to delayed effects Need long-term planning 4

70 Shortest Path V * (b) = 5 V * (d) = 2 V * (g) = 0 V * (s 0 ) = 6 s 0 b 4 3 d 3 f g 2 c V * (c) = 4 2 e V * (e) = 2 V * (f) = Bellman Equation V * (d) = min{3 + V * (g), 2 + V * (f)} Greedy is suboptimal due to delayed effects Need long-term planning 4

71 Shortest Path s 0 b 4 4 d 3 f g 2 c 2 e 5

72 Stochastic Shortest Path s 0 2 b c e d f g 6

73 Stochastic Shortest Path Markov Decision Process (MDP) state s 0 action 2 b c e d f transition distribution g 6

74 Stochastic Shortest Path s 0 2 b c e d f g Greedy is suboptimal due to delayed effects Need long-term planning 7

75 Stochastic Shortest Path s 0 2 b c e d f g Bellman Equation V * (c) = min{ V * (d) V * (e), 2 + V * (e)} 7 Greedy is suboptimal due to delayed effects Need long-term planning

76 Stochastic Shortest Path V * (g) = 0 s 0 2 b c e d f g Bellman Equation V * (c) = min{ V * (d) V * (e), 2 + V * (e)} 7 Greedy is suboptimal due to delayed effects Need long-term planning

77 Stochastic Shortest Path V * (b) = 6 V * (d) = 2 V * (g) = 0 V * (s 0 ) = 6.5 s 0 2 b c V * (c) = e d V * (e) = f V * (f) = g Bellman Equation V * (c) = min{ V * (d) V * (e), 2 + V * (e)} 7 Greedy is suboptimal due to delayed effects Need long-term planning

78 Stochastic Shortest Path V * (s 0 ) = 6.5 s 0 2 V * (b) = 6 V * (d) = 2 b c V * (c) = d e 2 V * (e) = 2.5 Bellman Equation V * (c) = min{ V * (d) V * (e), 2 + V * (e)} 3 f V * (f) = V * (g) = 0 g : optimal policy π * Greedy is suboptimal due to delayed effects Need long-term planning 7

79 Stochastic Shortest Path via trial-and-error s 0 2 b c e? d f g 8

80 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g 8

81 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g 8

82 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g 8

83 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g 8

84 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g 8

85 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g 8

86 Stochastic Shortest Path via trial-and-error s 0 2 b c d e f g Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 9

87 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 9

88 Stochastic Shortest Path via trial-and-error s 0 b 4 4 d 3 f g 2 c 2 e Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 20

89 Stochastic Shortest Path via trial-and-error s 0 b 4 4 d 3 f g 2 c 2 e Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 20

90 Stochastic Shortest Path via trial-and-error s 0 b d 3 f g 2 c 2 e Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 20

91 Stochastic Shortest Path via trial-and-error s 0 2 b c e d f g Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 20

92 Stochastic Shortest Path via trial-and-error s 0 Model-based RL 2 b c e d f g Trajectory : s 0 c d g Trajectory 2: s 0 c e f g 20

93 Stochastic Shortest Path via trial-and-error s 0 Model-based RL 2 b c e d f g Trajectory : s 0 c d g Trajectory 2: s 0 c e f g How many trajectories do we need to compute a near-optimal policy? sample complexity 20

94 Stochastic Shortest Path via trial-and-error s 0 b 4 4 d 3 f g 2 c 2 e How many trajectories do we need to compute a near-optimal policy? Assume states & actions are visited uniformly 2

95 Stochastic Shortest Path via trial-and-error s 0 b 4 4 d 3 f g 2 c 2 e How many trajectories do we need to compute a near-optimal policy? Assume states & actions are visited uniformly #trajectories needed n (#state-action pairs) #samples needed to estimate a multinomial distribution 2

96 Stochastic Shortest Path via trial-and-error s 0 2 b c 4 2 4?? e d?? 3 f g How many trajectories do we need to compute a near-optimal policy? Assume states & actions are visited uniformly #trajectories needed n (#state-action pairs) #samples needed to estimate a multinomial distribution 2

97 Stochastic Shortest Path via trial-and-error s 0 2 b c 4 2 4?? e d?? 3 f g Nontrivial! Need exploration How many trajectories do we need to compute a near-optimal policy? Assume states & actions are visited uniformly #trajectories needed n (#state-action pairs) #samples needed to estimate a multinomial distribution 2

98 Video game playing state s t S 22

99 Video game playing reward r t = R (s t, a t ) state s t S +20 action a t A 22

100 Video game playing reward r t = R (s t, a t ) state s t S +20 action a t A e.g., random spawn of enemies transition dynamics P ( s t, a t ) (unknown) 22

101 Video game playing reward r t = R (s t, a t ) state s t S +20 e.g., random spawn of enemies policy π: S A action a t A transition dynamics P ( s t, a t ) (unknown) objective: maximize E P H t= r t (or E P t= t r t ) 22

102 Video game playing reward r t = R (s t, a t ) state s t S +20 e.g., random spawn of enemies policy π: S A action a t A transition dynamics P ( s t, a t ) (unknown) objective: maximize E P H t= r t (or E P t= t r t ) H =5 γ = H = γ =

103 23 Video game playing

104 Video game playing Need generalization 23

105 Video game playing Need generalization Value function approximation 23

106 Video game playing f (x;θ) state features x Need generalization Value function approximation 24

107 Video game playing f (x;θ) state features x state features x θ f (x;θ) Need generalization Value function approximation 24

108 Video game playing f (x;θ) 24 state features x θ f (x;θ) Find θ s.t. f (x;θ) r + γ Ex x [ f (x ;θ)] state features x Need generalization Value function approximation f ( ; θ) V *

109 Video game playing x f (x;θ) +r x 24 state features x θ f (x;θ) Find θ s.t. f (x;θ) r + γ Ex x [ f (x ;θ)] state features x Need generalization Value function approximation f ( ; θ) V *

110 Adaptive medical treatment value State: diagnosis Action: treatment Reward: progress in recovery state features x 25

111 A Machine Learning view of RL

112 Computer Science Engineering Mathematics Optimal Control Operations Research Machine Learning Reinforcement Learning Bounded Rationality Reward System Classical/Operant Conditioning Neuroscience Psychology Economics 27 slide credit: David Silver

113 Supervised Learning Given {(x (i), y (i) )}, learn f : x y 28

114 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner 28

115 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) 28

116 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) 28

117 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) receives y (t) 28

118 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) receives y (t) Want to maximize # of correct predictions 28

119 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) receives y (t) Want to maximize # of correct predictions e.g., classifies if an image is about a dog, a cat, a plane, etc. (multi-class classification) 28

120 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) receives y (t) Want to maximize # of correct predictions e.g., classifies if an image is about a dog, a cat, a plane, etc. (multi-class classification) Dataset is fixed for everyone 28

121 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) receives y (t) Want to maximize # of correct predictions e.g., classifies if an image is about a dog, a cat, a plane, etc. (multi-class classification) Dataset is fixed for everyone Full information setting 28

122 Supervised Learning Given {(x (i), y (i) )}, learn f : x y Online version: for round t =, 2,, the learner observes x (t) predicts y ^ (t) receives y (t) Want to maximize # of correct predictions e.g., classifies if an image is about a dog, a cat, a plane, etc. (multi-class classification) Dataset is fixed for everyone Full information setting Core challenge: generalization 28

123 Contextual bandits For round t =, 2,, the learner 29

124 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A 29

125 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A Receives reward r (t) ~ R(x (t), a (t) ) (i.e., can be random) 29

126 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A Receives reward r (t) ~ R(x (t), a (t) ) (i.e., can be random) Want to maximize total reward 29

127 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A Receives reward r (t) ~ R(x (t), a (t) ) (i.e., can be random) Want to maximize total reward You generate your own dataset {(x (t), a (t), r (t) )}! 29

128 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A Receives reward r (t) ~ R(x (t), a (t) ) (i.e., can be random) Want to maximize total reward You generate your own dataset {(x (t), a (t), r (t) )}! e.g., for an image, the learner guesses a label, and is told whether correct or not (reward = if correct and 0 otherwise). Do not know what s the true label. 29

129 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A Receives reward r (t) ~ R(x (t), a (t) ) (i.e., can be random) Want to maximize total reward You generate your own dataset {(x (t), a (t), r (t) )}! e.g., for an image, the learner guesses a label, and is told whether correct or not (reward = if correct and 0 otherwise). Do not know what s the true label. e.g., for an user, the website recommends a movie, and observes whether the user likes it or not. Do not know what movies the user really want to see. 29

130 Contextual bandits For round t =, 2,, the learner Given x i, chooses from a set of actions a i A Receives reward r (t) ~ R(x (t), a (t) ) (i.e., can be random) Want to maximize total reward You generate your own dataset {(x (t), a (t), r (t) )}! e.g., for an image, the learner guesses a label, and is told whether correct or not (reward = if correct and 0 otherwise). Do not know what s the true label. e.g., for an user, the website recommends a movie, and observes whether the user likes it or not. Do not know what movies the user really want to see. Partial information setting 29

131 Contextual bandits Contextual Bandits (cont.) 30

132 Contextual bandits Contextual Bandits (cont.) Simplification: no x, Multi-Armed Bandits (MAB) 30

133 Contextual bandits Contextual Bandits (cont.) Simplification: no x, Multi-Armed Bandits (MAB) Bandit is a research area by itself. we will not do a lot of bandits but may go through some material that have important implications on general RL (e.g., lower bounds) 30

134 RL For round t =, 2,, 3

135 RL For round t =, 2,, For time step h=, 2,, H, the learner 3

136 RL For round t =, 2,, For time step h=, 2,, H, the learner Observes x h (t) 3

137 RL For round t =, 2,, For time step h=, 2,, H, the learner Observes x h (t) Chooses a h (t) 3

138 RL For round t =, 2,, For time step h=, 2,, H, the learner Observes x h (t) Chooses a h (t) Receives r h (t) ~ R(x h (t), a h (t)) 3

139 RL For round t =, 2,, For time step h=, 2,, H, the learner Observes x h (t) Chooses a h (t) Receives r h (t) ~ R(x h (t), a h (t)) Next x h+ (t) is generated as a function of x h (t) and a h (t) (or sometimes, all previous x s and a s within round t) 3

140 RL For round t =, 2,, For time step h=, 2,, H, the learner Observes x h (t) Chooses a h (t) Receives r h (t) ~ R(x h (t), a h (t)) Next x h+ (t) is generated as a function of x h (t) and a h (t) (or sometimes, all previous x s and a s within round t) Bandits + Delayed rewards/consequences 3

141 RL For round t =, 2,, For time step h=, 2,, H, the learner Observes x h (t) Chooses a h (t) Receives r h (t) ~ R(x h (t), a h (t)) Next x h+ (t) is generated as a function of x h (t) and a h (t) (or sometimes, all previous x s and a s within round t) Bandits + Delayed rewards/consequences The protocol here is for episodic RL (each t is an episode). 3

142 32 Why statistical RL?

143 Why statistical RL? Two types of scenarios in RL research 32

144 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach 32

145 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics 32

146 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics - Transition dynamics (Go rules) known, but too many states 32

147 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics - Transition dynamics (Go rules) known, but too many states - Run the simulator to collect data 32

148 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics - Transition dynamics (Go rules) known, but too many states - Run the simulator to collect data 2. Solving a learning problem 32

149 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics - Transition dynamics (Go rules) known, but too many states - Run the simulator to collect data 2. Solving a learning problem - e.g., adaptive medical treatment 32

150 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics - Transition dynamics (Go rules) known, but too many states - Run the simulator to collect data 2. Solving a learning problem - e.g., adaptive medical treatment - Transition dynamics unknown (and too many states) 32

151 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach - e.g., AlphaGo, video game playing, simulated robotics - Transition dynamics (Go rules) known, but too many states - Run the simulator to collect data 2. Solving a learning problem - e.g., adaptive medical treatment - Transition dynamics unknown (and too many states) - Interact with the environment to collect data 32

152 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach 2. Solving a learning problem 33

153 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach 2. Solving a learning problem I am more interested in #2. More challenging in some aspects. 33

154 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach 2. Solving a learning problem I am more interested in #2. More challenging in some aspects. Data (real-world interactions) is highest priority. Computation second. 33

155 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach 2. Solving a learning problem I am more interested in #2. More challenging in some aspects. Data (real-world interactions) is highest priority. Computation second. Even for #, sample complexity lower-bounds computational complexity statistical-first approach is reasonable. 33

156 Why statistical RL? Two types of scenarios in RL research. Solving a large planning problem using a learning approach 2. Solving a learning problem I am more interested in #2. More challenging in some aspects. Data (real-world interactions) is highest priority. Computation second. Even for #, sample complexity lower-bounds computational complexity statistical-first approach is reasonable. caveat to this argument: you can do a lot more in a simulator; see 33

157 MDP Planning

158 35 Infinite-horizon discounted MDPs

159 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) 35

160 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. 35

161 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. Action space A. 35

162 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. Action space A. We will only consider discrete and finite spaces in this course. 35

163 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. Action space A. We will only consider discrete and finite spaces in this course. Transition function P : S A (S). (S) is the probability simplex over S, i.e., all non-negative vectors of length S that sums up to 35

164 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. Action space A. We will only consider discrete and finite spaces in this course. Transition function P : S A (S). (S) is the probability simplex over S, i.e., all non-negative vectors of length S that sums up to Reward function R: S A R. (deterministic reward function) 35

165 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. Action space A. We will only consider discrete and finite spaces in this course. Transition function P : S A (S). (S) is the probability simplex over S, i.e., all non-negative vectors of length S that sums up to Reward function R: S A R. (deterministic reward function) Discount factor γ [0,) 35

166 Infinite-horizon discounted MDPs An MDP M = (S, A, P, R, γ) State space S. Action space A. We will only consider discrete and finite spaces in this course. Transition function P : S A (S). (S) is the probability simplex over S, i.e., all non-negative vectors of length S that sums up to Reward function R: S A R. (deterministic reward function) Discount factor γ [0,) The agent starts in some state s, takes action a, receives reward r 2 ~ R(s, a ), transitions to s 2 ~ P(s, a ), takes action a 2, so on so forth the process continues indefinitely 35

167 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] 36

168 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act 36

169 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act Often assume boundedness of rewards: r t [0, R max ] 36

170 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act Often assume boundedness of rewards: What s the range of E [? t= γt r t ] r t [0, R max ] 36

171 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act Often assume boundedness of rewards: What s the range of E [? t= γt r t ] r t [0, R max ] [ 0, R max γ ] 36

172 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act Often assume boundedness of rewards: What s the range of E [? t= γt r t ] A (deterministic) policy ": S A describes how the agent acts: at state s t, chooses action a t = "(s t ). r t [0, R max ] [ 0, R max γ ] 36

173 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act Often assume boundedness of rewards: What s the range of E [? t= γt r t ] A (deterministic) policy ": S A describes how the agent acts: at state s t, chooses action a t = "(s t ). r t [0, R max ] [ 0, R max γ ] More generally, the agent may choose actions randomly (": S (A)), or even in a way that varies across time steps ( nonstationary policies ) 36

174 Value and policy Want to take actions in a way that maximizes value (or return): E [ t= γt r t ] This value depends on where you start and how you act Often assume boundedness of rewards: What s the range of E [? t= γt r t ] A (deterministic) policy ": S A describes how the agent acts: at state s t, chooses action a t = "(s t ). r t [0, R max ] [ 0, R max γ ] More generally, the agent may choose actions randomly (": S (A)), or even in a way that varies across time steps ( nonstationary policies ) Define V π (s) = E [ t= γt r t s = s, π ] 36

175 <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> " X V (s) =E " Bellman equation for policy evaluation t= = E r + # t r t s = s, # X t r t s = s, t=2 = R(s, (s)) + X " X P (s 0 t s, (s)) E 2 r t s = s, s 2 = s 0, s 0 2S t=2 = R(s, (s)) + X " # X P (s 0 t s, (s)) E 2 r t s 2 = s 0, s 0 2S t=2 = R(s, (s)) + X " X P (s 0 s, (s)) E s 0 2S t= = R(s, (s)) + X s 0 2S P (s 0 s, (s)) V (s 0 ) # t r t s = s 0, # = R(s, (s)) + hp ( s, (s)), V ( )i 37

176 <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> <latexit sha_base64="eoquviw3eaqrldri0wywmhhoohu=">aaafnniczvrbaxnbfj52v63xluqjlwedjse07azbxwpfexzwiv6sfjljmjuzbiboxpizfcj2+6d88xf4hcffphvn+dsjoayvqufwyfhd+fyfd85nb0/evxpxznz2ltss5evbf2txlt+4+at6vbtvoptsvmpxikwbz5rtpci9ttxgh0kkphqf2zfp3xwxpffmal4hl3vs4qnqxjefmip0cblbvsv+6mmjzxvqcbswonjamah0vpfz57nwlcjhmcvhl6md98hhk00tpaaqldmsr0jpud9dqcy58hr6a84cofhhewoyjetdvq4wrv4gwpuah/ctoxjqalgyvg6plyqab5hjyqqyuyjyn7koxqbqn40rykzc6tpksff5twovua8jqfuf8/rz5t2yrbhstcubfnh4pb7nq9+tdqqajamaoc6tjwkzitvaay2dr7lct0rpz2k554lthlowawpyuv/2ixzfnqxzpkohsa9dj9dajunmqwf7bqwijoyckyanjrirkapivv30o94ndjnymhtpkl2rfrkjlzqfvsks5qlwy4xzrnggzmnw4xhsapzrodeksajqf4q2dmjanazixbqorgk9apkyrq89juzbdc9zzpg/o23xa7qthtb2ni3fsobvohmogfzge+gf6qieotz768t6bh2xp9if7k/2t3nq5sai5g765djffwcc5qb0</latexit> " X V (s) =E " Bellman equation for policy evaluation t= = E r + # t r t s = s, # X t r t s = s, t=2 = R(s, (s)) + X " X P (s 0 t s, (s)) E 2 r t s = s, s 2 = s 0, s 0 2S t=2 = R(s, (s)) + X " # X P (s 0 t s, (s)) E 2 r t s 2 = s 0, s 0 2S t=2 = R(s, (s)) + X " X P (s 0 s, (s)) E s 0 2S t= = R(s, (s)) + X s 0 2S P (s 0 s, (s)) V (s 0 ) # t r t s = s 0, # = R(s, (s)) + hp ( s, (s)), V ( )i 37

177 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S 38

178 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S R " as the vector [R(s, "(s))] s S 38

179 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S R " as the vector [R(s, "(s))] s S P " as the matrix [P(s s, "(s))] s S, s S 38

180 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S R " as the vector [R(s, "(s))] s S P " as the matrix [P(s s, "(s))] s S, s S V π = R π + γp π V π 38

181 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S R " as the vector [R(s, "(s))] s S P " as the matrix [P(s s, "(s))] s S, s S V π = R π + γp π V π (I γp π )V π = R π 38

182 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S R " as the vector [R(s, "(s))] s S P " as the matrix [P(s s, "(s))] s S, s S V π = R π + γp π V π (I γp π )V π = R π V π = (I γp π ) R π 38

183 Bellman equation for policy evaluation V π (s) = R(s, π(s)) + γ P( s, π(s)), V π ( ) Matrix form: define V " as the S vector [V " (s)] s S R " as the vector [R(s, "(s))] s S P " as the matrix [P(s s, "(s))] s S, s S V π = R π + γp π V π (I γp π )V π = R π 38 V π = (I γp π ) R π This is always invertible. Proof?

184 State occupancy (I γp π ) 39

185 State occupancy (I γp π ) Each row (indexed by s) is the discounted state occupancy d s ", whose (s )-th entry is 39

186 State occupancy (I γp π ) Each row (indexed by s) is the discounted state occupancy d " s, whose (s )-th entry is ds π(s ) = E γ [ t I[s t = s ] s = s, π t= ] 39

187 State occupancy (I γp π ) Each row (indexed by s) is the discounted state occupancy d " s, whose (s )-th entry is ds π(s ) = E γ [ t I[s t = s ] s = s, π t= ] Each row is like a distribution vector except that the entries sum up to /(-γ). Let denote the normalized vector. η π s = ( γ) dπ s 39

188 State occupancy (I γp π ) Each row (indexed by s) is the discounted state occupancy d " s, whose (s )-th entry is ds π(s ) = E γ [ t I[s t = s ] s = s, π t= ] Each row is like a distribution vector except that the entries sum up to /(-γ). Let denote the normalized vector. V " (s) is the dot product between and reward vector η π s = ( γ) dπ s d π s 39

189 State occupancy (I γp π ) Each row (indexed by s) is the discounted state occupancy d " s, whose (s )-th entry is Each row is like a distribution vector except that the entries sum up to /(-γ). Let denote the normalized vector. V " (s) is the dot product between and reward vector ds π(s ) = E γ [ t I[s t = s ] s = s, π t= ] η π s = ( γ) dπ s Can also be interpreted as the value function of indicator reward function d π s 39

190 Optimality For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) 40

191 Optimality For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) Let "* denote this optimal policy, and V* := V "* 40

192 For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) Let "* denote this optimal policy, and V* := V "* Bellman Optimality Equation: Optimality V (s) = max a A (R(s, a) + γe s P(s,a) [V (s )]) 40

193 For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) Let "* denote this optimal policy, and V* := V "* Bellman Optimality Equation: Optimality V (s) = max a A (R(s, a) + γe s P(s,a) [V (s )]) If we know V*, how to get "*? 40

194 For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) Let "* denote this optimal policy, and V* := V "* Bellman Optimality Equation: Optimality V (s) = max a A (R(s, a) + γe s P(s,a) [V (s )]) If we know V*, how to get "*? 40

195 For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) Let "* denote this optimal policy, and V* := V "* Bellman Optimality Equation: Optimality V (s) = max a A (R(s, a) + γe s P(s,a) [V (s )]) If we know V*, how to get "*? Easier to work with Q-values: Q*(s, a), as π (s) = arg max a A Q (s, a) 40

196 For infinite-horizon discounted MDPs, there always exists a stationary and deterministic policy that is optimal for all starting states simultaneously Proof: Puterman 94, Thm (reference due to Shipra Agrawal) Let "* denote this optimal policy, and V* := V "* Bellman Optimality Equation: Optimality V (s) = max a A (R(s, a) + γe s P(s,a) [V (s )]) If we know V*, how to get "*? Easier to work with Q-values: Q*(s, a), as π (s) = arg max a A Q (s, a) 40 Q (s, a) = R(s, a) + γe s P(s,a) [ max a A Q (s, a ) ]

197 4 Ad Hoc Homework

198 Ad Hoc Homework uploaded on course website 4

199 Ad Hoc Homework uploaded on course website help understand the relationships between alternative MDP formulations 4

200 Ad Hoc Homework uploaded on course website help understand the relationships between alternative MDP formulations more like readings w/ questions to think about 4

201 Ad Hoc Homework uploaded on course website help understand the relationships between alternative MDP formulations more like readings w/ questions to think about no need to submit 4

202 Ad Hoc Homework uploaded on course website help understand the relationships between alternative MDP formulations more like readings w/ questions to think about no need to submit will go through in class next week 4