Course basics CSE 190: Reinforcement Learning: An Introduction The website for the class is linked off my homepage. Grades will be based on programming assignments, homeworks, and class participation. Homeworks will be turned in, but not graded, as we will discushe answers in class in small groups. Turning it in means I can see that you are holding up your end of the conversation (this is a major part of class participation) Programming assignments will be graded Acknowledgment: A good number of these slides are cribbed from Rich Sutton Any email sent to me about the course should have CSE 190 in the subject line 2 Course goals After taking this course you should: Understand what is unique about Reinforcement Learning Understand the tradeoff between exploration and exploitation Be conversant in Markov Decision Problems (MDPs) Know the various solution methods for solving the RL problem: Dynamic programming (value iteration, policy iteration, etc.) Monte Carlo TD learning Know what an eligibility trace is Be aware of several well-known applications of RL Be able to read papers in the field and understand 75% of each paper. Last Time Difference from other forms of learning: Learning by interaction with environment, which leado The exploration/exploitation tradeoff: An agent learning by interacting with the environment must: Exploit its knowledge to maximize reward Explore the environment to ensure that its knowledge is correct The agent must try everything while favoring, over time, the most rewarding actions. Elements of RL: a policy a reward function a value function optionally, a model of the environment. 3 4
Last Time Elements of RL: a policy: a mapping from stateo actions, possibly stochastic a reward function: given as part of the problem a value function: A prediction of reward from a state optionally, a model of the environment. This should (almost) all be familiar from your programming assignment. Last Time: Elements of RL a policy: A mapping from stateo actions: (s) = a Stochastic policy: (s,a) = P(a s) a reward function: Specified in the environment, not in the agent Usually a scalar value at each state a value function: A mapping from stateo expected total rewards from this state if we follow policy Written: V (s) A model of the environment: Something that tells us what to expect if we take an action in a state: a P ss ' i.e., the probability of getting to state s from state s if we take action a. 5 6 Elements of RL A model of the environment supports planning through simulating the future (if I do this, then he ll do that ) In general, RL agents can span the gamut from reactive to far-sighted. Example 1: Tic-tac-toe Since one can always play to a draw, let s assume an imperfect opponent. Reinforcement learning approach: Set up a table, V[s i ], i=1 n, where n ihe number of possible states of the board, and s i is a state Each entry of V[] is an estimate of the probability of a win from that state: the value of that state Assume we always play X s - Initialize V as: V[s i = a state with three X s in a row] = 1 V[s i = a state with three O s in a row] = -1 V[s i = all other states] = 0.5 7 8
Example 1: Tic-tac-toe Reinforcement learning approach: Set up a table, V[s i ], i=1 n, where n ihe number of possible states of the board, and s i is a state Play many games against our imperfect opponent to learn values of states How do we play? We need a policy. Let s use one called -greedy For each move, of the time, we pick a move uniformly at random from the possible moves. Otherwise, we pick the move that gets uo the state with the highest value: V[s i ] (greedy). Example 1: Tic-tac-toe -greedy policy (details): % s k randomly, uniformly over states s k reachable from s i, with probability " (s i ) = $ arg maxv(s k ) over states s k reachable from s i, with probability 1-" k & % The case is exploration. The other (greedy, i.e., take the highest value state) case is exploitation 9 10 Example 1: Tic-tac-toe Example 1: Tic-tac-toe How do we update previous values? V(s i ) = V(s i ) + [V( s ") V(s i )] where s ihe state reached after the opponent s move and is a learning rate. This is called a temporal difference method because it uses values of states from two different time steps. It provably convergeo the optimal policy. Why do I say that updating the value estimates changehe policy? 11 12
Example 1: Tic-tac-toe Why does updating the V s change "? V(s i ) = V(s i ) + [V( s ") V(s i )] Because the policy is a function of V: % s k randomly, uniformly over states s k reachable from s i, with probability " (s i ) = $ arg maxv(s k ) over states s k reachable from s i, with probability 1-" k & % As we get better value estimates, we make better choices. 13 Example 2: n-armed bandit n = 10 possible actions Each Q * (a i ) is chosen randomly from a normal distrib.: Q * (a i ) ~N (0,1) each r t is also normal: r t ~N (Q * (a i ),1) Estimate Q-values using a running average: Q t (a) = r 1 + r 2 +...+ r ka k a Policy is -greedy Use 1000 plays repeat the whole thing 2000 times and average the results Hopefully, this is what you did for your programming assignment 14 Example 2: n-armed bandit We compare three policies: greedy, -greedy with = 0.1 and 0.01. Note that greedy never explores. Example 2: n-armed bandit We compare three policies: greedy, -greedy with = 0.1 and 0.01. Note that greedy never explores. And this is your programming assignment 15 16
Another approach to exploration: The Softmax policy Another approach to exploration: The Softmax policy P(a) = " e Q t (a ) n b=1 e Q t (b ) P(a) = " e Q t (a ) P(a) ihe probability of taking action a Q t (a) ihe current estimate of Q * (a) The higher Q t (a) is, the more likely we will choose action a. ihe temperature: this is annealed (starts hot and slowly cools over trials). As $0, the policy becomes deterministic. n b=1 e Q t (b ) 17 18 Another approach to exploration: The Softmax policy P(a) = " e Q t (a ) b=1 e Q t (b ) Why use this? Note that -greedy keeps exploring randomly long after it has learned what the right values are. Softmax quickly gives states or actions with better evidence higher priority, while still exploring. n Incremental estimation of Q s Recall (what you should have implemented in your programming assignment:) the sample average method for estimating Q k (a): Q t (a) = r 1 + r 2 +... + r ka k a Can we do this incrementally (without storing all of the rewards)? (a) = Q k (a) + 1 [ k + 1 r k +1 Q k (a)] Where k ihe number of times we have tried action a 19 20
Incremental estimation of Q s (a) = Q k (a) + 1 [ k + 1 r Q (a) k +1 k ] Note that this is a common form of update rules: New Estimate = Old Estimate + step-size[target-old Estimate] Note in the version above, the step size changes over time. 21 Tracking nonstationary problems Choosing Q k to be a sample average is appropriate in a stationary problem, i.e., when none of the Q * (a) change over time. When the Q * (a) change over time, this is called a non-stationary problem. In this case, the following is better: (a) = Q k (a) + [ r k +1 " Q k (a)] Where % is a constant between zero and one. (a) = (1 ")Q k (a) + "r k +1 (a) = (1 ") k Q 0 (a) + i=1 k "(1 ") k i r i 22 Tracking nonstationary problems Optimistic Initial Values The rule for non-stationary problems: (a) = Q k (a) + [ r k +1 " Q k (a)] (a) = (1 ") k Q 0 (a) + I.e., a recency-weighted exponential average. k i=1 "(1 ") k i r i All of the methods above are biased by the initial values, although, in the one we just looked at, the dependence on Q 0 clearly decreases over time: (a) = (1 ") k Q 0 (a) + This is bad: more parametero pick This is good: prior knowledge can be incorporated into the initial Q-values. k i=1 "(1 ") k i r i 23 24
Optimistic Initial Values One way to biahe model io use optimistic initial values. Suppose instead of 0 for the n-armed bandit, we used +5. Now everything looks good, especially to a greedy method So this encourages exploration: Optimistic Initial Values This looks good, but why doehe greedy method oscillate in the beginning? 25 26 Pause for effect Chapter 3: The Reinforcement Learning Problem Objectives of what I will talk about from this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which we have precise theoretical results; introduce key components of the mathematics: value functions and Bellman equations; describe trade-offs between applicability and mathematical tractability. 28
Chapter 3: The Reinforcement Learning Problem Objectives of what I will talk about from this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which we have precise theoretical results; introduce key components of the mathematics: value functions and Bellman equations; describe trade-offs between applicability and mathematical tractability. The Agent-Environment Interface Agent and environment interact at discrete time steps : t = 0, 1, 2, K Agent observes state at step t : S produces action at step t : a t A( ) gets resulting reward : r t +1 " and resulting next state : +1 29... r t +1 r +2 a t +1 s r t +3 a t +2 st t +1 a +3... t +2 t a t +3 30 The Agent Learns a Policy Policy at step t, t : a mapping from stateo action probabilities t (s,a) = probability that a t = a when = s Reinforcement learning methods specify how the agent changes its policy as a result of experience. Roughly, the agent s goal io get as much reward as it can over the long run. Getting the Degree of Abstraction Right Time steps need not refer to fixed intervals of real time. Actions can be low level (e.g., voltageo motors), or high level (e.g., accept a job offer), mental (e.g., shift in focus of attention), etc. States can low-level sensations, or they can be abstract, symbolic, based on memory, or subjective (e.g., the state of being surprised or lost ). An RL agent is not necessarily like a whole animal or robot. Rewards are in the agent s environment because the agent cannot change it arbitrarily - otherwise, it could simply reward itself and call it a day The environment is not necessarily unknown to the agent, only incompletely controllable. 31 32
Goals and Rewards Is a scalar reward signal an adequate notion of a goal? maybe not, but it is surprisingly flexible. A goal should specify what we want to achieve, not how we want to achieve it. A goal must be outside the agent s direct control thus outside the agent. The agent must be able to measure success: explicitly; frequently during its lifespan. The reward hypothesis The reward hypothesis: All of what we mean by goals and purposes can be thought of ahe maximization of the cumulative sum of a received scalar signal (reward) A sort of null hypothesis. Probably ultimately wrong, but so simple we have to disprove it before considering anything more complicated 33 34 Returns Returns for Continuing Tasks Suppose the sequence of rewards after step t is: r t +1, r t +2, r t + 3, What do we want to maximize? In general, we want to maximize the expected return, E{ R t }, for each step t. Episodic tasks: interaction breaks naturally into episodes, e.g., plays of a game, triphrough a maze. R t = r t +1 + r t +2 ++ r T, where T is a final time step at which a terminal state is reached, ending an episode. Continuing tasks: interaction does not have natural episodes. Instead, we use the Discounted return: R t = r t +1 + r t +2 + 2 r t + 3 + = k r t + k +1, " k =0 where, 0 $ $ 1, ihe discount rate. This ensurehat the expected reward converges. shortsighted 0 " 1 farsighted 35 36
An Example Avoid failure: the pole falling beyond a critical angle or the cart hitting end of track. Another Example Get to the top of the hill as quickly as possible. As an episodic task where episode ends upon failure: reward = +1 for each step before failure return = number of steps before failure As a continuing task with discounted return: reward = 1 upon failure; 0 otherwise " return = k, for k steps before failure In either case, return is maximized by avoiding failure for as long as possible. 37 reward = 1 for each step where not at top of hill " return = number of steps before reaching top of hill Return is maximized by minimizing number of steps to reach the top of the hill. 38 A Unified Notation In episodic tasks, we number the time steps of each episode starting from zero. We usually do not have to distinguish between episodes, so we write instead of, j for the state at step t of episode j. Think of each episode as ending in an absorbing state that always produces reward of zero: We can cover all cases by writing R t = k r t + k +1, where can be 1 only if a zero reward absorbing state is always reached. k =0 " 39 The Markov Property By the state at step t, the book means whatever information is available to the agent at step t about its environment. The state can include immediate sensations, highly processed sensations, and structures built up over time from sequences of sensations. Ideally, a state should summarize past sensations so ao retain all essential information, i.e., it should have the Markov Property: Pr +1 = s,r t +1 = r,a t,r t, "1,a t "1,,r 1,s 0,a 0 { } = Pr{ +1 = s,r t +1 = r,a t } for all s, r, and histories,a t,r t, "1,a t "1,,r 1, s 0,a 0. 40
Markov Decision Processes An Example Finite MDP If a reinforcement learning task hahe Markov Property, it is basically a Markov Decision Process (MDP). If state and action sets are finite, it is a finite MDP. To define a finite MDP, you need to give: state and action sets one-step dynamics defined by transition probabilities: P a s s = Pr{ +1 = s = s,a t = a} for all s, s "S, a "A(s). Recycling Robot At each step, robot hao decide whether it should (1) actively search for a can, (2) wait for someone to bring it a can, or (3) go to home base and recharge. Searching is better but runs down the battery; if runs out of power while searching, hao be rescued (which is bad). Decisions made on basis of current energy level: high, low. Reward = number of cans collected reward probabilities: a R s s = E{ r t +1 = s,a t = a, +1 = s } for all s, s "S, a "A(s). 41 42 Recycling Robot MDP Value Functions { } { } { } S = high, low A(high) = search, wait A(low) = search, wait, recharge R search = expected no. of cans while searching R wait = expected no. of cans while waiting R search > R wait The value of a state ihe expected return starting from that state; depends on the agent s policy: State - value function for policy : { } = E & $ " k r t +k +1 = s V (s) = E R t = s The value of taking an action in a state under policy ihe expected return starting from that state, taking that action, and thereafter following : % ' k =0 ( ) * 43 Action- value function for policy : { } = E & $ " k r t + k +1 = s,a t = a Q (s, a) = E R t = s, a t = a CSE 190: Reinforcement Learning, k Lecture = 0 2 % ' ( ) * 44
Bellman Equation for a Policy More on the Bellman Equation The basic idea: R t = r t +1 + r t +2 + 2 r t + 3 + 3 r t + 4 ( ) = r t +1 + r t +2 + r t + 3 + 2 r t + 4 ( V a (s) = (s,a) P s s" $% R a s s" + V ( s ")& ' a ( s" This is a set of equations (in fact, linear), one for each state. The value function for is its unique solution. = r t +1 + R t +1 So: V (s) = E R t = s { } { } = E r t +1 + " V ( +1 ) = s Backup diagrams: Or, without the expectation operator: ( ( V a (s) = (s,a) P s s" $% R a s s" + V ( s ")& ' a s" 45 for V for Q 46 Gridworld Golf Actions: north, south, east, west; deterministic. If would take agent off the grid: no move but reward = 1 Other actions produce reward = 0, except actionhat move agent out of special states A and B as shown. State-value function for equiprobable random policy; & = 0.9 State is ball location Reward of 1 for each stroke until the ball is in the hole Value of a state? Actions: putt (use putter) driver (use driver) putt succeeds anywhere on the green 47 48
Optimal Value Functions Optimal Value Function for Golf For finite MDPs, policies can be partially ordered: " if and only if V (s) " V (s) for all s $S There are always one or more policiehat are better than or equal to all the others. These are the optimal policies. We denote them all *. Optimal policies share the same optimal state-value function: V (s) = maxv " (s) for all s S " Optimal policies also share the same optimal action-value function: Q (s,a) = maxq " (s,a) for all s S and a A(s) " This ihe expected return for taking action a in state s and thereafter following an optimal policy. 49 We can hit the ball farther with driver than with putter, but with less accuracy Q*(s,driver) givehe value or using driver first, then using whichever actions are best 50 Bellman Optimality Equation for V* The value of a state under an optimal policy must equal the expected return for the best action from that state: V (s) = max a"a(s) Q (s,a) { } = max E r t +1 + $ V ( +1 ) = s,a t = a a"a(s) a = max & P s s% '( R a s s% + $ V ( s %)) * a"a(s) s% The relevant backup diagram: Bellman Optimality Equation for Q* Q (s,a) = E{ r t +1 + " maxq ( +1, a ) = s,a t = a} ( a = P $ s s % s a R s s The relevant backup diagram: a + " maxq ( s, a ) a & ' V * ihe unique solution of this system of nonlinear equations. 51 Q * ihe unique solution of this system of nonlinear equations. 52
Why Optimal State-Value Functions are Useful Any policy that is greedy with respect to V* is an optimal policy. E.g., back to the gridworld: Therefore, given V*, one-step-ahead search producehe long-term optimal actions. What About Optimal Action-Value Functions? Q * Given, the agent does not even have to do a one-step-ahead search: " (s) = arg max aa(s) Q" (s, a) 53 54 Solving the Bellman Optimality Equation Finding an optimal policy by solving the Bellman Optimality Equation requirehe following: accurate knowledge of environment dynamics; we have enough space and time to do the computation; the Markov Property. How much space and time do we need? polynomial in number of states (via dynamic programming methods; Chapter 4), BUT, number of states is often huge (e.g., backgammon has about 10 20 states). We usually have to settle for approximations. Many RL methods can be understood as approximately solving the Bellman Optimality Equation. 55 Agent-environment interaction States Actions Rewards Policy: stochastic rule for selecting actions Return: the function of future rewards agent trieo maximize Episodic and continuing tasks Markov Property Markov Decision Process Transition probabilities Expected rewards Summary Value functions State-value function for a policy Action-value function for a policy Optimal state-value function Optimal action-value function Optimal value functions Optimal policies Bellman Equations The need for approximation 56
END