Internet Monetization March May, 2013
Discrete time Finite A decision process (MDP) is reward process with decisions. It models an environment in which all states are and time is divided into stages. Definition A Process is a tuple S, A, P, R, γ, µ S is a (finite) set of states A is a (finite) set of actions P is a state transition probability matrix, P(s s, a) R is a reward function, R(s, a) = E[r s, a] γ is a discount factor, γ [0, 1] a set of initial probabilities µ 0 i = P(X 0 = i) for all i
Example: Student MDP
Goals and Rewards Is a scalar reward an adequate notion of a goal? Sutton hypothesis: That all of what we mean by goals and purposes can be well thought of as the maximization of the cumulative sum of a received scalar signal (reward) Probably ultimately wrong, but so simple and flexible we have to disprove it before considering anything more complicated A goal should specify what we want to achieve, not how we want to achieve it The same goal can be specified by (infinite) different reward functions A goal must be outside the agent s direct control thus outside the agent The agent must be able to measure success: explicitly frequently during her lifespan
Return Time horizon finite: finite and fixed number of steps indefinite: until some stopping criteria is met (absorbing states) infinite: forever Cumulative reward total reward: V = average reward: i=1 r 1 + + r n V = lim n n discounted reward: V = γ i 1 r i mean variance reward i=1 r i
Infinite horizon Discounted Return Definition The return v t is the total discounted reward from time step t. v t = r t+1 + γr t+2 + = γ k r t+k+1 k=0 The discount γ [0, 1) is the present value of future rewards The value of receiving reward r after k + 1 time steps is γ k r Immediate reward vs delayed reward γ close to 0 leads to myopic evaluation γ close to 1 leads to far sighted evaluation γ can be also interpreted as the probability that the process will go on
Why discount? Most reward (and decision) processes are discounted, why? Mathematically convenient to discount rewards Avoids infinite returns in cyclic processes Uncertainty about the future may not be fully represented If the reward is financial, immediate rewards may earn more interest than delayed rewards Animal/human behavior shows preference for immediate reward It is sometimes possible to use undiscounted reward processes (i.e. γ = 1), e.g. if all sequences terminate
Policies A policy, at any given point in time, decides which action the agent selects A policy fully defines the behavior of an agent Policies can be: ian History dependent Deterministic Stochastic Stationary Non stationary
Stationary Stochastic ian Policies Definition A policy π is a distribution over actions given the state: π(a s) = P[a s] MDP policies depend on the current state (not the history) i.e., Policies are stationary (time independent) Given an MDP M = S, A, P, R, γ, µ and a policy π The state sequence s 1, s 2,... is a process S, P π, µ The state and reward sequence s 1, r 2, s 2,... is a reward process S, P π, R π, γ, µ, where P π = π(a s)p(s s, a) R π = π(a s)r(s, a) a A a A
Value Functions Given a policy π, it is possible to define the utility of each state: Policy Evaluation Definition The state value function V π (s) of an MDP is the expected return starting from state s, and then following policy π V π (s) = E π[v t s t = s] For control purposes, rather than the value of each state, it is easier to consider the value of each action in each state Definition The action value function Q π (s, a) is the expected return starting from state s, taking action a, and then following policy π Q π (s, a) = E π[v t s t = s, a t = a]
Example: Value Function of Student MDP
Bellman Expectation Equation The state value function can again be decomposed into immediate reward plus discounted value of successor state, V π (s) = E π [r t+1 + γv π (s t+1 ) s t = s] ( = a A π(a s) R(s, a) + γ s S P(s s, a)v π (s ) The action-value function can similarly be decomposed ) Q π (s, a) = E π [r t+1 + γq π (s t+1, a t+1 ) s t = s, a t = a] = R(s, a) + γ P(s s, a)v π (s ) s S = R(s, a) + γ P(s s, a) π(a s )Q π (s, a ) s S a A
Bellman Expectation Equation (Matrix Form) The Bellman expectation equation can be expressed concisely using the induced MRP with direct solution V π = R π + γp π V π V π = (I γp π ) 1 R π
Bellman operators for V π Definition The Bellman operator for V π is defined as T π : R S R S (maps value functions to value functions): ( ) (T π V π )(s) = a A π(a s) R(s, a) + γ s S P(s s, a)v π (s ) Using Bellman operator, Bellman expectation equation can be compactly written as: T π V π = V π V π is a fixed point of the Bellman operator T π This is a linear equation in V π and T π If 0 < γ < 1 then T π is a contraction w.r.t. the maximum norm
Bellman operators for Q π Definition The Bellman operator for Q π is defined as T π : R S A R S A (maps action value functions to action value functions): (T π Q π )(s, a) = R(s, a) + γ s S P(s s, a) a A π(a s ) (Q π (s, a )) Using Bellman operator, Bellman expectation equation can be compactly written as: T π Q π = Q π Q π is a fixed point of the Bellman operator T π This is a linear equation in Q π and T π If 0 < γ < 1 then T π is a contraction w.r.t. the maximum norm
Example: Value Function of Student MDP Matlab code % Problem definition gamma = 1; R = [-0.5;-1.5;-1;5.5]; P = [0.5 0.5 0 0;0.5 0 0.5 0; 0 0 0 0.5; 0 0.1 0.2 0.2]; % Closed form computation of the value function I = eye(4); Vcf = inv(i-gamma*p)*r % Iterative computation of the value function V = zeros(4,1); for i = 1:10 V = R + gamma*p*v; error(i) = max(abs(vcf - V)); end % Plotting how the error decrese with the number of iterations plot([1:10],error);
Optimal Value Function Definition The optimal state value function V (s) is the maximum value function over all policies V (s) = max V π (s) π The optimal action value function Q (s, a) is the maximum action value function over all policies Q (s, a) = max Q π (s, a) π The optimal value function specifies the best possible performance in the MDP An MDP is solved when we know the optimal value function
Example: Optimal Value Function of Student MDP
Optimal Policy Value functions define a partial ordering over policies Theorem π π if V π (s) V π (s), For any Process s S There exists an optimal policy π that is better than or equal to all other policies π π, π All optimal policies achieve the optimal value function, V π (s) = V (s) All optimal policies achieve the optimal action value function, Q π (s) = Q (s) There is always a deterministic optimal policy for any MDP A deterministic optimal policy can be found by maximizing over Q (s, a) { π 1 if a = arg max a A Q (s, a) (a s) = 0 otherwise
Example: Optimal Policy for Student MDP
Bellman Optimality Equation Bellman Optimality Equation for V V (s) = max a = max a Q (s, a) { R(s, a) + γ s S P(s s, a)v (s ) } Bellman Optimality Equation for Q Q (s) = R(s, a) + γ s S P(s s, a)v (s ) = R(s, a) + γ s S P(s s, a) max a Q (s, a )
Example: Bellman Optimality Equation in Student MDP
Bellman Optimality Operator Definition The Bellman optimality operator for V is defined as T : R S R S (maps value functions to value functions): ( ) (T V )(s) = max a A R(s, a) + γ s S P(s s, a)v (s ) Definition The Bellman optimality operator for Q is defined as T : R S A R S A (maps action value functions to action value functions): (T Q )(s) = R(s, a) + γ s S P(s s, a) max a Q (s, a )
Properties of Bellman Operators Monotonicity: if f 1 f 2 component wise T π f 1 T π f 2, T f 1 T f 2 Max Norm Contraction: for two vectors f 1 anf f 2 T π f 1 T π f 2 γ f 1 f 2 T f 1 T f 2 γ f 1 f 2 V π is the unique fixed point of T π V is the unique fixed point of T For any vector f R S and any policy π, we have lim k (T π ) k f = V π, lim k (T ) k f = V
Solving the Bellman Optimality Equation Bellman optimality equation is non linear No closed form solution for the general case Many iterative solution methods Linear Reinforcement Learning Q learning SARSA
Extensions to MDP Undiscounted, average reward MDPs Infinite and continuous MDPs Partially observable MDPs Semi-MDPs Non stationary MDPs, games
Exercises Model the following problems But who s counting Blackjack
Brute Force Solving an MDP means finding an optimal policy A naive approach consists of enumerating all the deterministic policies evaluate each policy return the best one The number of policies is exponential: A S Need a more intelligent search for best policies restrict the search to a subset of the possible policies using stochastic optimization algorithms
What is? : sequential or temporal component to the problem : optimizing a program, i.e., a policy c.f. linear programming A method for solving complex problems By breaking them down into subproblems Solve the subproblems Combine solutions to subproblems
Requirements for is a very general solution method for problems which have two properties: Optimal substructure Principle of optimality applies Optimal solution can be decomposed into subproblems Overlapping subproblems Subproblems recur many times Solutions can be cached and reused decision processes satisfy both properties Bellman equation gives revursive decomposition Value function stores and reuses solutions
Planning by assumes full knowledge of the MDP It is used for planning in an MDP Prediction Input: MDP S, A, P, R, γ, µ and policy π (i.e., MRP S, P π, R π, γ, µ ) Output: value function V π Control Input: MDP S, A, P, R, γ, µ Output: value function V and optimal policy π
Other Applications of is used to solve many other problems: Scheduling algorithms String algorithms (e.g., sequence alignment) Graph algorithms (e.g., shortest path algorithms) Graphical models (e.g., Viterbi algorithm) Bioinformatics (e.g., lattice models)
Finite Horizon Principle of optimality: the tail of an optimal policy is optimal for the tail problem Backward induction Backward recursion V k (s) = max a A k R k (s, a) + P k (s s, a)v k+1 (s ) s, k = N 1,..., 0 S k+1 Optimal policy π k (s) arg max a A k R k (s, a) + P k (s s, a)v k+1 (s ) s, k = 0,..., N 1 S k+1 Cost: N S A vs A N S of brute force policy search From now on, we will consider infinite horizon discounted MDPs
Policy Evaluation For a given policy π compute the state value function V π Recall State value function for policy π: { } V π (s) = E γ t r t s 0 = s Bellman equation for V π : [ V π (s) = a A π(a s) t=0 R(s, a) + γ s S P(s s, a)v π (s ) ] A system of S simultaneous linear equations Solution in matrix notation (complexity O(n 3 )): V π = (I γp π ) 1 R π
Iterative Policy Evaluation Iterative application of Bellman expectation backup V 0 V 1 V k V k+1 V π A full policy evaluation backup: V k+1 (s) [ π(a s) R(s, a) + γ ] P(s s, a)v k (s ) a A s S A sweep consists of applying a backup operation to each state Using synchronous backups At each iteration k + 1 For all states s S Update V k+1 (s) from V k (s )
Example Small Gridworld Undiscounted episodic MDP γ = 1 All episodes terminate in absorbing terminal state Transient states 1,..., 14 One terminal state (shown twice as shaded squares) Actions that would take agent off the grid leave state unchanged Reward is 1 until the terminal state is reached
Policy Evaluation in Small Gridworld
Policy Evaluation in Small Gridworld
Policy Improvement Consider a deterministic policy π For a given state s, would it better to do an action a π(s)? We can improve the policy by acting greedily π (s) = arg max a A Qπ (s, a) This improves the value from any state s over one step Q π (s, π (s)) = max a A Qπ (s, a) Q π (s, π(s)) = V π (s)
Policy Improvement Theorem Theorem Let π and π be any pair of deterministic policies such that Q π (s, π (s)) V π (s), s S Then the policy π must be as good as, or better than π V π (s) V π (s), s S Proof. V π (s) Q π (s, π (s)) = E π [r t+1 + γv π (s t+1 ) s t = s] [ E π rt+1 + γq π (s t+1, π (s t+1 )) s t = s ] [ ] E π r t+1 + γr t+2 + γ 2 Q π (s t+2, π (s t+2 )) s t = s E π [r t+1 + γr t+2 +... s t = s] = V π (s)
What if improvements stops (V π = V π )? Q π (s, π (s)) = max a A Qπ (s, a) = Q π (s, π(s)) = V π (s) But this is the Bellman optimality equation Therefore V π (s) = V π (s) = V (s) for all s S So π is an optimal policy! π 0 V π 0 π 1 V π 1 π V π
Modified Does policy evaluation need to converge to V π? Or should we introduce a stopping condition e.g., ɛ convergence of value function Or simply stop after k iterations of iterative policy evaluation? For example, in the small gridworld k = 3 was sufficient to achieve optimal policy Why not update policy every iteration? i.e. stop after k = 1
Generalized Policy evaluation: Estimate V π e.g., Iterative policy evaluation Policy improvement: Generate π π e.g., Greedy policy improvement
Determinisitic If we know the solution to subproblems V (s ) Then it is easy to construct the solution to V (s) V (s) max a A R(s, a) + V (s ) The idea of value iteration is to apply these updates iteratively e.g., Starting from the goal and working backward
in MDPs Many MDPs don t have a finite horizon They are tipically loopy So there is no end to work backwards from However, we can still propagate information backwards Using Bellman optimality equation to backup V (s) from V (s ) Each subproblem is easier due to discount factor γ Iterate until convergence
Principle of Optimality in MDPs Theorem A policy π(a s) achieves the optimal value from state s, V π (s) = V (s), if and only if For any state s reachable from s π achieves the optimal value from state s, V π (s ) = V (s ) So an optimal policy π (a s) must consist of: an optimal first action a followed by an optimal policy from successor state s V (s) = max R(s, a) + γ P(s s, a)v (s ) a A s S
Problem: find optimal policy π Solution: iterative application of Bellman optimality backup V 1 V 2 V Using synchronous backups At each iteration k + 1 For all states s S Update V k+1 (s) from V k (s ) Unlike policy iteration there is no explicit policy Intermediate value functions may not correspond to any policy demo: http://www.cs.ubc.ca/ poole/demos/mdp/vi.html
Convergence and Contractions Define the max norm: V = max s V (s) Theorem converges to the optimal state-value function lim k V k = V Proof. V k+1 V = T V k T V γ V k V γ k+1 V 0 V Theorem V i+1 V i < ɛ V i+1 V < 2ɛγ 1 γ
Synchronous Algorithms Problem Bellman Equation Algorithm Prediction Bellman Expectation Equation Policy Evaluation (Iterative) Control Bellman Expectation + Greedy Policy Improvement Control Bellman Optimality Equation Algorithms are based on state value function V π (s) or V (s) Complexity O(mn 2 ) per iteration, for m actions and n states Could also apply to action value function Q π (s, a) or Q (s, a) Complexity O(m 2 n 2 ) per iteration