Markov Decision Processes Chapter 17. Mausam

Transcription

1 Markov Decision Processes Chapter 17 Mausam

2 Planning Agent Static vs. Dynamic Fully vs. Partially Observable Environment What action next? Deterministic vs. Stochastic Perfect vs. Noisy Instantaneous vs. Durative Percepts Actions 2

3 Classical Planning Static Environment Fully Observable Perfect What action next? Deterministic Instantaneous Percepts Actions 3

4 Stochastic Planning: MDPs Static Environment Fully Observable Perfect What action next? Stochastic Instantaneous Percepts Actions 4

5 MDP vs. Decision Theory Decision theory episodic MDP -- sequential 5

6 Markov Decision Process (MDP) S: A set of states A: A set of actions T(s,a,s ): transition model C(s,a,s ): cost model G: set of goals s 0 : start state : discount factor R(s,a,s ): reward model factored Factored MDP absorbing/ non-absorbing 6

7 Objective of an MDP Find a policy : S A which optimizes minimizes maximizes maximizes discounted or undiscount. expected cost to reach a goal expected reward expected (reward-cost) given a horizon finite infinite indefinite assuming full observability 7

8 Role of Discount Factor ( ) Keep the total reward/total cost finite useful for infinite horizon problems Intuition (economics): Money today is worth more than money tomorrow. Total reward: r 1 + r r 3 + Total cost: c 1 + c c 3 + 8

9 Examples of MDPs Goal-directed, Indefinite Horizon, Cost Minimization MDP <S, A, T, C, G, s 0 > Most often studied in planning, graph theory communities Infinite Horizon, Discounted Reward Maximization MDP <S, A, T, R, > Most often studied in machine learning, economics, operations research communities most popular Oversubscription Planning: Non absorbing goals, Reward Max. MDP <S, A, T, G, R, s 0 > Relatively recent model 9

10 Acyclic vs. Cyclic MDPs a P b a P b Q R S T R S T c c c c c c c G C(a) = 5, C(b) = 10, C(c) =1 Expectimin works V(Q/R/S/T) = 1 V(P) = 6 action a G Expectimin doesn t work infinite loop V(R/S/T) = 1 Q(P,b) = 11 Q(P,a) =???? suppose I decide to take a in P Q(P,a) = * Q(P,a) 10 = 13.5

11 Brute force Algorithm Go over all policies ¼ How many? A S finite Evaluate each policy V ¼ (s) Ã expected cost of reaching goal from s how to evaluate? Choose the best We know that best exists (SSP optimality principle) V ¼ *(s) V ¼ (s) 11

12 Policy Evaluation Given a policy ¼: compute V ¼ V ¼ : cost of reaching goal while following ¼ 12

13 Deterministic MDPs Policy Graph for ¼ ¼(s 0 ) = a 0 ; ¼(s 1 ) = a 1 C=5 C=1 s 0 s 1 s a g 0 a 1 V ¼ (s 1 ) = 1 V ¼ (s 0 ) = 6 add costs on path to goal 13

14 Acyclic MDPs Policy Graph for ¼ s 0 V ¼ (s 1 ) = 1 V ¼ (s 2 ) = 4 Pr=0.6 C=5 s 1 a 0 a 1 Pr=0.4 C=2 s 2 C=1 C=4 V ¼ (s 0 ) = 0.6(5+1) + 0.4(2+4) = 6 a 2 s g backward pass in reverse topological order 14

15 General MDPs can be cyclic! s 0 Pr=0.6 C=5 s 1 a 0 a 1 Pr=0.4 C=2 s 2 a 2 C=1 Pr=0.7 C=4 Pr=0.3 C=3 s g cannot do a simple single pass V ¼ (s 1 ) = 1 V ¼ (s 2 ) =?? (depends on V ¼ (s 0 )) V ¼ (s 0 ) =?? (depends on V ¼ (s 2 )) 15

16 General SSPs can be cyclic! s 0 Pr=0.6 C=5 s 1 a 0 a 1 Pr=0.4 C=2 s 2 V ¼ (g) = 0 V ¼ (s 1 ) = 1+V ¼ (s g ) = 1 V ¼ (s 2 ) = 0.7(4+V ¼ (s g )) + 0.3(3+V ¼ (s 0 )) V ¼ (s 0 ) = 0.6(5+V ¼ (s 1 )) + 0.4(2+V ¼ (s 2 )) a 2 C=1 Pr=0.7 C=4 Pr=0.3 C=3 s g a simple system of linear equations 16

17 Policy Evaluation (Approach 1) Solving the System of Linear Equations V ¼ (s) = 0 if s 2 G = X s 0 2S T (s; ¼(s); s 0 ) [C(s; ¼(s); s 0 ) + V ¼ (s 0 )] S variables. O( S 3 ) running time 17

18 Iterative Policy Evaluation V ¼ (s 2 ) s 0 Pr=0.6 C=5 Pr=0.4 C=2 s 1 s 2 a 2 C=1 a 0 a V ¼ (s 0 ) Pr=0.7 C=4 Pr=0.3 C=3 s g 18

19 Policy Evaluation (Approach 2) V ¼ (s) = X s 0 2S T (s; ¼(s); s 0 ) [C(s; ¼(s); s 0 ) + V ¼ (s 0 )] iterative refinement V ¼ n (s) Ã X s 0 2S T (s; ¼(s); s 0 ) C(s; ¼(s); s 0 ) + V ¼ n 1(s 0 ) 19

20 Iterative Policy Evaluation iteration n ²-consistency termination condition 20

21 Convergence & Optimality For a proper policy ¼ Iterative policy evaluation converges to the true value of the policy, i.e. lim n!1 V ¼ n = V ¼ irrespective of the initialization V 0 21

22 Policy Evaluation Value Iteration (Bellman Equations for MDP 1 ) <S, A, T, C,G, s 0 > Define V*(s) {optimal cost} as the minimum expected cost to reach a goal from this state. V* should satisfy the following equation: V (s) = 0 if s 2 G X = min T (s; a; s 0 ) [C(s; a; s 0 ) + V (s 0 )] a2a s 0 2S Q*(s,a) V*(s) = min a Q*(s,a) 22

23 Bellman Equations for MDP 2 <S, A, T, R, s 0, > Define V*(s) {optimal value} as the maximum expected discounted reward from this state. V* should satisfy the following equation: 23

24 Fixed Point Computation in VI V (s) = min a2a X s 0 2S T (s; a; s 0 ) [C(s; a; s 0 ) + V (s 0 )] iterative refinement V n (s) Ã min a2a X s 0 2S T (s; a; s 0 ) [C(s; a; s 0 ) + V n 1 (s 0 )] non-linear 24

25 Example s 0 a 20 a a 00 s 2 s 40 4 a 41 a 21 a 1 a C=2 3 a 01 s 1 s 3 C=5 Pr=0.6 Pr=0.4 s g 25

26 Bellman Backup s 4 s 3 a 41 a 3 C=2 a 40 C=5 Pr=0.6 Pr=0.4 s g min Q 1 (s 4,a 40 ) = Q 1 (s 4,a 41 ) = = 2.8 V 1 = 2.8 s 4 a greedy = a 41 C=5 C=2 a 40 a 41 s g V 0 = 0 s 3 V 0 = 2

27 Value Iteration [Bellman 57] No restriction on initial value function iteration n ²-consistency termination condition 27

28 Example (all actions cost 1 unless otherwise stated) s 0 a 20 a a 00 s 2 s 40 4 a 41 a 21 a 1 a C=2 3 a 01 s 1 s 3 C=5 Pr=0.6 Pr=0.4 s g n V n (s 0 ) V n (s 1 ) V n (s 2 ) V n (s 3 ) V n (s 4 )

29 Comments Decision-theoretic Algorithm Dynamic Programming Fixed Point Computation Probabilistic version of Bellman-Ford Algorithm for shortest path computation MDP 1 : Stochastic Shortest Path Problem Time Complexity one iteration: O( S 2 A ) number of iterations: poly( S, A, 1/², 1/(1- )) Space Complexity: O( S ) 31

30 Monotonicity For all n>k V k p V * V n p V* (V n monotonic from below) V k p V * V n p V* (V n monotonic from above) 32

31 Changing the Search Space Value Iteration Search in value space Compute the resulting policy Policy Iteration Search in policy space Compute the resulting value 40

32 Policy iteration [Howard 60] assign an arbitrary assignment of 0 to each state. repeat costly: O(n 3 ) Policy Evaluation: compute V n+1 : the evaluation of n Policy Improvement: for all states s compute n+1 (s): argmin a2 Ap(s) Q n+1 (s,a) until n+1 = n Advantage searching in a finite (policy) space as opposed to uncountably infinite (value) space convergence in fewer number of iterations. all other properties follow! Modified Policy Iteration approximate by value iteration using fixed policy 41

33 Modified Policy iteration assign an arbitrary assignment of 0 to each state. repeat Policy Evaluation: compute V n+1 the approx. evaluation of n Policy Improvement: for all states s compute n+1 (s): argmax a2 Ap(s) Q n+1 (s,a) until n+1 = n Advantage probably the most competitive synchronous dynamic programming algorithm. 42

34 Applications Stochastic Games Robotics: navigation, helicopter manuevers Finance: options, investments Communication Networks Medicine: Radiation planning for cancer Controlling workflows Optimize bidding decisions in auctions Traffic flow optimization Aircraft queueing for landing; airline meal provisioning Optimizing software on mobiles Forest firefighting 43