Module 8 Linear Programming. CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo

Transcription

1 Module 8 Linear Programming CS 886 Sequential Decision Making and Reinforcement Learning University of Waterloo

2 Policy Optimization Value and policy iteration Iterative algorithms that implicitly solve an optimization problem Can we explicitly write down this optimization problem? Yes, it can be formulated as a linear program 2

3 Primal Linear Program primallp(mdp) min V s w(s)v(s) subject to V s R s, a + γ Pr s s, a V s s, a s return V Variables: V s s Objective: min s w(s)v(s) where w(s) is a weight assigned to state s Constraints: V s R s, a + γ Pr s s, a V s s, a s 3

4 Objective Why do we minimize a weighted combination of the values? Shouldn t we maximize value? Value functions V that satisfy the constraints are upper bounds on the optimal value function V V s V s s Minimizing value ensures that we choose the lowest upper bound min V V(s) = V s s 4

5 Upper bound Theorem: Value functions V that satisfy V s R s, a + γ s Pr s s, a V s s, a are upper bounds on the optimal value function V V s V s s Proof: Since V s R s, a + γ s Pr s s, a V s s, a Then V s max R s, a + γ Pr a s s s, a V s s = H (V)(s) s Furthermore V H V H (H V H V = V 5

6 Weight function (initial state) How do we choose the weight function? If the policy always starts in the same initial state s 0, then set w s = 1 s = s 0 0 otherwise This ensures that w s V s = V (s 0 ) s 6

7 Weight function (any state) If the policy may start in any state, then assign a positive weight to each state, i.e. w s > 0 s This ensures that V is minimized at each s and therefore V s = V s s The magnitude of the weight doesn t matter when the LP is solved exactly. We will revisit the choice of w(s) when we discuss approximate linear programming. 7

8 Optimal Policy Linear program finds V We can extract π from V as usual: π s argmax a R s, a + γ Pr s s, a V (s ) s Or check the active constraints For each s, check which a leads to equality V s = R s, a + γ s Pr s s, a V(s ) V s R s, a + γ s Pr s s, a V s a Set π s a 8

9 Direct Policy Optimization The optimal solution to the primal linear program is V, but we still have to extract π Could we directly optimize π? Yes, by considering the dual linear program 9

10 Dual Linear Program duallp(mdp) max s,a y s, a R(s, a) y subject to a y s, a = b s + γ s,a Pr (s s, a)y s, a y s, a 0 s, a Let π a s = Pr a s = y(s, a)/ a y(s, a) return π s Variables: y s, a s, a frequency of each s, a -pair (proportional to π) Objective: max s,a y s, a R(s, a) y Constraints: a y s, a = b s + γ s,a Pr (s s, a)y s, a 10

11 Duality For every primal linear program in the form min c T x x s. t. Ax b There is an equivalent dual linear program in the form max y bt y s. t. A T y = c and y 0 Interpretation: c = w x = V y π A = I γt a a b = [R a ] a Where min x c T x = max y bt y 11

12 State Frequency Let f(s) be the frequency of s under policy π. 0 step: f 0 s = w(s) 1 step: f 1 s = w s + γ Pr (s s, π s )w s s 2 steps: f 2 s = w s + γ Pr (s s, π s )w s s +γ 2 Pr s s, π s Pr s s, π s w(s) s,s n steps: f n s n = w s n + γ Pr s n s n 1, π s n 1 s n 1 f n 1 (s n 1 ) steps: f s = w s + γ Pr s s, π(s) s f(s) 12

13 State-Action Frequency Let y s, a be the state-action frequency y s, a = π a s f s where π a s = Pr a s is a stochastic policy Then the following equations are equivalent f s = w s + γ s Pr s s, π(s) f(s) a π(a s ) f π s = w s + s Pr s s, a π a s f π (s) a y(s, a ) = w s + s Pr s s, a y(s, a) Constraint of dual LP 13

14 Policy We can recover π from y. y s, a = π a s f s (by definition) π a s = y s,a f s (isolate π) π a s = y s,a a y s,a (by definition) π may be stochastic Actions with non-zero probability are necessarily optimal 14

15 Objective Duality theory guarantees that the objectives of the primal and dual LPs are equal max y s,a y s, a R s, a = min V s w(s) V(s) This means that s,a y s, a R s, a implicitly measures the value of the optimal policy. 15

16 Solution Algorithms Two broad classes of algorithms: Simplex (corner search) Interior point methods (interior iterative methods) Polynomial complexity (MDP is in P, not NP) Many packages for linear programming CPLEX (robust, efficient and free for academia) 16