Controlled Diffusions and Hamilton-Jacobi Bellman Equations

Transcription

1 Controlled Diffusions and Hamilton-Jacobi Bellman Equations Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Winter 2014 Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

2 Continuous-time formulation Notation and terminology: x (t) 2 R n u (t) 2 R m ω (t) 2 R k dx = f (x, u) dt + G (x, u) dω Σ (x, u) = G (x, u) G (x, u) T ` (x, u) 0 q T (x) 0 π (x) 2 R m v π (x) 0 π (x), v (x) state vector control vector Brownian motion (integral of white noise) continuous-time dynamics noise covariance cost for choosing control u in state x (optional) scalar cost at terminal states x 2 T control law value/cost-to-go function optimal control law and its value function Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

3 Stochastic differential equations and integrals Ito diffusion / stochastic differential equation (SDE): dx = f (x) dt + g (x) dω This cannot be written as ẋ = f (x) + g (x) ω because ω does not exist. The SDE means that the time-integrals of the two sides are equal: x (T) x (0) = Z T 0 Z T f (x (t)) dt + g (x (t)) dω (t) 0 The last term is an Ito integral. For an Ito process y (t) adapted to ω (t), i.e. depending on the sample path only up to time t, this integral is Definition (Ito integral) Z T y (t) dω (t), lim N 1 0 N! i=0 y (t i) (ω (t i+1 ) ω (t i )) 0=t 0 <t 1 <<t N =T Replacing y (t i ) with y ((t i+1 + t i ) /2) yields the Stratonovich integral. Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

4 Stochastic chain rule and integration by parts A twice-differentiable function a (x) of an Ito diffusion dx = f (x) dt + g (x) dω is an Ito process (not necessarily a diffusion) which satisfies: Lemma (Ito) da (x (t)) = a 0 (x (t)) dx (t) a00 (x (t)) g (x (t)) 2 dt This is the stochastic version of the chain rule. Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

5 Stochastic chain rule and integration by parts A twice-differentiable function a (x) of an Ito diffusion dx = f (x) dt + g (x) dω is an Ito process (not necessarily a diffusion) which satisfies: Lemma (Ito) da (x (t)) = a 0 (x (t)) dx (t) a00 (x (t)) g (x (t)) 2 dt This is the stochastic version of the chain rule. There is also a stochastic version of integration by parts: x (T) y (T) x (0) y (0) = Z T 0 Z T x (t) dy (t) + y (t) dx (t) + [x, y] T 0 The last term (which would be 0 if x (t) or y (t) were differentiable) is Definition (quadratic covariation) [x, y] T, lim N 1 N! i=0 (x (t i+1) x (t i )) (y (t i+1 ) y (t i )) 0=t 0 <t 1 <<t N =T For a diffusion with constant noise amplitude we have [x, x] T = g 2 T. Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

6 Forward and backward equations, generator Let p (y, sjx, t), s t denote the transition probability density under the Ito diffusion dx = f (x) dt + g (x) dω. Then p satisfies the following PDEs: Theorem (Kolmogorov equations) forward (FP) equation backward equation s p = y (fp) y 2 g 2 p t p = f x (p) g2 (p) = L [p (y, sj, t)] 2 x2 Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

7 Forward and backward equations, generator Let p (y, sjx, t), s t denote the transition probability density under the Ito diffusion dx = f (x) dt + g (x) dω. Then p satisfies the following PDEs: Theorem (Kolmogorov equations) forward (FP) equation backward equation s p = y (fp) y 2 g 2 p t p = f x (p) g2 (p) = L [p (y, sj, t)] 2 x2 The operator L which computes expected directional derivatives is called the generator of the stochastic process. It satisfies (in the vector case): Theorem (generator) E L [v ()] (x), x(0)=x [v (x ( ))] v (x) lim = f (x) T v x (x) + 1!0 2 tr (Σ (x) v xx (x)) Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

8 Discretizing the time axis Consider the explicit Euler discretization with time step : x (t + ) = x (t) + f (x (t), u (t)) + p G (x (t), u (t)) ε (t) where ε (t) N (0, I). The term p appears because the variance grows linearly with time. Thus the transition probability p (x 0 jx, u) is Gaussian, with mean x + f (x, u) and covariance matrix Σ (x, u). The one-step cost is ` (x, u). Now we can apply the Bellman equation (in the finite horizon setting): v (x, t) = min n ` (x, u) + E u x 0 p(jx,u) v x 0, t + o = n o min ` (x, u) + E u dn( f(x,u), Σ(x,u)) [v (x + d, t + )] Next we use the Taylor-series expansion of v... Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

9 Hamilton-Jacobi-Bellman (HJB) equation v (x + d, t + ) = v (x, t) + v t (x, t) + o 2 + +d T v x (x, t) dt v xx (x, t) d + o d 3 h i Using the fact that E d T Md = tr (cov [d] M) + o 2, the expectation is E d [v (x + d, t + )] = v (x, t) + v t (x, t) + o f (x, u) T v x (x, t) + 2 tr (Σ (x, u) v xx (x, t)) Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

10 Hamilton-Jacobi-Bellman (HJB) equation v (x + d, t + ) = v (x, t) + v t (x, t) + o 2 + +d T v x (x, t) dt v xx (x, t) d + o d 3 h i Using the fact that E d T Md = tr (cov [d] M) + o 2, the expectation is E d [v (x + d, t + )] = v (x, t) + v t (x, t) + o 2 + Substituting in the Bellman equation, v (x, t) = min u + f (x, u) T v x (x, t) + 2 tr (Σ (x, u) v xx (x, t)) ` (x, u) + v (x, t) + vt (x, t) + o f (x, u) T v x (x, t) + 2 tr (Σ (x, u) v xx (x, t)) Simplifying, dividing by and taking! 0 yields the HJB equation v t (x, t) = min ` (x, u) + f (x, u) T v x (x) + 1 u 2 tr (Σ (x, u) v xx (x)) Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

11 HJB equations for different problem formulations Definition (Hamiltonian) H [x, u, v ()], ` (x, u) + f (x, u) T v x (x) tr (Σ (x, u) v xx (x)) = ` + L [v] The HJB equations for the optimal cost-to-go v are Theorem (HJB equations) first exit 0 = min u H [x, u, v ()] v (x 2 T ) = q T (x) finite horizon v t (x, t) = min u H [x, u, v (, t)] v (x, T) = q T (x) discounted average 1 τ v (x) = min u H [x, u, v ()] c = min u H [x, u, ev ()] Discounted cost-to-go: v π (x) = E R 0 exp ( t/τ) ` (x (t), u (t)) dt. Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

12 Existence and uniqueness of solutions The HJB equation has at most one classic solution (i.e. a function which satisfies the PDE everywhere.) If a classic solution exists then it is the optimal cost-to-go function. The HJB equation may not have a classic solution; in that case the optimal cost-to-go function is non-smooth (e.g. bang-bang control.) The HJB equation always has a unique viscosity solution which is the optimal cost-to-go function. Approximation schemes based on MDP discretization (see below) are guaranteed to converge to the unique viscosity solution / optimal cost-to-go function. Most continuous function approximation schemes (which scale better) are unable to represent non-smooth solutions. All examples of non-smoothness seem to be deterministic; noise tends to smooth the optimal cost-to-go function. Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

13 Example of noise smoothing Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

14 More tractable problems Consider a restricted family of problems with dynamics and cost dx = (a (x) + B (x) u) dt + C (x) dω ` (x, u) = q (x) ut R (x) u For such problems the Hamiltonian can be minimized analytically w.r.t. u. Suppressing the dependence on x for clarity, we have min H = min q + 12 ut Ru + (a + Bu) T v x + 12 CC tr T v xx u u The minimum is achieved at u = min H = q + a T v x + 1 u 2 tr CC T v xx R 1 B T v x and the result is 1 2 vt x BR 1 B T v x Thus the HJB equations become 2nd-order quadratic PDEs, no longer involving the min operator. Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

15 More tractable problems (generalizations) Allowing control-multiplicative noise: Σ (x, u) = C 0 (x) C 0 (x) T + K k=1 C k (x) uu T C k (x) T The optimal control law becomes: u = R + K k=1 CT k v xxc k 1 B T v x Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

16 More tractable problems (generalizations) Allowing control-multiplicative noise: Σ (x, u) = C 0 (x) C 0 (x) T + K k=1 C k (x) uu T C k (x) T The optimal control law becomes: u = R + K k=1 CT k v xxc k 1 B T v x Allowing more general control costs: ` (x, u) = q (x) + i r (u i ), r : convex The optimal control law becomes: n o u = arg min i r (u i ) + u T B T v x = r 0 1 B T v x u Z juj r (u) = s atanh 0 w u max dw =) u = u max tanh s 1 B T v x Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

17 Pendulum example θ = k sin (θ) + u Stochastic dynamics: dx = a (x) dt + B (udt + σdω) Cost and optimal control: First-order form: x1 x = = x 2 θ θ x a (x) = 2 k sin (x 1 ) 0 B = 1 ` (x, u) = q (x) + r 2 u2 u (x) = r 1 v x2 (x) HJB equation (discounted): 1 τ v = q + x 2v x1 + k sin (x 1 ) v x2 + σ2 2 v x 2 x 2 1 2r v2 x 2 Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

18 Pendulum example continued Parameters: k = σ = r = 1, τ = 0.3, q = 1 exp 2θ 2, β = 0.99 Dicretize state space, approximate derivatives via finite differences, iterate: v (n+1) = βv (n) + (1 β) τ min H (n) u Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

19 MDP discretization Define discrete state and control spaces X (h) R n, U (h) R m and discrete time step (h), where h is a "coarseness" parameter and h! 0 corresponds to infinitely dense discretization. Construct p (h) (x 0 (h) jx (h), u (h)) s.t. Definition (local consistency) d, x 0 (h) x (h) E [d] = (h) f(x (h), u (h) ) + o( (h) ) cov [d] = (h) Σ(x (h), u (h) ) + o( (h) ) In the limit h! 0 the MDP solution v (h) converges to the solution v of the continuous problem, even when v is non-smooth (Kushner and Dupois) Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

20 Constructing the MDP For each x (h), u (h) choose vectors fv i g i=1k such that all possible next states are x 0 (h) = x (h) + hv i. Then compute p i (h) = p (h)(x (h) + hv i jx (h), u (h) ) as: Find w i, y i s.t. i w i v i = f i y i v i v T i = Σ i y i v i = 0 and set i w i = 1, w i 0 i y i = 1, y i 0 p i (h) = hw i + y i h + 1 h (h) = 2 h + 1 Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

21 Constructing the MDP For each x (h), u (h) choose vectors fv i g i=1k such that all possible next states are x 0 (h) = x (h) + hv i. Then compute p i (h) = p (h)(x (h) + hv i jx (h), u (h) ) as: Find w i, y i s.t. i w i v i = f i y i v i v T i = Σ i y i v i = 0 i w i = 1, w i 0 i y i = 1, y i 0 and set Set (h) = h and minimize Σ h i p i (h) (v i f) (v i f) T 2 s.t. i p i (h) v i = f i p i (h) = 1, p i (h) 0 p i (h) = hw i + y i h + 1 h (h) = 2 h + 1 Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16

22 Constructing the MDP For each x (h), u (h) choose vectors fv i g i=1k such that all possible next states are x 0 (h) = x (h) + hv i. Then compute p i (h) = p (h)(x (h) + hv i jx (h), u (h) ) as: Find w i, y i s.t. i w i v i = f i y i v i v T i = Σ i y i v i = 0 and set i w i = 1, w i 0 i y i = 1, y i 0 p i (h) = hw i + y i h + 1 h (h) = 2 h + 1 Set (h) = h and minimize Σ h i p i (h) (v i f) (v i f) T 2 s.t. i p i (h) v i = f i p i (h) = 1, p i (h) 0 Set (h) = h and p i (h) x N (h) + hv i ; hf, hσ Emo Todorov (UW) AMATH/CSE 579, Winter 2014 Winter / 16