Numerical schemes of resolution of stochastic optimal control HJB equation

Transcription

1 Numerical schemes of resolution of stochastic optimal control HJB equation Elisabeth Ottenwaelter Journée des doctorants 7 mars 2007 Équipe COMMANDS Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

2 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

3 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

4 Model problem T Min IE l(t, y(t), u(t))dt + l F (y(t )); τ { (P τ,x ) dy(t) = f (t, y(t), u(t))dt + σ(t, y(t), u(t))dw(t), y(τ) = x, u(t) U, τ [0, T ], t [τ, T ]. y(t) IR n, u(t) IR m state and control variable, l and l F, real functions, running and final cost, f : IR IR n IR m IR n the drift, σ mapping into the space of n r matrices, w standard r dimensional Brownian motion, l, l F, f and σ Lipschitz and bounded. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

5 HJB Equation V value function of problem (P τ,x ), unique viscosity solution of v t (t, x) = inf {l(t, x, u) + f (t, x, u) v x(t, x) + a(t, x, u) v xx (t, x)}, u U for all t, x [0, T ] IR n. v(t, x) = l F (x), for all x IR n. (HJB) a(t, x, u) := 1 2 σ(t, x, u)σ(t, x, u) covariance matrix, A B := n i,j=1 A ijb ij scalar product associated with the Frobenius norm, Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

6 Previous results Lions and Mercier (1980), Menaldi (1989) : Classical finite difference methods, Kushner (1977), Kushner and Dupuis (1992) : Markov chain approximation, Camilli and Falcone (1995) : methods based on a priori time discretization (and the related dynamic programming principle for discrete time problems), Krylov (2000) : an error estimate of a large class of discretization schemes. Barles and Jakobsen (2002, 2003) : improvements of the error estimates. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

7 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

8 Discretization Space and time discretization h 1,..., h n space discretization steps, x k := n i=1 k ie i point of the state space, k Z n, Q N number of time steps, h 0 := T /Q time step, t q := qh 0, q = 0,..., Q. v q k approximation of the value function V at (t, x) = (t q, x k ). Upwind finite difference operator ( D ± ϕ k )i = ϕ k+e i ϕ k h i if f (t q, x k, u) i 0, ϕ k ϕ k ei h i if not. ϕ = {ϕ k } real valued function over Z n. Second order finite difference operator associated with ξ Z n ξ ϕ k := ϕ k+ξ 2ϕ k + ϕ k ξ Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

9 Discretization Space and time discretization h 1,..., h n space discretization steps, x k := n i=1 k ie i point of the state space, k Z n, Q N number of time steps, h 0 := T /Q time step, t q := qh 0, q = 0,..., Q. v q k approximation of the value function V at (t, x) = (t q, x k ). Upwind finite difference operator ( D ± ϕ k )i = ϕ k+e i ϕ k h i if f (t q, x k, u) i 0, ϕ k ϕ k ei h i if not. ϕ = {ϕ k } real valued function over Z n. Second order finite difference operator associated with ξ Z n ξ ϕ k := ϕ k+ξ 2ϕ k + ϕ k ξ Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

10 Discretization Space and time discretization h 1,..., h n space discretization steps, x k := n i=1 k ie i point of the state space, k Z n, Q N number of time steps, h 0 := T /Q time step, t q := qh 0, q = 0,..., Q. v q k approximation of the value function V at (t, x) = (t q, x k ). Upwind finite difference operator ( D ± ϕ k )i = ϕ k+e i ϕ k h i if f (t q, x k, u) i 0, ϕ k ϕ k ei h i if not. ϕ = {ϕ k } real valued function over Z n. Second order finite difference operator associated with ξ Z n ξ ϕ k := ϕ k+ξ 2ϕ k + ϕ k ξ Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

11 Strong consistency Stencil S : finite subset of Z n \ {0}, a h := {a ij /h i h j } : scaled covariance matrix, Approximation of the second-order term in the HJB equation αq,k,ξ u ξv q k, are to be set ξ S where αu q,k,ξ Strongly consistent (1) approximation of a(t, x, u) Dxx 2 if αq,k,ξ u ξξ = a h (t q, x k, u), for all k Z n. ξ S (1) Consistency of generalized finite difference schemes for the stochastic HJB equation. Bonnans, Zidani (2003) Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

12 Explicit backward scheme v q 1 k v q k = inf h 0 u U v Q k = l F l(t q, x k, u) + f (t q, x k, u) Dq,k u v q k + αq,k,ξ u ξv q k ξ S If the coefficients αq,k,ξ u are nonnegative and n i=1 the scheme is monotone. f i h i + 2 trace a h 1 h 0. When min i h i 0, we may take h 0 = C min i ( h 2 i ), for C > 0 small enough (depending on f and a). Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

13 Scheme properties Strong consistency and monotonicity GFD are a particular case of consistent Markov chain approximations convergent in view of Kushner and Dupuis (1992), monotonicity, stability and consistence convergence (2), Krylov, Barles and Jacobsen hypotheses satisfied, the error estimates apply. (2) Barles and Souganidis (1991) Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

14 Computation of coefficients α u q,k,ξ To be done at each point of the grid, for each time step, possibly for each control, Solution of a linear program, Expensive if the stencil is large. size(s) := max{ ξ ; ξ S} Main result (3) For 2D problems : an algorithm for computing the coefficients in O(size(S)) operations, For nonconsistent problems : computation of the closest consistent matrix in O(size(S)) operations. The closest consistent matrix for stencil of size p max is computed in O(1) operations after having obtained the closest consistent matrix for stencil of size p max 1. (3) A fast algorithm for the two dimensional HJB equation of stochastic control, Bonnans, Ottenwaelter, Zidani (2004) Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

15 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

16 Cone of positive semidefinite matrices z 3 Q IV D H 13 Q III 1 Ω H 2 Q II C ( ) a11 a 12 a 12 a 22 a 11 2a12 a 22 O 1 Q I z 1 z 2 Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

17 Cone of positive semidefinite matrices ξ 2 z 2 0 q III q II Q II Q I z 1 < z 3 Ω z 1 z 3 q IV q I Q III Q IV O ξ 1 z 2 < 0 Figure: Correspondence of regions ξ D IR 2 ξξ C Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

18 Stencil O Figure: Family relations in regular grid Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

19 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

20 Stern-Brocot tree Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

21 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

22 Algorithm DECOMP Ω 1 0 Figure: Correspondence of directions p max S pmax ( Polyhedral ) cone C(S pmax ) Data : p max N, ε= dist a a, C(S p max ) relative error. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

23 Main result Theorem Algorithm DECOMP provides a decomposition of a h with a relative error lower than ε, and stops after at most p max iterations. The cost of each iteration is O(1) operations, and hence, its total cost is no more than O(p max ). Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

24 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

25 Projection error Lemma The distance from a PSD matrix a to C(S pmax ) is at most ε pmax a, and ε pmax = p 2 max + 1 p max 2 2 p 2 max p 2 max. Conversely, given ε > 0, the distance from a to C(S pmax ) is at most ε when p max p ε, with 1 ε p ε := 2 ε 2 ε. 1 ε 2 Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

26 Solution error Remark If consistency does not hold, and ε = 0, then algorithm DECOMP computes the decomposition of the projection of a(t, x, u) onto C(S pmax ). In that case, the numerical scheme can be interpreted as a consistent approximation for the perturbed HJB equation. v, v : resp. the solution of the HJB equation and the (well-defined) solution of the perturbed HJB equation, When the step size vanishes, the limit of error between the two solutions is v v, Combining Jakobsen and Karlsen estimates of this error (2002) with the previous Lemma, we obtain v v C a a 1/2 C ε 1/2 p max where C and C do not depend on p max. C /p max Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

27 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

28 An uncontrolled problem Test function 1 { W (t, x1, x 2 ) = (1 + t) sin x 1 sin x 2 0 x 1 π; 0 x 2 π; 0 t 1. ( sin(x1 + x where σ(t, x 1, x 2 ) = 2 ) β 0 cos(x 1 + x 2 ) 0 β f (t, x 1, x 2 ) = 0 ) Ex 1 : p max = 5, β 2 = 0.1 a C Ex 2 : p max = 5, β 2 = 0 a C Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

29 Numerical results log e Ex 2 5 Ex log x Figure: Error vs discretization step, p max = 5 Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

30 Optimal control Test function 2 { W (t, x1, x 2 ) = (1 + t) sin x 1 sin x 2 1 x 1 1; 1 x 2 1; 0 t 0.5 where σ(t, x 1, x 2 ) = ( 2 sin(x1 + x 2 ) 2 cos(x1 + x 2 ) ) does not depend on the control, f (t, x, u) = u, u u2 2 1 is the control. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

31 Numerical results log10 e 1.6 p = p = p = log10 x p = Figure: Error vs discretization step, optimal control, p max = 1, 2, 4, 10 Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

32 Outline 1 Previous results Model problem Generalized Finite Differences (GFD) 2 A fast algorithm for the 2D HJB equation of stochastic control Structure of 2D diffusion matrices Stern-Brocot tree Decomposition of the scaled diffusion matrix Projection error Numerical results 3 Other schemes : implicit and splitting Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

33 Implicit and splitting schemes Explicit Euler scheme needs t = O ( ( x) 2). Implicit Euler scheme allows large time steps, but large scale linear systems to solve at each time step. Since our scheme expresses the evolution operator as a sum of rank-one diffusion operators, whose directions are given by the stencil, a natural alternative is to use a splitting decomposition method, for which up to S (or 2 S for second-order schemes) tridiagonal linear systems have to be solved at each time step. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

34 Cost For 2D systems : on a square grid, in the case h 1 = h 2, N h = O(1/h 1 ) unknowns per column, the bandwidth is as much as p max N h, in the case of stencil S pmax. The cost of factorization is, when p max N h, of the order of p 2 max N 4 h For the Laplace operator we have p max = 1 and solving the implicit scheme is already expensive. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

35 Implicit scheme where and V (t + h 0, x) V (t, x) h 0 = inf u U {F(t + h 0, x, u, V )}, t h 0 {1,..., N t }, x IR N, V (0, x) = Φ(x), x IR N, F (t, x, u, V ) = l(t, x, u) + F 0 (t, x, u, V ) := F i (t, x, u, V ) := F i (t, x, u, V ) := 1 i S F i (t, x, u, V ) l(t, x, u), UW i (f, V )(t, x, u) + α ξ i (t, x, u) ξ i V (t, x), 1 i N α ξ i (t, x, u) ξ i V (t, x), N < i S UW i (f, V ): upwind finite difference first order operator associated to f. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

36 Implicit splitting scheme Minimization w.r.t. the control u : Howard algorithm. Loop on the stencil directions inside a time step. V (0, x) = Φ(x), x IR N, and t h 0 {1,..., N t } : û(0, x) argmin F (0, x, u, V ) u U V 0 (x) = V (t, x) V i+1 (x) = V i (x) + h 0 F i (t + h 0, x, û(t, x), V i+1 ), V (t + h 0, x) = V S +1 (x) û(t + h 0, x) argmin {F (t + h 0, x, u, V )}. u U i = 0,..., S, Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

37 Scheme properties Since monotonicity and consistency convergence (Barles and Souganidis, 1991), we study the conditions of monotonicity and consistency for the splitting scheme inside a time step, for a loop on the directions. Monotonicity For each direction the implicit scheme is monotone, The splitting scheme is monotone as a composition of monotone schemes. Consistency : open problem. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

38 Numerical test Test function : 2D Gaussian function W (t, x 1, x 2 ) = 1 1 2π(2t + 1) e 2(2t + 1) ( a22 x 2 1 2a 12x 1 x 2 + a 11 x 2 2 ) (a 11 a 22 a 2 12 ) Solution of the PDE W t (t, x 1, x 2 ) = a W xx (t, x 1, x 2 ) where a = ξξ ξ 2 + ξ ξ ξ 2, t 0 = 0.5. Tests with ξ = ( 1 0 ) ( 2, ξ = 1 ). Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

39 Numerical results : simple splitting Simple splitting N x Error Order Time N t sec sec sec sec sec sec 642 Remark t = O( x) Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38

40 Numerical results : Strang splitting N x Strang splitting Error Order Time N t sec sec sec sec sec sec 642 Remark Strang splitting divides the error by 2. Elisabeth Ottenwaelter (INRIA et CMAP) Numerical schemes for HJB 7 mars / 38