Suboptimal feedback control of PDEs by solving Hamilton-Jacobi Bellman equations on sparse grids

Suboptimal feedback control of PDEs by solving Hamilton-Jacobi Bellman equations on sparse grids Jochen Garcke joint work with Axel Kröner, INRIA Saclay and CMAP, Ecole Polytechnique Ilja Kalmykov, Universität Bonn (who is looking for a PhD position) Department Numerical Data-Driven Prediction Institute for Numerical Simulation

Outline 1 Optimal control 2 Sparse Grids 3 Numerical Results 4 Higher Order Methods

Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete control problem HJB-equation sparse grids feedback operator

Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete control problem HJB-equation sparse grids feedback operator

Optimal Feedback Control optimal feedback control of a dynamical system min J(u) = u U ad T 0 l(y(t), u(t)) dt, ẏ(t) = f (y(t), u(t)), t > 0 y(0) = y 0 s.t. state y(t) R d, initial state y 0 R d control u(t) U m R m (often called action) Lipschitz continuous dynamics f : R d R m R d running cost with polynomial growth l : R d R m R set of admissible controls U ad = {u L 2 ([0, T ]; U m ) U m R m compact} aim: feedback law u = K(t, y (t))

Semi-Lagrangian scheme from DPP one derives semi-lagrangian scheme { } v k (x) = min v k+1 (y x ( t)) + t l(x, u) u U (SL) for k = N 1,..., 0 with v N (x) = 0, time step t > 0, x R d y x ( t) state obtained by time discretization scheme from x evaluation of the right hand side in (SL), either: (potentially expensive) comparison over a finite set U finite U in linear quadratic case, i.e. f (x, u) = Ax + Bu, l(x, u) = 1 2 A R d d, B R d m ( x T Mx + u T Ru ), M R d d, R R m m the optional feedback control is given by (needing approx. for v) u (t) = P U ( R 1 B T v(y (t), t) )

Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete control problem HJB-equation sparse grids feedback operator

Application to a PDE Control Problem wave equation (following Kröner, Kunisch, Zidani (2015)) ŷ tt c ŷ = Bu in (0, T ) Ω, ŷ(0) = ŷ 0, ŷ t (0) = ŷ 1 in Ω, ŷ = 0 on (0, T ) Ω initial state and velocity ŷ 0 H 1 (Ω) and ŷ 1 L 2 (Ω) control operator B := (sin(πx),..., sin(mπy)) (can be generalized) formulate as first order system in time with y 1 = y, y 2 = ẏ which we can write as y 1 t = y 2, y 2 t y 1 = Bu, y 1 (0) = ŷ 0, y 2 (0) = ŷ 1, y t + Ay = (0, Bu) T, y(0) = y 0 ( ) 0 Id with y = (y 1, y 2 ), y 0 = (ŷ 0, ŷ 1 ) T Y 1, and A = c 0

Semi-Discrete Formulation of Control Problem semi-discrete formulation of control problem by method of lines for a given basis we define b := (ϕ 1,..., ϕ d ), d N, with ϕ i : Ω R A := (( ϕ i (x), ϕ j (x)) i,j=1,...,d ) M := ((ϕ i (x), ϕ j (x)) i,j=1,...,d ) (stiffness matrix) (mass matrix) in our numerical examples we later choose ϕ i (x) := sin(iπx), i = 1,..., d, and obtain A = diag((1/2(iπ) 2 ) i=1,...,d ), M = diag((1/2) i=1,...,d )

Resulting Semi-Discrete PDE Control Problem this now fits into control formulation from beginning min J(u) = u U ad T 0 l(y(t), u(t)) dt, ẏ(t) = f (y(t), u(t)), t > 0 y(0) = y 0 for initial value y 0 R 2d and t [0, T ). dynamics f (x, u) = Ax + Bu with running cost l(x, u) := β x x1 T Mx 1 + β u u T u if value function is differentiable in x, t optimal feedback control u (y (t), t) = P U ( 1 ) B T x v(y (t), t) β u where P U projection on set of admissible controls Hamilton-Jacobi Bellman equation with dimension 2d s.t.

Optimal Control of Low Dim. Approx. of PDEs continuous problem described by PDE semi-discretization in space model reduction semi-discrete problem HJB-equation sparse grids feedback operator

Interpolation with Hierarchical Basis φ φ φ 3,1 3,2 φ 3,4 3,3 φ φ3,6 3,5 φ3,7 nodal basis V 1 V 2 V 3 φ φ φ 3,1 2,1 φ 1,1 3,3 φ 3,5 φ 2,3 φ 3,7 hierarchical basis V 3 = W 3 W2 V1

Hier. Basis Functions in Higher Dimensions d-dimensional piecewise d-linear functions d φ l,j (x) := φ lt,jt (x t ) t=1 hierarchical difference space W l (e t is t-th unit vector) W l := V l \ d V l et, hier. diff. space represented by W l = span{φ l,j j B l } with { B l := j N d j t = 1,..., 2 lt 1, j t odd, t = 1,..., d, if l t > 1, j t = 0, 1, 2, t = 1,..., d, if l t = 1 t=1 full grid space in hierarchical basis V s n := l n W l }.

Hierarchical Subspaces W l W 1,1 W 1,2 W 1,3 W 1,4 W 2,1 W 2,2 W 2,3 W 2,4 W 3,1 W 3,2 W 3,3 W 3,4 W 4,1 W 4,2 W 4,3 W 4,4

Sparse Grids we define the sparse grid function space Vn s V n as Vn s := l 1 n+d 1 W l every f Vn s can now be represented as fn s (x) = α l,j φ l,j (x) j B l l 1 n+d 1 approximation property in H 2 mix sparse grid needs O(h 1 n f f s n 2 = O(h 2 n log(h 1 n ) d 1 ) (log(h 1 )) d 1 ) points n

Sparse Grids in two and three dimensions

Problems with Sparse Grids: Monotonicity interpolation of peaked Gaussian fct. with sparse grid n = 2 f (x 1, x 2 ) := exp ( 100(x 1 0.5) 2) exp ( 100(x 2 0.5) 2) 0-level set of interpolant is pink in the right picture sparse grid interpolation does not preserve positivity

Spatially Adaptive Sparse Grids to approximate functions which either do not fulfil smoothness condition at all or strongly vary due to finite but locally large derivatives adaptive refinement may be used start with a regular grid of level 2 (left) to populate index set I refine one grid point by creating all children (middle) to keep grid consistent, missing parents are created (right) usually hierarchical surplus α l,j is used as refinement indicator

Basic Sparse Grid SL Scheme evaluate for x Q I, t > 0, K = T / t, k = K 1,..., 0, ( ) v k (x) = min tl(x, u) + v k+1 (y x ( t)), u U v k (x) = 0 y x ( t) state obtained by time discretization scheme from x Algorithm 1: Adaptive SL-SG scheme Data: refinement constant ε, coarsening constant η Result: sequence of adaptive sparse grid solutions v k V I(k) initialize I(K) for k = K 1,..., 0 do iterate in time with t = T /K initialize I(k 1) with I(k) adaptively interpolate min u U ( vk (y x ( t)) + tl(x, u) ) coarsen v k 1 V I(k 1) see Bokanowski, G., Griebel, and Klompmaker (2013) compute v k 1

Further Discretization Aspects to determine minimizing control within the SL-scheme we use minimization by comparison over finite subset U σ U or gradient of the value function u n (x, s) = P U ( 1 ) B T h v n+1 (x, s) β u per finite differences ( h v n+1 (x, s) ) i := v n+1 (x + h e i, s) v n+1 (x h e i, s) 2h for computation of y x ( t) we use second order Heun scheme in our experiments we focus on discretization error in space, while using time resolution which is good enough reference solution v r in 2D computed with a higher order finite difference code on a uniform mesh by an ENO scheme compute reference trajectories y r in state space and u r in control space using a Riccati approach

Example 2D: Simplified Semi-Discrete Wave Equation dynamics f (x, u) = Ax + Bu with running cost l(x, u) := β x x T 1 Mx 1 + β u u T u example based on harmonic oscillator β x = 2, β u = 0.1, T = 1, t = 0.01, ( ) ( ) 0 1 0 A =, B =, 1 0 1 initial data x R 2, domain Q = [ 1, 1] 2, U = [ 3.5, 3.5].

Example 2D: Convergence of value function 0.47 normal hat (gradient) fold out hat (gradient) normal hat (compare) fold out hat (compare) v v L 2 10 2 0.53 10 3 10 2 10 3 ε 10 4

Example 2D: Convergence of value function normal hat (adaptive) fold out hat (adaptive) normal hat (regular) fold out hat (regular) 0.96 v v L 2 10 2 0.57 10 3 10 1 10 2 10 3 10 4 nodes (end)

Example 2D: Value function 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.1 1.0 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0.1 1.0 0.5 0.5 1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 1.0 0.5 0.0 0.5 1.0 1.0 0.5 0.0 (a) ε = 1.95 4 (b) ε = 1.95 4 Figure: adaptive sparse grid with normal and fold out hat functions

Example 2D: Convergence in the Trajectory 10 1 0.51 normal hat (gradient) fold out hat (gradient) normal hat (compare) fold out hat (compare) 10 2 10 1 0.62 normal hat (level) fold out hat (level) normal hat (eps) fold out hat (eps) y y L 10 0 0.49 y y L 10 0 0.6 10 2 10 3 10 4 ε (a) L error in the trajectory vs. ε 10 1 10 2 10 3 10 4 nodes (b) L error in the trajectory vs. nodes Figure: Comparing the scheme on adaptive sparse grids using normal hat and fold out basis functions

Example 2D: Convergence in the Control 10 0 0.52 normal hat (gradient) fold out hat (gradient) normal hat (compare) fold out hat (compare) 10 1 10 0 0.62 normal hat (level) fold out hat (level) normal hat (eps) fold out hat (eps) u u L 0.56 u u L 0.63 10 2 10 2 10 3 10 3 10 2 10 3 ε 10 4 (a) L error in the control vs. ε 10 1 10 2 10 3 10 4 nodes (b) L error in the control vs. nodes Figure: Comparing the scheme using the gradient approach on adaptive and regular sparse grids using normal hat and fold out basis functions

Example: Semi-Discrete Wave Equation (4D) reminder: wave equation as first order system in time ẏ(t) = f w (y(t), u(t)), t > 0, y(0) = y 0 with dynamics f w : R 2d R m R, f w (x, u) := Ax + Bu where ( A := 0 I d cm 1 A 0 ), B := ( ) 0, b R m d, y b 0 R 2d cost consider setup l w (x, u) := β x x T 1 Mx 1 + β u u T u β x = 2, β u = 0.1, T = 4, t = 0.01, c = 0.05.

Example: Convergence in the Control 10 1 10 0 0.65 normal hat fold out hat 10 2 10 1 0.53 normal hat (eps) fold out hat (eps) fold out hat (level) u u L 0.63 u u L 10 0 10 2 10 2 1.41 10 2 10 3 10 4 ε (a) L error in the control vs. ε 10 3 10 1 10 2 10 3 10 4 10 5 nodes (b) L error in the control vs. nodes Figure: Error in the control comparing the scheme based on sparse grids using normal hat and fold out basis functions, regular and adaptive sparse grids (4D).

Example: semi-discrete wave equation (4D) 0.4 0.1 0.5 0.3 0.2 0.1 0 SL-SG Riccati 0-0.1-0.2-0.3 SL-SG Riccati 0-0.5-1 SL-SG Riccati -0.1 0 1 2 3 4 time (a) y 1-0.4 0 1 2 3 4 time (b) y 3-1.5 0 1 2 3 4 time (c) u 1 0.6 0.2 1 0.4 0.2 0 SL-SG Riccati 0-0.2-0.4-0.6 SL-SG Riccati 0.5 0-0.5-1 SL-SG Riccati -0.2 0 1 2 3 4 time (d) y 2-0.8 0 1 2 3 4 time (e) y 4-1.5 0 1 2 3 4 time (f) u 2

Example: semi-discrete wave equation (6D) 10 1 fold out hat (gradient) fold out hat (gradient) 0.62 10 0 u u L 10 0 u u L 10 2 10 3 10 4 ε (a) error in the control vs. ε 10 2 10 3 10 4 nodes (b) error in the control vs. nodes Figure: The adaptive sparse grids scheme using fold out basis functions (6D). Need to decrease time step to t = 0.0025 (larger entries in stiffness matrix).

Example: semi-discrete wave equation (8D) 10 1 fold out hat (gradient) fold out hat (gradient) 0.57 10 0 u u L 10 0 u u L 10 2 10 3 10 4 ε (a) error in the control vs. ε 10 2 10 3 10 4 nodes (b) error in the control vs. nodes Figure: The adaptive sparse grids scheme using fold out basis functions (8D). Need to decrease time step to t = 0.00125 (larger entries in stiffness matrix).

Bilinear System of Schrödinger type we write a Schrödinger type equation as a real-valued system f s : R 2d R R, f s (x, u) := Ax + uby with ( 0 cm A = 1 ) A cm 1, B = M 1 ˆB, M = A 0, as before discrete cost functional is given by J(u, y) := T 0 l(y(t), u(t)) dt running cost l s (x, u) := β x x T Mx + β u u T u, where x = (x 1, x 2 ) R 2d, u U ( ) M 0, ˆB R 2d 2d 0 M

Example 2D setup A = ( ) 0 0.5, B = 0.5 0 ( ) 0 0.5, M = 0.5 0 with T = 1, t = 1/500, β x = 2, β u = 0.1, U = [ 4, 4] in this case the control bounds are active ( ) 1 0 0 1

2D-Bilinear Schrödinger Type Problem 10 0 normal hat fold out hat error 0.59 0.43 10 2 10 1 10 2 10 3 10 4 nodes (end) (a) error in the value function vs. nodes 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 3 2 1 0 1 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 (b) fold-out basis functions and ε = 7.78 10 4 5 4 Figure: Bilinear equation: Sparse grid (2D).

Higher Order Methods we aimed to investigate (adaptive) time stepping in particular in combination with adaptive sparse grids but got side-tracked into investigating higher order time stepping which resulted into also looking at higher order (B-Splines) sparse grids we can give so far just preliminary results

Higher Order Runge-Kutta Methods look at our control formulation from beginning T min J(u) = l(y(t), u(t)) dt, s.t. u U ad 0 ẏ(t) = f (y(t), u(t)), t > 0, y(0) = y 0 we need time discretisation for y(t) and quadrature rule to evaluate min u Uad J(u) with higher order RK-scheme this could look like (misusing notation) ( ) v k (x) = min t c i l(x τ i, u i ) + v k+1 (ŷ x ( t)) u 1,u 2,u 3,... i where ŷ x takes into account actions u i and RK-scheme with intermediate steps τ i therefore RK4 would have O(dc 4 ) complexity if complexity for computing one control is d c see also Falcone, Ferretti (1994).

Simplified Semi-Discrete Wave - Using Heun v v L2 10 2 l = 4 and 2 control points l = 6 and 2 control points l = 8 and 2 control points l = 4 and 1 control point l = 6 and 1 control point l = 8 and 1 control point v v L2 10 2 l = 4 and 2 control points l = 6 and 2 control points l = 8 and 2 control points l = 4 and 1 control point l = 6 and 1 control point l = 8 and 1 control point 10 3 10 3 10 0 t 10 2 10 3 10 2 10 3 10 4 10 5 10 6 Function evaluations (a) error in value function vs. time steps (b) error in value function vs. evaluations Figure: Error to the FD reference solution for Heun time integrator and different control strategies. Evaluation of control using compare approach with 40 points.

Structure Preserving Runge-Kutta Methods observe that for explicit or diagonally implicit RK schemes, the last element u n s of discrete control vector affects only the last RK-stage k s in fact, condition on RK-coefficients can be formulated, so that control optimization can be applied separately to each stage class of RK methods, which fulfils this are diagonally implicit symplectic Runge-Kutta (DISRK) schemes re-use implicit collocation RK-scheme for quadrature originally developed for long time integration of Hamiltonian systems are equivalent to composition of implicit midpoint schemes Ψ h = Φ bsh Φ b2 h Φ b1 h we can construct a SL scheme, which has O(d c s) complexity for the minimization problem problem for s > 1 DISRK goes backwards in time for some steps!?

Simplified Semi-Discrete Wave - Revisited v v L2 10 2 SG level 2 SG level 4 SG level 6 SG level 8 SG level 10 SG level 12 v v L2 10 2 SG level 2 SG level 4 SG level 6 SG level 8 SG level 10 SG level 12 10 3 10 3 10 0 t 10 2 10 3 10 0 t 10 2 10 3 (a) Heun, d 2 c cost (b) implicit midpoint, 2 d c cost Figure: Error against FD reference, T = 1.0

Simplified Semi-Discrete Wave - Revisited 10 4 10 5 l = 6 l = 8 l = 10 l = 12 l = 14 10 4 10 5 l = 10 eval. on [ 1.0, 1.0] 2 l = 10 eval. on [ 0.5, 0.5] 2 l = 12 eval. on [ 1.0, 1.0] 2 l = 12 eval. on [ 0.5, 0.5] 2 v v L2 10 6 v v L2 10 6 2.07 10 7 10 7 10 8 10 2 t 10 3 10 4 10 2 t 10 3 10 4 (a) implicit midpoint / DISRK1 (b) implicit midpoint / DISRK1 Error against Riccati-Solution, T = 0.1 measured on full domain [ 1, 1] 2 and smaller [ 0.5, 0.5] 2 similarly we observed for DISRK3 an order 3 with error to 10 6

Simplified Semi-Discrete Wave Higher Order Discretization v v L2 10 5 10 6 10 7 10 8 10 9 l = 10 modified linear l = 4 mod. BSpline deg. 3 l = 5 mod. BSpline deg. 3 l = 6 mod. BSpline deg. 3 l = 7 mod. BSpline deg. 3 0 1 10 2 t 10 3 10 4 (c) linear SG vs. B-Splines -SG results using Sparse Grids with B-Splines using DISRK1, measure error on smaller domain

Conclusion optimal feedback control PDE problems lead to HJB equations use model reduction to reduce complexity SL-scheme on sparse grids for HJB equations value function in coefficients of spectral basis can be smooth higher order time stepping schemes and higher order discretisation can be effective for smooth problems sparse grids with B-splines converge nicer, but have higher computing costs due to need to solve a linear equation system are there are other composite methods, which are better suitable?

Key References O. Bokanowski, J. G., M. Griebel, and I. Klompmaker An adaptive sparse grid semi-lagrangian scheme for first order Hamilton-Jacobi Bellman equations, J. of Sci. Comp., (2013) M. Falcone, R. Ferretti, Discrete time high-order schemes for viscosity solutions of Hamilton-Jacobi-Bellman equations, Numerische Mathematik. (1994). J. G., A. Kröner, Suboptimal feedback control of PDEs by solving HJB equations on adaptive sparse grids, J. of Sci. Comp., (2016). A. Kröner, K. Kunisch, H. Zidani, Optimal feedback control for the undamped wave equation equation, ESAIM: Control Optim. Calc. Var. (2015).