Symplectic Möbius integrators for LQ optimal control problems

Transcription

1 Symplectic Möbius integrators or LQ optimal control problems Jason Frank Abstract The paper presents symplectic Möbius integrators or Riccati equations arising in linear quadratic optimal control problems. All proposed methods preserve symmetry, positivity and quadratic invariants or the Riccati equations, and non-statinary Lyapunov unctions. In addition, an eicient and numerically stable discretization procedure based on reinitialization or the associated linear Hamiltonian system is proposed. An eicacy o the proposed methodology is illustrated by applying an implicit midpoint version o the Möbius integrator to a Riccati equation associated with Galerkin projection o a linear hyperbolic equation. I. INTRODUCTION Optimal control problems or linear dierential equations with quadratic cost unctions are extensively studied rom both theoretical and numerical standpoints. In particular, it is well-known that under some assumptions the optimal control is represented as a solution o the Pontryagin maximum principle [12]. On the other hand, an optimal control in the eedback orm may be represented by means o the Riccati matrix which solves a non-linear matrix dierential Riccati equation [13]. This latter representation plays an important role in the state estimation and H 2 - iltering problems which are dual to optimal control problems [1]. In this paper we propose a class o numerical methods or solving the dierential Riccati equation. This in turn allows one to construct an accurate numerical solution o the optimal control problem in eedback orm and compute an observer or state estimator or a non-stationary linear system. The literature on numerical methods or Riccati equations is very rich. Without claiming completeness we mention a class o methods based on backward dierentiation ormulas (BDF) Utrecht University, The Netherlands, j.e.rank@uu.nl IBM Research, Dublin, Ireland, sergiy.zhuk@ie.ibm.com Sergiy Zhuk or symmetric and non-symmetric non-stationary dierential Riccati equations [4]. These methods require on solving an algebraic Riccati equation (ARE) at each time-step. The latter is done either using Newton s method [2] or a Schur decomposition method [9]. Another class o methods is represented by exact discretization o stationary dierential Riccati equations [7]. The class o methods proposed here is based on so-called Möbius integrators or Riccati equations [14]. The basic idea behind the Möbius transormation is to make use o the act that the solution o the Riccati equation induces a low on a Grassmannian maniold. This low is called a Möbius transormation (see or instance [14]). It may be constructed by solving an associated linear Hamiltonian system which has the same orm as Euler-Lagrange equations associated to the Pontryagin maximum principle. We propose to construct a numerical approximation o the Möbius transorm by using symplectic Runge- Kutta methods o order p and thus avoid numerical instabilities associated with Möbius transorm by means o a reinitialization. The resulting symplectic Möbius integrator is stable and preserves symmetry and positivity o the Riccati matrix. Also it preserves all quadratic invariants and a non-stationary variant o a Lyapunov quadratic orm (associated with the inverse Riccati matrix). The latter plays an important role in state estimation as it deines a decay rate o the state estimation error over time. We note that the algorithm proposed in [6] appears to be quite close to the symplectic Möbius integrators proposed in this paper: namely, the authors propose a numerically stable version o the so-called Davidson-Maki algorithm which represents the solution o the symmetric Riccati equation by means o Bernoulli substitution. The numerical stability o the substitution is achieved through the reinitialization over the course o computing the approximate solution o the linear Hamiltonian

2 system. In act, Bernoulli substitution represents a Möbius transorm. Unlike [6] where explicit RK, linear multistep and BDF time integrators where applied to get numerical Möbius transorm we construct a symplectic RK method to approximate the latter. This allows us to preserve symmetry and positivity. We illustrate the method on a numerical example which comes rom the discretization o the linear non-stationary hyperbolic equation by means o the Galerkin method. The experiment shows that our implementation with reinitialization is stable unlike a standard implicit mid-point approximation o a Möbius transorm and preserves symmetry and positivity as well as a non-stationary Lyapunov quadratic orm. The latter is o major importance in practice o using on-line estimators in the orm o ilters[17], [15] as the ilter does not require relaunching when the new observation Y (t) arrives and, thus, it can be integrated or a long time. In this case, the structure preserving discretisation becomes necessary to guarantee that the error estimates hold true or the discrete system. This paper is organized as ollows. Section II-.1 reviews LQ control problems, introduces linear Hamiltonian systems and Riccati equations, and discusses the relations to the dual state estimation problem. Section III introduces symplectic Möbius integrators. Section IV contains numerical assessment o the implicit midpoint version o the Möbius integrator or non-stationary linear equations. Section V contains concluding remarks. II. REVIEW OF LQ OPTIMAL CONTROL PROBLEMS 1) LQ problem: The linear quadratic (LQ) optimal control problem is stated as ollows. 1 On the time interval t (, t ) determine x L 2 (, t, R n ) and u L 2 (, t, R m ) to mini- 1 Analogously, one could deine the dual problem, in which the initial condition x 0 is explicit, and the end condition x enters the cost unction. The problems are equivalent up to reversal o time. mize the cost unctional L[x, u] = x T 0 Q 0 x 0 + t t + u T (t)r(t)u(t) dt. subject to the constraint: x T (t)q(t)x(t)dt ẋ(t) = A(t)x(t) + B(t)u(t), x(t ) = x, (1) where x(0) = x 0, Q 0 = Q T 0 0, Q(t) = Q T (t) 0 and R(t) = R T (t) > 0 are matrices o appropriate dimensions. It is well known [12] that the solution o the LQ problem is in the orm: û = R(t) 1 B(t) T λ where the Lagrange multiplier λ L 2 (, t, R n ) solves the ollowing linear Hamiltonian two-point boundary value problem: ẋ = A(t)x + B(t)R(t) 1 B(t) T λ, x(t ) = x, λ = Q(t)x A(t) T λ, λ( ) = Q 0 x( ). On the other hand, the optimal control û admits a so-called eedback representation: û = R(t) 1 B(t) T P (t)x(t) where P solves Riccati equation: P ( ) = Q 0 and dp dt = A(t)P P AT (t) + Q(t) P B(t)R 1 (t)b T (t)p. (2) It is well-known [13, p.121, Lemma 4.1] that under our assumptions on Q, R and Q 0 there exist U(t) and V (t) such that U( ) = I and V ( ) = Q 0 and: du dt = A(t)U + B(t)R 1 (t)b T (t)v, dv dt = Q(t)U AT (t)v, (3) and P (t) = V (t)u 1 (t). In particular, x(t) = U(t)U 1 (t )x and λ(t) = V (t)x 0 so that λ(t) = P (t)x(t). This is remarkable because in general the coupled dynamics (x(t), λ(t)) is described by a 2n 2n matrix. Here the dimension reduction ollows rom the coupled boundary condition at t =. The consequence is that our solution (x(t), λ(t)) evolves on the space o n- dimensional subspaces o R 2n, the Grassmannian Gr(2n, n), as described in Section.

3 Now, substituting û(t) into the cost and using We compute: (3), the cost unction can be simpliied along our (e T (t)p 1 (t)e(t)) solution as L[x, λ] = 1 [ t ] = e T (P 1 Q(t)P 1 + H T (t)r 1 (t)h(t))e, 2 xt 0 Q 0 x 0 + x T d 0 dt (U(t)T V (t)) dt xso 0 that or t < t we get: = 1 2 xt 0 U(t ) T V (t )x 0 = 1 2 xt P (t )x representing the minimal value o the cost. 2) Dual estimation problem: The LQ problem arises in many applications. For example, it is the key object in H 2 -iltering (Kalman iltering [1]) and minimax state estimation (see or instance [10], [3], [11] and [8]). Namely, the cost unction L represents, in particular, a worstcase estimation error σ or the ollowing state estimation problem: given an output y(t) R p o a linear system dp dt = AT (t)p(t) + (t), p( ) = 0, (4) in the orm y(t) = B T (t)p(t) + η(t) ind û L 2 (, t, R p ) such that: σ(û) σ(u), u L 2 (, t, R p ) where σ(u) := sup (l T p(t ) 0,,e assuming that T 0 Q t t u T (t)y(t)dt) 2, T (t)q 1 (t)(t)dt + η T (t)r 1 (t)η(t)dt 1. In act, L(u, x) = σ(u) and so, by minimizing L(u) one inds the estimate û o l T p(t ) with the minimal worst-case error σ(û). It turns out that t u T (t)y(t)dt = l T ˆp(t ) where ˆp solves the so-called ilter equation: dˆp dt = ( AT (t) P (t)b(t)r 1 (t)b T (t))ˆp(t) + P (t)b(t)r 1 (t)y(t), ˆp( ) = 0. Let us now assume that the estimation error e(t) := p(t) ˆp(t) equals e 0 at time instant t = t and (t) = 0, η(t) = 0 or t > t. Then one may write an equation or e: e(t ) = e 0 and ė = ( A T (t) P (t)b(t)r 1 (t)b T (t))e(t). (5) (e T (t )P 1 (t )e(t )) (e T (t)p 1 (t)e(t)) (A). We would like to prove that this decay o a nonstationary variant o a Lyapunov quadratic orm along the trajectory e(t) holds or the discrete dynamics which to be presented in the ollowing sections. III. MÖBIUS INTEGRATORS Each real n m matrix Y deines a subspace o R m+n as ollows. Partition z R m+n as z = ( u v ), where u R m and v R n. Then consider the set o all such z satisying v = Y u. This set deines an m-dimensional subspace o R m+n, a basis or which can easily be constructed as the column space o the matrix Z = [ Im Y ], where I m denotes the m-dimensional identity matrix. Note that Z indeed has ull rank m, independent o the rank o Y. As noted in [14], not all m dimensional subspaces o R m+n can be constructed this way, but a dense open subset o them can. The set o all m dimensional subspaces o R m+n that do have this property is called the Grassmannian, denoted Gr(m + n, m), which can be given topological structure and is compact in R m+n. Now consider the action o the Lie group GL(m+ n) on Gr(m+n, m). Let A GL(m+n) be close to the identity, and partitioned as [ ] a b A = I + h, c d where h is a small parameter and a R m m, d R n n, etc., and consider the action o A on the basis Z. Let Z = AZ, Z = [ U V ], U = I m + h(a + by ) V = Y + h(c + dy )

4 Under what condition does the column space o Z deine a subspace in Gr(m + n, m)? In this case there exists Y such that V = Y U, hence Y + h(c + dy ) = Y [I + h(a + by )] Y = [Y + h(c + dy )] [I + h(a + by )] 1. For h small enough, the inverse exists, and hence ininitessimal generators in GL(m + n) preserve the Grassmann maniold. What is more, Y = [Y + h(c + dy )] [ I h(a + by ) + O(h 2 ) ] = Y + h(c + dy Y by Y a) + O(h 2 ), and in the limit h 0, we see that GL(m + n) induces a low on Gr(m + n, m) corresponding to the Riccati equation or Y = V U 1 dy dt = c + dy Y by Y a. (6) For more general elements o GL(n + m), we have ( ) U = α + βy α β V = γ + δy, det 0, γ δ and we have V = Y U : γ+δy = Y (α+βy ) Y = (γ+δy )(α+βy ) 1, (7) provided the inverse exists. Such a transormation is reerred to as a generalized Möbius transormation in [14]. Schi & Shnider [14] propose constructing Möbius integrators or the Riccati equation (6) by solving the related linear equation dγ dt = [ ] a b Γ, Γ(0) = I (8) c d over a short time interval (0, h) using a numerical method or an (approximate) matrix exponential. The exact (or approximate, or that matter) solution ( ) ( ) α β a b Γ(h) = = I + h + O(h 2 ) γ δ c d can be used to deine a Möbius transormation (7) to propagate Y over (t n, t n+1 ), i.e., by taking Y = Y n, Y n+1 = Y. This approach can be easily generalized to nonautonomous Riccati equations, but requires solving (8) at each time step. An advantage o the approach is that it avoids representation singularities. The eiciency o the approach o [14] can be improved slightly by directly approximating [ ] [ ] dz a b dt = Z, Z(0) = Z n Im = c d Y n whose solution is Z n+1 = [ ] α + βyn, γ + δy n the numerator and denominator o the desired Möbius transormation. The stability o Möbius integrators can be directly analyzed in the context o standard linear stability theory or numerical methods applied to the auxiliary problem (8). A. Symplectic Möbius integrator or Riccati equations We recall that the solution P o the Riccati equation (2) may be derived rom Hamiltonian system (3). Indeed, P = V U 1 and (3) may be written as ollows: ( ) [ ] [ ] ( ) du/d I Q A T U = dv/dt I 0 A BR 1 B T, V Thus, numerical approximation o the Riccati equation may be conducted by constructing a numerical method or the Hamiltonian system (3). Current wisdom [5] suggests using symplectic integrators or Hamiltonian systems. I A, B, Q and R are sparse, then partitioned RK methods or splitting methods may be an interesting alternative, but in general we must consider the class o symplectic Runge-Kutta (RK) methods. We stress that P (t) is symmetric and non-negative i the initial condition P (0) is symmetric and non-negative. Thus, the corresponding numerical method should preserve these properties together with the a non-stationary variant o a Lyapunov quadratic orm ( decay property (A) mentioned above). In what ollows we propose s-stage implicit RK method [5, p.29] that preserves symmetry, positivity and a non-stationary Lyapunov quadratic orm o the continuous Riccati matrix.

5 Following [18] we introduce a uniorm grid t n := nh, n = 1,..., L, h := t L on (, t ) and let {a ij } s i,j=1, {b i} s i=1 denote the coeicients o s-stage implicit RK method o order p [5, p.29] or s 1. Let us also set c i := s j=1 a ij. Our main result the symplectic Möbius integrator or the auxiliary system (3) is ormulated in the ollowing theorem. Theorem 1: Assume that M jk := b j b k b k a kj b j a jk = 0 or 1 j, k s. Deine P n = V n Un 1 or n > 0 and P 0 := Q 0 and set P in = V in U 1 in where U n, V n and U in, V in are deined rom the ollowing equations: U n+1 = U n + h U in = U n + h V n+1 = V n + h V in = V n + h s b i δu in, i=1 s a ij δu jn, U n = I, j=1 s b i δv in, i=1 s a ij δv jn, V n = P n, j=1 δu in = A in U in + B in R 1 in BT inv in, δv in = A T inv in + Q in U in. (9) where A in := A(t n + c i h) and B in, Q in, R in are deined analogously. Then P n is a symmetric non-negative matrix and P n P (t n ) O(h p ) or the continuous Riccati matrix P (t). Moreover, or stationary A, B, Q and R such that A, B is stabilizable and Q 0 = 0 the decay property (A) is preserved as well. Proo: Let us irst justiy the reinitialization proposed in (9), that is V n = P n and U n = I. This may be easily explained or the case o continuous time. Namely, we note that under the change o variables U(t) := Û(t)X, V (t) := V (t)x, where Û, V solve (3), one would get that P (t) = V (t) Û 1 (t) = V (t)u 1 (t) = P (t). Thereore, we are ree to re-initialize U n, V n at each time-step t n. That is, we can compute P n+1 as P n+1 = V n+1 Un+1 1 n+1, U n+1 are obtained through (9) with V n = P n and U n = I. The reinitialization has one major advantage: it keeps U n close to the identity and wellconditioned numerically, which, in turn, allows us to use standard error estimates or RK methods o order p. These imply that P (t ) P t = O(h p ) or P (t ) = V (t )U 1 (t ) and P t = V t Ut 1. Let us now prove that P n is symmetric and nonnegative. To this end we apply one o the key properties o symplectic RK methods (those with M ij = 0), namely the ollowing discrete version o the integration by parts ormula: U T t V t = U T 0 V 0 +h L n=1 i=1 s ( b i δu T in V in + UinδV T ) in. By substituting expressions or δu in and δv in rom (9) into the above ormula and multiplying both sides o the resulting equality by Ut T := (Ut T ) 1 rom the let and Ut 1 rom the right we ind: P t Γ := = V t Ut 1 L s n=1 i=1 = Ut T (Q 0 + hγ)ut 1, b i ( V T in B in R 1 in BT inv in + U T inq in U in ). It is clear that Γ = Γ T. On the other hand, b i 0 or all i or the symplectic Gauss-Legendre RK methods and so Γ is a symmetric non-negative matrix. Using the above representation it is easy to derive that P t = P T t 0. Let us inally prove the decay property (A). Since the pair A, B is stabilizable and the noises are trivial ater t > t (by assumption (A)) it ollows that lim t e(t) = 0. Indeed, let P denote the unique equilibrium solution o the algebraic Riccati equation corresponding to (2). Then A T P BR 1 B T is stable and since lim P (t) = P it ollows that e(t) 0 monotonically at least ater some t t. On the other hand P (t) is a non-decreasing matrix-valued unction (see or instance [16, p.218]). Finally, e T t n Pt 1 n e tn is a non-increasing unction or t > t as symplectic RK methods preserve quadratic invariants [5]. We leave the proo o the decay property (A) in the general case o non-stationary matrices A, B or the urther research. Nevertheless we stress that the property (A) takes place in the

6 numerical experiment conducted in the ollowing section even though all the matrices are nonstationary. Gain 1 IV. NUMERICAL EXPERIMENTS In this experiment we approximated Riccati equation (2) using the reinitialized implicit midpoint method representing a symplectic Möbius integrator o order 2 (i.e. the method (9) with s=1, a 11 = b 1 = 1/2) and compared it to the standard iterated midpoint method applied to (3). The matrix A(t) represents the stiness matrix o the Galerkin method applied to the ollowing 2D linear transport problem: t I + u x I + v y I = 0, I(x, y, 0) = I 0 (x, y), I = 0 on Ω, (10) where the low (u(x, y, t), v(x, y, t)) is given. The transport equation was projected onto a inite dimensional subspace generated by the eigenunction o the Laplacian on Ω = [0, 2π] 2, namely by ϕ ks := sin( kx 2 ) sin( sy 2 ). We intentionally took a physical example in order to show that our method outperorms standard numerical integration approaches in a situation which is quite common in practice. We reer the reader to [18] or the detailed description o the Galerkin projection or the above equations. For the simulation we took C(t) = 1 diag(1, 0,..., 0), Q 0 = Q(t) = 100 I, B = diag(1, 0,..., 0), h = 0.002, = 0 and advanced Riccati equation (2) to the time-instant t = 5500h by using the Möbius integrator in the orm o reinitialized implicit midpoint method and the non-reinitialized midpont method. The igures below represent the simulation results. Clearly, both methods coincide or n < 4500 but then (or n > 4500) the non-reinitialized midpoint version o the numerical Riccati matrix has growing oscillations (see IV-IV). This shows that the reinitialization together with symmetry and positivity preservation plays a vital role or long simulations (like iltering problems where one does not reitialize the ilter when new observations appear). We also approximated the solution o (5) using the implicit midpoint method where the numerical P (t) was represented by the Fig. 1. The Frobenius norm o the Riccati matrix P (t) computed by means o the Möbius integrator in the orm o reinitialized implicit midpoin method Gain Fig. 2. The Frobenius norm o the Riccati matrix computed by means o the Möbius integrator in the orm o a standard midpoint. reinitialized implicit midpoint. We observe on IV that the decay property (A) holds true even though A(t) is non-stationary. V. CONCLUSION The paper presents a class o symplectic Möbius integrators o order p. The main advantages o the methods within this class over standard time integrators or Riccati equations are the ollowing: Möbius integrators allow to integrate Riccati equation through singularities which are related only to the local coordinates and are not present in the exact low map over the Grassmanian maniold; symplectic Möbius integrators with reinitialization deliver stable numerical integration schemes o higher order;

7 Gain 1 Gain 2 Gain 3 the proposed class o methods preserves symmetry, positivity and quadratic invariants or the Riccati equations; Decay o a non-stationary Lyapunov quadratic orm along the trajectory o a state estimation error ( see equation (A)) is preserved REFERENCES Fig. 3. 1st element o the diagonal o the Riccati matrix P (t) computed by means o the proposed method (Gain 1) and standard midpoint (Gain 2) or the irst 2500 steps Gain 1 Gain Fig. 4. 1st element o diagonal o the Riccati matrix P (t) computed by means o the proposed method (Gain 1) and standard midpoint (Gain 2) or the last 1000 steps x 10 5 Decay o the error Fig. 5. Non-stationary Lyapunov quadratic orm e T n Pn 1 e n decreases along the trajectory o a discretized state estimation error e n. [1] K. Åström. Introduction to Stochastic Control. Dover, [2] P. Benner, J. Li, and T. Penzl. Numerical solution o large-scale Lyapunov equations, Riccati equations, and linear-quadratic optimal control problems. Numer. Linear Algebra Appl., 15:755? 777, [3] F. L. Chernousko. State Estimation or Dynamic Systems. Boca Raton, FL: CRC, [4] L. Dieci. Numerical integration o dierential Riccati equation and some related issues. SIAM Journal on numerical analysis, 29(3), [5] E. Hairer, C. Lubich, and G. Wanner. Geometric Numerical Integration. Springer, 2nd edition, [6] C. Kenny and R. Leipnik. Numerical integration o the dierential matrix Riccati equation. IEEE TAC, AC- 30(10), [7] K. Kittipeerachon, N. Hori, and Y. Tomita. Exact discretization o a matrix dierential Riccati equation with constant coeicients. IEEE TAC, 54(5), [8] Alexander Kurzhanski and István Vályi. Ellipsoidal Calculus or Estimation and Control. Systems & Control: Foundations & Applications. Birkhäuser Boston Inc., Boston, MA, [9] A. Laub. A Schur method or solving algebraic Riccati equations. IEEE TAC, AC-24: , [10] Mario Milanese and Roberto Tempo. Optimal algorithms theory or robust estimation and prediction. IEEE Trans. Automat. Control, 30(8): , [11] A. Nakonechny. A minimax estimate or unctionals o the solutions o operator equations. Arch. Math. (Brno), 14(1):55 59, [12] L. Pontryagin, V. Boltyanskii, R. Gamkrelidze, and E. Mishchenko. The Mathematical Theory o Optimal Processes. English translation: Interscience, [13] T. Reid. Riccati Dierential Equations. Academic Press, [14] J. Schi and S. Shnider. A natural approach to the numerical integration o Riccati dierential equations. SIAM J. Numer. Anal., 36(5): , [15] T. Tchrakian and S. Zhuk. A macroscopic traic data assimilation ramework based on ourier-galerkin method and minimax estimation. submitted., 2014.

8 [16] H.L. Trentelman, A. A. Stoorvogel, and M. Hautus. Control theory o linear systems. Springer, [17] S. Zhuk. Minimax state estimation or linear discretetime dierential-algebraic equations. Automatica J. IFAC, 46(11): , [18] S. Zhuk, J. Frank, I. Herlin, and R. Shorten. Data assimilation or linear parabolic equations: minimax projection method. submitted., 2013.