Solving Large-Scale Matrix Equations: Recent Progress and New Applications

Transcription

1 19th Conference of the International Linear Algebra Society (ILAS) Seoul, August 6 9, 2014 Solving Large-Scale Matrix Equations: Recent Progress and New Applications Peter Benner Max Planck Institute for Dynamics of Complex Technical Systems Computational Methods in Systems and Control Theory Magdeburg, Germany benner@mpi-magdeburg.mpg.de Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 1/52

2 Overview 1 Introduction 2 Applications 3 Solving Large-Scale Sylvester and Lyapunov Equations 4 Solving Large-Scale Lyapunov-plus-Positive Equations 5 References Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 2/52

3 Overview 1 Introduction Classification of Linear Matrix Equations Existence and Uniqueness of Solutions 2 Applications 3 Solving Large-Scale Sylvester and Lyapunov Equations 4 Solving Large-Scale Lyapunov-plus-Positive Equations 5 References Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 3/52

4 Introduction Linear Matrix Equations/Men with Beards Sylvester equation James Joseph Sylvester (September 3, 1814 March 15, 1897) AX + XB = C. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 4/52

5 Introduction Linear Matrix Equations/Men with Beards Sylvester equation Lyapunov equation James Joseph Sylvester (September 3, 1814 March 15, 1897) AX + XB = C. Alexander Michailowitsch Ljapunow (June 6, 1857 November 3, 1918) AX + XA T = C, C = C T. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 4/52

6 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Generalized Sylvester equation: AXD + EXB = C. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

7 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Generalized Sylvester equation: Generalized Lyapunov equation: AXD + EXB = C. AXE T + EXA T = C, C = C T. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

8 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Generalized Sylvester equation: Generalized Lyapunov equation: AXD + EXB = C. AXE T + EXA T = C, C = C T. Stein equation: X AXB = C. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

9 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Generalized Sylvester equation: Generalized Lyapunov equation: Stein equation: AXD + EXB = C. AXE T + EXA T = C, C = C T. X AXB = C. (Generalized) discrete Lyapunov/Stein equation: EXE T AXA T = C, C = C T. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

10 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Generalized Sylvester equation: Generalized Lyapunov equation: Stein equation: AXD + EXB = C. AXE T + EXA T = C, C = C T. X AXB = C. (Generalized) discrete Lyapunov/Stein equation: Note: EXE T AXA T = C, C = C T. Consider only regular cases, having a unique solution! Solutions of symmetric cases are symmetric, X = X T R n n ; otherwise, X R n l with n l in general. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

11 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Bilinear Lyapunov equation/lyapunov-plus-positive equation: AX + XA T + m N k XNk T = C, C = C T. k=1 Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

12 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Bilinear Lyapunov equation/lyapunov-plus-positive equation: AX + XA T + Bilinear Sylvester equation: m N k XNk T = C, C = C T. k=1 AX + XB + m N k XM k = C. k=1 Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

13 Introduction Generalizations of Sylvester (AX + XB = C) and Lyapunov (AX + XA T = C) Equations Bilinear Lyapunov equation/lyapunov-plus-positive equation: AX + XA T + Bilinear Sylvester equation: m N k XNk T = C, C = C T. k=1 AX + XB + m N k XM k = C. k=1 (Generalized) discrete bilinear Lyapunov/Stein-minus-positive eq.: EXE T AXA T m N k XNk T = C, C = C T. k=1 Note: Again consider only regular cases, symmetric equations have symmetric solutions. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 5/52

14 Introduction Existence of Solutions of Linear Matrix Equations I Exemplarily, consider the generalized Sylvester equation AXD + EXB = C. (1) Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 6/52

15 Introduction Existence of Solutions of Linear Matrix Equations I Exemplarily, consider the generalized Sylvester equation AXD + EXB = C. (1) Vectorization (using Kronecker product) representation as linear system: ( D T A + B T E Ax = c. } {{ } =:A ) vec(x ) }{{} =:x = vec(c) }{{} =:c Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 6/52

16 Introduction Existence of Solutions of Linear Matrix Equations I Exemplarily, consider the generalized Sylvester equation AXD + EXB = C. (1) Vectorization (using Kronecker product) representation as linear system: ( D T A + B T E Ax = c. } {{ } =:A ) vec(x ) }{{} =:x = vec(c) }{{} =:c = (1) has a unique solution A is nonsingular Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 6/52

17 Introduction Existence of Solutions of Linear Matrix Equations I Exemplarily, consider the generalized Sylvester equation AXD + EXB = C. (1) Vectorization (using Kronecker product) representation as linear system: ( D T A + B T E Ax = c. } {{ } =:A ) vec(x ) }{{} =:x = vec(c) }{{} =:c = (1) has a unique solution A is nonsingular Lemma Λ (A) = {α j + β k α j Λ (A, E), β k Λ (B, D)}. Hence, (1) has unique solution = Λ (A, E) Λ (B, D) =. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 6/52

18 Introduction Existence of Solutions of Linear Matrix Equations I Exemplarily, consider the generalized Sylvester equation AXD + EXB = C. (1) Vectorization (using Kronecker product) representation as linear system: ( D T A + B T E Ax = c. } {{ } =:A ) vec(x ) }{{} =:x = vec(c) }{{} =:c = (1) has a unique solution A is nonsingular Lemma Λ (A) = {α j + β k α j Λ (A, E), β k Λ (B, D)}. Hence, (1) has unique solution = Λ (A, E) Λ (B, D) =. Example: Lyapunov equation AX + XA T = C has unique solution µ C : ±µ Λ (A). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 6/52

19 Introduction The Classical Lyapunov Theorem Theorem (Lyapunov 1892) Let A R n n and consider the Lyapunov operator L : X AX + XA T. Then the following are equivalent: (a) Y > 0: X > 0: L(X ) = Y, (b) Y > 0: X > 0: L(X ) = Y, (c) Λ (A) C := {z C Rz < 0}, i.e., A is (asymptotically) stable or Hurwitz. A. M. Lyapunov. The General Problem of the Stability of Motion (in Russian). Doctoral dissertation, Univ. Kharkov English translation: Stability of Motion, Academic Press, New-York & London, P. Lancaster, M. Tismenetsky. The Theory of Matrices (2nd edition). Academic Press, Orlando, FL, [Chapter 13] Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 7/52

20 Introduction The Classical Lyapunov Theorem Theorem (Lyapunov 1892) Let A R n n and consider the Lyapunov operator L : X AX + XA T. Then the following are equivalent: (a) Y > 0: X > 0: L(X ) = Y, (b) Y > 0: X > 0: L(X ) = Y, (c) Λ (A) C := {z C Rz < 0}, i.e., A is (asymptotically) stable or Hurwitz. The proof (c) (a) is trivial from the necessary and sufficient condition for existence and uniqueness, apart from the positive definiteness. The latter is shown by studying z H Yz for all eigenvectors z of A. A. M. Lyapunov. The General Problem of the Stability of Motion (in Russian). Doctoral dissertation, Univ. Kharkov English translation: Stability of Motion, Academic Press, New-York & London, P. Lancaster, M. Tismenetsky. The Theory of Matrices (2nd edition). Academic Press, Orlando, FL, [Chapter 13] Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 7/52

21 Introduction The Classical Lyapunov Theorem Theorem (Lyapunov 1892) Let A R n n and consider the Lyapunov operator L : X AX + XA T. Then the following are equivalent: (a) Y > 0: X > 0: L(X ) = Y, (b) Y > 0: X > 0: L(X ) = Y, (c) Λ (A) C := {z C Rz < 0}, i.e., A is (asymptotically) stable or Hurwitz. Important in applications: the nonnegative case: L(X ) = AX + XA T = WW T, where W R n n W, n W n. A Hurwitz unique solution X = ZZ T for Z R n n X with 1 n X n. A. M. Lyapunov. The General Problem of the Stability of Motion (in Russian). Doctoral dissertation, Univ. Kharkov English translation: Stability of Motion, Academic Press, New-York & London, P. Lancaster, M. Tismenetsky. The Theory of Matrices (2nd edition). Academic Press, Orlando, FL, [Chapter 13] Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 7/52

22 Introduction Existence of Solutions of Linear Matrix Equations II For Lyapunov-plus-positive-type equations, the solution theory is more involved. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 8/52

23 Introduction Existence of Solutions of Linear Matrix Equations II For Lyapunov-plus-positive-type equations, the solution theory is more involved. Focus on the Lyapunov-plus-positive case: m AX + XA }{{ T + N } k XNk T = C, C = C T 0. =:L(X ) k=1 }{{} =:P(X ) Note: The operator P(X ) m N k XNk T is nonnegative in the sense that P(X ) 0, whenever X 0. j=1 Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 8/52

24 Introduction Existence of Solutions of Linear Matrix Equations II For Lyapunov-plus-positive-type equations, the solution theory is more involved. Focus on the Lyapunov-plus-positive case: m AX + XA }{{ T + N } k XNk T = C, C = C T 0. =:L(X ) k=1 }{{} =:P(X ) Note: The operator P(X ) m N k XNk T is nonnegative in the sense that P(X ) 0, whenever X 0. This nonnegative Lyapunov-plus-positive equation is the one occurring in applications like model order reduction. j=1 Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 8/52

25 Introduction Existence of Solutions of Linear Matrix Equations II For Lyapunov-plus-positive-type equations, the solution theory is more involved. Focus on the Lyapunov-plus-positive case: m AX + XA }{{ T + N } k XNk T = C, C = C T 0. =:L(X ) k=1 }{{} =:P(X ) Note: The operator P(X ) m N k XNk T is nonnegative in the sense that P(X ) 0, whenever X 0. This nonnegative Lyapunov-plus-positive equation is the one occurring in applications like model order reduction. If A is Hurwitz and the N k are small enough, eigenvalue perturbation theory yields existence and uniqueness of solution. This is related to the concept of bounded-input bounded-output (BIBO) stability of dynamical systems. j=1 Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 8/52

26 Introduction Existence of Solutions of Linear Matrix Equations II Theorem (Schneider 1965, Damm 2004) Let A R n n and consider the Lyapunov operator L : X AX + XA T and a nonnegative operator P (i.e., P(X ) 0 if X 0). The following are equivalent: (a) Y > 0: X > 0: L(X ) + P(X ) = Y, (b) Y > 0: X > 0: L(X ) + P(X ) = Y, (c) Y 0 with (A, Y ) controllable: X > 0: L(X ) + P(X ) = Y, (d) Λ (L + P) C := {z C Rz < 0}, (e) Λ (L) C and ρ(l 1 P) < 1, where ρ(t ) = max{ λ λ Λ (T )} = spectral radius of T. T. Damm. Rational Matrix Equations in Stochastic Control. Number 297 in Lecture Notes in Control and Information Sciences. Springer-Verlag, H. Schneider. Positive operators and an inertia theorem. Numerische Mathematik, 7:11 17, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 8/52

27 Overview 1 Introduction 2 Applications Stability Theory Biochemical Engineering Fractional Differential Equations Some Classical Applications 3 Solving Large-Scale Sylvester and Lyapunov Equations 4 Solving Large-Scale Lyapunov-plus-Positive Equations 5 References Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 9/52

28 Applications Stability Theory I Classical From Lyapunov s theorem, immediately obtain characterization of asymptotic stability of linear dynamical systems ẋ(t) = Ax(t). (2) Theorem (Lyapunov) The following are equivalent: For (2), the zero state is asymptotically stable. The Lyapunov equation AX + XA T = Y has a unique solution X = X T > 0 for all Y = Y T < 0. A is Hurwitz. A. M. Lyapunov. The General Problem of the Stability of Motion (In Russian). Doctoral dissertation, Univ. Kharkov English translation: Stability of Motion, Academic Press, New-York & London, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 10/52

29 Applications Stability Theory II Detecting Hopf Bifurcations Detecting instability in large-scale dynamical systems caused by Hopf bifurcations identifying the rightmost pair of complex eigenvalues of large sparse generalized eigenvalue problems. [Meerbergen/Spence 2010] suggest Lyapunov inverse iteration for the dynamical system with parameter µ R Mx t = f (x; µ). Task: Identify critical points (x, µ ) where the steady-state solution (i.e., x t 0) changes from being stable to unstable. Their continuation algorithm involves solution of generalized Lyapunov equation AX j+1 M T + MX j+1 A T = F j F (X j ), where A = D xf ( x; µ) and ( x; µ) is current estimate of critical point. K. Meerbergen, A. Spence. Inverse iteration for purely imaginary eigenvalues with application to the detection of Hopf bifurcations in large-scale problems. SIAM Journal on Matrix Analysis and Applications, 31: , H.C. Elman, K. Meerbergen, A. Spence, M. Wu. Lyapunov inverse iteration for identifying Hopf bifurcations in models of incompressible flow. SIAM Journal on Scientific Computing, 34(3):A1584-A1606, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 10/52

30 Applications Stability Theory III Metastable Equilibria of Stochastic Systems Metastable states of stochastic processes Figure: Metastable states (red dashed) and path of of a 1-dimensional stochastic ODE. This is Fig. 2.2(c) of [Kuehn 2012]. C. Kuehn. Deterministic continuation of stochastic metastable equilibria via Lyapunov equations and ellipsoids. SIAM Journal on Scientific Computing, 34(3):A1635-A1658, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 10/52

31 Applications Stability Theory III Metastable Equilibria of Stochastic Systems Tracking (w.r.t. a parameter µ R) metastable equilibrium points of stochastic differential equations (SDEs) via continuation methods: Let x R n and consider the SDE dx t = f (x t; µ)dt + σf (x t; µ)dw t, where W t = k-dimensional Brownian motion, σ > 0 controls the noise level and f, F sufficiently smooth. For metastable equilibrium points x := x (µ), stochastic paths with high probability stay in regions characterized by covariance matrix C of x t, linearized at x, defined by Lyapunov equation A(x ; µ)c + CA(x ; µ) T + σ 2 F (x ; µ)f (x ; µ) T = 0. where A(x; µ) := (D xf )(x; µ). C. Kuehn. Deterministic continuation of stochastic metastable equilibria via Lyapunov equations and ellipsoids. SIAM Journal on Scientific Computing, 34(3):A1635-A1658, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 10/52

32 Applications Biochemical Engineering Biochemical reaction networks under certain assumptions can be described by ċ(t) = Sv(c(t), q), (2) where S R n m is the stoichiometric matrix, c(t) R n denotes the species concentrations, v(t) R m the reaction rates, and q the rate constants. In order to take molecular fluctuations (or intrinsic noise) due the stochasticity of the biochemical reactions into account, need the covariance matrix C R n n of the concentrations. With the diffusion matrix D R n n reflecting the randomness of the reaction events, and the drift matrix A = v c (c0 ) R n n denoting the Jacobian of (2) along the macroscopic state trajectory at the equilibrium state c 0, C is determined by the Lyapunov equation AC + CA T + D = 0. P. Kügler, W. Yang. Identification of alterations in the Jacobian of biochemical reaction networks from steady state covariance data at two conditions. Journal of Mathematical Biology, 68: , Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 11/52

33 Applications Fractional Differential Equations Fractional partial differential equations have received recent interest in various fields, e.g., viscoelasticity (e.g., Kelvin-Voigt fractional derivative model), image processing, electro-analytical chemistry, biomedical engineering. T. Breiten, V. Simoncini, M. Stoll. Fast iterative solvers for fractional differential equations. Max Planck Institute Magdeburg Preprints MPIMD/14-02, January Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 12/52

34 Applications Fractional Differential Equations Fractional partial differential equations have received recent interest in various fields, e.g., viscoelasticity (e.g., Kelvin-Voigt fractional derivative model), image processing, electro-analytical chemistry, biomedical engineering. Definition (Caputo derivative) Given f C n (a, b), α [n 1, n), Caputo derivative of real order α is defined by: C a Dt α 1 t f (n) (s) f (t) := ds. Γ(n α) (t s) α n+1 a T. Breiten, V. Simoncini, M. Stoll. Fast iterative solvers for fractional differential equations. Max Planck Institute Magdeburg Preprints MPIMD/14-02, January Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 12/52

35 Applications Fractional Differential Equations Fractional partial differential equations have received recent interest in various fields, e.g., viscoelasticity (e.g., Kelvin-Voigt fractional derivative model), image processing, electro-analytical chemistry, biomedical engineering. Definition (Riemann-Liouville derivative) Given integrable f (t) with t [a, b], β [n 1, n), left sided Riemann-Liouville derivative of real order β is defined by: ( ) n RL a D β 1 d t f (s) t f (t) := ds. Γ(n β) dt (t s) β n+1 a T. Breiten, V. Simoncini, M. Stoll. Fast iterative solvers for fractional differential equations. Max Planck Institute Magdeburg Preprints MPIMD/14-02, January Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 12/52

36 Applications Fractional Differential Equations Consider fractional heat equation C 0 Dt α u(x, t) RL a Dx β u(x, t) = f (x, t). For discretization use Grünwald-Letnikov formula (β (1, 2)) RL a Dx β 1 u(x, t) = lim M h α and as an approximation get RL a Dx β u n+1 i 1 hx β M g β,k u(x (k 1)h, t) k=0 i+1 k=0 g β,k u n+1 i k+1. T. Breiten, V. Simoncini, M. Stoll. Fast iterative solvers for fractional differential equations. Max Planck Institute Magdeburg Preprints MPIMD/14-02, January Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 12/52

37 Applications Fractional Differential Equations Consider fractional heat equation C 0 Dt α u(x, t) RL a Dx β u(x, t) = f (x, t). For discretization use Grünwald-Letnikov formula (β (1, 2)) RL a Dx β 1 u(x, t) = lim M h α M g β,k u(x (k 1)h, t) k=0 and as an approximation get in (Toeplitz) matrix form h β x g β,1 g β,0 0 g β,2 g β,1 g β,0 g β,3 g β,2 g β,1 g β, gβ,2 g β, gβ, gβ,2 g β,1 g β,0 g β,nx g β,nx g β,2 g β,1 } {{ } T nx β u n+1 1 u n+1 2 u n+1 3. u n+1 nx 1 u n+1 nx Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 12/52.

38 Applications Fractional Differential Equations For fractional heat equation equation C 0 Dt α u(x, t) RL a Dx β u(x, t) = f (x, t) get ( (T nt α I nx ) (I nt L nx β ) ) u = f where Tα nt and L nx β are Toeplitz matrices. With u = vec(u) and dropping all superscripts this corresponds to the Sylvester equation UT T α L β U = F. T. Breiten, V. Simoncini, M. Stoll. Fast iterative solvers for fractional differential equations. Max Planck Institute Magdeburg Preprints MPIMD/14-02, January Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 12/52

39 Some Classical Applications Algebraic Riccati Equations (ARE) Solving AREs by Newtons s Method Feedback control design often involves solution of A T X + XA XGX + H = 0, G = G T, H = H T. In each Newton step, solve Lyapunov equation (A GX j ) T X j+1 + X j+1 (A GX j ) = X j GX j H. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 13/52

40 Some Classical Applications Algebraic Riccati Equations (ARE) Solving AREs by Newtons s Method Feedback control design often involves solution of A T X + XA XGX + H = 0, G = G T, H = H T. In each Newton step, solve Lyapunov equation (A GX j ) T X j+1 + X j+1 (A GX j ) = X j GX j H. Decoupling of dynamical systems, e.g., in slow/fast modes, requires solution of nonsymmetric ARE AX + XF XGX + H = 0. In each Newton step, solve Sylvester equation (A X j G)X j+1 + X j+1 (F GX j ) = X j GX j H. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 13/52

41 Some Classical Applications Model Reduction Model Reduction via Balanced Truncation For linear dynamical system ẋ(t) = Ax(t) + Bu(t), y(t) = Cx r (t), x(t) R n find reduced-order system x r (t) = A r x r (t) + B r u(t), y r (t) = C r x r (t), x(t) R r, r n such that y(t) y r (t) < δ. The popular method balanced truncation requires the solution of the dual Lyapunov equations AX + XA T + BB T = 0, A T Y + YA + C T C = 0. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 13/52

42 Overview This part: joint work with Patrick Kürschner and Jens Saak (MPI Magdeburg) 1 Introduction 2 Applications 3 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics LR-ADI Derivation Low-Rank Structure of the Residual Realification of LR-ADI Self-generating Shifts The New LR-ADI Applied to Lyapunov Equations 4 Solving Large-Scale Lyapunov-plus-Positive Equations 5 References Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 14/52

43 Solving Large-Scale Sylvester and Lyapunov Equations The Low-Rank Structure Sylvester Equations Find X R n m solving AX X B = F G T, where A R n n, B R m m, F R n r, G R m r. If n, m large, but r n, m X has a small numerical rank. [Penzl 1999, Grasedyck 2004, Antoulas/Sorensen/Zhou 2002] rank(x, τ) = f min(n, m) singular values of example 10 0 σ(x ) σ 155 u u Compute low-rank solution factors Z R n f, Y R m f, D R f f, such that X ZDY T with f min(n, m). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 15/52

44 Solving Large-Scale Sylvester and Lyapunov Equations The Low-Rank Structure Lyapunov Equations Find X R n n solving where A R n n, F R n r. If n large, but r n X has a small numerical rank. [Penzl 1999, Grasedyck 2004, Antoulas/Sorensen/Zhou 2002] rank(x, τ) = f n AX +X A T = F F T, u Compute low-rank solution factors Z R n f, D R f f, such that X ZDZ T with f n. singular values of example σ 155 u σ(x ) Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 15/52

45 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics Sylvester equation AX XB = FG T is equivalent to linear system of equations ( Im A B T I n ) vec(x) = vec(fg T ). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 16/52

46 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics Sylvester equation AX XB = FG T is equivalent to linear system of equations ( Im A B T I n ) vec(x) = vec(fg T ). This cannot be used for numerical solutions unless nm 100 (or so), as it requires O(n 2 m 2 ) of storage; Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 16/52

47 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics Sylvester equation AX XB = FG T is equivalent to linear system of equations ( Im A B T I n ) vec(x) = vec(fg T ). This cannot be used for numerical solutions unless nm 100 (or so), as it requires O(n 2 m 2 ) of storage; direct solver needs O(n 3 m 3 ) flops; Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 16/52

48 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics Sylvester equation AX XB = FG T is equivalent to linear system of equations ( Im A B T I n ) vec(x) = vec(fg T ). This cannot be used for numerical solutions unless nm 100 (or so), as it requires O(n 2 m 2 ) of storage; direct solver needs O(n 3 m 3 ) flops; low (tensor-)rank of right-hand side is ignored; Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 16/52

49 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics Sylvester equation AX XB = FG T is equivalent to linear system of equations ( Im A B T I n ) vec(x) = vec(fg T ). This cannot be used for numerical solutions unless nm 100 (or so), as it requires O(n 2 m 2 ) of storage; direct solver needs O(n 3 m 3 ) flops; low (tensor-)rank of right-hand side is ignored; in Lyapunov case, symmetry and possible definiteness are not respected. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 16/52

50 Solving Large-Scale Sylvester and Lyapunov Equations Some Basics Sylvester equation AX XB = FG T is equivalent to linear system of equations ( Im A B T I n ) vec(x) = vec(fg T ). This cannot be used for numerical solutions unless nm 100 (or so), as it requires O(n 2 m 2 ) of storage; direct solver needs O(n 3 m 3 ) flops; low (tensor-)rank of right-hand side is ignored; in Lyapunov case, symmetry and possible definiteness are not respected. Possible solvers: Standard Krylov subspace solvers in operator from [Hochbruck, Starke, Reichel, Bao,... ]. Block-Tensor-Krylov subspace methods with truncation [Kressner/Tobler, Bollhöfer/Eppler, B./Breiten,... ]. Galerkin-type methods based on (extended, rational) Krylov subspace methods [Jaimoukha, Kasenally, Jbilou, Simoncini, Druskin, Knizhermann,... ] Doubling-type methods [Smith, Chu et al., B./Sadkane/El Khoury,... ]. ADI methods [Wachspress, Reichel et al., Li, Penzl, B., Saak, Kürschner,... ]. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 16/52

51 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Sylvester and Stein equations Let α β with α / Λ(B), β / Λ(A), then AX XB = FG T }{{} Sylvester equation X = A X B + (β α )F G H }{{} Stein equation with the Cayley like transformations A := (A β I n ) 1 (A α I n ), B := (B α I m ) 1 (B β I m ), F := (A β I n ) 1 F, G := (B α I m ) H G. fix point iteration X k = A X k 1 B + (β α )F G H for k 1, X 0 R n m. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 17/52

52 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Sylvester and Stein equations Let α k β k with α k / Λ(B), β k / Λ(A), then AX XB = FG T }{{} Sylvester equation X = A k X B k + (β k α k )F k G k H }{{} Stein equation with the Cayley like transformations A := (A β k I n ) 1 (A α k I n ), B := (B α k I m ) 1 (B β k I m ), F := (A β k I n ) 1 F, G := (B α k I m ) H G. alternating directions implicit (ADI) iteration X k = A k X k 1 B k + (β k α k )F k G H k for k 1, X 0 R n m. [Wachspress 1988] Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 17/52

53 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Sylvester ADI iteration [Wachspress 1988] X k = A k X k 1 B k + (β k α k )F k G H k, A k := (A β k I n ) 1 (A α k I n ), B k := (B α k I m ) 1 (B β k I m ), F k := (A β k I n ) 1 F R n r, G k := (B α k I m ) H G C m r. Now set X 0 = 0 and find factorization X k = Z k D k Y H k X 1 = A 1 X 0 B 1 + (β 1 α 1 )F 1 G H 1, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 18/52

54 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Sylvester ADI iteration [Wachspress 1988] X k = A k X k 1 B k + (β k α k )F k G H k, A k := (A β k I n ) 1 (A α k I n ), B k := (B α k I m ) 1 (B β k I m ), F k := (A β k I n ) 1 F R n r, G k := (B α k I m ) H G C m r. Now set X 0 = 0 and find factorization X k = Z k D k Yk H X 1 = (β 1 α 1 )(A β 1 I n ) 1 F G T (B α 1 I m ) 1 V 1 := Z 1 = (A β 1 I n ) 1 F R n r, D 1 = (β 1 α 1 )I r R r r, W 1 := Y 1 = (B α 1 I m ) H G C m r. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 18/52

55 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Sylvester ADI iteration [Wachspress 1988] X k = A k X k 1 B k + (β k α k )F k G H k, A k := (A β k I n ) 1 (A α k I n ), B k := (B α k I m ) 1 (B β k I m ), F k := (A β k I n ) 1 F R n r, G k := (B α k I m ) H G C m r. Now set X 0 = 0 and find factorization X k = Z k D k Yk H X 2 = A 2 X 1 B 2 + (β 2 α 2 )F 2 G2 H =... = V 2 = V 1 + (β 2 α 1 )(A + β 2 I ) 1 V 1 R n r, W 2 = W 1 + (α 2 β 1 )(B + α 2 I ) H W 1 R m r, Z 2 = [Z 1, V 2 ], D 2 = diag (D 1, (β 2 α 2 )I r ), Y 2 = [Y 1, W 2 ]. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 18/52

56 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Algorithm [B. 2005, Li/Truhar 2008, B./Li/Truhar 2009] Algorithm 1: Low-rank Sylvester ADI / factored ADI (fadi) Input : Matrices defining AX XB = FG T and shift parameters {α 1,..., α kmax }, {β 1,..., β kmax }. Output: Z, D, Y such that ZDY H X. 1 Z 1 = V 1 = (A β 1 I n ) 1 F, 2 Y 1 = W 1 = (B α 1 I m ) H G. 3 D 1 = (β 1 α 1 )I r 4 for k = 2,..., k max do 5 V k = V k 1 + (β k α k 1 )(A β k I n ) 1 V k 1. 6 W k = W k 1 + (α k β k 1 )(B α k I n ) H W k 1. 7 Update solution factors Z k = [Z k 1, V k ], Y k = [Y k 1, W k ], D k = diag (D k 1, (β k α k )I r ). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 19/52

57 Solving Large-Scale Sylvester and Lyapunov Equations ADI Shifts Optimal Shifts Solution of rational optimization problem min max α j C λ Λ(A) j=1 β j C µ Λ(B) k (λ α j )(µ β j ) (λ β j )(µ α j ), for which no analytic solution is known in general. Some shift generation approaches: generalized Bagby points, [Levenberg/Reichel 1993] adaption of Penzl s cheap heuristic approach available [Penzl 1999, Li/Truhar 2008] approximate Λ(A), Λ(B) by small number of Ritz values w.r.t. A, A 1, B, B 1 via Arnoldi, just taking these Ritz values alone also works well quite often. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 20/52

58 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Disadvantages of Low-Rank ADI as of 2012: 1 No efficient stopping criteria: Difference in iterates norm of added columns/step: not reliable, stops often too late. Residual is a full dense matrix, can not be calculated as such. 2 Requires complex arithmetic for real coefficients when complex shifts are used. 3 Expensive (only semi-automatic) set-up phase to precompute ADI shifts. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 21/52

59 Solving Large-Scale Sylvester and Lyapunov Equations LR-ADI Derivation Disadvantages of Low-Rank ADI as of 2012: 1 No efficient stopping criteria: Difference in iterates norm of added columns/step: not reliable, stops often too late. Residual is a full dense matrix, can not be calculated as such. 2 Requires complex arithmetic for real coefficients when complex shifts are used. 3 Expensive (only semi-automatic) set-up phase to precompute ADI shifts. Will show: none of these disadvantages exists as of today = speed-ups old vs. new LR-ADI can be up to 20! Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 21/52

60 Solving Large-Scale Sylvester and Lyapunov Equations Low-Rank Structure of the Residual Low-rank Structure of S k in LR-ADI [B./Kürschner 2013] S k := A ( Z k D k Yk H ) ( Zk D k Yk H ) B FG T = Q k Uk H C n m, }{{} large, dense n m matrix Q k = Q k 1 + (β k α k )V k C n r, U k = U k 1 (β k α k )W k C m r. rank(s k ) r. Moreover, with Q 0 = F, U 0 = G it holds for the LR-ADI iterations V k = (A α k I n ) 1 Q k 1, W k = (B β k I m ) H U k 1, k 1. Holds also similarly in LR-ADI for Lyapunov equations. [B./Kürschner/Saak 2013] Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 22/52

61 Solving Large-Scale Sylvester and Lyapunov Equations Low-rank Sylvester ADI reloaded [B./Kürschner 2013] Algorithm 2: Reformulated Factored ADI iteration (fadi 2.0) Input : Matrices defining AX XB = FG T and shift parameters {α 1,..., α kmax }, {β 1,..., β kmax }, tolerance τ. Output: Z, Y, D such that ZDY H X. 1 Q 0 = F, U 0 = G, k = 1. 2 while Q k 1 Uk 1 H τ FG T do 3 γ k = β k α k. 4 V k = (A β k I n ) 1 Q k 1, W k = (B α k I m ) H U k 1, 5 Q k = Q k 1 + γ k V k, U k = U k 1 γ k W k. 6 Update solution factors Z k = [Z k 1, V k ], Y k = [Y k 1, W k ], D k = diag (D k 1, γ k I r ). 7 k + + Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 23/52

62 Solving Large-Scale Sylvester and Lyapunov Equations Computing the Residual Norm Low-rank factors Q k, U k of the residual S k now integral part of the iteration. Allows a cheap computation of S k 2 via, e.g., S k 2 = Q k U H k 2 = U k R H k 2, Q k = H k R k, H H k H k = I r requires thin QR factorization of an n r matrix and 2 computation of an r r matrix. Much cheaper than the traditional approach: apply Lanczos to Sk HS k to get S k 2 = λ max (Sk HS k) requires several matrix vector products with A, B (and A T, B T ) and additional scalar products. Note: In Lyapunov case, residual evaluation is almost free as no QR factorization is required. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 24/52

63 Solving Large-Scale Sylvester and Lyapunov Equations Low-Rank Structure of the Residual Example I: 5-point discretizations of the operator [Jbilou 2006] L(x) := x v. x f (ξ 1, ξ 2 )x on Ω = (0, 1) 2 for x = x(ξ 1, ξ 2 ), homogeneous Dirichlet BC. A: 150 grid points, v = [e ξ1+ξ2, 1000ξ 2 ], f (ξ 1, ξ 2 ) = ξ 1, B: 120 grid points, v = [sin(ξ 1 + 2ξ 2 ), 20e ξ1+ξ2 ], f (ξ 1, ξ 2 ) = ξ 1 ξ 2. n = 22500, m = 14400, F, G random with r = 4 columns. Shifts: 10 Ritz values w.r.t. A, A 1, B, B 1 yield 20 α- shifts, 20 β- shifts Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 25/52

64 Solving Large-Scale Sylvester and Lyapunov Equations Low-Rank Structure of the Residual Example I: 5-point discretizations of the operator [Jbilou 2006] L(x) := x v. x f (ξ 1, ξ 2 )x on Ω = (0, 1) 2 for x = x(ξ 1, ξ 2 ), homogeneous Dirichlet BC. Sk fadi ( sec.) iteration number k comp. time for Sk fadi 2.0 (61.1 sec.) sec. ˆ=80.5% sec. ˆ=0.005% iteration number k Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 25/52

65 Solving Large-Scale Sylvester and Lyapunov Equations Realification of LR-ADI We have real matrices A, B, F, G defining the Sylvester equation. If Λ(A), Λ(B) C some α k, β k might be complex complex operations in LR-ADI Z, D, Y complex. To generate real solution factors we need that {α k }, {β k } form Proper and suitably ordered sets of shifts If α k C then α k+1 = α k and either β k, β k+1 = β k C or β k, β k+1 R. If β k C then β k+1 = β k and either α k, α k+1 = α k C or α k, α k+1 R. No restriction, since ADI is independent of the order of shifts. Can be achieved by simple permutation of the sets of shifts. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 26/52

66 Solving Large-Scale Sylvester and Lyapunov Equations Realification of LR-ADI Relation of Iterates [B./Kürschner 2013] If α k, α k+1 = α k C and β k, β k+1 = β k C then V k+1 = V k + β k γ k Im (β k ) Im (V k), W k+1 = W k + β k γ k Im (α k ) Im (W k). Linear systems with A α k I n, B β k I m not required, low-rank factors always augmented by real data: Z k+1 = [ Z k 1, [Re (V k ), Im (V k )] R n 2r], Y k+1 = [ Y k 1, [Re (W k ), Im (W k )] R m 2r], D k+1 = diag ( D k 1, [ ] R 2r 2r), similar relations for residual factors Q k+1 R n r, U k+1 R m r and for the other shift sequences. (Generalization of Lyapunov case as in [B./Kürschner/Saak 2012/13].) Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 27/52

67 Solving Large-Scale Sylvester and Lyapunov Equations Realification of LR-ADI Example I, cont.: Shifts: 10 Ritz values w.r.t. A, A 1, B, B 1 yield 20 α- shifts (4 real, 8 complex), 20 β- shifts (12 real, 4 complex) fadi 2.0 (61.1 sec.) fadi R 2.0 (36.9 sec.) 100 Sk iteration number k spy(d k ) of fadi R. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 28/52

68 Solving Large-Scale Sylvester and Lyapunov Equations Self-generating Shifts Problems with heuristic shifts: low-rank structure of solution not embraced, no known rules for the numbers k max of α / β shifts, Ritz values (i.e., Arnoldi steps) k A +, k A, k B +, k B w.r.t. A, A 1, B, B 1, Arnoldi process brings additional costs. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 29/52

69 Solving Large-Scale Sylvester and Lyapunov Equations Self-generating Shifts Problems with heuristic shifts: low-rank structure of solution not embraced, no known rules for the numbers k max of α / β shifts, Ritz values (i.e., Arnoldi steps) k A +, k A, k B +, k B w.r.t. A, A 1, B, B 1, Arnoldi process brings additional costs. Observation: Even small changes in these numbers can lead to significantly different convergence results. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 29/52

70 Solving Large-Scale Sylvester and Lyapunov Equations Self-generating Shifts A cheap but powerful way out [Hund 2012, B./Kürschner/Saak 2013] 1 Choose initial shifts, e.g, Q = orth(f ), {α} = Λ( Q T A Q), Ũ = orth(g), {β} = Λ(ŨT BŨ). 2 If these are depleted during ADI compute new shifts via {α new } = Λ( Q T A Q), {β new } = Λ(Ũ T BŨ), where Variant 1: Q = orth(re (V ), Im (V )), Ũ = orth(re (W ), Im (W )) (iterates), Variant 2: Q = orth(q), Ũ = orth(u) (residual factors). Works surprisingly well although no setup parameters are needed. Theoretical foundation is current research. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 30/52

71 Solving Large-Scale Sylvester and Lyapunov Equations Self-generating Shifts Example I, cont.: 10 3 Ritz values (36.9 sec.) Variant 1 (20.56 sec.) Variant 2 (17.24 sec.) Sk tol iteration number k Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 31/52

72 Projection-Based Lyapunov Solvers for Lyapunov equation 0 = AX + XA T + BB T Projection-based methods for Lyapunov equations with A + A T < 0: 1 Compute orthonormal basis range (Z), Z R n r, for subspace Z R n, dim Z = r. 2 Set  := Z T AZ, ˆB := Z T B. 3 Solve small-size Lyapunov equation  ˆX + ˆX  T + ˆB ˆB T = 0. 4 Use X Z ˆX Z T. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 32/52

73 Projection-Based Lyapunov Solvers for Lyapunov equation 0 = AX + XA T + BB T Projection-based methods for Lyapunov equations with A + A T < 0: 1 Compute orthonormal basis range (Z), Z R n r, for subspace Z R n, dim Z = r. 2 Set  := Z T AZ, ˆB := Z T B. 3 Solve small-size Lyapunov equation  ˆX + ˆX  T + ˆB ˆB T = 0. 4 Use X Z ˆX Z T. Examples: Krylov subspace methods, i.e., for m = 1: Z = K(A, B, r) = span{b, AB, A 2 B,..., A r 1 B} [Saad 1990, Jaimoukha/Kasenally 1994, Jbilou ]. Extended Krylov subspace method (EKSM) [Simoncini 2007], Z = K(A, B, r) K(A 1, B, r). Rational Krylov subspace methods (RKSM) [Druskin/Simoncini 2011]. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 32/52

74 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

75 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. EKSM requires only one (or two in the presence of a mass matrix) factorizations in total, LR-ADI needs a new factorization for each new shift. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

76 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. EKSM requires only one (or two in the presence of a mass matrix) factorizations in total, LR-ADI needs a new factorization for each new shift. EKSM requires deflation strategy for MIMO systems MIMO case requires no extra treatment in LR-ADI. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

77 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. EKSM requires only one (or two in the presence of a mass matrix) factorizations in total, LR-ADI needs a new factorization for each new shift. EKSM requires deflation strategy for MIMO systems MIMO case requires no extra treatment in LR-ADI. Both methods have low-rank expressions for the residual, enabling residual-based stopping criteria. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

78 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. EKSM requires only one (or two in the presence of a mass matrix) factorizations in total, LR-ADI needs a new factorization for each new shift. EKSM requires deflation strategy for MIMO systems MIMO case requires no extra treatment in LR-ADI. Both methods have low-rank expressions for the residual, enabling residual-based stopping criteria. Both methods can be run fully automatic (LR ADI requires self-generating shifts for this). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

79 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. EKSM requires only one (or two in the presence of a mass matrix) factorizations in total, LR-ADI needs a new factorization for each new shift. EKSM requires deflation strategy for MIMO systems MIMO case requires no extra treatment in LR-ADI. Both methods have low-rank expressions for the residual, enabling residual-based stopping criteria. Both methods can be run fully automatic (LR ADI requires self-generating shifts for this). EKSM requires dissipativity of A, i.e., A + A T < 0, to guarantee convergence, ADI only needs Λ (A) C. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

80 Solving Large-Scale Sylvester and Lyapunov Equations The New LR-ADI Applied to Lyapunov Equations Comparison of the new LR-ADI and EKSM Both methods require a system solve and several matvecs per iteration. EKSM requires only one (or two in the presence of a mass matrix) factorizations in total, LR-ADI needs a new factorization for each new shift. EKSM requires deflation strategy for MIMO systems MIMO case requires no extra treatment in LR-ADI. Both methods have low-rank expressions for the residual, enabling residual-based stopping criteria. Both methods can be run fully automatic (LR ADI requires self-generating shifts for this). EKSM requires dissipativity of A, i.e., A + A T < 0, to guarantee convergence, ADI only needs Λ (A) C. If it converges, EKSM is usually faster for SISO systems with A = A T < 0. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 33/52

81 The New LR-ADI Applied to Lyapunov Equations Example II: an ocean circulation problem [Van Gijzen et al. 1998] FEM discretization of a simple 3D ocean circulation model (barotropic, constant depth) stiffness matrix A with n = 42, 249, choose artificial constant term B = rand(n,5). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 34/52

82 The New LR-ADI Applied to Lyapunov Equations Example II: an ocean circulation problem [Van Gijzen et al. 1998] FEM discretization of a simple 3D ocean circulation model (barotropic, constant depth) stiffness matrix A with n = 42, 249, choose artificial constant term B = rand(n,5). Convergence history: R / B T B TOL LR-ADI with adaptive shifts vs. EKSM coldim(z) LR-ADI EKSM Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 34/52

83 The New LR-ADI Applied to Lyapunov Equations Example II: an ocean circulation problem [Van Gijzen et al. 1998] FEM discretization of a simple 3D ocean circulation model (barotropic, constant depth) stiffness matrix A with n = 42, 249, choose artificial constant term B = rand(n,5). Convergence history: R / B T B TOL LR-ADI with adaptive shifts vs. EKSM coldim(z) CPU times: LR-ADI 110 sec, EKSM 135 sec. LR-ADI EKSM Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 34/52

84 The New LR-ADI Applied to Lyapunov Equations Example III: the triple-chain-ocillator [Truhar/Veselic 2009] Standard vibrational system second-order system with n = 21, 001, linearization n = 42, 002, Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 35/52

85 The New LR-ADI Applied to Lyapunov Equations Example III: the triple-chain-ocillator [Truhar/Veselic 2009] Standard vibrational system second-order system with n = 21, 001, linearization n = 42, 002, Again, artificial constant term: B = rand(n,5). Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 35/52

86 The New LR-ADI Applied to Lyapunov Equations Example III: the triple-chain-ocillator [Truhar/Veselic 2009] Standard vibrational system second-order system with n = 21, 001, linearization n = 42, 002, Again, artificial constant term: B = rand(n,5). Convergence history: 10 4 LR-ADI with adaptive shifts vs. EKSM R / B T B 10 4 TOL coldim(z) LR-ADI EKSM Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 35/52

87 Solving Large-Scale Sylvester and Lyapunov Equations Summary & Outlook Numerical enhancements of low-rank ADI for large Sylvester/Lyapunov equations: 1 low-rank residuals, reformulated implementation, 2 compute real low-rank factors in the presence of complex shifts, 3 self-generating shift strategies (quantification in progress). Recall the example: sec. down to sec. acceleration by factor almost 20. Generalized version enables derivation of low-rank solvers for various generalized Sylvester equations. Ongoing work: Apply LR-ADI in Newton methods for algebraic Riccati equations N (X ) = AX + XB + FG T XST T X = 0, D(X ) = AXA T EXE T + SS T + A T XF (I r + F T XF ) 1 F T XA = 0. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 36/52

88 Overview This part: joint work with Tobias Breiten (KFU Graz, Austria) 1 Introduction 2 Applications 3 Solving Large-Scale Sylvester and Lyapunov Equations 4 Solving Large-Scale Lyapunov-plus-Positive Equations Application: Balanced Truncation for Bilinear Systems Existence of Low-Rank Approximations Generalized ADI Iteration Bilinear EKSM Tensorized Krylov Subspace Methods Comparison of Methods 5 References Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 37/52

89 Solving Large-Scale Lyapunov-plus-Positive Equations Application: Balanced Truncation for Bilinear Systems Bilinear control systems: m ẋ(t) = Ax(t) + N i x(t)u i (t) + Bu(t), Σ : i=1 y(t) = Cx(t), x(0) = x 0, where A, N i R n n, B R n m, C R q n. Properties: Approximation of (weakly) nonlinear systems by Carleman linearization yields bilinear systems. Appear naturally in boundary control problems, control via coefficients of PDEs, Fokker-Planck equations,... Due to the close relation to linear systems, a lot of successful concepts can be extended, e.g. transfer functions, Gramians, Lyapunov equations,... Linear stochastic control systems possess an equivalent structure and can be treated alike [B./Damm 11]. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 38/52

90 Solving Large-Scale Lyapunov-plus-Positive Equations Application: Balanced Truncation for Bilinear Systems The concept of balanced truncation can be generalized to the case of bilinear systems, where we need the solutions of the Lyapunov-plus-positive equations: AP + PA T + A T Q + QA T + m N i PA T i + BB T = 0, i=1 m Ni T QA i + C T C = 0. i=1 Due to its approximation quality, balanced truncation is method of choice for model reduction of medium-size biliner systems. For stationary iterative solvers, see [Damm 2008], extended to low-rank solutions recently by [Szyld/Shank/Simoncini 2014]. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 39/52

91 Solving Large-Scale Lyapunov-plus-Positive Equations Application: Balanced Truncation for Bilinear Systems The concept of balanced truncation can be generalized to the case of bilinear systems, where we need the solutions of the Lyapunov-plus-positive equations: Further applications: AP + PA T + A T Q + QA T + m N i PA T i + BB T = 0, i=1 m Ni T QA i + C T C = 0. i=1 Analysis and model reduction for linear stochastic control systems driven by Wiener noise [B./Damm 2011], Lévy processes [B./Redmann 2011]. Model reduction of linear parameter-varying (LPV) systems using bilinearization approach [B./Breiten 2011]. Model reduction for Fokker-Planck equations [Hartmann et al. 2013]. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 39/52

92 Solving Large-Scale Lyapunov-plus-Positive Equations Some basic facts and assumptions AX + XA T + m N i XNi T + BB T = 0. (3) i=1 Need a positive semi-definite symmetric solution X. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 40/52

93 Solving Large-Scale Lyapunov-plus-Positive Equations Some basic facts and assumptions AX + XA T + m N i XNi T + BB T = 0. (3) i=1 Need a positive semi-definite symmetric solution X. As discussed before, solution theory for Lyapuonv-plus-positive equation is more involved then in standard Lyapuonv case. Here, existence and uniqueness of positive semi-definite solution X = X T is assumed. Max Planck Institute Magdeburg P. Benner, Large-Scale Matrix Equations 40/52