Computational methods for large-scale matrix equations: recent advances and applications. V. Simoncini

Transcription

1 Computational methods for large-scale matrix equations: recent advances and applications V. Simoncini Dipartimento di Matematica Alma Mater Studiorum - Università di Bologna valeria.simoncini@unibo.it 1

2 Some matrix equations Sylvester matrix equation AX+XB +D = 0 Eigenvalue pbs and tracking, Control, MOR, Assignment pbs, Riccati eqn Lyapunov matrix equation AX+XA +D = 0, D = D Stability analysis in Control and Dynamical systems, Signal processing, eigenvalue computations Multiterm matrix equation A 1 XB 1 +A 2 XB A l XB l = C (Stochastic) PDEs Focus: All or some of the matrices are large (and possibly sparse) 2

3 Some matrix equations Sylvester matrix equation AX+XB +D = 0 Eigenvalue pbs and tracking, Control, MOR, Assignment pbs, Riccati eqn Lyapunov matrix equation AX+XA +D = 0, D = D Stability analysis in Control and Dynamical systems, Signal processing, eigenvalue computations Multiterm matrix equation A 1 XB 1 +A 2 XB A l XB l = C (Stochastic) PDEs Focus: All or some of the matrices are large (and possibly sparse) 3

4 Some matrix equations Sylvester matrix equation AX+XB +D = 0 Eigenvalue pbs and tracking, Control, MOR, Assignment pbs, Riccati eqn Lyapunov matrix equation AX+XA +D = 0, D = D Stability analysis in Control and Dynamical systems, Signal processing, eigenvalue computations Multiterm matrix equation A 1 XB 1 +A 2 XB A l XB l = C Control, (Stochastic) PDEs,... Focus: All or some of the matrices are large (and possibly sparse) 4

5 Some matrix equations Sylvester matrix equation AX+XB +D = 0 Eigenvalue pbs and tracking, Control, MOR, Assignment pbs, Riccati eqn Lyapunov matrix equation AX+XA +D = 0, D = D Stability analysis in Control and Dynamical systems, Signal processing, eigenvalue computations Multiterm matrix equation A 1 XB 1 +A 2 XB A l XB l = C Control, (Stochastic) PDEs,... Survey article: V.Simoncini, SIAM Review

6 More matrix equations Systems of linear matrix equations: A 2 X+XA 1 +B T P = F 1 A 1 Y +YA 2 + PB = F 2 BX+YB T = F 3 Riccati equation: Find X R n n such that AX+XA XBB X+C C = 0 workhorse in Control Theory Focus: All or some of the matrices are large (and possibly sparse) 6

7 More matrix equations Systems of linear matrix equations: A 2 X+XA 1 +B T P = F 1 A 1 Y +YA 2 + PB = F 2 BX+YB T = F 3 Riccati equation: Find X R n n such that AX+XA XBB X+C C = 0 workhorse in Control Theory Focus: All or some of the matrices are large (and possibly sparse) 7

8 More matrix equations Systems of linear matrix equations: A 2 X+XA 1 +B T P = F 1 A 1 Y +YA 2 + PB = F 2 BX+YB T = F 3 Riccati equation: Find X R n n such that AX+XA XBB X+C C = 0 workhorse in Control Theory Focus: All or some of the matrices are large (and possibly sparse) 8

9 Solving the Lyapunov equation. The problem Approximate X in: AX+XA +BB = 0 A R n n neg.real B R n p, 1 p n Time-invariant linear dynamical system: x (t) = Ax(t)+Bu(t), x(0) = x 0 Closed form solution: X = 0 e ta BB e ta dt = 0 xx dt with x = exp( ta)b. X symmetric semidef. see, e.g., Antoulas 05, Benner 06 9

10 Solving the Lyapunov equation. The problem Approximate X in: AX+XA +BB = 0 A R n n neg.real B R n p, 1 p n Time-invariant linear system: x (t) = Ax(t)+Bu(t), x(0) = x 0 Closed form solution: X = 0 e ta BB e ta dt X symmetric semidef. see, e.g., Antoulas 05, Benner 06 10

11 Linear systems vs linear matrix equations Large linear systems: Ax = b, A R n n Krylov subspace methods (CG, MINRES, GMRES, BiCGSTAB, etc.) Preconditioners: find P such that AP 1 x = b x = P 1 x is easier and fast to solve Large linear matrix equation: AX+XA +BB = 0 No preconditioning to preserve symmetry X is a large, dense matrix low rank approximation X X = ZZ, Ztall 11

12 Linear systems vs linear matrix equations Large linear systems: Ax = b, A R n n Krylov subspace methods (CG, MINRES, GMRES, BiCGSTAB, etc.) Preconditioners: find P such that AP 1 x = b x = P 1 x is easier and fast to solve Large linear matrix equations: AX+XA +BB = 0 No preconditioning - to preserve symmetry X is a large, dense matrix low rank approximation X X = ZZ, Ztall 12

13 Linear systems vs linear matrix equations Large linear systems: Ax = b, A R n n Krylov subspace methods (CG, MINRES, GMRES, BiCGSTAB, etc.) Preconditioners: find P such that AP 1 x = b x = P 1 x is easier and fast to solve Large linear matrix equations: AX+XA +BB = 0 Kronecker formulation: (A I +I A)x = b x = vec(x) 13

14 Projection-type methods Given an low dimensional approximation space K, X X m col(x m ) K Galerkin condition: R := AX m +X m A +BB K VmRV m = 0 K = Range(V m ) Assume VmV m = I m and let X m := V m Y m Vm. Projected Lyapunov equation: Vm(AV m Y m Vm +V m Y m VmA + BB )V m = 0 (VmAV m )Y m +Y m (VmA V m ) + VmBB V m = 0 Early contributions: Saad 90, Jaimoukha & Kasenally 94, for K = K m (A,B) = Range([B,AB,...,A m 1 B]) 14

15 Projection-type methods Given an low dimensional approximation space K, X X m col(x m ) K Galerkin condition: R := AX m +X m A +BB K VmRV m = 0 K = Range(V m ) Assume VmV m = I m and let X m := V m Y m Vm. Projected Lyapunov equation: Vm(AV m Y m Vm +V m Y m VmA +BB )V m = 0 (VmAV m )Y m +Y m (VmA V m )+VmBB V m = 0 Early contributions: Saad 90, Jaimoukha & Kasenally 94, for K = K m (A,B) = Range([B,AB,...,A m 1 B]) 15

16 Projection-type methods Given an low dimensional approximation space K, X X m col(x m ) K Galerkin condition: R := AX m +X m A +BB K VmRV m = 0 K = Range(V m ) Assume VmV m = I m and let X m := V m Y m Vm. Projected Lyapunov equation: Vm(AV m Y m Vm +V m Y m VmA +BB )V m = 0 (VmAV m )Y m +Y m (VmA V m )+VmBB V m = 0 Early contributions: Saad 90, Jaimoukha & Kasenally 94, for K = K m (A,B) = Range([B,AB,...,A m 1 B]) 16

17 Projection-type methods Given an low dimensional approximation space K, X X m col(x m ) K Galerkin condition: R := AX m +X m A +BB K VmRV m = 0 K = Range(V m ) Assume VmV m = I m and let X m := V m Y m Vm. Projected Lyapunov equation: Vm(AV m Y m Vm +V m Y m VmA +BB )V m = 0 (VmAV m )Y m +Y m (VmA V m )+VmBB V m = 0 Early contributions: Saad 90, Jaimoukha & Kasenally 94, for K = K m (A,B) = Range([B,AB,...,A m 1 B]) 17

18 More recent options as approximation space Enrich space to decrease space dimension Extended Krylov subspace K = EK := K m (A,B)+K m (A 1,A 1 B), that is, K = Range([B,A 1 B,AB,A 2 B,A 2 B,A 3 B,...,]) (Druskin & Knizhnerman 98, Simoncini 07) Rational Krylov subspace K = K := Range([B,(A s 1 I) 1 B,...,(A s m I) 1 B]) usually, {s 1,...,s m } C + chosen either a-priori or dynamically In both cases, for Range(V m ) = K, projected Lyapunov equation: (VmAV m )Y m +Y m (VmA V m )+VmBB V m = 0 X m = V m Y m Vm 18

19 More recent options as approximation space Enrich space to decrease space dimension Extended Krylov subspace K = EK := K m (A,B)+K m (A 1,A 1 B), that is, K = Range([B,A 1 B,AB,A 2 B,A 2 B,A 3 B,...,]) (Druskin & Knizhnerman 98, Simoncini 07) Rational Krylov subspace K = K := Range([B,(A s 1 I) 1 B,...,(A s m I) 1 B]) usually, {s 1,...,s m } C + chosen either a-priori or dynamically In both cases, for Range(V m ) = K, projected Lyapunov equation: (VmAV m )Y m +Y m (VmA V m )+VmBB V m = 0 X m = V m Y m Vm 19

20 More recent options as approximation space Enrich space to decrease space dimension Extended Krylov subspace K = EK := K m (A,B)+K m (A 1,A 1 B), that is, K = Range([B,A 1 B,AB,A 2 B,A 2 B,A 3 B,...,]) (Druskin & Knizhnerman 98, Simoncini 07) Rational Krylov subspace K = K := Range([B,(A s 1 I) 1 B,...,(A s m I) 1 B]) usually, {s 1,...,s m } C + chosen either a-priori or dynamically In both cases, for Range(V m ) = K, projected Lyapunov equation: (VmAV m )Y m +Y m (VmA V m )+VmBB V m = 0 X m = V m Y m Vm 20

21 Bilinear systems of matrix equations Find X R n 1 n 2 and P R m n 2 such that A 1 X+XA 2 +B T P = F 1 BX = F 2 with A i R n i n i, B R m n 1, F 1 R n 1 n 2, F 2 R m n 2, m n 1 Emerging matrix formulation of different application problems Constraint control Mixed formulations of stochastic diffusion problems Discretized deterministic/stochastic (Navier-)Stokes equations... 21

22 An example. Mixed FE formulation of stochastic Galerkin diffusion pb c 1 u p = 0, u = f, l Assume that c 1 = c 0 + λr c r ( x)ξ r (ω) and that an appropriate class of r=1 finite elements is used for the discretization of the problem (see, e.g., the derivation in Elman & Furnival & Powell, 2010) After discretization the problem reads: For l = 1 we obtain l G 0 K 0 + λgr K r G T 0 BT 0 u r=1 = p G 0 B 0 0 f K 0 XG 0 +K 1 XG 1 +B T 0 PG 0 = 0, B 0 XG 0 = F 22

23 An example. Mixed FE formulation of stochastic Galerkin diffusion pb c 1 u p = 0, u = f, l Assume that c 1 = c 0 + λr c r ( x)ξ r (ω) and that an appropriate class of r=1 finite elements is used for the discretization of the problem (see, e.g., the derivation in Elman & Furnival & Powell, 2010) After discretization the problem reads: For l = 1 we obtain l G 0 K 0 + λgr K r G T 0 BT 0 u r=1 = p G 0 B 0 0 f K 0 XG 0 +K 1 XG 1 +B T 0 PG 0 = 0, B 0 XG 0 = F 23

24 The bilinear case. Computational strategies A 1 X+XA 2 +B T P = F 1 BX = F 2 Kronecker formulation (monolithic): A BT B O x = p f 1, A = I A 1 +A T 2 I, B = B I f 2 with x = vec(x), p = vec(p), f 1 = vec(f 1 ) and f 2 = vec(f 2 ) Extremely rich literature from saddle point algebraic linear systems Problem: Coefficient matrix has size (n 1 n 2 +mn 2 ) (n 1 n 2 +mn 2 ) 24

25 The bilinear case. Computational strategies. Cont d A 1 X+XA 2 +B T P = F 1 BX = F 2 Derive numerical strategies that directly work with the matrix equations: Small scale: Null space method Small and medium scale: Schur complement method (also directly applicable to trilinear case) Large scale: Iterative method for low rank F i, i = 1,2 Small and medium scale actually means Large scale for the Kronecker form! 25

26 Large scale problem. Iterative method. 1/3 A 1 X+XA 2 +B T P = F 1 BX = F 2 Rewrite as B T A 1 B 0 with X + P I n 1 0 X A 2 = 0 0 P F 1, MZ+D 0 ZA 2 = F F 2 M,D 0 R (n 1+m) (n 1 +m) A 2 R n 2 n 2 nonsingular D 0 highly singular If F low rank, exploit projection-type strategies for Sylvester equations 26

27 Large scale problem. Iterative method. 2/3 A 1 B 0 B T X + P I n 1 0 X A 2 = 0 0 P F 1 MZ+D 0 ZA 2 = F F 2 with F low rank. We rewrite the matrix equation as a Sylvester equation: ZA 1 2 +M 1 D 0 Z = F with F = M 1 FA 1 2 of low rank if F is of low rank, F = Fl FT r Z Z k = V k Z k W T k with Range(V k ),Range(W k ) appropriate approximation spaces of small dimensions 27

28 Large scale problem. Iterative method. 3/3 Galerkin-Projection method ZA 1 2 +M 1 D 0 Z = F l FT r Z Z k = V k Z k W T k Choice of V k,w k. A possible strategy: W k = EK k (A T 2, F r ), Extended Krylov subspace V k = K k (M 1 D 0, F l ) K k ((M 1 D 0 +σi) 1, F l ) Augmented Krylov subspace, σ R (see, e.g., Shank & Simoncini, 2013) Note: M has size (n 1 +m) (n 1 +m) (Compare with (n 1 n 2 +mn 2 ) (n 1 n 2 +mn 2 ) of the Kronecker form) 28

29 Numerical experiments A 1 X XA 2 +B T P = 0, vs Az = f BX = F 2 A 1 L 1 = u xx u yy, F 2 rank-1 A 2 L 1 = (e 10xy u x ) x (e 10xy u y ) y +10(x+y)u x B = bidiag( 1,1) R (n 2 n 1 ) n 2, params: tol=10 6, σ = 10 2 n 1 n 2 size(a) Monolithic Matrix eqns Elapsed Time Elapsed Time , e e-02 (4) , e e-02 (4) , e e-02 (4) , e e-01 (5) , e e-01 (5) ,962, e e+00 (5) Monolithic: direct solver (iterative not competitive) 29

30 H BT B 0 Numerical experiments. 1D stochastic Stokes problem. x = f 1, H = (ν 0 G 0 +ν 1 G 1 ) A x, B = G 0 B x p f 2 where ν = ν 0 +ν 1 ξ(ω) uncertain viscosity, ξ random variable Then A x XG 0 ν 0 +A x XG 1 ν 1 +B T xpg 0 = F 1, B x X = F 2 with G 0 = I. This corresponds to B T x ν 0A x B x 0 X + P ν 1A x X G 1 = 0 P F1 F2 that is MZ+D 0 ZG 1 = F 30

31 Numerical experiments. 1D stochastic Stokes problem. MZ+D 0 ZG 1 = F vs Az = f ν 0 = 1/10,ν 1 = 3ν 0 /10 Powell & Silvester, 2012 Elapsed Time n 1 n 2 size(a) Monolithic Matrix eqns , (2) , (2) , (2) n 1 n 2 size(a) Monolithic Matrix eqns , (2) , (2) , (2) n 2 could be much larger, n 2 = O(10 3 ) Memory requirements are limited, Z = Z 1 Z T 2 of very low rank 31

32 Numerical experiments. 1D stochastic Stokes problem. MZ+D 0 ZG 1 = F vs Az = f ν 0 = 1/10,ν 1 = 3ν 0 /10 Powell & Silvester, 2012 Elapsed Time n 1 n 2 size(a) Monolithic Matrix eqns , (2) , (2) , (2) n 1 n 2 size(a) Monolithic Matrix eqns , (2) , (2) , (2) n 2 could be much larger, n 2 = O(10 3 ) Memory requirements are limited, Z = Z 1 Z T 2 of very low rank 32

33 Numerical experiments. 1D stochastic Stokes problem. MZ+D 0 ZG 1 = F vs Az = f ν 0 = 1/10,ν 1 = 3ν 0 /10 Powell & Silvester, 2012 Elapsed Time n 1 n 2 size(a) Monolithic Matrix eqns , (2) , (2) , (2) n 1 n 2 size(a) Monolithic Matrix eqns , (2) , (2) , (2) n 2 could be much larger, n 2 = O(10 3 ) Memory requirements are limited, Z = Z 1 Z T 2 of very low rank 33

34 Multiterm linear matrix equation A 1 XB 1 +A 2 XB A l XB l = C Applications: Control (Stochastic) PDEs Matrix least squares... Main device: Kronecker formulation ( B 1 A B l A l ) x = c Iterative methods: matrix-matrix multiplications and rank truncation (Benner, Breiten, Bouhamidi, Chehab, Damm, Grasedyck, Jbilou, Kressner, Matthies, Onwunta, Raydan, Stoll, Tobler, Zander, and many others...) 34

35 Multiterm linear matrix equation A 1 XB 1 +A 2 XB A l XB l = C Applications: Control (Stochastic) PDEs Matrix least squares... Main device: Kronecker formulation ( B 1 A B l A l ) x = c Iterative methods: matrix-matrix multiplications and rank truncation (Benner, Breiten, Bouhamidi, Chehab, Damm, Grasedyck, Jbilou, Kressner, Matthies, Onwunta, Raydan, Stoll, Tobler, Zander, and many others...) 35

36 Multiterm linear matrix equation A 1 XB 1 +A 2 XB A l XB l = C Applications: Control (Stochastic) PDEs Matrix least squares... Alternative approaches: Projection onto rich approximation space Compression to two-term matrix equation Splitting strategy towards two-term matrix equation... 36

37 PDEs on uniform grids and separable coeffs ε u+φ 1 (x)ψ 1 (y)u x +φ 2 (x)ψ 2 (y)u y +γ 1 (x)γ 2 (y)u = f (x,y) Ω φ i,ψ i,γ i, i = 1,2 sufficiently regular functions + b.c. Problem discretization by means of a tensor basis Multiterm linear equation: εt 1 U εut 2 +Φ 1 B 1 UΨ 1 +Φ 2 UB 2 Ψ 2 +Γ 1 UΓ 2 = F Finite Diff.: U i,j = U(x i,y j ) approximate solution at the nodes 37

38 PDEs with random inputs Stochastic steady-state diffusion eqn: Find u : D Ω R s.t. P-a.s., (a(x,ω) u(x,ω)) = f(x) in D u(x,ω) = 0 on D f: deterministic; a: random field, linear function of finite no. of real-valued random variables ξ r : Ω Γ r R Common choice: truncated Karhunen Loève (KL) expansion, a(x,ω) = µ(x)+σ m λr φ r (x)ξ r (ω), r=1 µ(x): expected value of diffusion coef. σ: std dev. (λ r,φ r (x)) eigs of the integral operator V wrto V(x,x ) = 1 σ 2 C(x,x ) (λ r ց C : D D R covariance fun. ) 38

39 Discretization by stochastic Galerkin Approx with space in tensor product form a X h S p Ax = b, A = G 0 K 0 + m G r K r, b = g 0 f 0, r=1 x: expansion coef. of approx to u in the tensor product basis {ϕ i ψ k } K r R n x n x, FE matrices (sym) G r R n ξ n ξ, r = 0,1,...,m Galerkin matrices associated w/ S p (sym.) g 0 : first column of G 0 f 0 : FE rhs of deterministic PDE n ξ = dim(s p ) = (m+p)! m!p! n x n ξ huge a S p set of multivariate polyn of total degree p 39

40 The matrix equation formulation (G 0 K 0 +G 1 K G m K m )x = g 0 f 0 transforms into K 0 XG 0 +K 1 XG K m XG m = F, F = f 0 g0 (G 0 = I) Solution strategy. Conjecture: {K r } from trunc d Karhunen Loève (KL) expansion X X low rank, X = X1 X T 2 (Possibly extending results of Grasedyck, 2004) 40

41 Matrix Galerkin approximation of the deterministic part. 1 Approximation space K k and basis matrix V k : X X k = V k Y V k R k = 0, R k := K 0 X k +K 1 X k G K m X k G m f 0 g 0 Computational challenges: Generation of K k involved m+1 different matrices {K r }! Matrices K r have different spectral properties n x,n ξ so large that X k,r k should not be formed! (Powell & Silvester & Simoncini, SISC 2017) For a full account attend Catherine Powell s talk, M12, 16:45 Thu 41

42 Matrix Galerkin approximation of the deterministic part. 1 Approximation space K k and basis matrix V k : X X k = V k Y V k R k = 0, R k := K 0 X k +K 1 X k G K m X k G m f 0 g 0 Computational challenges: Generation of K k involved m+1 different matrices {K r }! Matrices K r have different spectral properties n x,n ξ so large that X k,r k should not be formed! (Powell & Silvester & Simoncini, SISC 2017) For a full account attend Catherine Powell s talk, M12, 16:45 Thu 42

43 Not discussed but in this category Sylvester-like linear matrix equations AX +f(x)b = C typically (but not only!): f(x) = X, f(x) = X, or f(x) = X (Bevis, Braden, Byers, Chiang, De Terán, Dopico, Duan, Feng, Gonzalez, Guillery, Hall, Hartwig, Ikramov, Kressner, Montealegre, Reyes, Schröder, Vorntsov, Watkins, Wu,...) Linear systems with complex tensor structure Ax = b with A = k I n1 I nj 1 A j I nj+1 I nk. j=1 Dolgov, Grasedyck, Khoromskij, Kressner, Oseledets, Tobler, Tyrtyshnikov, and many more... 43

44 Not discussed but in this category Sylvester-like linear matrix equations AX +f(x)b = C typically (but not only!): f(x) = X, f(x) = X, or f(x) = X (Bevis, Braden, Byers, Chiang, De Terán, Dopico, Duan, Feng, Gonzalez, Guillery, Hall, Hartwig, Ikramov, Kressner, Montealegre, Reyes, Schröder, Vorntsov, Watkins, Wu,...) Linear systems with complex tensor structure Ax = b with A = k I n1 I nj 1 A j I nj+1 I nk. j=1 Dolgov, Grasedyck, Khoromskij, Kressner, Oseledets, Tobler, Tyrtyshnikov, and many more... 44

45 Conclusions Large-scale (Multiterm) linear equations are a new computational tool Great advances in solving really large linear matrix equations Second order (matrix) challenges rely on strength and maturity of linear system solvers Low-rank tensor formats is the new generation of approximations Reference for linear matrix equations: V. Simoncini, Computational methods for linear matrix equations, SIAM Review, Sept