Exploiting off-diagonal rank structures in the solution of linear matrix equations

Transcription

1 Stefano Massei Exploiting off-diagonal rank structures in the solution of linear matrix equations Based on joint works with D. Kressner (EPFL), M. Mazza (IPP of Munich), D. Palitta (IDCTS of Magdeburg) and L. Robol (CNR of Pisa) Lyon, 15 May 2018 Speaker: Stefano Massei 1 / 35

2 Introduction In many different settings, such as problems of control and PDEs we deal with issues like solving AX + XA = C AX + XB = C Lyapunov equation, Sylvester equation, where A, B, C, X C m m. A Sylvester equation is equivalent to the m 2 m 2 linear system (I A + B T I)vec(X) = vec(c) Speaker: Stefano Massei 2 / 35

3 Introduction In the small scale scenario, the state of the art techniques, e.g. the Bartels and Stewart algorithm, require to compute the Schur forms of A and B by a QR method. This has a complexity O(m 3 ) for the flops and O(m 2 ) for the storage. Much better than O(m 6 ) that would be required by usual direct methods on the big linear system! In the case of large scale matrices (m 10 4 ) it is essential to exploit structure in the coefficients and in the solution X. Speaker: Stefano Massei 3 / 35

4 Introduction A favorable case is when the right hand side C has low rank and the spectra of A and B are well separated (for example separated by a line). In fact, in this situation the solution X exhibits a low numerical rank X UV T, U, V C m k, k m. In many applications matrices A and B are sparse and positive definite, which implies the separation of the spectra. Under these assumptions, we can employ low-rank iterative algorithms like Krylov methods which have O(m) cost in flops and storage. Speaker: Stefano Massei 4 / 35

5 Rank structure in the solution Singular values Singular values Singular values decay in the solution of AX + XB = C with rank(c) = 1, for two different configurations of the spectra of A and B. Speaker: Stefano Massei 5 / 35

6 Rank structure in the solution Theorem (Beckermann-Townsend) Let X be such that AX + XB = C where C has rank k and let A, B be normal matrices. If E and F are two sets which contain the spectra of A and B, respectively, then the singular values of X verify σ 1+lk (X) X 2 Z l (E, F ) := inf r R l,l max E r(z) min F r(z), l 1, where R l,l is the set of rational functions of degree at most (l, l). Z l (E, F ) are known in the literature as Zolotarev numbers Normal hypothesis on A and B can be relaxed, switching to numerical ranges If E and F are separated by a line this result ensures a fast decay in the singular values of the solution Exact rank in the right hand side can be replaced by numerical rank Speaker: Stefano Massei 6 / 35

7 Zolotarev numbers decay Z l (E, F ) := max E r(z) inf r R l,l min F r(z) If A = B, A symmetric positive definite then E = [a, b], F = [ b, a] and ( ) Z l ([a, b], [ b, a]) 4ρ 2l π 2, ρ = exp 2 log(4 b a ), 0 < a < b <. For more general cases, one can link the decay with logarithmic capacity of condenser plates: Z l (E, F ) 4ρ 2l, ρ = Cap(E, F ) 1. Not very informative unless you know something about E, F, but some cases can be solved explicitly, like E = S 1, F = R \ S 1. Speaker: Stefano Massei 7 / 35

8 Motivation Recently, some attention has been payed to the case in which A, B and C are banded [1,2]. In particular, it has been shown that if A and B are additionally well conditioned then the solution X of the Sylvester equation is numerically banded. [1] A. Haber, M. Verhaegen. Sparse solution of the Lyapunov equation for large-scale interconnected systems, Automatica [2] D. Palitta, V. Simoncini. Numerical methods for Lyapunov equations with banded symmetric data, Speaker: Stefano Massei 8 / 35

9 Band structure in the solution Log-scale plot of the solution of AX + XB = C. A, B R tridiagonal positive definite, κ(a), κ(b) < 50, C symmetric tridiagonal. Speaker: Stefano Massei 9 / 35

10 Band structure in the solution Consider the following experiment: m = 300, A = B = trid( 1, 2, 1) R m m, C random m m diagonal matrix, X solution of AX + XA = C. We study the decay in the bandwidth of X plotting the quantity max diag( X, l), l = 1,..., m. Moreover, we plot the distribution of the singular values σ l of the sub diagonal block X( m : m, 1 : m 2 ) Speaker: Stefano Massei 10 / 35

11 Band structure in the solution σ l decay in the band l Speaker: Stefano Massei 11 / 35

12 Quasiseparable matrices Definition A R m m has quasiseparable rank k if the maximum rank among the off diagonal submatrices of A is k. Properties: (i) q rank (A + B) q rank (A) + q rank (B) (ii) q rank (A B) q rank (A) + q rank (B) (iii) q rank (A) = q rank (A 1 ) Speaker: Stefano Massei 12 / 35

13 ɛ-quasiseparable matrices Definition A R m m has ɛ-quasiseparable rank k if for every off-diagonal block Y of A it holds σ k+1 (Y ) ɛ. Lemma Let A R m m be of ɛ-quasiseparable k. Then there exists δa such that q rank (A + δa) k, δa 2 2ɛ m. Speaker: Stefano Massei 13 / 35

14 Quasiseparable structure in the solution Assume A, B and C to have a low quasiseparable rank and consider the following partitioning for the Sylvester equation AX + XB = C [ ] [ ] [ ] [ ] [ ] A11 A 12 X11 X 12 X11 X + 12 B11 B 12 C11 C = 12. A 21 A 22 X 21 X 22 X 21 X 22 B 21 B 22 C 21 C 22 Looking at the (2, 1) block we get the equation A 22 X 21 + X 21 B 11 = C 21 A 21 X 11 X 22 B 21. X 21 solves another Sylvester equation with a low-rank RHS. The same holds for X 12. If diagonal blocks have well separated spectra then X 21, X 12 have low numerical ranks. Speaker: Stefano Massei 14 / 35

15 Quasiseparable structure in the solution [ ] [ ] [ ] [ ] [ ] A11 A 12 X11 X 12 X11 X + 12 B11 B 12 C11 C = 12 A 21 A 22 X 21 X 22 X 21 X 22 B 21 B 22 C 21 C 22 (1) Theorem Let A, B be Hermitian positive definite matrices with spectra contained in [a, b] and with quasiseparable rank k A and k B, respectively. Let C be quasiseparable of rank k C, then for the solution of (1) it holds σ 1+lk (X 21 ) X 21 2 Z l ([a, b], [ b, a]), k := k A + k B + k C. Speaker: Stefano Massei 15 / 35

16 10 1 Singular values of X 21 Bound from Zolotarev 10 5 σ Lyapunov equation AX + XA = C with matrices of size m = 300. A is random tridiagonal positive definite with spectrum in [0.9, 3.5], C is a random diagonal matrix. Speaker: Stefano Massei 16 / 35

17 Representing Quasiseparable matrices Every quasiseparable matrix can be block partitioned as: [ ] [ ] [ ] A11 A A = 12 A11 A = A 21 A 22 A 22 A 21 }{{}}{{} Block quasisep. low-rank Speaker: Stefano Massei 17 / 35

18 Representing Quasiseparable matrices Every quasiseparable matrix can be block partitioned as: [ ] [ ] [ ] A11 A A = 12 A11 A = A 21 A 22 A 22 A 21 }{{}}{{} Block quasisep. low-rank Simple idea: store low-rank blocks as outer products, and diagonal ones recursively (H-matrices, HODLR). More refined idea: Represent interactions between levels as well, by means of nested bases (H 2 -matrices, HSS). Speaker: Stefano Massei 17 / 35

19 Representing Quasiseparable matrices Every quasiseparable matrix can be block partitioned as: [ ] [ ] [ ] A11 A A = 12 A11 A = A 21 A 22 A 22 A 21 }{{}}{{} Block quasisep. low-rank Simple idea: store low-rank blocks as outer products, and diagonal ones recursively (H-matrices, HODLR). More refined idea: Represent interactions between levels as well, by means of nested bases (H 2 -matrices, HSS). Speaker: Stefano Massei 17 / 35

20 Representing Quasiseparable matrices Within these formats: Storage complexity is either O(m log m) (HODLR) or O(m) (HSS). All the matrix operations cost either O(m log α m) (HODLR) or O(m) (HSS). Operations can be performed adaptively into the rank of the off-diagonal blocks, by adding a re-compression stage. Re-compression does not change the complexity good for handling ɛ-quasiseparability. Speaker: Stefano Massei 18 / 35

21 Solving linear matrix equations If A, B are positive definite, an easy way to get a fast solver for AX + XB = C is to combine the fast HODLR/HSS arithmetic with the classical strategies: exploit the relation ([ ]) A C sign = 0 B [ ] I 2X, 0 I by using the matrix sequences arising from Newton s method A j+1 = 1 2 (A j+a 1 j ), B j+1 = 1 2 (B j+b 1 j ), C j+1 = 1 2 (C j+a 1 j C j B 1 j ). Discretize the closed formula X = + 0 e ta Ce tb dt s w j e θj A Ce θj B and evaluate the exponential functions with a rational approximant, e.g. Padé with scaling and squaring. j=1 Speaker: Stefano Massei 19 / 35

22 Laplace equation We consider the Laplace equation on the unit square: { 2 u x 2 u 2 y = f (x, y) (x, y) Ω, 2 u(x, y) = 0 (x, y) Ω., Ω = [0, 1]2, that provides the Lyapunov equation AX + XA = C, 2 1 A = h , C ij = f (x i, y j ). 1 2 We choose f (x, y) = log(0.1 + x y ) which generates a right hand side with low quasiseparable rank. Speaker: Stefano Massei 20 / 35

23 Laplace equation Time (s) Sign+HODLR Exp+HODLR lyap (dense) O(n log 2 n) n size q rank res.( ) Speaker: Stefano Massei 21 / 35

24 Room for improvement Even if the two methods scale nicely with the dimension, they rely heavily on the re-compression steps required by the fast HOLDR/HSS arithmetic. This suggests that we are not exploiting the information about the ɛ-quasiseparable rank of the final solution, that is provided by the theory. For this reason we came out with another idea that fits more naturally with the structure in the data. Speaker: Stefano Massei 22 / 35

25 Updating a linear matrix equation Suppose that we have already computed X 0 that solves A 0 X 0 + X 0 B 0 = C 0 and that we are interested in finding X which verifies (A 0 + δa)x + X(B 0 + δb) = C 0 + δc. Speaker: Stefano Massei 23 / 35

26 Updating a linear matrix equation Suppose that we have already computed X 0 that solves A 0 X 0 + X 0 B 0 = C 0 and that we are interested in finding X which verifies (A 0 + δa)x + X(B 0 + δb) = C 0 + δc. Can we do something better than starting the computation from scratch? Speaker: Stefano Massei 23 / 35

27 Updating a linear matrix equation Suppose that we have already computed X 0 that solves A 0 X 0 + X 0 B 0 = C 0 and that we are interested in finding X which verifies (A 0 + δa)x + X(B 0 + δb) = C 0 + δc. Can we do something better than starting the computation from scratch? If δa, δb and δc are rank structured then yes! Speaker: Stefano Massei 23 / 35

28 Updating a linear matrix equation Let us denote with δx := X X 0, then (A 0 + δa)(x 0 + δx) + (X 0 + δx)(b 0 + δb) = C 0 + δc. (2) By subtracting A 0 X 0 + X 0 B 0 = C 0 from equation (2), we get (A 0 + δa)δx + δx(b 0 + δb) = δc δax 0 X 0 δb. (3) Speaker: Stefano Massei 24 / 35

29 Updating a linear matrix equation Let us denote with δx := X X 0, then (A 0 + δa)(x 0 + δx) + (X 0 + δx)(b 0 + δb) = C 0 + δc. (2) By subtracting A 0 X 0 + X 0 B 0 = C 0 from equation (2), we get (A 0 + δa)δx + δx(b 0 + δb) = δc δax 0 X 0 δb. (3) If δa, δb and δc are low-rank matrices then the same hold for the right hand side UV := δc δax 0 X 0 δb, in particular, both U and V have at most rank δa + rank δb + rank δc columns. Finally, if A 0 + δa and (B 0 + δb) have well separated spectra then δx is numerically low-rank. Speaker: Stefano Massei 24 / 35

30 Updating a linear matrix equation Algorithm 1 Solving (A 0 + δa)(x 0 + δx) + (X 0 + δx)(b 0 + δb) = C + δc 1: X 0 solve Sylv(A 0, B 0, C 0 ) 2: Compute U, V such that δc δax 0 X 0 δb = UV 3: δx low rank Sylv(A 0 + δa, B 0 + δb, U, V ) 4: return X 0 + δx The procedure low rank Sylv can be any low-rank solver. For the experiments shown in this presentation we employed the Extended Krylov method which project the equation on the tensorized subspace U t V t where U t : = span{u, A 1 U, AU, A 2 U,..., A t 1 U, A t U}, V t : = span{v, B 1 V, BV, B 2 V,..., B t 1 U, B t U}, with A = A 0 + δa and B = B 0 + δb. Speaker: Stefano Massei 25 / 35

31 A divide and conquer method Suppose that every off-diagonal block of A, B and C has rank (at most) k and consider the following partitioning for the Sylvester equation AX + XB = C: [ ] [ ] [ ] [ ] [ ] A11 A 12 X11 X 12 X11 X + 12 B11 B 12 C11 C = 12. A 21 A 22 X 21 X 22 X 21 X 22 B 21 B 22 C 21 C 22 Splitting A, B and C into their block diagonal and antidiagonal parts, leads to two equations [ [ ] [ A11 B11 C11 A22]X 0 + X 0 =, B22 C22] [ ] [ ] [ ] C A δx + δx B = 12 A 12 B X C 21 A 0 X 12 0, 21 B 21 one with block diagonal coefficients and the other with low-rank right hand side. Speaker: Stefano Massei 26 / 35

32 A divide and conquer method In particular, the equation with block diagonal coefficients can be decoupled in two equations of dimension n 2 while the other provides a contribution of (numerical) low-rank. Expanding the recursion we get: Speaker: Stefano Massei 27 / 35

33 A divide and conquer method In particular, the equation with block diagonal coefficients can be decoupled in two equations of dimension n 2 while the other provides a contribution of (numerical) low-rank. Expanding the recursion we get: Speaker: Stefano Massei 27 / 35

34 A divide and conquer method In particular, the equation with block diagonal coefficients can be decoupled in two equations of dimension n 2 while the other provides a contribution of (numerical) low-rank. Expanding the recursion we get: Speaker: Stefano Massei 27 / 35

35 A divide and conquer method In particular, the equation with block diagonal coefficients can be decoupled in two equations of dimension n 2 while the other provides a contribution of (numerical) low-rank. Expanding the recursion we get: Speaker: Stefano Massei 27 / 35

36 A divide and conquer method In particular, the equation with block diagonal coefficients can be decoupled in two equations of dimension n 2 while the other provides a contribution of (numerical) low-rank. Expanding the recursion we get: Speaker: Stefano Massei 27 / 35

37 A divide and conquer method It is crucial to ensure the separation of the spectra up to the lower levels of recursion. This is given if the separation holds for the numerical ranges of A and B. The rank in the smallest off-diagonal blocks of the solution seems to grow logarithmically. This is not the case when the coefficients are quasiseparable, so it is advisable to re-compress after each sum. If A and B are sparse (e.g. banded) the Krylov subspaces can be generated using sparse arithmetic. Speaker: Stefano Massei 28 / 35

38 Algorithm 2 Solving AX + XB = C with A, B and C HODLR matrices 1: procedure D&C Sylv(A, B, C) 2: if A, B are small matrices then 3: return solve Sylv(A, B, C) 4: else 5: Decompose A = [ ] A δa, B = 0 A 22 [ ] [ ] B11 0 C δb, C = + δc 0 B 22 0 C 22 6: X 11 D&C Sylv(A 11, B 11, C 11 ) 7: X 22 D&C [ Sylv(A 22 ], B 22, C 22 ) X11 0 8: Set X 0 0 X 22 9: Compute U and V such that UV = δc δax 0 X 0 δb 10: δx low rank Sylv(A, B, U, V ) 11: return X 0 + δx 12: end if Speaker: Stefano Massei 29 / 35

39 Numerical results: convection diffusion We consider the convection-diffusion equation { u + v u = f (x, y) (x, y) Ω := [0, 1], u(x, y) = 0 (x, y) Ω where v = [10, 10] and f (x, y) = log(1 + x y ). A finite difference discretization leads to the Lyapunov equation AX + XA = C with the nonsymmetric matrix A = (n + 1) (n + 1) and the matrix C with off-diagonal blocks of numerically low-rank. Speaker: Stefano Massei 30 / 35

40 Numerical results: convection diffusion 10 4 Sign+HODLR D&C+HOLDR O(n log n) n Res Sign Res D&C rank , , , , , , , On the left, timings of the two methods with respect to the size of the coefficients. On the right, residual and maximal rank in the off-diagonal blocks of the solution. Speaker: Stefano Massei 31 / 35

41 Numerical results: temperature model We consider the Lyapunov equation AX + XA = C coming from a model describing the temperature change of a thermally actuated deformable mirror used in extreme ultraviolet lithography [1]. S m = trid(1, 0, 1) R m m, 1 m m = ones(m,m) A = I n ( 1.36 I S 6 ) S n I 6, C = I n ( I 6 ) 0.1 S n The coefficients are block tridiagonal, with bandwidth 6 and 11, respectively nz = nz = 820 [1] A. Haber, M. Verhaegen. Sparse solution of the Lyapunov equation for large-scale interconnected systems, Automatica Speaker: Stefano Massei 32 / 35

42 Numerical results: temperature model Time (s) CG D&C+HSS O(n log n) Memory (KB) Sparse HSS O(n) n n The CG (considered in [2]) exploits the sparsity of the coefficient matrix and of the RHS in (I A + A I)x = vec(c). Both methods are stopped when the relative residue is Speaker: Stefano Massei 33 / 35

43 Other applications Non local operators: with fractional derivatives, we swap banded matrices for rank structured ones no matter which discretization we choose: quasiseparable rank is log(m) log(ɛ 1 ). Rank structures give fast methods, especially when treating separable 2D problems. CAREs: One way for solving the continuous-time algebraic Riccati equation AX + XA XBX = C, is to apply the Newton s method. This provides the matrix sequence {X k } defined by the recurrence relation: (A X k B)X k+1 + X k+1 (A X k B) = C X k BX k. Under the common assumption that B is low-rank, we have a sequence of linear matrix equations with perturbed coefficients, and all the perturbations are low-rank. The updating approach manage to speed up the process after the first iteration. Speaker: Stefano Massei 34 / 35

44 Conclusions & References Under reasonable assumptions, off-diagonal rank structures in the coefficients are likely to be present in the solution of a linear matrix equation. Low rank perturbations in the coefficients of a linear matrix equation often translate into numerically low-rank variations of the solution. The use of low-rank updates can help in designing fast solvers for equation with hierarchically low-rank coefficients. Can we deal with 3D problems? Which tensorial format is the most suitable? Full stories: S. M., M. Mazza, L. Robol. Fast solvers for 2D fractional diffusion equations using rank structured matrices, ArXiv, D.Kressner, S.M., L. Robol. Low-rank updates and a divide and conquer method for linear matrix equations, ArXiv, S.M., D. Palitta, L.Robol. Solving rank structured Sylvester and Lyapunov equations, ArXiv, Speaker: Stefano Massei 35 / 35