S N. hochdimensionaler Lyapunov- und Sylvestergleichungen. Peter Benner. Mathematik in Industrie und Technik Fakultät für Mathematik TU Chemnitz

Transcription

1 Ansätze zur numerischen Lösung hochdimensionaler Lyapunov- und Sylvestergleichungen Peter Benner Mathematik in Industrie und Technik Fakultät für Mathematik TU Chemnitz S N SIMULATION benner@mathematik.tu-chemnitz.de Hagen, 28. Oktober 2004

2 Outline Outline Linear matrix equations Applications Direct Methods: Hessenberg-Schur, Bartels-Stewart, Hammarling Sign function method Low-rank approximation Large, sparse Lyapunov equations ADI and Smith iterations H-matrix sign function method Conclusions SIMULATIONSN Peter Benner 2

3 Linear matrix equations Linear Matrix Equations equations without symmetry Sylvester equation AX + XB = W discrete Sylvester equation AXB X = W with data A R n n, B R m m, W R n m and unknown X R n m. equations with symmetry Lyapunov equation AX + XA T = W Stein equation (discrete Lyapunov equation) AXA T X = W with data A R n n, W = W T R n n and unknown X R n n. Here: focus on Sylvester and Lyapunov equations; analogous results and methods for discrete versions exist. SIMULATIONSN Peter Benner 3

4 Linear matrix equations Theoretical Background Using the Kronecker (tensor) product, AX + XB = W is equivalent to ( (Im A) + ( B T I n )) vec (X) = vec (W ). SIMULATIONSN Peter Benner 4

5 Linear matrix equations Theoretical Background Using the Kronecker (tensor) product, AX + XB = W is equivalent to ( (Im A) + ( B T I n )) vec (X) = vec (W ). Hence, Sylvester equation has a unique solution M := (I m A) + ( ) B T I n is invertible. SIMULATIONSN Peter Benner 4

6 Linear matrix equations Theoretical Background Using the Kronecker (tensor) product, AX + XB = W is equivalent to ( (Im A) + ( B T I n )) vec (X) = vec (W ). Hence, Sylvester equation has a unique solution M := (I m A) + ( ) B T I n is invertible. 0 σ (M) = σ ( (I m A) + (B T I n ) ) = {λ j + µ k, λ j σ (A), µ k σ (B)}. SIMULATIONSN Peter Benner 4

7 Linear matrix equations Theoretical Background Using the Kronecker (tensor) product, AX + XB = W is equivalent to ( (Im A) + ( B T I n )) vec (X) = vec (W ). Hence, Sylvester equation has a unique solution M := (I m A) + ( ) B T I n is invertible. 0 σ (M) = σ ( (I m A) + (B T I n ) ) = {λ j + µ k, λ j σ (A), µ k σ (B)}. σ (A) σ ( B) = SIMULATIONSN Peter Benner 4

8 Linear matrix equations Complexity Issues Solving the Sylvester equation via the linear system of equations AX + XB = W ( (Im A) + ( B T I n )) vec (X) = vec (W ) requires LU factorization of n m n m matrix; for n m, the complexity is O(n 6 ); storing n m unknowns; i.e., for n m we have n 2 storage requirement for X. Example: n = m = 1000 Gaussian elimination on a Pentium IV (2 GHz) would take YEARS. SIMULATIONSN Peter Benner 5

9 Applications Applications Stability analysis of linear dynamical systems (Lyapunov s first method): ẋ = Ax is asymptotically (exponentially, Lyapunov) stable AP + P A T + I n = 0 has solution X > 0. V (x) := x T P x is a Lyapunov function for ẋ = Ax. Feedback stabilization for ẋ = Ax + Bu. Model reduction based on balancing and truncation Block-diagonalization of matrices decoupling techniques for ordinary and partial differential equations Image restoration and registration Linear vibration analysis and damper optimization... SIMULATIONSN Peter Benner 6

10 Applications Model Reduction Given linear (control) system ẋ(t) = Ax(t) + Bu(t), y(t) = Cx(t), x(t) R n, with A stable (σ (A) C ), find reduced-order model x(t) = Ã x(t) + Bu(t), ỹ(t) = C x(t), x(t) R l of degree l n such that y ỹ small. Balanced truncation: compute (Ã, B, C) = (ZAT, ZB, CT ) where Z := Σ 1/2 1 V T 1 R, T := ST U 1 Σ 1/2 1 are defined via the SVD»» SR T Σ1 0 V T = [U 1 U 2 ] 1 0 Σ 2 V T 2 and S, R satisfy the controllability and observability Lyapunov equations AS T S + S T SA T + BB T = 0, A T R T R + R T RA + C T C = 0. SIMULATIONSN Peter Benner 7

11 Direct methods Direct Methods Bartels-Stewart method for Sylvester and Lyapunov equation; Hessenberg-Schur method for Sylvester equations; Hammarling s method for stable Lyapunov equations AX + XA T + GG T = 0. All based on the fact that if A, B T are in Schur form, then M = ( (I m A) + ( B T I n )) is block-upper triangular. Hence, M x = b can be solved by back-substitution. Clever (recursive) implementation of back-substitution process requires nm(n + m) flops. For Sylvester equations, B in Hessenberg form is enough ( Hessenberg-Schur method). Hammarling s method modifies recursive process to compute Cholesky factor Y of X directly. All methods require Schur decomposition of A and Schur or Hessenberg decomposition of B need QR algorithm which requires 25n 3 flops for Schur decomposition. Not feasible for large-scale problems (n > 1000). SIMULATIONSN Peter Benner 8

12 Sign function method The Sign Function Method [Roberts 71] For Z R n n with σ (Z) ır = and Jordan canonical form Z = S 1 [ J J ] S = sign (Z) := S [ Ik 0 0 I n k ] S 1. (J ± = Jordan blocks corresponding to σ (Z) C ± ) sign (Z) is root of I n = use Newton s method to compute it: Z 0 Z, Z j+1 1 ( c j Z j + 1 ) Z 1 j, j = 1, 2,... 2 c j = sign (Z) = lim j Z j. (c j > 0 is scaling parameter for convergence acceleration and rounding error minimization.) SIMULATIONSN Peter Benner 9

13 Sign function method Solving Sylvester Equations with the Sign Function Method Consider Lyapunov equation AX + XB + W = 0, A, B stable. As sign ( S 1 ZS ) = S 1 sign (Z) S and [ ] [ ] [ ] [ In 0 A 0 In 0 A 0 = X I m W B X I m 0 B it follows that ([ ]) [ ] A 0 In 0 sign =. W B 2X I m ] Solution of Sylvester (Lyapunov) equation can be read off sign ([ A W ]) 0 B. SIMULATIONSN Peter Benner 10

14 Sign function method The Sign Function Method for Sylvester Equations Apply sign function Newton iteration Z j+1 (Z j + Z 1 j )/2 to Z = [ A W ] 0 B. = sign (Z) = lim j Z j = [ In 2X 0 I n ]. Newton iteration (with scaling) is equivalent to A 0 A, B 0 B, W 0 W for j = 0, 1, 2,... A j+1 1 ( Aj + c 2 2c ja 1 ) j, j B j+1 1 ( Bj + c 2 2c jb 1 ) j, j W j+1 1 ( Wj + c 2 2c ja 1 j W j B 1 ) j. j = X = 1 2 lim j W j Requires explicit inverses not feasible for large-scale problems (n > 1000). SIMULATIONSN Peter Benner 11

15 Direct methods vs. sign function Comparison Compare Bartels-Stewart method as implemented in the Matlab Control Toolbox function lyap, Hessenberg-Schur method as implemented in the SLICOT-based Matlab function slsylv sign function method for random Sylvester equations Bartels Stewart Hessenberg Schur sign function Bartels Stewart Hessenberg Schur sign function 40 Relative Residual CPU time (sec) Problem size n=m Problem size n=m SIMULATIONSN Peter Benner 12

16 Low-rank approximation Low-Rank Approximation Consider the stable Lyapunov equation A T X + XA + C T C = 0, A stable, C R p n. unique solution X 0 = X = Y T Y for Y R n Y n, n Y n. In almost all applications, p n. Often, X has low (numerical) rank. Example: random matrix A, stability margin 0.055, n = 500, m = 10, numerical rank = 31. λ k Eigenvalues in decreasing order eps k SIMULATIONSN Peter Benner 13

17 Low-rank approximation Computation of Y Standard approach: ] Y = R n n, Cholesky factor of X, n Y = n. [ Can be computed by Hammarling s method. Approach here: Y R rank(x) n, full rank factor of X, n Y n. Advantages: more reliable if Cholesky factor is numerically singular; more efficient if rank (X) n (in particular for subsequent computations as in balanced truncation model reduction); SIMULATIONSN Peter Benner 14

18 Factorized sign function method Sign Function Method Again Want factor Y of solution of A T X + XA + C T C = 0. For W 0 = C T 0 C 0 := C T C, C R p n obtain W j+1 = 1 ( Wj + c 2 2c ja T j j W j A 1 ) 1 j = 2c j [ C j c j C j A 1 j ] T [ C j c j C j A 1 j ]. = re-write W j iteration: C 0 := C, C j+1 := [ 1 2cj C j c j C j A 1 j ]. Problem: C j R p j n = C j+1 R 2p j n, i.e., the necessary workspace doubles in each iteration. Two approaches in order to limit work space... SIMULATIONSN Peter Benner 15

19 Factorized sign function method Compute Cholesky factor Y c of X Require p j n: for j > log 2 (n/p) compute QR factorization» 1 p 2cj C j c j C j A 1 j = U j» Ĉj 0, Ĉ j = h i R n n. = W j = ĈT j Ĉj, Y c = 1 2 lim j Ĉ j Compute full-rank factor Y f of X In every step compute rank-revealing QR factorization:» 1 p 2cj C j c j C j A 1 j ([ where R j+1 R p j+1 p j+1, p j+1 = rank = U j+1 " Rj+1 T j+1 0 S j+1 C j c j C j A 1 j # Π j+1, ]). Then C j+1 := [ R j+1 T j+1 ]Π j+1, W j+1 = C T j+1c j+1, Y f = 1 2 lim j C j SIMULATIONSN Peter Benner 16

20 Factorized sign function method Parallelization Newton iteration for sign function easy to parallelize need basic linear algebra (systems of linear equations, matrix inverse, matrix addition, matrix product). Use MPI, BLACS for communication, PBLAS and ScaLAPACK for numerical linear algebra portable code. Development of software library PLiC. Testing on PC Cluster (Linux) with 32 Intel Xeon-2.4GHz processors with workspace/processor: 1 GByte, Myrinet Switch, bandwidth 1.4 Gbit/sec, shows good efficiency and high scalability. SIMULATIONSN Peter Benner 17

21 Factorized solution of Sylvester equations Factorized Solution of Sylvester Equations Let A R n n, B R m m stable, F R n p, G R p m and consider AX + XB + F G = 0. Idea: often, rank (X) min{n, m} or X X ( ) with rank X n. rank (X) = r = Y, Z T R n r with X = Y Z (full-rank factorization, FRF ). Computing FRF with standard methods, Bartels-Stewart method Hessenberg-Schur method would require to compute X R n m and then use SVD or SVD-like factorization. Here: compute FRF directly from F, G using sign function iteration. SIMULATIONSN Peter Benner 18

22 Factorized solution of Sylvester equations Computing Full-Rank Factors of Solution = re-write (unscaled) W j iteration converging to 2X as F j+1 G j+1 = W j ( Wj + A 1 j W j B 1 ) 1 j = 2 ( Fj G j + (A 1 j F j )(G j B 1 j ) ), yielding F 0 := F, F j+1 = 1 2 [ Fj, A 1 j F j ], G 0 := G, G j+1 = 1 2 [ G j G j B 1 j ]. Again: F j R n p j = F j+1 R n 2p j, i.e., workspace doubles per iteration. In order to limit workspace during iteration, compute full-rank factorization in each iteration step directly from F j+1, G j+1. SIMULATIONSN Peter Benner 19

23 Factorized solution of Sylvester equations Keeping the Rank Small ALGORITHM»» G j R1 1. Compute rank-revealing QR factorization (RRQR) G j B 1 =: U Π j 0 G, with U orthogonal, Π G a permutation matrix, R 1 R r m upper triangular of full row-rank. h i» 2. Compute RRQR F j, A 1 T1 j F j U = V Π 0 F, with V orthogonal, Π F a permutation matrix, T 1 R t 2p j upper triangular of full row-rank. 3. Partition V = [ V 1, V 2 ], V 1 R n t, and compute [ T 11, T 12 ] := T 1 Π F, where T 11 R t r. 4. Set F j+1 := V 1 T 11, G j+1 := R 1 Π G. THEOREM W j+1 = F j+1 G j+1, Y := 1 lim F j and Z := 1 lim G j 2 j 2 j exist, and X = Y Z is FRF. SIMULATIONSN Peter Benner 20

24 Factorized solution of Sylvester equations Numerical Example Numerical examples n = 50 : 50 : 2000, n = 500 : 500 : 4000, m = 1, sep(a, B) = 2 n. Compare sign function based method and SLICOT-based implementation of the Hessenberg- Schur method. Use Matlab Release 14 on a Xeon 3GHz workstation with 8GB Ram iterations for sign function iteration. For n = 4000, Y, Z T R , i.e., 1.4MB instead of 122MB storage required for solution. A X + X B + CD F Residuals sign function Matlab/SLICOT A X + X B + F G F 10 8 Residuals sign function Matlab/SLICOT n n SIMULATIONSN Peter Benner 21

25 Factorized solution of Sylvester equations Timings Numerical examples sign function Matlab/SLICOT CPU Times sign function Matlab/SLICOT Execution Times 6000 sec CPU time [sec] n n SIMULATIONSN Peter Benner 22

26 Methods for Large, Sparse Lyapunov Equations Apply adapted splitting methods (Gauß-Seidel, SSOR, etc.) Apply adapted iterative solvers (GMRES, TFQMR, etc.) X V XV T for solution X of projected Lyapunov equation (V T AV ) X + X(V T AV ) T = V T W V, V R n l. Can be computed via Krylov subspace methods. None of these approaches is efficient in general. Idea: try to compute low-rank approximation to X via (approximation to) full-rank factor Y. SIMULATIONSN Peter Benner 23

27 ADI method ADI Method for Lyapunov Equations For A R n n stable, W R n w (w n), consider Lyapunov equation A T X + XA = BB T. ADI Iteration: [Wachspress 88] (A T + p k I)X (j 1)/2 = BB T X k 1 (A p k I) (A T T + p k I)X k = BB T X (j 1)/2 (A p k I) with parameters p k C and p k+1 = p k if p k R. For X 0 = 0 and proper choice of p k : lim X k = X superlinear. k Re-formulation using X k = Y k Y T k yields iteration for Y k... SIMULATIONSN Peter Benner 24

28 ADI method Set X k = Y k Y T k Factored ADI Iteration [Penzl 97, Li/Wang/White 99, B./Li/Penzl], some algebraic manipulations = V 1 p 2Re (p 1 )(A T + p 1 I) 1 B, Y 1 V 1 FOR j = 2, 3,... r V k `I (pk + p k 1 )(A T + p k I) 1 V k 1, Y k ˆ Y k 1 V k where Re (p k ) Re (p k 1 ) Y kmax = [ V 1... V kmax ] V k = C n w and Y kmax Yk T max X Note: Implementation in real arithmetic possible by combining two steps. SIMULATIONSN Peter Benner 25

29 ADI method Low-rank ADI in Action Numerical example Solve AX +XA T +BB T = 0. Finite differences method for 2D heat equation with boundary control Convergence history of low rank ADI iteration n = 10, 000, m = 1. Use lyapack implementation [Penzl 99]. Normalized residual norm Need k max = 57 iterations, i.e., Y kmax R 10, ca. 25 sec. on Centrino 1.4 GHz notebook Iteration steps SIMULATIONSN Peter Benner 26

30 Smith iteration Numerical example Smith Iteration For Stein equations F XF H X = G (1) with F < 1, the fix point iteration X k+1 = F X k F H G, k = 0, 1,... (X 0 = 0) converges to the unique solution X. Smith iteration for Lyapunov equations: if X is a solution of then it is a solution of (1) with AX + XA T = W, F = (I n + µa) 1 (I n µa), G = 2(µ + µ)(i n + µa) 1 W (I n + µa) H. Choosing µ such that σ (F ) { z < 1}, we can compute X as the limit of X k+1 = (I n + µa) 1 ( (I n µa)x k (I n µa) H + 2(µ + µ)w ) (I n + µa) H. SIMULATIONSN Peter Benner 27

31 Smith iteration Details and Extensions Numerical example If A is stable, any µ C can be used. Select µ in order to optimize convergence rate (ρ(f ) 2 ). For σ (A) R, µ = λ max λ min is optimal. [Varga 62, Rosen/Wang 95] Convergence can be accelerated by changing the shift µ in each iteration; in order to limit work needed for factorizations, use finite number l of different shifts and apply them cyclically, store and re-use the l different factorizations if workspace permits. (cyclic) Smith(l) method. [Penzl 00]. Low-rank version possible if W = BB T, mathematically equivalent to low rank ADI with parameters applied cyclically. [Penzl 00]. Number of columns in Y k (X k = Y k Y H ) can be reduced by column compression k technique. [Antoulas/Sorensen/Gugercin 03]. SIMULATIONSN Peter Benner 28

32 H-matrix sign function method H-Matrix Implementation of the Sign Function Method Recall: solution of the Lyapunov equation AX + XA T + W = 0 with the sign function method: A 0 = A, W 0 = BB T A j+1 = 1 2 (A j + A 1 j ) W j+1 = 1 2 (W j + A T j W j A 1 j ) involves the inversion, addition and multiplication of n n matrices complexity: O(n 3 ) Approximation of A in H-matrix format, use of the formatted H-matrix arithmetic complexity: O(n log 2 n). SIMULATIONSN Peter Benner 29

33 H-matrix sign function method Hierarchical Matrices: Short Introduction Hierarchical (H-)matrices are a data-sparse approximation of large, dense matrices arising from the discretization of non-local integral operators occurring in BEM, as inverses of FEM discretized elliptic differential operators, but can also be used to represent FEM matrices directly. Properties of H-matrices: only few data are needed for the representation of the matrix, matrix-vector multiplication can be performed in almost linear complexity (O(n log n)), building sums, products, inverses is of almost linear complexity. SIMULATIONSN Peter Benner 30

34 H-matrix sign function method Hierarchical Matrices: Construction Consider matrices over a product index set I I. Partition I I by the H-tree T I I, where a problem dependent admissibility condition is used to decide whether a block t s I I allows for a low rank approximation. Definition: Hierarchical matrices (H-matrices) The set of the hierarchical matrices is defined by H(T I I, k) := {M R I I rank(m t s ) k admissible leaves t s of T I I } Submatrices of M H(T I I, k) corresponding to inadmissible leaves are stored as dense blocks whereas those corresponding to admissible leaves are stored in factorized form as rank-k matrices, called R k -format. SIMULATIONSN Peter Benner 31

35 H-matrix sign function method Example Stiffness matrix of 2D heat equation with distributed control and isolation BC n = 1024 and k = 4 SIMULATIONSN Peter Benner 32

36 H-matrix sign function method Hierarchical Matrices: Formatted Arithmetic H(T I I, k) is not a linear subspace of R I I formatted arithmetics projection of the sum, product and inverse into H(T I I, k) 1. Formatted Addition ( ) with complexity N H H = O(nk 2 log n)) (for sparse T I I ) Corresponds to best approximation (in the Frobenius-norm). 2. Formatted Multiplication ( ) N H H = O(nk 2 log 2 n) (under some technical assumptions on T I I ) 3. Formatted inversion (Ĩnv) N H, g Inv = O(nk 2 log 2 n) (under some technical assumptions on T I I ) SIMULATIONSN Peter Benner 33

37 H-matrix sign function method H-Matrix Sign Function Method Sign function iteration with formatted H-matrix arithmetic: Ã 0 = (A) H, Ã j+1 = 1 2 (Ãj Ĩnv(A j)) Accuracy control for iterates k = O(log 1 δ + log 1 ρ )), where (Ã 1 j (Ã 1 j + Inv(Ãj)) (Ã 1 j Inv(Ãj)) 2 δ Inv(Ãj)) 2 ρ = forward error bound (assuming c j (δ + ρ) A 1 j 2 < 1 j): Ãj A j 2 c j (δ + ρ), where c 0 = Ã0 A 2 (δ + ρ) 1, c j+1 = 1 2 ( ) A 1 j c j + c j 1 c j (δ + ρ) A 1 j 2. SIMULATIONSN Peter Benner 34

38 H-matrix sign function method Numerical Performance Solve AX + XA T + BB T = 0. FEM discretization of 2D heat equation with boundary control. Accuracy and rank of computed factor n r AX+XA T +BB T 2 2 A 2 X 2 + BB T For n = 262, 144 (that is, 34 billion unknowns in X) we get r = 21 5MB for solution instead of 64GB! Workspace (KB) CPU time 10 8 Storage Requirements H matrix based sign function n log 2 (n) n 10 8 Computing Time H matrix based sign function n log 2 (n) n results by U. Baur SIMULATIONSN Peter Benner 35

39 Conclusions Conclusions Large-scale dense problems can be efficiently solved using parallel implementation of the sign function method. Large-scale dense, but data-sparse problems can be efficiently solved using H-matrix implementation of the sign function method. The most promising methods for large-scale sparse problems are low-rank Smith and ADI methods; selection of acceleration shifts remains a tricky issue. Krylov subspace methods and splitting methods are not competitive in general. To-Do: H-matrix sign function for Sylvester equations Low-rank ADI method for Sylvester equations Low-rank Smith method for Sylvester equations SIMULATIONSN Peter Benner 36