FEM and sparse linear system solving

Transcription

1 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Lecture 9, Nov 17, 2017: Krylov space methods Peter Arbenz Computer Science Department, ETH Zürich

2 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Survey on lecture Survey on lecture The finite element method Direct solvers for sparse systems Iterative solvers for sparse systems Stationary iterative methods, preconditioning Steepest descent and conjugate gradient methods Krylov space methods for nonsymmetric systems GMRES, MINRES Preconditioning Nonsymmetric Lanczos iteration based methods Bi-CG, QMR, CGS, BiCGstab Multigrid (preconditioning)

3 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Survey on lecture Outline of this lecture 1. Krylov (sub)spaces 2. Orthogonal bases for Krylov spaces 3. GMRES 4. MINRES Krylov space methods are matrix-free : only a MatVec function is necessary, that computes y Ax.

4 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Krylov spaces In steepest descent and CG we have rk = rk 1 α k 1 Apk 1, pk = rk + β k 1 pk 1, where rk = b Axk is the residual and pk is the search direction. Since pk depends on pk 1 and rk we can write rk = r0 + k c j A j r0 rk = p k (A)r0, p k (0) = 1, j=1 Thus, the residuals of these methods can be written as a k-th order polynomial in A applied to the initial residual r0.

5 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Krylov spaces (cont.) Moreover, k c j A j rj = rk r0 j=1 = (b Axk) (b Ax0) = A(xk x0) such that k 1 xk = x0 c j+1 A j r0. j=0

6 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Krylov spaces (cont.) Definition For given A, the m-th Krylov space generated by the vector r is given by K m = K m (A, r) := span { r, Ar, A 2 r,..., A m 1 r}. We can also write K m (A, r) = {p(a)r p P m 1 }. where P d denotes the set of polynomials of degree at most d. For the residuals rm of stationary or CG iterations we have rm r0 + AK m 1 (A, r0) K m (A, r0).

7 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Cayley Hamilton theorem Theorem: Let χ A (ζ) := det(ζi A) denote the characteristic polynomial of A R n n. Then χ A (A) = O. Corollary: Let A be nonsingular and χ A (ζ) = ζ n + a n 1 ζ n 1 + a n 2 ζ n a 0, with a 0 = det(a) 0, then A 1 = 1 [ A n 1 + a n 1 A n 2 + a n 2 A n a 1 I ]. a 0

8 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Cayley Hamilton theorem (cont.) Consequence: If A nonsingular, then A 1 is a polynomial in A of degree at most n 1. If Ax = b then x = (A n 1 b + a n 1 A n 2 b + a n 2 A n 3 b + + a 1 b)/( a 0 ). As Ae0 = r0 we have e0 = (A n 1 r0 + a n 1 A n 2 r0 + a n 2 A n 3 r0 + + a 1 r0)/( a 0 ). x x0 + K n (A, r0). Therefore it makes sense to look for approximate solutions in Krylov spaces.

9 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Minimal polynomial Definition For A R n n, the unique monic polynomial µ A of minimal degree d satisfying µ A (A) = 0 is called the minimal polynomial of A. Theorem: Let A have distinct eigenvalues λ 1,..., λ k, then µ A (ζ) = k (ζ λ i ) r i, where r i is the order of the largest Jordan block of A for λ i. Note: i=1 The degree of µ A is at most n (Cayley Hamilton). e0 K d, interesting when d n Ideal: few distinct eigenvalues, small Jordan blocks!

10 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Dependence on right-hand side What does it mean when K 1 K 2 K p = K p+1 =? A p r0 = α 0 r0 + α 1 Ar0 + α 2 A 2 r0 + + α p 1 A p 1 r0 Multiplying by A 1 and noting that A 1 r0 = e0 gives A p 1 r0 = α 0 e0 + α 1 r0 + α 2 Ar0 + + α p 1 A p 2 r0 Therefore e0 K p (A, r0) or x x0 + K p (A, r0). We get the solution after precisely p steps. Note that p depends on r0 (and thus on x0).

11 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces Dependence on right-hand side (cont.) Let A be diagonalizable with A = UΛU 1. This means that there n linearly independent vectors (eigenvectors) with Aui = uiλ i, 1 i n. If r0 = p i=1 α i ui then A j r0 = p i=1 α iλ j ui. So, if r0 span{u1,..., up} with Aui = uiλ i, then K p = K p+1 means that the solution is found in p iteration steps. This is an uncommon situation! Usually, p n and it is not practical to iterate until the Krylov space is exhausted. We try to get a good approximation of x after m n iteration steps.

12 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Krylov spaces What is the best xm from K m? Want to best approximate x = A 1 b from x0 + K m (A, r0). Various answers lead to different methods: Ideal but not practical: minimize em 2 = x xm 2. When A is SPD, then we can minimize em A = rm A 1. Conjugate Gradients Can minimize rm 2. For symmetric A: MINimum RESidual, for nonsymmetric A: Generalized Minimum RESidual Can enforce Galerkin condition rm K m or Petrov-Galerkin condition rm L m, e.g. BiConjugate Gradients and Quasi Minimum Residual, these are m constraints to compute m parameters α 0,..., α m 1 from xm = x0 + α 0 r0 + α 1 Ar0 + α 2 A 2 r0 + + α m 1 A m 1 r0.

13 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis An orthogonal basis for K m Problem: The matrix K m (A, r0) := r0 Ar0 A m 1 r0 becomes more and more ill-conditioned as k increases. (Remember vector iteration for computing largest eigenvalue.) Solution: We have to find a well-conditioned basis of K m.

14 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis Arnoldi & Lanczos algorithms Task: For j = 1, 2,..., m, compute orthonormal bases {v1,..., vj} for the Krylov spaces K j = span { r0, Ar0, A 2 r0,..., A j 1 r0}. The algorithms that do this are Lanczos algorithm for A symmetric/hermitian. Arnoldi algorithm for A nonsymmetric. In principle, Lanczos and Arnoldi algorithms implement the Gram Schmidt orthogonalization procedure. Difficulty: Because of ill-conditioning, we do not want to explicitly form r0, Ar0,..., A j r0.

15 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis Arnoldi & Lanczos algorithms (cont.) Instead of using A j r0 we proceed with Avj. (Notice that Avi K i+1 K j for all i < j.) Orthogonalize Avj against v1,..., vj by the Gram Schmidt procedure: j wj = Avj vih ij. wj points in the desired new direction (unless it is 0). Therefore, i=1 vj+1 = wj/ wj. Q: What happens if wj = 0?

16 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis Arnoldi: Classical and modified Gram Schmidt v1 = r0/ r0 2 for j:=1 to m do wj = Avj; for i:=1 to j do Orthogonalize Avj against v1,..., vj: { hi,j := (Avj) vi; (Classical Gram Schmidt) h i,j := w j v i; (Modified Gram Schmidt) wj := wj h i,j vi; end for h j+1,j = wj 2. (Stop if h j+1,j is zero.) vj+1 = wj/h j+1,j ; end for In practice: MGS is more numerically stable than CGS

17 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis Remark: Classical vs. modified Gram Schmidt Given vectors a and v1,..., vj with v T k v l = 0 for all 1 k, l j. Task: Compute z s.t. a = i Classical Gram Schmidt z = a; for i:=1 to j do α i = v i T a; z = z viα i ; end for MGS is more stable than CGS. α i vi + z with z T vl = 0 for all l. Modified Gram Schmidt z = a; for i:=1 to j do α i = v i T z; z = z viα i ; end for There are other solutions as, e.g., Householder reflectors. See Golub van Loan: Matrix Computations.

18 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis Arnoldi relation Define V m := [v1,..., vm]. Then we get the Arnoldi relation AV m = V m H m + wme T m = V m+1 Hm. H m A V m = V m + O w m

19 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 Orthogonal basis Arnoldi relation (cont.) Here, h 11 h 12 h 1,m h 21 h 22 h 2,m H m = h 3,2 h 3,m.... R (m+1) m h m+1,m The square matrix H m R m m is obtained from H m by deleting the last row. Notice that H m = V T m AV m. Therefore, if A is symmetric H m T m is tridiagonal! The Lanczos relation is AV m = V m T m + wme T m = V m+1 T m.

20 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES GMRES The GMRES (Generalized Minimal Residual) algorithm computes xm x0 + K m (A, r0) that leads to the smallest residual exploiting the Arnoldi relation. The MINRES algorithm does the same for symmetric matrices using the Lanczos relation. Goal: Cheaply minimize rm 2 = b Axm 2, xm x0 + K m (A, r0), using the Arnoldi relation and R(V m ) = K m (A, r0). AV m = V m+1 H m

21 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES A Hessenberg Least-Squares problem Arnoldi basis expansion xm = x0 + V m y gives with β := r0 2 : min rm 2 = min b A(x0 + V m y) 2 y = min r0 AV m y 2 y = min r0 V m+1 Hm y 2 y ( = min V m+1 βe 1 H m y) 2 y = min βe1 H m y 2. y Hessenberg least squares problem of dimension (m + 1) m. QR factorization of H m can be cheaply computed using Givens rotations.

22 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES QR factorization of H m using m Givens row rotations h 11 h 12 h 13 h 14 h 21 h 22 h 23 h 24 h 32 h 33 h 34 h 43 h 44 h 54 r 11 r 12 r 13 r 14 0 r 22 r 23 r 24 0 r 33 r 34 h 43 h 44 h 54 r 11 r 12 r 13 r 14 0 r 22 r 23 r 24 h 32 h 33 h 34 h 43 h 44 h 54 r 11 r 12 r 13 r 14 r 0 r 22 r 23 r r 12 r 13 r 14 0 r 33 r 34 0 r 44 0 r 22 r 23 r 24 0 r 33 r 34 0 r 44 h 54 0 Apply same rotations to rhs & solve triangular m m system: min rm 2 = min βe1 H m y 2 = min βz R m y 2.

23 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Residual results Monotonicity: rm+1 2 rm 2 because K m K m+1. Residual polynomial: Since rm = b Axm r0 + AK m, we have rm = p m (A)r0. p m is a polynomial of degree m and p m (0) = 1. Denote the set of all such polynomials by P m. Theorem Let A = QΛQ 1 be diagonalizable. Then at step m of the GMRES iteration, the residual rm satisfies rm 2 r0 2 inf p m P m p m (A) 2 κ(q) inf p m P m max p m(λ). λ σ(a)

24 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Residual results (cont.) We have seen a similar result for the CG algorithm. There κ(q) = 1 and σ(a) [α, β], α > 0. In more general situations, the distribution of A s eigenvalues is more complicated. One may replace σ(a) by a simpler shaped set D σ(a) to simplify the analysis. E.g., D may be an ellipse for which more general Chebyshev polynomials have been developed. But 0 must never be contained in D!

25 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Sample matrices & convergence Remember: minimal polynomial µ A, assume that A diagonalizable all Jordan blocks size one Ideally: few distinct eigenvalues Heuristically: eigenvalues should best be squeezed in small ball Perfect: A is (multiple of) identity: one point spectrum Visual interpretation of preconditioning goal Compare convergence of GMRES for two matrices with quite different spectra Pictures from Trefethen Bau, Numerical Linear Algebra, SIAM 1997, Lecture 35.

26 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Matrix 1 spectrum Matrix A is obtained in Matlab by m = 200; A = 2*eye(m) + 0.5*randn(m)/sqrt(m);. A has a spectrum in a ball with radius 1/2 centered at 2 Let us choose p n (z) = (1 z/2) n. On the spectrum 1 z/2 1/4 and therefore p n (z) (1/4) n. The convergence in this case is extraordinarily steady at a rate 4 n, see next slide.

27 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Matrix 1 GMRES convergence

28 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Matrix 2 - spectrum A = A + D where A is matrix 1 and D is the diagonal matrix with d k = ( 2+2 sin θ k )+i cos θ k, θ k = kπ/(m+1), 0 k < m. A has a spectrum that is surrounding the origin. The eigenvalues lie in a semicircular cloud that bends around the origin. The convergence rate is much worse than in example 1. The condition of Q is not large. The slow convergence is due to the location of the eigenvalues.

29 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Matrix 2 GMRES convergence

30 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES Practical issues The GMRES algorithm with m steps needs memory space for m + O(1) vectors of length n plus the matrices A and M. CG needed just 4 vectors Slow convergence may entail expensive memory requirements. Recipe: limit m. Execute m steps of GMRES. Then restart with xm as initial approximation of the next GMRES cycle. GMRES does not care about symmetries. There is no gain in choosing a symmetric preconditioner M when solving M 1 Ax = M 1 b. Incomplete LU factorizations (ILU) are popular preconditioners.

31 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 GMRES The GMRES(m) algorithm Set an initial guess x0 until convergence do Compute initial residual, r0 = b Ax0 Solve for preconditioned residual, Mz0 = r0 Normalize residual: v1 = z0/ z0 for k = 1, 2,..., m do Compute matrix-vector product rk = Avk Solve Mzk = rk Orthonormalize zk with respect to all vj, j = 1,..., k by modified Gram-Schmidt procedure to obtain vk+1 end Solve GMRES minimization problem. Copy solution xm into x0. Check convergence, if satisfied leave GMRES. end

32 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 MINRES Krylov space methods for symmetric systems Let A be symmetric indefinite. The CG method does not work since A does not induce a norm: x T Ax and x T A 1 x can be negative and it does not make sense to minimize r A 1. Then the minimal residual (MINRES) algorithm is a feasible solution method. MINRES is essentially GMRES adapted to symmetric systems. Two things are worth noting: Avk does not need to be orthogonalized against vj for j < k 1: (Avk) T vj = vk T AT A=A T vj = vk T Avj = 0 if j + 1 < k. }{{} K j+1 We have a short recurrence.

33 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 MINRES Krylov space methods for symmetric systems (cont.) Hm =: T m is tridiagonal. t 11 t 12 t 21 t 22 t 23. T m = t..... [ 3,2... = tm 1,m 1 t m 1,m t m,m 1 t m,m t m+1,m T m t m+1,m e T m Note that the submatrix T m is symmetric. Operating with Givens rotations on T m zeros the lower off-diagonal elements at the expense of an additional upper off-diagonal. Of course the resulting matrix is upper-triangular. ]

34 The updates of xk and rk can be done by 3-term recurrencies. Preconditioners for A must be SPD in order to use this procedure. (Why?) FEM & sparse linear system solving, Lecture 9, Nov 19, /36 MINRES The (unpreconditioned) MINRES algorithm Choose x0. Compute r0 = b Ax0. β = r0. Set v0 = r0/β. for k = 1, 2,... do wk := Avk 1; if k > 1 then wk = wk vk 2t k,k 1 t k,k := w T k v k 1; wk = wk vk 1t k,k t k+1,k := wk 2 ; yk = wk/t k+1,k ; Update the QR factorization T k = Q k R k. Solve R k hk = βq T k e 1. xk := V k hk. rk := b Axk. Check for convergence. end do

35 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 MINRES Note on CG CG MINRES In both methods: xm x0 + K m (A, r0) MINRES: Choose xm such that rm 2 is minimized. min rm 2 = min r0 AV m y 2 = min β e1 T m y 2. y y CG: Choose xm such that the Galerkin condition rm K m is satisfied. 0 = V T m (b Axm) = V T m (r0 AV m ym) = β e1 T m ym. (β = r0 2 )

36 FEM & sparse linear system solving, Lecture 9, Nov 19, /36 MINRES Note on CG Main practical issues for Krylov space methods Select reasonable convergence criterion, and reasonable accuracy threshold τ. Without a good preconditioner, the method might not converge within a reasonable number of iteration steps. Cannot keep increasing the dimension of K, not enough memory! = need to decide how and when to restart.