Scientific Computing WS 2018/2019 Lecture 9 Jürgen Fuhrmann juergen.fuhrmann@wias-berlin.de Lecture 9 Slide 1
Lecture 9 Slide 2 Simple iteration with preconditioning Idea: Aû = b iterative scheme û = û M 1 (Aû b) u k+1 = u k M 1 (Au k b) (k = 0, 1... ) 1. Choose initial value u 0, tolerance ε, set k = 0 2. Calculate residuum r k = Au k b 3. Test convergence: if r k < ε set u = u k, finish 4. Calculate update: solve Mv k = r k 5. Update solution: u k+1 = u k v k, set k = i + 1, repeat with step 2. Lecture 5 Slide 42
Lecture 9 Slide 3 The Jacobi method Let A = D E F, where D: main diagonal, E: negative lower triangular part F : negative upper triangular part Preconditioner: M = D, where D is the main diagonal of A ( ) u k+1,i = u k,i 1 a iju k,j b i (i = 1... n) a ii j=1...n Equivalent to the succesive (row by row) solution of a iiu k+1,i + a iju k,j = b i (i = 1... n) j=1...n,j i Already calculated results not taken into account Alternative formulation with A = M N: u k+1 = D 1 (E + F )u k + D 1 b = M 1 Nu k + M 1 b Variable ordering does not matter Lecture 5 Slide 43
Lecture 9 Slide 4 The Gauss-Seidel method Solve for main diagonal element row by row Take already calculated results into account a iiu k+1,i + a iju k+1,j + a iju k,j = b i (i = 1... n) j<i j>i (D E)u k+1 Fu k = b May be it is faster Variable order probably matters Preconditioners: forward M = D E, backward: M = D F Splitting formulation: A = M N forward: N = F, backward: M = E Forward case: u k+1 = (D E) 1 Fu k + (D E) 1 b = M 1 Nu k + M 1 b Lecture 5 Slide 44
Lecture 9 Slide 5 Block methods Jacobi, Gauss-Seidel, (S)SOR methods can as well be used block-wise, based on a partition of the system matrix into larger blocks, The blocks on the diagonal should be square matrices, and invertible Interesting variant for systems of partial differential equations, where multiple species interact with each other Lecture 5 Slide 48
Lecture 9 Slide 6 Convergence Let û be the solution of Au = b. Let e k = u j û be the error of the k-th iteration step u k+1 = u k M 1 (Au k b) = (I M 1 A)u k + M 1 b u k+1 û = u k û M 1 (Au k Aû) = (I M 1 A)(u k û) = (I M 1 A) k (u 0 û) resulting in e k+1 = (I M 1 A) k e 0 So when does (I M 1 A) k converge to zero for k? Lecture 5 Slide 49
Lecture 9 Slide 7 Spectral radius and convergence Definition The spectral radius ρ(a) is the largest absolute value of any eigenvalue of A: ρ(a) = max λ σ(a) λ. Theorem (Saad, Th. 1.10) lim A k = 0 ρ(a) < 1. k Proof, : Let u i be a unit eigenvector associated with an eigenvalue λ i. Then so we must have ρ(a) < 1 Au i = λ iu i A 2 u i = λ ia iu i = λ 2 u i. A k u i = λ k u i therefore A k u i 2 = λ k and lim λ k = 0 k Lecture 5 Slide 52
Lecture 9 Slide 8 Corollary from proof Theorem (Saad, Th. 1.12) lim k Ak 1 k = ρ(a) Lecture 5 Slide 54
Lecture 9 Slide 9 Back to iterative methods Sufficient condition for convergence: ρ(i M 1 A) < 1. Lecture 5 Slide 55
Eigenvalue analysis for more general matrices For 1D heat conduction we used a very special regular structure of the matrix which allowed exact eigenvalue calculations Generalizations to tensor product is possible Generalization to varying coefficients, unstructured grids... what can be done for general matrices? Lecture 9 Slide 10
The Gershgorin Circle Theorem (Semyon Gershgorin,1931) (everywhere, we assume n 2) Theorem (Varga, Th. 1.11) Let A be an n n (real or complex) matrix. Let Λ i = a ij j=1...n j i If λ is an eigenvalue of A then there exists r, 1 r n such that λ a rr Λ r Proof Assume λ is eigenvalue, x a corresponding eigenvector, normalized such that max i=1...n x i = x r = 1. From Ax = λx it follows that (λ a ii )x i = a ij x j λ a rr = j=1...n j i j=1...n j r a rj x j j=1...n j r a rj x j j=1...n j r a rj = Λ r Lecture 9 Slide 11
Gershgorin Circle Corollaries Corollary: Any eigenvalue of A lies in the union of the disks defined by the Gershgorin circles λ {µ V : µ a ii Λ i } Corollary: i=1...n ρ(a) max i=1...n ρ(a) max j=1...n n a ij = A j=1 n a ij = A 1 i=1 Proof n µ a ii Λ i µ Λ i + a ii = a ij j=1 Furthermore, σ(a) = σ(a T ). Lecture 9 Slide 12
Gershgorin circles: example 1.9 1.8 3.4 A = 0.4 1.8 0.4, λ 1 = 1, λ 2 = 2, λ 3 = 3, Λ 1 = 5.2, Λ 2 = 0.8, λ 3 = 0.15 0.05 0.1 2.3 Im 6 5 4 3 2 1 0 4 3 2 1 0 1 2 3 4 5 6 7 8 Re 1 2 3 4 5 6 Lecture 9 Slide 13
Gershgorin circles: heat example I A = B = (I D 1 A) = 2 h 2 h 1 0 2 1 1 2 0 2 2 h............ 1 1 2 0 2 2 h............ 2 h 1 1 2 0 2 1 1 2 0 2 1 2 0 2 h We have b ii = 0, Λ i = { 1 2, i = 1, n 1 i = 2... n 1 estimate λ i 1 Lecture 9 Slide 14
Gershgorin circles: heat example II Let n=11, h=0.1: ( ) ihπ λ i = cos 1 + 2h (i = 1... n) Im 1 0.8 0.6 0.4 0.2 0 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 Re 0.2 0.4 0.6 0.8 1 the Gershgorin circle theorem is too pessimistic... Lecture 9 Slide 15
Weighted directed graph representation of matrices Define a directed graph from the nonzero entries of a matrix A = (a ik ): 4 Nodes: N = {N i } i=1...n Directed edges: E = { N k N l a kl 0} Matrix entries weights of directed edges 1. 0. 0. 2. 0. 3. 4. 0. 5. 0. A = 6. 0. 7. 8. 9. 0. 0. 10. 11. 0. 0. 0. 0. 0. 12. N 5 12 1 N 1 9 N 2 3 5 2 6 8 N 3 7 11 N 4 10 1:1 equivalence between matrices and weighted directed graphs Convenient e.g. for sparse matrices Lecture 9 Slide 16
Reducible and irreducible matrices Definition A is reducible if there exists a permutation matrix P such that PAP T = ( ) A11 A 12 0 A 22 A is irreducible if it is not reducible. Theorem (Varga, Th. 1.17): A is irreducible the matrix graph is connected, i.e. for each ordered pair (N i, N j ) there is a path consisting of directed edges, connecting them. Equivalently, for each i, j there is a sequence of consecutive nonzero matrix entries a ik1, a k1k 2, a k2k 3..., a kr 1k r a kr j. Lecture 9 Slide 17
Taussky theorem (Olga Taussky, 1948) Theorem (Varga, Th. 1.18) Let A be irreducible. Assume that the eigenvalue λ is a boundary point of the union of all the disks λ {µ C : µ a ii Λ i } i=1...n Then, all n Gershgorin circles pass through λ, i.e. for i = 1... n, λ a ii = Λ i Lecture 9 Slide 18
Taussky theorem proof Proof Assume λ is eigenvalue, x a corresponding eigenvector, normalized such that max i=1...n x i = x r = 1. From Ax = λx it follows that (λ a rr )x r = j=1...n j r λ a rr j=1...n j r a rj x j (1) a rj x j j=1...n j r a rj = Λ r (2) λ is boundary point λ a rr = Λ r For all p r with a rp 0, x p = 1. Due to irreducibility there is at least one such p. For this p, equation (2) is valid (with p in place of r) λ a pp = Λ p Due to irreducibility, this is true for all p = 1... n. Lecture 9 Slide 19
Consequences for heat example from Taussky theorem B = I D 1 A We had b ii = 0, Λ i = { 1 2, i = 1, n 1 i = 2... n 1 estimate λ i 1 Assume λ i = 1. Then λ i lies on the boundary of the union of the Gershgorin circles. But then it must lie on the boundary of both circles with radius 1 2 and 1 around 0. Contradiction λ i < 1, ρ(b) < 1! Lecture 9 Slide 20
Diagonally dominant matrices Definition Let A = (a ij ) be an n n matrix. A is diagonally dominant if (i) for i = 1... n, a ii j=1...n j i a ij A is strictly diagonally dominant (sdd) if (i) for i = 1... n, a ii > a ij j=1...n j i A is irreducibly diagonally dominant (idd) if (i) A is irreducible (ii) A is diagonally dominant for i = 1... n, a ii j=1...n j i a ij (iii) for at least one r, 1 r n, a rr > j=1...n j r a rj Lecture 9 Slide 21
A very practical nonsingularity criterion Theorem (Varga, Th. 1.21): Let A be strictly diagonally dominant or irreducibly diagonally dominant. Then A is nonsingular. If in addition, a ii > 0 is real for i = 1... n, then all real parts of the eigenvalues of A are positive: Reλ i > 0, i = 1... n Lecture 9 Slide 22
A very practical nonsingularity criterion, proof I Proof: Assume A strictly diagonally dominant. Then the union of the Gershgorin disks does not contain 0 and λ = 0 cannot be an eigenvalue A is nonsingular. As for the real parts, the union of the disks is i=1...n {µ C : µ a ii Λ i } and Reµ must be larger than zero if µ should be contained. Lecture 9 Slide 23
A very practical nonsingularity criterion, proof I Assume A irreducibly diagonally dominant. Then, if 0 is an eigenvalue, it sits on the boundary of one of the Gershgorin disks. By Taussky theorem, we have a ii = Λ i for all i = 1... n. This is a contradiction as by definition there is at least one i such that a ii > Λ i Assume a ii > 0, real. All real parts of the eigenvalues must be 0. Therefore, if a real part is 0, it lies on the boundary of at least one disk. By Taussky theorem it must be contained at the same time in the boundary of all the disks and in the imaginary axis. This contradicts the fact that there is at least one disk which does not touch the imaginary axis as by definition there is at least one i such that a ii > Λ i Lecture 9 Slide 24
Corollary Theorem: If A is complex hermitian or real symmetric, sdd or idd, with positive diagonal entries, it is positive definite. Proof: All eigenvalues of A are real, and due to the nonsingularity criterion, they must be positive, so A is positive definite. Lecture 9 Slide 25
Heat conduction matrix α + 1 h A = 2 h 2 h............ 1 2 h h 1 2 h h 1 h 1 h + α A is idd A is nonsingular diaga is positive real eigenvalues of A have positive real parts A is real, symmetric A is positive definite Lecture 9 Slide 26
Perron-Frobenius Theorem (1912/1907) Definition: A real n-vector x is positive (x > 0) if all entries of x are positive nonnegative (x 0) if all entries of x are nonnegative Definition: A real n n matrix A is positive (A > 0) if all entries of A are positive nonnegative (A 0) if all entries of A are nonnegative Theorem(Varga, Th. 2.7) Let A 0 be an irreducible n n matrix. Then (i) A has a positive real eigenvalue equal to its spectral radius ρ(a). (ii) To ρ(a) there corresponds a positive eigenvector x > 0. (iii) ρ(a) increases when any entry of A increases. (iv) ρ(a) is a simple eigenvalue of A. Proof: See Varga. Lecture 9 Slide 27
Perron-Frobenius for general nonnegative matrices Each n n matrix can be brought to the normal form R 11 R 12... R 1m PAP T 0 R 22... R 2m =.... 0 0... R mm where for j = 1... m, either R jj irreducible or R jj = (0). Theorem(Varga, Th. 2.20) Let A 0 be an n n matrix. Then (i) A has a nonnegative eigenvalue equal to its spectral radius ρ(a). This eigenvalue is positive unless A is reducible and its normal form is strictly upper triangular (ii) To ρ(a) there corresponds a nonzero eigenvector x 0. (iii) ρ(a) does not decrease when any entry of A increases. m Proof: See Varga; σ(a) = σ(r jj ), apply irreducible Perron-Frobenius to R jj. j=1 Lecture 9 Slide 28
Theorem on Jacobi matrix Theorem: Let A be sdd or idd, and D its diagonal. Then ρ( I D 1 A ) < 1 Proof: Let B = (b ij ) = I D 1 A. Then b ij = If A is sdd, then for i = 1... n, { 0, i = j aij a ii, i j j=1...n b ij = j=1...n j i a ij a ii = Λ i a ii < 1 Therefore, ρ( B ) < 1. Lecture 9 Slide 29
Theorem on Jacobi matrix II If A is idd, then for i = 1... n, j=1...n j=1...n b ij = j=1...n j i a ij a ii = Λ i a ii 1 b rj = Λ r < 1 for at least one r a rr Therefore, ρ( B ) <= 1. Assume ρ( B ) = 1. By Perron-Frobenius, 1 is an eigenvalue. As it is in the union of the Gershgorin disks, for some i, λ = 1 Λ i a ii 1 and it must lie on the boundary of this union. By Taussky then one has for all i λ = 1 Λ i a ii = 1 which contradicts the idd condition. Lecture 9 Slide 30
Jacobi method convergence Corollary: Let A be sdd or idd, and D its diagonal. Assume that a ii > 0 and a ij 0 for i j. Then ρ(i D 1 A) < 1, i.e. the Jacobi method converges. Proof In this case, B = B. Lecture 9 Slide 31
Regular splittings A = M N is a regular splitting if M is nonsingular M 1, N are nonnegative, i.e. have nonnegative entries Regard the iteration u k+1 = M 1 Nu k + M 1 b. We have I M 1 A = M 1 N. Lecture 9 Slide 32
Convergence theorem for regular splitting Theorem: Assume A is nonsingular, A 1 0, and A = M N is a regular splitting. Then ρ(m 1 N) < 1. Proof: Let G = M 1 N. Then A = M(I G), therefore I G is nonsingular. In addition A 1 N = (M(I M 1 N)) 1 N = (I M 1 N) 1 M 1 N = (I G) 1 G By Perron-Frobenius (for general matrices), ρ(g) is an eigenvalue with a nonnegative eigenvector x. Thus, 0 A 1 Nx = ρ(g) 1 ρ(g) x Therefore 0 ρ(g) 1. As I G is nonsingular, ρ(g) < 1. Lecture 9 Slide 33
Convergence rate comparison Corollary: ρ(m 1 N) = Proof: Rearrange τ = τ 1+τ where τ = ρ(a 1 N). ρ(g) 1 ρ(g) Corollary: Let A 0, A = M 1 N 1 and A = M 2 N 2 be regular splittings. If N 2 N 1 0, then 1 > ρ(m 1 2 N 2 ) ρ(m 1 1 N 1 ). Proof: τ 2 = ρ(a 1 N 2 ) ρ(a 1 N 1 ) = τ 1 But τ 1+τ is strictly increasing. Lecture 9 Slide 34
M-Matrix definition Definition Let A be an n n real matrix. A is called M-Matrix if (i) a ij 0 for i j (ii) A is nonsingular (iii) A 1 0 Corollary: If A is an M-Matrix, then A 1 > 0 A is irreducible. Proof: See Varga. Lecture 9 Slide 35
Main practical M-Matrix criterion Corollary: Let A be sdd or idd. Assume that a ii > 0 and a ij 0 for i j. Then A is an M-Matrix. Proof: We know that A is nonsingular, but we have to show A 1 0. Let B = I D 1 A. Then ρ(b) < 1, therefore I B is nonsingular. We have for k > 0: I B k+1 = (I B)(I + B + B 2 + + B k ) (I B) 1 (I B k+1 ) = (I + B + B 2 + + B k ) The left hand side for k converges to (I B) 1, therefore (I B) 1 = k=0 B k As B 0, we have (I B) 1 = A 1 D 0. As D > 0 we must have A 1 0. Lecture 9 Slide 36
Application Let A be an M-Matrix. Assume A = D E F. Jacobi method: M = D is nonsingular, M 1 0. N = E + F nonnegative convergence Gauss-Seidel: M = D E is an M-Matrix as A M and M has non-positive off-digonal entries. N = F 0. convergence Comparison: N J N GS Gauss-Seidel converges faster. More general: Block Jacobi, Block Gauss Seidel etc. Lecture 9 Slide 37
Intermediate Summary Given some matrix, we now have some nice recipies to establish nonsingularity and iterative method convergence: Check if the matrix is irreducible. This is mostly the case for elliptic and parabolic PDEs. Check if the matrix is strictly or irreducibly diagonally dominant. If yes, it is in addition nonsingular. Check if main diagonal entries are positive and off-diagonal entries are nonpositive. If yes, in addition, the matrix is an M-Matrix, its inverse is nonnegative, and elementary iterative methods converge. These critera do not depend on the symmetry of the matrix! Lecture 9 Slide 38
Example: 1D finite difference matrix: α + 1 h Au = 2 h 2 h............ 1 2 h h 1 2 h h 1 h 1 h + α u 1 u 2 u 3. u N 2 u N 1 u N = f = αv 1 hf 2 hf 3. hf N 2 hf N 1 αv n idd positive main diagonal entries, nonpositive off-diagonal entries A is nonsingular, has the M-property, and we can e.g. apply the Jacobi and Gauss-Seidel iterative method to solve it (ok, in 1D we already know this is a bad idea... ). for f 0 and v 0 it follows that u 0. heating and positive environment temperatures cannot lead to negative temperatures in the interior. Lecture 9 Slide 39
Incomplete LU factorizations (ILU) Idea (Varga, Buleev, 1960): fix a predefined zero pattern apply the standard LU factorization method, but calculate only those elements, which do not correspond to the given zero pattern Result: incomplete LU factors L, U, remainder R: A = LU R Problem: with complete LU factorization procedure, for any nonsingular matrix, the method is stable, i.e. zero pivots never occur. Is this true for the incomplete LU Factorization as well? Lecture 9 Slide 40
Comparison of M-Matrices Theorem(Saad, Th. 1.33): Let A, B n n matrices such that (i) A B (ii) b ij 0 for i j. Then, if A is an M-Matrix, so is B. Proof: For the diagonal parts, one has D B D A > 0, D A A D B B 0 Therefore I D 1 A A D 1(D B B) D 1 (D B B) = I D 1 B =: G 0. A B B Perron-Frobenius ρ(g) = ρ(i D 1 B B) ρ(i D 1 A A) < 1 I G is nonsingular. From the proof of the M-matrix criterion, B = (I G) 1 = k=0 G k 0. As D B > 0, we get B 0. D 1 B Lecture 9 Slide 41
M-Property propagation in Gaussian Elimination Theorem:(Ky Fan; Saad Th 1.10) Let A be an M-matrix. Then the matrix A 1 obtained from the first step of Gaussian elimination is an M-matrix. Proof: One has aij 1 = a ij ai1a1j a 11, a ij, a i1, a 1j 0, a 11 > 0 aij 1 0 for i j 1 0... 0 a 12 a 11 1... 0 A = L 1 A 1 with L 1 =.... 0 a 1n a 11 0... 1 A 1 nonsingular nonsingular, nonnegative Let e 1... e n be the unit vectors. Then A 1 1 e 1 = 1 a e 11 1 0. For j > 1, A 1 1 e j = A 1 L 1 e j = A 1 e j 0. A 1 1 0 Lecture 9 Slide 42
Stability of ILU Theorem (Saad, Th. 10.2): If A is an M-Matrix, then the algorithm to compute the incomplete LU factorization with a given nonzero pattern A = LU R is stable. Moreover, A = LU R is a regular splitting. Lecture 9 Slide 43
Stability of ILU decomposition II Proof Let à 1 = A 1 + R 1 = L 1 A + R 1 where R 1 is a nonnegative matrix which occurs from dropping some off diagonal entries from A 1. Thus, à 1 A 1 and Ã1 is an M-matrix. We can repeat this recursively à k = A k + R k = L k A k 1 + R k = L k L k 1 A k 2 + L k R k 1 + R k = L k L k 1... L 1 A + L k L k 1... L 2 R 1 + + R k Let L = (L n 1... L 1 ) 1, U = à n 1. Then U = L 1 A + S with S = L n 1 L n 2... L 2 R 1 + +R n 1 = L n 1 L n 2... L 2 (R 1 +R 2 +... R n 1 ) Let R = R 1 + R 2 +... R n 1, then A = LU R where U 1 L 1, R are nonnegative. Lecture 9 Slide 44
ILU(0) Special case of ILU: ignore any fill-in. Representation: M = ( D E) D 1 ( D F ) D is a diagonal matrix (wich can be stored in one vector) which is calculated by the incomplete factorization algorithm. Setup: for(int i=0;i<n;i++) d(i)=a(i,i) for(int i=0;i<n;i++) { d(i)=1.0/d(i) for (int j=i+1;j<n;j++) d(j)=d(j)-a(i,j)*d(i)*a(j,i) } Lecture 9 Slide 45
ILU(0) Solve Mu = v for(int i=0;i<n;i++) { double x=0.0; for (int j=0;j<i;i++) x=x+a(i,j)*u(j) u(i)=d(i)*(v(i)-x) } for(int i=n-1;i>=0;i--) { double x=0.0 for(int j=i+1;j<n;j++) x=x+a(i,j)*u(j) u(i)=u(i)-d(i)*x } Lecture 9 Slide 46
ILU(0) Generally better convergence properties than Jacobi, Gauss-Seidel One can develop block variants Alternatives: ILUM: ( modified ): add ignored off-diagonal entries to D ILUT: zero pattern calculated dynamically based on drop tolerance Dependence on ordering Can be parallelized using graph coloring Not much theory: experiment for particular systems I recommend it as the default initial guess for a sensible preconditioner Incomplete Cholesky: symmetric variant of ILU Lecture 9 Slide 47