Iterative Methods and Multigrid Part 1: Introduction to Multigrid 2000 Eric de Sturler 1 12/02/09 MG01.prz
Basic Iterative Methods (1) Nonlinear equation: f(x) = 0 Rewrite as x = F(x), and iterate x i+1 = F(x i ) (fixed-point iteration) Converges locally if F (x) < 1 in neighborhood of solution Linear system: Ax = b Matrix splitting: A = M N (M N)x = b w Mx= Nx+ b w x = M 1 Nx+ M 1 b Iterate: x k+1 =(I P 1 A)x k + P 1 b or Converges iff (if and only if) (M 1 N)<1 Sol ve Mx k+1 = Nx k + b Methods: Jacobi iteration, Gauss-Seidel, (S)SOR,... Fixed-point: x = M 1 Nx+ M 1 b w M 1 Ax = M 1 b Fixed-point is solution of the preconditioned system: M 1 Ax = M 1 b 2 12/02/09 MG01.prz
Basic Iterative Methods (2) x k+1 = M 1 Nx k + M 1 b = M 1 (M A)x k + M 1 b = x k + M 1 b M 1 Ax k Linear System: Ax = b Residual: r k = b Ax k Prec. system: M 1 Ax = M 1 b Prec. residual: r k = M 1 b M 1 Ax k x k+1 = x k + r k u x k+1 = x 0 + r 0 + r 1 + + r k Update: x k+1 x 0 = r 0 + r 1 + + r k r k+1 = M 1 b M 1 Ax k+1 = M 1 b M 1 Ax k M 1 Ar k = r k M 1 Ar k r k+1 = (I M 1 A)r k = (M 1 N) k+1 r 0 3 12/02/09 MG01.prz
Basic Iterative Methods (3) Solution to : Error: Ax = bxˆ k = xˆ x k Residual and error: r k = b Ax k = Axˆ Ax k = A k ( r k = M 1 A k ) Theorem: xˆ is a fixed point of x k+1 = M 1 Nx k + M 1 biff xˆ is the solution of M 1 Ax = M 1 b (g Ax = b) Proof: x =(I M 1 A)x+ M 1 b = x M 1 Ax+ M 1 b M 1 Ax = M 1 b w k+1 = xˆ x k+1 = M 1 Nxˆ + M 1 b M 1 Nx k M 1 b = M 1 N k k+1 = M 1 N k = (M 1 N) k+1 0 and r k+1 = (M 1 N) k+1 r 0 4 12/02/09 MG01.prz
Matrix Splittings (1) Some well-known choices for M: A 11 A 1p A = A p1 A pp M = diag(a 11, A 22,, A pp ) (point/block) Jacobi M = A 11 A p1 A pp (point/block) Gauss-Seidel M = 1 A 11 A p1 1 A pp (point/block) SOR x SOR k+1 = x GS SOR k+1 + (1 )x k 5 12/02/09 MG01.prz
Review Basic Iterative Methods Solving Au = f Write A = D L U, where U is the strict upper triangular part, L is the strict lower triangular part, and D is the diagonal. Jacobi iteration: Du k+1 = (L + U)u k + f Gauss-Seidel iteration: (D L )u k+1 = Uu k + f Block versions possible. Basic iteration: u k+1 = P 1 (P A)u k + P 1 f = R x u k + P 1 f Convergence iff and asymptotic convergence rate:. (R x ) < 1 (R x ) R J = D 1 (L + U) R GS = (D L ) 1 U 6 12/02/09 MG01.prz
Use of Relaxation Parameter We can use a relaxation parameter to improve convergence: Jacobi iteration: ũ k = D 1 (L + U)u k + D 1 f u k+1 = (1 )u k + ũ k Alternatively: u k+1 = u k + r k with r k = D 1 (f Au k ) R J, = (1 )I + R J Gauss-Seidel iteration: ũ k = (D L ) 1 Uu k + (D L ) 1 f u k+1 = (1 )u k + ũ k Alternatively: u k+1 = u k + r k with r k = (D L ) 1 (f Au k ) R GS, = (1 )I + R GS 7 12/02/09 MG01.prz
Model Problem Consider the following model problem, u xx = f for xc (0, 1) with Dirichlet boundary conditions u = 0 for x = 0 and x = 1. We discretize the problem (see Meurant 1.9) using n + 1 grid points. This gives h = n 1 and x j = jh for j = 0 n. We use the standard central finite difference discretization u xx = u i 1 2u i +u i+1. h + O(h 2 ) 2 1 1 1 This gives the matrix A = tridiag 2 2 2 2. 1 1 1 This gives eigenvectors u (k) j = sin jk n and eigenvalues k h ( k) = 4sin 2 ( k. (see Meurant 1.10) 2n ) 8 12/02/09 MG01.prz
Model Problem (Jacobi) Let s consider pointwise algorithm for model problem, for, and u xx + u = f 0 < x < 1 u(0) = u(1) = 0 Equidistant grid points: and. Initially. x i = ih u i = u(x i ) = 0 Centered finite differences: where. u i 1 + 2u i u i+1 = h 2 f i i = 1 n 1 Jacobi iteration: u ( k+1) = D 1 (L + U)u ( k) + D 1 f Update at each point: u (k+1) i = 1 2 h 2 f i + u (k) (k) i 1 + u i+1 = u i (k) + 1 2r i (k) Update can be done for all points at once (no dependencies). With relaxation parameter : u (k+1) i = u (k) i + 1 (k) 2 r i 9 12/02/09 MG01.prz
Model problem (Gauss-Seidel) Let s consider pointwise algorithm for model problem, for, and u xx + u = f 0 < x < 1 u(0) = u(1) = 0 Equidistant grid points: x i = ih and u i = u(x i ). Centered finite differences: u i 1 + 2u i u i+1 = h 2 f i where i = 1 n 1. Gauss-Seidel iteration: u ( k+1) = (D L ) 1 Uu ( k) + (D L ) 1 f Update at each point: u (k+1) i = 1 2 h 2 f i + u (k+1) (k) i 1 + u i+1 = u i (k) + 1 2r i (k) where r (k) i h h 2 f i + u (k+1) i 1 + u (k) (k) i+1 2u i : the residual evaluated after updating u i 1 d u i 1 (k+1) With relaxation parameter : u (k+1) i = u (k) i + 1 (k) 2 r i 10 12/02/09 MG01.prz
Point Jacobi for Model Problem Now consider the rate of convergence of the point Jacobi iteration. We take,, and. M = 2I N = M A = 2I A M 1 N = I 1 2 A The iteration matrix, G h M 1 N, has the same eigenvectors as A. For its eigenvalues we get G = 1 1 2 A. We have A,k = 4sin 2 (k 2n ) = 4sin 2 (k n ), and so G,k = 1 2sin 2 (k ) 2n = 1 2sin 2 (k n ) For we get k = 1 G,1 = 1 2sin 2 ( n ) l 1 2 n 2 = 1 2 2n 2 Likewise, we get G,n 1 = 1 2sin 2 ((n 1) n ) l 1 + 2 2n 2 Clearly, for, (and fairly rapidly so) n d (G) d 1 11 12/02/09 MG01.prz
Error Smoothing Convergence: e ( k) = G k e ( 0), where G is iteration matrix G = (I P 1 A). u ( k+1) = u ( k) + P 1 r ( k) = u ( k) + P 1 (f Au ( k) ) = (I P 1 A)u ( k) + P 1 f Jacobi iteration matrix: (1 )I + R J = (1 )I + D 1 (D A) This gives: I 2 A Eigenvalues : I 2 A (I 2A) = 1 2 (A) Eigenvalues of A 12 12/02/09 MG01.prz
Error Smoothing Eigenvalues of : A Assume eigenvector close to physical eigenvector: v k = sin kj n Apply pointwise rule for Jacobi: Av j,k = sin k (j 1) n + 2sin kj n sin k (j+1) n = 2sin kj n 2sin kj n (1 cos k n ) (so eigenvector indeed) 2sin kj n cos k n = 1 cos k n = 1 cos 2 k 2n = 1 (1 2sin 2 ( k 2n )) = 2sin 2 ( k 2n ) 2sin kj n (1 cos k n ) = sin kj n * 4sin 2 ( k 2n ) This gives for the weighted Jacobi method: I 2A (RJ, where ) = 1 2 sin 2 ( k 2n ) 0 < [ 1 Always converges. Poor convergence for which modes? 13 12/02/09 MG01.prz
Point Gauss-Seidel for Model Problem Next we consider the rate of convergence of the point Gauss-Seidel method (in natural ordering) for the model problem. We solve Ax = b. Let A = L + D + U, where D is diagonal of A, and L and U are strictly lower and upper triangular part of A. Take M = L + D and N = U. Iterate Mx k+1 = Nx k + b. Eigenvectors of M 1 N? M 1 Nw = w g Nw = Mw Model problem: with. 2 w j w j 1 = w j+1 w 0 = w n = 0 Recurrence w j+1 2 w j + w j 1 = 0. Substitute w j = z j ; this gives z 2 2 z + = 0 d z 1,2 =! 2. Real gives either strictly increasing or decreasing (or constant, i.e. zero). So we need z complex, and therefore 0 < < 1. 14 12/02/09 MG01.prz
Point Gauss-Seidel for Model Problem j j General solution w j = 1 z 1 + 2 z 2, where z 1 = + i 2 and z 2 = i 2 Note z 1 = 1/2 d z 1,2 = 1/2 1/2! i 1/2 2 = 1/2 e i. This gives 1/2 = cos w 0 = 1 + 2 = 0 d 2 = 1 w n = 1 cos n e in 1 cos n e in = 2i 1 cos n sin n = 0 So we must have = k = k n and we can take w k,j = cos k n sin jk n = cos k sin j k (why?) For the eigenvalue we therefore get k = cos 2 k for k = 1 n 1. This gives = 1 = cos 2 n = 1 sin 2 n l 1 2. n How does this relate to the convergence rate for 2 Jacobi iteration? 15 12/02/09 MG01.prz
Red-Black ordering for Gauss-Seidel Gauss-Seidel depends on ordering. Often-used special ordering: Red-Black ordering, where red unknowns reference only black unknowns and vice versa. In this case, order even grid points first and then odd grid points Red/Even update: where r (k) 2i h h 2 (k) f 2i + u 2i 1 u (k+1) 2i = 1 2 h2 (k) f 2i + u 2i 1 (k) + u 2i+1 (k) 2u 2i (k) + u 2i+1 With relaxation parameter : u (k+1) 2i = u (k) 2i + 1 2 r (k) 2i = u (k) 2i + 1 2 r (k) 2i Black/Odd update: u (k+1) 2i 1 = 1 2 h2 f 2i 1 + u (k+1) (k+1) 2i 2 + u 2i+2 where r (k+1) 2i 1 h h 2 f 2i 1 + u (k+1) 2i 2 + u (k+1) (k) 2i+2 2u 2i 1 (k) = u 2i 1 + 1 2 r (k+1) 2i 1 With relaxation parameter : u (k+1) (k) 2i 1 = u 2i 1 + 1 2 r (k+1) 2i 1 16 12/02/09 MG01.prz
Error Smoothing 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Initial Solution=-Error 0 10 20 30 40 50 60 70 80 90 100 DCT: [4.9, 0.27, 0.16, 0.061, -2.4,...], O(0.1) or O(0.2) 17 12/02/09 MG01.prz
Error Smoothing 0.8 Solution=-Error 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 90 100 After 20 Jacobi iterations with =1 18 12/02/09 MG01.prz
Error Smoothing 0.7 Solution=-Error 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 90 100 After 50 Jacobi iterations with =1 19 12/02/09 MG01.prz
Error Smoothing 0.45 Solution=-Error 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0 10 20 30 40 50 60 70 80 90 100 After 1000 Jacobi iterations with =1 DCT: [2.4,, -1.1,,...,, 0.35], =O(0.01), mainly O(1e-3) 20 12/02/09 MG01.prz
Error Smoothing 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 10 20 30 40 50 60 70 80 90 100 ω = 2/3 21 12/02/09 MG01.prz
Error Smoothing 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 10 20 30 40 50 60 70 80 90 100 22 12/02/09 MG01.prz
Error Smoothing 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 10 20 30 40 50 60 70 80 90 100 23 12/02/09 MG01.prz
Error Smoothing 0.45 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0 10 20 30 40 50 60 70 80 90 100 24 12/02/09 MG01.prz