An Introduction of Multigrid Methods for Large-Scale Computation

Transcription

1 An Introduction of Multigrid Methods for Large-Scale Computation Chin-Tien Wu National Center for Theoretical Sciences National Tsing-Hua University 01/4/005

2 How Large the Real Simulations Are? Large-scale Simulation of Polymer Electrolyte Fuel Cells by parallel Computing (Hua Meng and Chao-Yang Wang, 004) Three-Dimensional Finite Element Modeling of Human Ear for Sound Transmission (R. Z. Gan, B. Feng and Q. Sun, 004) FEM model with O(10 6 ) nodes FEM model with 10 5 ~10 6 nodes Car engine with O(10 5 ) nodes Commercial Aircraft: 10 7 nodes FINFET transistor: 10 5 nodes

3 What do we need in order to simulate? Our goal is to introduce multigrid methods for solving sparse linear systems. Why multigrid? 1. Computation cost of multigrid is proportional to problem sizes.. Multigrid is easy to be parallelized.

4 Outlines Stationary Iterative Methods Some finite element error estimates Multigrid Algebraic Multigrid Nonlinear Multigrid (FAS) Multigrid Parallelization Reference: 1. An introduction to multilevel methods (Jinchao Xu). Multigrid Methods (Stephen F. McCormick) 3. A multigrid tutorial (William L. Briggs) 4. Matrix iterative analysis (Richard S. Varga) 5. The mathematical theory of finite element methods (Brenner and Scott) 6. Introduction to Algebraic Multigrid (Christian Wagner)

5 Solving Linear System Ax=b by Iterative Methods Methods: Stationary Methods: Jacobi, Gauss Seidel (GS), SOR. Krylov Subspace Methods: Conjugate gradient, GMRES, by Saad and Schultz 1986, and MINRES, by Paige and Saunders Multigrid Methods: Geometric multigrid (MG), by Fedorenko 1961, and algebraic multigrid (AMG), by Ruge and Stüben 1985.

6 Basic questions and some definitions Basic questions are 1. How do we iterate?. For what category of matrices A, the iteration converge? 3. What is the convergence rate? Some definitions: A is irreducible if there is no permutation P such that P T AP= A 1,1 A 1, 0 A, A is non-negative (denoted as A 0) if a i,j 0, for all 1 i,j n A is an M-matrix if A is nonsingular, a i,j 0 for i j, and A -1 0 A is irreducibly diagonally dominant if A is irreducible, diagonally dominant with a i,i > a i,j for some i. n j=1,j i A=M-N is a regular splitting of A if M is nonsingular and M -1 0

7 1. r old = f Au old. Solve e=b -1 r old 3. update u new = u old + e Theorem: Let A 0 be an irreducible matrix. Then 1. A has a positive real eigenvalue equal to its spetral radius. There is an eigenvector x>0 corresponds to ρ( A) 3. ρ( A) increases when any entry of A increases. 4. ρ A is a simple eigenvalue of A. e new = e old B 1 ( f Au old ) = e old B 1 A( u u old ) = ( I - B -1 A)e old B is called an iterator or preconditioner of A. E B = I - B -1 A is called the error reduction operator of the iterator B Perron-Frobenius Theorem Stationary Iterative Methods

8 Some Well Known Iterative Methods Suppose A = D L U, Richardson: Jacobi: Damped Jacobi: Gauss-Seidel: B = 1, where 0<ω < ω ρ A B = D where D is the diagonal, L and U are lower and upper triangular parts, respectively.. B = 1 ω D, where 0<ω < ρ D -1 A B = ( D L). SOR: B = 1 ( ω D ω L ), where 0<ω <.

9 Jacobi and Gauss-Seidel Jacobi: x i (m+1) = n j =1 j i a i, j a i,i x (m) j + r i a i,i Gauss-Seidel: x i (m+1) = i 1 j =1 a i, j a i,i x j (m+1) n j =i+1 a i, j a i,i x j (m) + r i a i,i HW1: Write down a formula for SOR HW: x x 1 Write a program to solve 1 1 = x x Jacobi and Gauss-Seidel, starting with initial x (0) = [ 0,0,0,0]. by

10 and E GS = I D L Let E J = I D 1 A ( 1 A). Since the solution of HW is x= [ 1,1,1,1] and e 0 = x x (0) = [1,1,1,1]. Clearly, we have e m J = ( E J ) m e 0 and e m GS = ( E GS ) m e 0, One can easily check that 1 e m J = 1 1 and e m m 1 GS = 1. Thus, e m 4 m 1 J = 1 > e m m 1 GS = m 1 1 You might get a feeling that Gauss-Seidel method is faster than Jacobi method. Stein-Rosenberg Theorem Theorem: Let B J = L + U be the Jacobi matrix and B GS = I L 1 U be the Gauss- Seidel matrix. Then one and only one of the following relations is vaild: 1) ρ B J = ρ( B GS ) = 0. < ρ( B J ) < 1. = ρ( B GS ) = 1. < ρ( B GS ). ) 0 < ρ B GS 3) ρ B J 4) 1 < ρ B J

11 Convergence of Jacobi, Gauss-Seidel and SOR Iterative Methods Lemma 1. If A= a i,j n 0 is irreducible then either a i,j min 1 i n n j=1 a i, j < ρ A < max 1 i n n j=1 a i, j j=1 =ρ( A) or (1). Proof: Case(1): All row sums of A are equal (=σ ): Let ζ = [ 1,1,,1]. Clearly, Aζ =σζ and σ ρ( A). However, the Gerchgorin's Theorem implies ρ A Case(): Not all row sums of A are equal: Construct B= b i,j respectively, such that n j=1 σ. Hence, ρ( A) = σ. 0 and C= ( c i,j ) 0,by decreasing and increasing some entries of A, b, j = α = min 1 i n n j=1 a i, j n and c, j = β = By Perron-Frobenius theorem, we have ρ B j=1 max 1 i n n j=1. ρ( A) ρ C Clearly, from the result of Case(1), the inequality (1) holds. a i, j, for all 1 n. Lemma. Let A and B be two matrices with 0 B A. Then ρ( B) ρ( A)

12 Theorem 1. Let A= ( a ) i,j be a strictly or irreducibly diagonally dominant matrix then the Jacobi and Gauss-Seidel iterative methods converge. 0 i = j Proof: Recall that E J = I-D -1 A = D -1 ( L + U ) = ( b i, j ), where b i,j = a i, j. From Lemma, i j a i,i it is clear that ρ( B) ρ( B ). Since A is strictly diagonally dominant, clearly, we have n b i, j < 1 for all 1 i n. Therefore, Lemma 1 implies ρ B j=1 we have shown the Jacobi iterative method converge from ρ B Now, since E GS =I-( D-L) -1 A= ( D-L) -1 U= I-D -1 L < 1. As a result, ρ B <1. -1 D -1 U. Let L=D -1 L and U=D -1 U. We have ( I-L ) 1 U ( I-L ) 1 U ( n 1 I+ L + L + + L ) U = I- L Now consider B J = L + U and B GS = ρ( B J ) < 1, the Stein-Rosenberg theorem implies ρ( B GS ) < 1. 1 U. ( I- L ) 1 U. Since we have already shown Therefore, we conclude the Gauss-Seidel iterative method converges.

13 Theorem. Let A=D-E-E * and D be Hermitian matrices, where D is positive definite, and D-ωE is non-singular for 0 ω. Let E SOR = I ω ( D ωe) 1 A. Then ρ E SOR positive definite and 0<ω <. < 1 if only if A is Proof: First, assume e 0 is a nonzero vector, the SOR iteration can be written as ( D ωe)e m+1 = ( ωe * + ( 1 ω ) D)e m, m () Let δ m = e m e m+1. Substracting ( D ωe)e m and ( ωe * + ( 1 ω ) D)e m+1 from both side of (), we have ( D ωe)δ m = ω Ae m ---- (3) and ω Ae m+1 = ( 1 ω ) D + ωe * δ m (4). From e * * m (3) e m+1 (4) and "simplifying the expression" (HW), one has ( -ω )δ * m Dδ m =ω e * * m Ae m e m+1 Ae m+1 { } (5). Assume A is positive definite and 0<ω < and let e 0 be any eigenvector of E SOR. We have e 1 =λe 0 and δ 0 = ( 1-λ)e 0 and (5) reduces to -ω ω 1 λ e * 0 De 0 = ( 1 λ )e * 0 Ae (6). Now, λ 1. Otherwise, δ 0 = 0 Ae 0 = 0 (by (3)) e 0 = 0 contradiction! Since A and D are positive definite and 0<ω <, (6) implies 1 λ > 0. Therefore, ρ E SOR Using similar arguments, one can show that the converse is also true. < 1.

14 For SOR, it is possible to determined the optimal value of ω for special type of matrices (p-cyclic). The optimal value ω b is precisely specified as the unique positive root (0<p/(p-1)) of the equation For p=, ρ( E J )ω b 1 p p = p p p 1 ρ E ω b = 1 + J ρ E J ( ω 1 b ), ( Varga 1959). ( Young 1950) Semi-Iterative Method: y m = v j ( m) x j where v j ( m) =1. We have e m = v j ( m)e m j =0 m j =0 m. In general, e m = P m ( E S )e 0 where P m is a polymonial and E S is the error reduction of an iterater S This is so-called polynomial acceleration method. The most important one is the Chebyshev polynomials. m j =0

15 Algorithm: y m+1 = ω m+1 Chebyshev Semi-Iterative Method { E S y m + f y m 1 } + y m 1, for m 1, where C m+1 ( 1 / ρ), C and C m-1 m+1 and y 0 = x 0. ω m+1 = 1 + C m 1 1 / ρ ρ = ρ E S Convergence: m / m e m ω b 1 1+ ω b 1 e 0, here ω b = ρ The convergence rate is accelerated as ρ 0. are Chebyshev polynomials, C 0 = 1, C 1 = x, C m+1 = xc m C m 1 Remark: There are cases that the polynomial acceleration does not improve asymptotic rate of convergence.

16 Some PDE and Finite Element Analysis Finite Element Solutions: Let I h be a given triangulation, V h = { v H 1 0 : v T P 1 ( T ), T I } h and π h :H 1 V h be the interpolation defined by π h ( u) ( N i ) = u( N i ). Consider the weak solution u H 1 satisfying a( u,v)= ( f,v), for all v H 1 0, a finite element solution u h V h satisfies a( u,v)= ( f,v), for all v V h. Interpolation Errors: H -Regularity: v-π h v 1 Ch r u r +1 and v-π h v 0 Ch r +1 u r +1. a, is said to be H -Regular if there exists a constant C such that for all f L u C f 0

17 Finite Element Solution is Quasi-Optimal Céa Theorem: u u h H 1 C α min v V h u v H 1 where, C is the continuity constant and α is the coercivity constant of a,. Proof: Step1: Step: HW4: = 0, for all v V h a( u u h,u u h ) = a( u u h,u v) + a( u u h,v u h ) =a( u u h,u v) C u u h u v H 1 H 1 a u u h,v α u u h H 1 If a(, ) is self-adjoint, show that u u = min h u v A v V A, h where v A = a( v,v) is the energy norm. Remark: finite element solution is the orthogonal projection of the exact solution with respect to the energy norm.

18 FEM Error Estimation Theorem 3: Assume the interpolation error estimations holds for the given I h and a, has the H Regularity. The following estimates hold. u u h H 1 Ch r u r (i) and u u h L Ch r +1 u r (ii) Proof: From the interpolation estimation and Céa Theorem, (i) is trivial. To prove (ii), we use the duality argument. Let w be the solution to the adjoint problem, a( v,w)= ( u-u h,v), for all v H 1. Choosing v=u-u h, we have ( u-u h,u-u h )= a( u-u h,w) = a( u-u h,w w h ), for any w h V h C u u h H 1 w w h H 1 Ch u u h H 1 w Ch u u h H 1 u u h L. Therefore, u u h L Ch r +1 u r +1.

19 Definition: mesh dependent norm v k,h = ( A k h v,v) h for v V h, k=0,1, where ( v,w) h = h v( N i ),w( N i ). Clearly, v 0,h v 0 and v 1,h v A. Lemma 3: Λ( A h ) Ch Proof: Let λ be an eigenvalue of A h with eigenvector φ. a( φ,φ)= ( A h φ,φ) h = λ( φ,φ) h = λ φ 0,h. λ C φ A Ch φ 0 Ch. φ 0,h φ 0,h Lemma 4: (Generalized Cauchy-Schwarz Inequality) a v,w v 1+t,h w 1 t,h v,w V h and t R.

20 Ideas: Multigrid Methods Approximate solutions on fine grid using iterative methods. Correct remaining errors from coarse grids. k-1ki Mesh Refinement Prolongation Restriction kk-1i Prolongation Restriction MG Coarsening MG V-cycle

21 Why Multigrid Works? 1. Relaxation methods converge slowly but smooth the error quickly. Ex1: consider Lu = u'' = λu finite difference -u j-1 + u j u j+1 h = λu j Eigenvalues λ k = 4 kπ h sin (N + 1) and eigenvectors φ kjπ k = sin j N + 1 here, k=1 N is the wave number and j is the node number. Richardson relaxation: E R = (I σ 1 A) where A h = 1 tridiag[-1-1]. h Fourier analysis: Choosing σ = 4 (largest eigenvalue). h e m = N k =1 1 λ k σ m N kπ N φ k = 1 sin (N + 1) φ k = α m k φ k k =1 m k =1, after m relaxation. α k m 0 more quickly for k close to N.

22 Damped Jacobi e = sin jπ N jπ sin N jπ sin N + 1 e m = ( I ω D 1 A) m e. Smooth error modes are more oscillatory on coarse grids. Smooth errors can be better corrected by relaxation on coarser grids. Smooth error on fine grid Smooth error on coarse grid Relaxation convergence rate on fine grid is 1-O(h ) Relaxation convergence rate on coarse grid: 1-O(4h ) Remember: α 1 1 π N + 1 = 1 O( h ) for N 1

23 3. The smooth error is corrected by coarse grid correction operator: E c = ( I I h H A 1 H I H h A ) h = ( A 1 h I h H A 1 H H I ) h A h, here I h H and I H h are called restriction and prolongation operator respectively. HhI hhi A H can be obtained from discretization on coarse grid A H = I H h A h I h H and I H h 0chHcHhhEIIAE == h =c I H T (Galerkin formulation) E c is an A-orthogonal projection A h E c e, I h H e = 0 N ( E c ) h = R( I ) H R( E c ) = N ( I H h A ) h and E c is identity on N ( I H h A ) h

24 H A Picture That Show How Multigrid Works! N ( I H h A h ) o relaxation L h R( I H ) H N ( I H h A h ) correction o L R I h ( H ) H N ( I H h A h ) H N ( I H h A h ) o relaxation L correction L h R( I H ) o h R( I H )

25 Consider I h H = [ 1,1, 1 ( ]T linear interpolation) and I H h = 1 [1,1, 1 ]. It is easy to check that A H =I h H A h I H h is the discretization of L on I H. Now, for any v V h, let f v = A h v. One can consider v and v H = I h H A 1 H I H h f v as finite element approximations of ˆv, the solution of a( ˆv,w)= ( f v,w). Then, from the FEM-error estimation and H -regularity, we have E c ( v) = ( A 1 h I h H A 1 H H I ) h A h v k = ˆv v H ( ˆv v) k k ( ) Ch k ˆv Ch k f v 0 = Ch k Av 0 Consider the eigenfunction φ k j, k N. φ k j is also an eigenfunction of A H We have E c ( k φ ) j Chλ k = O h 1. This concludes the coarse-grid correction fixes the low frequency errors. For k N, E c ( k φ ) j 4 C 1 h, the high frequency errors can be amplified by coarse-grid correction.

26 Multigrid (MG) Algorithm: 1. x k =w k. (pre-smoothing) x k =w k +M k -1 g k -A k x k 3. (restriction) g =I k-1 k g k k -A k x k 4. (correction) q i =MG k-1 (q i-1,g k ) for 1 i m, m=1 or and q 0 =0 5. (prolongation) q =I k m q k-1 m 6. set x k =x k +q m 7. (post-smoothing) x k =x k +M k -1 g k -A k x k 8. set MG k (w k,g k )=x k MG Error reduction operator: Multigrid Algorithm E mg = ( A 1 h I h H A 1 H H I )( h A h E s ) = E s ( I I h H A 1 H I H h A ) h Pre-smoothing only Post-smoothing only

27 Multigrid Cycles Multigrid V-cycle (m=1) Multigrid V-cycle (m=) Full Multigrid cycle (FMG)

28 Results provided by in NCTU

29 MG Convergence Smoothing property: A l E s η( m) A l, for all 0 m < and l > 0. Approximation property: A 1 l I H h A 1 l 1 I H 1 h C A A l, for all l > 0. Ideas for proving the approximation property is shown in P.5 (*) Proof of smoothing property: Consider E S =E R = I- 1 Λ A h. Let v V h and v m S = E m S ( v). From Fourier expansion v= v k φ k A h E R ( v) 0 Since v s A h E R, we have v m S = 1 1 Λ A m v = λ = v s 1 k Λ λ k v k =Λ 1 λ k Λ Λ sup 0 x 1 0 v 0 m {( 1 x) m x} v s A h v s m 1 λ k Λ v k φ k. Therefore, Ch - 1 for Richarson iteration, clearly, m m v s 0 λ k Λ v s λ k v k ( v) 0 1 m A h v 0 A h E R η( m) A h, η( m) 0 as m. HW5: Prove MG with Richardson smoother is convergent in -norm 1

30 Choices of Interpolations and Coarse Grids Linear interpolation: p = r = Operator-dependent interpolation: De Zeeuw 1990 The rule in chosing interpolation and restriction: m p + m r > m (Brandt 1977) where m is the order of the PDE, m p -1 is the degree of polynomials exactly interpolated by I H h and m r -1 is the degree of ploynomials exactly interpolated by ( H I ) T h. Coarse grid selection: regular coarsening, semi-coarsening, algebraic coarsening

31 Choices of Smoothers Stationary iterative methods : Jacobi, Gauss-Seidel, SOR, Block-type stationary iterative methods (blocks can be determined by the way we number the nodes) Vertical line ordering Horizontal line ordering Red-Black ordering T -I -I T -I A= -I T -I -I T, T=[-1,4,-1]: Matrix of -D Laplacian Matrix Pattern for line ordering Matrix Pattern for R-B ordering

32 HW6: Write a MG code for solving 1-D using linear interpolant I H h u = f = u( 1) = 0 u 0 Update even (Red) points first: x i = 0.5 f i + x i 1 + x i+1 Update odd (Black) points: x i = 0.5 f i+1 + x i + x i+ and I h H = 0.5 I H h Red-Black Gauss-Seidel smoother T and the Brandt s Local Mode Analysis For analyzing the robustness of a smoother. Brandt s local mode analysis is a useful tool. Here, we demonstrate the method by considering the Jacobi and Gauss-Seidel relaxation for the -D laplace equation with periodic boundary condition.

33 Brandt s smoothing factor Let ε be the error before relaxation. From discrete Fourier transform theory, ε can be written as ε i, j = ˆε θ φ i, j ( θ) (i), where θ= ( θ 1,θ ), φ i, j θ ˆε θ = Θ n = θ Θ n 1 ε ( n + 1) k,l φ k,l ( θ ), and 1 k,l n π n + 1 k,l n + 1 k,l n + 3, n is odd. = e i iθ 1 + jθ, Similarly, the error ε obtained after relaxation can be written as ε = ˆε θ φ i, j θ -----(ii). Let λ θ θ Θ n factor is defined as ρ=sup λ θ ˆε θ. Brandt's smoothing ε θ, π θ k π, k = 1,.

34 ( ) Recall that ε i, j = ε i, j ω 4 4ε ε + ε + ε + ε i, j i+1, j i 1, j i, j +1 i, j 1 into it, we have ˆε θ φ i, j ( θ) = { ˆε θ φ i, j ( θ) ω 4 4 ˆε θ φ i, j ( θ) ( ˆε θ φ i+1, j ( θ) + ˆε θ φ i 1, j θ θ Θ n θ Θ n + ˆε θ φ i+1, j ( θ)) }= ˆε θ φ i, j θ φ i, j that ρ = max 1 ω, 1 ω, 1 3ω minimize ρ is 4 5 Smoothing Factor of Damped Jacobi Iteration θ Θ n ω 4 4φ i, j ( θ) φ i, j θ ( θ)e -iθ }= ˆε θ 1 ω 1 cos θ 1 θ Θ n Therefore, λ( θ) = 1 ω 1 cos θ 1 + cos( θ ) + cos( θ ) φ i, j ( θ). It is easy to see. The optimal ω that and the smoothing factor ρ = 0.6 for such ω. Plug (i) and (ii) + ˆε θ φ i, j +1 ( θ) e iθ 1 φ i, j ( θ)e -iθ 1 φ i, j ( θ)e iθ HW7: Show that the smoothing factor of the Gauss-Seidel iteration is 0.5

35 Convergence: How Much Multigrid Costs? Stationary method 1-O(κ -1 ) 1-h Conjugate gradient 1-O(κ -1/ ) 1-h Multigrid O(1) independent with h How much each MG step cost? Ignore the cost associated with inter-grid transfer (typically within 10-0%). Computation cost of one MG V-cycle is cn d ( 1 + d + d + ) = cnd 1 d n d : total number of points d: dimension of the problem c: cost for updating a single unknown cn d : cost per relaxation sweep.

36 Standard MG can fail! The original PDE has poor coercivity or regularity (for example, crack problems, convection-diffusion problems, etc.) - Relaxation may not smooth the error. - coarse grid correction can only capture a small portion of the error or even worse! The left figure is a sketch to illustrate why MG slow convergence Next, let s consider the following example: H N I h H A h o o relaxation o o correction L h R( I H ) d dx c( x) du ε, 0 x i dx = f ( x) 0 h Ex:, here c( x) = 1, i 0 h < x i 1 h ---- (+) u( 0) = u( 1) = 0 ε, i 1 h < x 1

37 Discrete matrix of (+) Damped Jacobi E DJ matrix of A h ε ε ε ε ε ε 1+ ε A h = 1 1+ ε ε ε ε ε ε ε 1 1 / 1 / 1 1 / ε / (1+ ε) 1 1 / (1+ ε) 1 / 1 1 / I ω D 1 A h = I ω 1 / (1 + ε) 1 ε / (1 + ε) 1 / 1 1 / 1 / 1

38 For ε 0, the eigen vector corresponding to the largest eigenvalue λ ( 0) of E DJ converges toward to the vector e ( 0) while λ ( 0) 1, where ( e ( 0) ) = i ih, i 0 h, i 0 i 1 n 1 ih i 0 n x i 0 h i 0 h < x i 1 h i 1 n 1 h, i 1 h < x 1. + i 0 h + i 1 h Figure of e (0) Damped Jacobi fails to smooth the high frequency error! MG convergence is deteriorated as ε 0 A remedy of this is to use operator-dependent interpolation! Construct such interpolation is not easy. But, there is a easier and better way to do it!

39 Algebraic Multigrid MG 1. A priori generated coarse grids are needed. Coarse grids need to be generated based on geometric information of the domain. AMG 1. A priori generated coarse grids are not needed! Coarse grids are generated by algebraic coarsening from matrix on fine grid.. Interpolation operators are defined independent with coarsening process. 3. Smoother is not always fixed.. Interpolation operators are defined dynamically in coarsening process. 3. Smoother is fixed. Ideas: Fix the smoothing operator. Carefully select coarse grids and define interpolation weights

40 AMG Convergence Smoothing assumption: α>0 E s e 1 e 1 α e for all e Vh Approximation assumption: min e I h e H e H β e H 0 1 where β is independent with e. E c e = ( AE c e, E c e I h 1 H e ) H E c e E c e I h H e H β E c e E c e 0 1 E s E c E c e α E c α e β E c α e 1 1 β e 1,here v 0 = Dv,v, v 1 = Av,v, v = D -1 Av,Av, for v V h AMG works when A is a symmetric positive definite M-matrix. In the following, we assume that A is also weakly diagonally dominate

41 What Does the Smooth Assumption Tell? Smooth error is characterized by E s e s 1 e s 1, e s is very small e 1 D 1/ Ae D 1/ e = e e 0 e 1 << e 0 ( Ae,e) = 1 1 a i, j a i,i a i, j (e i e j ) + a i, j e << a i i,i e i i, j ( e i e j ) << a ii e i j a i, j ( e i e j ) j i e i Smoother errors vary slowly in the direction of strong connection, from e i to e j, where a i, j a i,i are large. AMG coarsening should be done in the direction of the strong connections. In the coarsening process, interpolation weights are computed so that the approximation assumption is satisfied. (detail see Ruge and Stüben 1985) i j i

42 What Does the Approximation Assumption Tell? Approximation assumption min e I h e H e H β e H 0 1 a ii e i w ik e k k C β 1 ( a i F ij )(e i e j ) + i, j Since a ii e i w ik e k i F k C i = a ii w ik (e i e k ) + (1 s i )e i i F k C j a ij a ii w ik (e i e k ) + (1 s i )e i, i F k C e i here w ik > 0 is the interpolation weight from node k to node i, and s i = clearly, if a ii w ik (e i e k ) β ( a i F k C ij )(e i e j ) i, j ( Θ) a ii (1 s i )e i β a ij i F j e, i i the approximation assumption holds. For Θ ( Ξ) 0 a ii w ik β a ik and 0 a ii ( 1 s i ) β a ik w ik < 1, k C to hold, we can simply require. k

43 Lemma 5: Given a β 1, suppose the coarse grid C is selected such that a i,i + a i, j = j C i j i j C i Then, the approximation assumption holds if the interpolation weights are defined as w i,k = a i,k a i, j 1 β a i,i where C i =N i C, C is the coarse grid and N i = neighbors of i-th node a i,i ω i,k = a i,i a i,i j C i a i, j ( Φ). a i,k a = a i,i i, j a a i,k β a i,k i, j j C i j C i ( 1 s i ) = a i,i 1 ω i,k k C i = a i,i 1 β j a i, j k C i j C i a i,k a i, j a i, j =a j i,i Therefore, Ξ holds. From the arguments in previous page, We can conclude the approximation holds. j C i a i, j

44 Smoothing property holds for GS Recall E S = I B 1 A. We have E GS e 1 = A( I B 1 A)e, ( I B 1 A)e AB 1 Ae,e = Ae,e ( Ae, B 1 Ae) + ( AB 1 Ae, B 1 Ae) ( BB 1 Ae, B 1 Ae) + ( AB 1 Ae, B 1 Ae) B 1 Ae, B 1 Ae = e 1 B 1 Ae, BB 1 Ae = e 1 ( B T + B A ). The smooth assumption α e (( B T + B A) B 1 Ae, B 1 Ae) ---- (Θ) Let e=b 1 Ae. Since e = ( D 1 Ae, Ae) = ( D 1 BB 1 Ae, BB 1 Ae), clearly, (Θ) α D 1 B e, B e ---- (ΘΘ). ( B T + B A) e, e Now consider B=D-L. Since B T + B A = D, we have (ΘΘ) α BT D 1 B e, e D e, e = α D 1 B T D 1 BD e, e D 1/ e, D 1/ e = αρ( D 1 B T D 1 B) 1. α 1 ρ D 1 B T D 1 B = α D 1 B T D 1 BD 1/ e, D 1/ e D 1/ e, D 1/ e ---- (ΘΘΘ).

45 Therefore, the smooth assumption holds for Gauss-Seidel iteration. If A is a diagonally dominant M-matrices, we can estimate α as follows: Since ρ( D 1 B T D 1 B) ρ( I D 1 L T )ρ( I D 1 L) ( 1 + ρ( D 1 L T ))( 1 + ρ( D 1 L) ), and ρ( D 1 L) max 1 i n n j =1, j i a i, j a i,i 1, clearly, we have ρ D 1 B T D 1 B Therefore, Gauss-Seidel iteration satisfies the smoothing property with α= 1 4. h A H = ( I H ) T h A h I H A h ( Ξ) and ( Φ).

46 AMG Coarsening Criteria First, let s define the following sets: N S { i = j : a i, j γ max( } a i,m ),0 < γ < 1 ( S N ) T i = { S j : i N } j m i S ( N i ) T N i S Here, S N i S ( N i ) T is the set of nodes that node i strongly connects to. is the set of nodes strongly connects to node i. C i -nodes should be chosen from N i S From convergence result, we want β close to 1. This suggests we need a larger set C i ( we need to choose a small γ ). We don t want C=ø but we want C as small as possible. Criterion 1. For each node i in F, node j in N i s should be either in C or strongly connected to at least one node in C i. Criterion. C should be a maximal subset of all nodes with the property that no two C-nodes are strongly connected to each other.

47 AMG Coarsening (I) +1-1 N S i -1 S ( N i ) T Fine points +-1

48 AMG Coarsening (II)

49 AMG Coarsening: Example 1 Laplace operator from Galerkin FEM Discretization: A very good MG and AMG tutorial resource (by Van Emden Henson) :

50 AMG Coarsening: Example Convection-Diffusion with Characteristic and downstream layers εδu + u y = if (y=0 x>0) or x=1, u Ω = 0 otherwise, where Ω=[-1,1] [-1,1] Solution from Galerkin discretization on 3x3 grid Solution from SDFEM discretization on 3x3 grid

51 SDFEM discretization with δ T = h yields the left matrix stencil: ε ε ε h 6 ε h ε h 3 6 ε 3 h 6 ε 3 h 3 ε 3 h 6 ε 3 AMG coarsening with strong connection parameter ε/h << β << 0.5 C-point C-point Coarse grids from GMG coarsening Coarse grids from AMG coarsening

52 Example : GMG v.s. AMG level GMG AMG level GMG AMG level GMG AMG On the uniform mesh: ε=10 - ε=10-3 ε=10-4 Level GMG AMG Level GMG AMG Level GMG AMG On the adaptive mesh: ε=10 - ε=10-3 ε=10-4

53 Nonlinear Multigrid Nonlinear problems: L( u) = f discretization One needs to solve A h ( u h ) = f h a 1 a a n ( u 1,u,,u n ) u 1,u,,u n u 1,u,,u n = f 1 f f n. u j u j D Du A u h 1 ( f A ( h u )) j

54 Nonlinear Relaxation: ( old old,,u ) n for all i=1,,,n Jacobi: a i u old 1,,u old i 1,u new i,u i+1 Gauss-Seidel: a i ( old old,,u ) n for all i=1,,,n u new 1,,u new i 1,u new i,u i+1 Solve local nonlinear problems iteratively. Example ( Nonlinear Gauss-Seidel ) : Discretiation: Newton iteration for each j, starting from j=1

55 Nonlinear defect correction: In linear case: r h (n) = A h u h A h u h (n) In nonlinear case: r (n) (n) h = A h ( u h ) A h u h (n) = A h ( u h - u h ) (n) A h ( u - u h ) Solving A H e H = I h H r h (n) does not give an approximation to e h = u h - u h (n). Now consider e h = u h - u (n) h where u h satisfies r (n) h = A h e H = u H - I H h u (n) h where u H satisfies I H h r (n) h = A H Observe that u h (n) u h u H I h H u h e H I h H e h. (Here, I h H In this point of view, e H is a reasonable approximation of e h. Now, we can write down the FAS algorithm: ( u h ) A h (n) u h A H I H (n) h u h u H can simply be an injection) FAS: 1. Nonlinear Relaxation. Restrict u n h and r n h by r H = I H h r n h and v = I H n h u h = r H + A H v 3. Solve A H u H 4. Compute e H = u H v 5. Update u h n u h n + I H h e H

56 - u(x, y) + γ u(x, y) e u(x,y) = f(x, y) in [0,1] [0,1], u(x, y) = ( x-x )sin 3π y Discretization: finite difference Interpolation and Restriction Relaxation: Nonlinear Gauss-Seidel: p = r = ( u i, j = u i, j h 4u i, j u i 1, j u i+1, j u i, j 1 u ) i, j +1 + u i, j e u i, j f i, j 4h + γ ( 1+ u i, j )e u i, j starting from ( i,j) = ( 1,1), ( 1, ),,(,1), (,),, ( n,n).

57 An example taking from Multigrid Tutorial (Briggs) Who is better Newton-MG or FAS? Not so sure but FAS is popular in CFD.

58 Multigrid Parallelization Parallelization: Using multiple computers to do the job! communication What need to be done? P P P P M M M M Multigrid is a scalable algorithm! (Jim E. Jones, CASC, Lawrence Livermore National Laboratory)

59 Domain Decomposition: D G D B FEM assembling in domains D G, D B, can be done simultaneously. Matrix-vector product A x can be computed indenpendently in each domain. Pass x G to D and x B to D 1. Jacobi and red-black Gauss-Seidel Relaxations can be done in parallel. Grid-Coarsening and refinement can be done in parallel (not quite easy need to keep tracking grid topology). Interpolations can be parallelized too.

60 Scalability T( N,P) : Time to solve a problems with N unknowns on P processors Speedup S(N,P)=T(N,1)/T(N,P). Perfect if S(N,P)=P; Scaled Efficiency: E(N,P)=T(N,1)/T(NP,P). Perfect if E(N,P)=1. Assume D problem of size (pn) is distributed to p processors. Number of unknowns in each processor N Time for relaxation on grid level k: T k = T comm 5 point stencils N + 5 Time for a V-cycle = T v T k 8αL + 16Nβ α = startup time, β = time to transfer a single double k k N k N f, T comm (n) = α + βn = communication time for transmitting n doubles to one processor. f = one floating point operation time. f,

61 Since MG has O(1) convergence rate, we can analyze the scaled efficiency as follows: 40 3 T v N,1 N f T v (( pn ), p) = 8α log ( pn ) + 16β ( N ) N f O 1 log p O( 1) as N E N,P E N,P as p We need to be careful. In IBM SP, α = β = f = Communication is expensive!