Multilevel spectral coarse space methods in FreeFem++ on parallel architectures

Transcription

1 Multilevel spectral coarse space methods in FreeFem++ on parallel architectures Pierre Jolivet Laboratoire Jacques-Louis Lions Laboratoire Jean Kuntzmann DD 21, Rennes. June 29, 2012 In collaboration with F. Nataf, N. Spillane, V. Dolean, F. Hecht, C. Prud homme.

2 Outline 1 Introduction Context Problem setting 2 Spectral coarse space A quick overview Construction of a suitable deflation matrix 3 Implementation and numerical results Implementation framework Computing resources Scalability tests 4 Conclusion 2 / 22 Pierre Jolivet DD 21

3 Context Solve large systems arising from the finite element method. 3 / 22 Pierre Jolivet DD 21

4 Context Solve large systems arising from the finite element method. What are the different alternatives? parallel direct solvers (MUMPS, SuperLU, PaStiX..), parallel iterative solvers (HIPS, Hypre..), hybrid solvers (BoomerAMG, ML, PARDISO..),... 3 / 22 Pierre Jolivet DD 21

5 Context Solve large systems arising from the finite element method. What are the different alternatives? parallel direct solvers (MUMPS, SuperLU, PaStiX..), parallel iterative solvers (HIPS, Hypre..), hybrid solvers (BoomerAMG, ML, PARDISO..),... = high-performance algorithms on massively parallel distributed memory multiprocessor architectures. Constraints: robust in size and heterogeneities, easy to maintain. 3 / 22 Pierre Jolivet DD 21

6 Boundary value problems Given a FE space V and f V, find u V such that: a(u, v) = f, v v V = Au = F where a Ω (u, v) = a Ω (u, v) = Ω Ω κ u v C : ε(u) : ε(v) = A is assumed to be SPD 4 / 22 Pierre Jolivet DD 21

7 Notation Ω is split into N overlapping subdomains {Ω i } N i=1 {V i } N i=1. Let: R i be the restriction from V to Vi (thus Ri T is the extension by 0 from V i to V ), A ij := R i ARj T, O i the set of neighboring subdomains (and O i = O i {i}), {χ i } N i=1 be a partition of unity: supp(χ i ) Ω i and N χ i = 1, i=1 {D i } N i=1 be the discretization of this partition: N Ri T D i R i = I. i=1 5 / 22 Pierre Jolivet DD 21

8 One-level Schwarz method Two classical preconditioners are: N MRAS 1 = Ri T i=1 N MASM 1 = Ri T i=1 D i A 1 ii R i A 1 ii R i Some properties obviously parallel, not scalable. 6 / 22 Pierre Jolivet DD 21

9 Outline 1 Introduction 2 Spectral coarse space A quick overview Construction of a suitable deflation matrix 3 Implementation and numerical results 4 Conclusion 7 / 22 Pierre Jolivet DD 21

10 Principle Let Z be a rectangular matrix so that the bad eigenvectors of M 1 A belong to the space spanned by its columns. Define: E := Z T AZ P := I AQ Q := ZE 1 Z T = the number of columns of Z is O(N). 8 / 22 Pierre Jolivet DD 21

11 Principle Let Z be a rectangular matrix so that the bad eigenvectors of M 1 A belong to the space spanned by its columns. Define: E := Z T AZ P := I AQ Q := ZE 1 Z T = the number of columns of Z is O(N). Two-level preconditioners P BNN := (I AQ) T M 1 (I AQ) + Q P A-DEF1 := M 1 (I AQ) + Q P A-DEF2 := (I AQ) T M 1 + Q Differences and similarities studied in (Tang et al. 2009). 8 / 22 Pierre Jolivet DD 21

12 Principle Let Z be a rectangular matrix so that the bad eigenvectors of M 1 A belong to the space spanned by its columns. Define: E := Z T AZ P := I AQ Q := ZE 1 Z T = the number of columns of Z is O(N). Two-level preconditioners P BNN := (I AQ) T M 1 (I AQ) + Q P A-DEF1 := M 1 (I AQ) + Q P A-DEF2 := (I AQ) T M 1 + Q Differences and similarities studied in (Tang et al. 2009). The real problem is: how to build Z? 8 / 22 Pierre Jolivet DD 21

13 A priori construction Based on an analysis of the underlying PDE, (Nataf et al. 2011; Spillane et al. 2011) M13P1. Generalized eigenvalue problem on the overlap On each Ω i, find the eigenpairs (Λ ik, λ ik ) k such that: a Ωi (Λ ik, v i ) = λ ik a Ω i (χ i Λ ik, χ i v i ) v i V i which after discretization yields: A i Λ ik = λ ik D i A i D i Λ ik 9 / 22 Pierre Jolivet DD 21

14 A threshold criterion selects only ν i eigenvectors associated to low frequency eigenvalues: W W Z = W N where W i = [ D i Λ i1 D i Λ i2 D i Λ iνi ] 10 / 22 Pierre Jolivet DD 21

15 Outline 1 Introduction 2 Spectral coarse space 3 Implementation and numerical results Implementation framework Computing resources Scalability tests 4 Conclusion 11 / 22 Pierre Jolivet DD 21

16 Domain Specific Language FreeFem++, with the help of: Metis (Karypis and Kumar 1998) SCOTCH (Chevalier and Pellegrini 2008) UMFPACK (Davis 2004) ARPACK (Lehoucq et al. 1998) SLEPc (Hernandez et al. 2005) BLOPEX (Knyazev et al. 2007) MPI (Snir et al. 1995) / 22 Pierre Jolivet DD 21

17 Domain Specific Language FreeFem++, with the help of: Metis (Karypis and Kumar 1998) SCOTCH (Chevalier and Pellegrini 2008) UMFPACK (Davis 2004) ARPACK (Lehoucq et al. 1998) SLEPc (Hernandez et al. 2005) BLOPEX (Knyazev et al. 2007) MPI (Snir et al. 1995)... Why use a DS(E)L instead of C/C++/Fortran/..? performances close to low-level language implementation, hard to beat something as simple as: varf a(u, v) = int3d(mesh)([dx(u), dy(u), dz(u)] * [dx(v), dy(v), dz(v)]) + int3d(mesh)(f * v) + on(boundary_mesh)(u = 0) 12 / 22 Pierre Jolivet DD 21

18 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k Cannot be done with FreeFem++. k O j 13 / 22 Pierre Jolivet DD 21

19 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap A jk D k R k u = R j R T k A kk D k R k u 13 / 22 Pierre Jolivet DD 21

20 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap A jk D k R k u = R j R T k A kk D k R k u 13 / 22 Pierre Jolivet DD 21

21 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap What does it mean? A jk D k R k u = R j R T k A kk D k R k u Au = black box plugged with a Krylov method 13 / 22 Pierre Jolivet DD 21

22 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap What does it mean? A jk D k R k u = R j R T k A kk D k R k u Au = N j=1 AR T j D j R j u 13 / 22 Pierre Jolivet DD 21

23 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap What does it mean? A jk D k R k u = R j R T k A kk D k R k u N R i Au = R i ARj T j=1 D j R j u 13 / 22 Pierre Jolivet DD 21

24 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap What does it mean? A jk D k R k u = R j R T k A kk D k R k u N R i Au = R i ARj T D j R j u = A ij D j R j u j=1 j O i = R i Rj T A jj D j R j u j O i 13 / 22 Pierre Jolivet DD 21

25 Reformulation Parallel FE computations implies parallel matrix assembly: A jk := R j AR T k k O j Cannot be done with FreeFem++. A simple transformation on the overlap What does it mean? A jk D k R k u = R j R T k A kk D k R k u N R i Au = R i ARj T D j R j u = A ij D j R j u j=1 j O i = R i Rj T A jj D j R j u j O i local unknowns on V j 13 / 22 Pierre Jolivet DD 21

26 Coarse space construction The same goes for the construction of E := Z T AZ. A double block matrix multiplication E ij = Λ T R(i) ϕ(i) D R(i) A R(i)R(j) D R(j) Λ R(j)ϕ(j) where { } i R : j max i : ν k < j k=1 ϕ : j j R(j) ν k k=1 A similar reformulation leads to: E ij = Λ T R(i) ϕ(i) D R(i) R i R T j A R(j)R(j) D R(j) Λ R(j)ϕ(j) 0 j O i 14 / 22 Pierre Jolivet DD 21

27 Heterogeneous architectures N cores Memory Peak performance Compilers titane@cea To 140 TFLOP/s Intel babel@idris To 139 TFLOP/s IBM + GNU curie@cea To 2 PFLOP/s Intel CUDA cores ANR Preconditioning scientific applications on petascale Heterogeneous machines PRACE High Performance Computing-PDE Bruyères-le-Châtel, France. Orsay, France. 15 / 22 Pierre Jolivet DD 21

28 Darcy pressure equation I κ(x, y) x y Fig: Two dimensional diffusivity κ 16 / 22 Pierre Jolivet DD 21

29 Darcy pressure equation II Number of d.o.f Number of iterations: 20 Setup time: 1min30s Solve time: 20s % 80 % 60 % 40 % 20 % 0 % Fig: 2D efficiency (weak scaling) P 3 FE, stopping criterion ε = / 22 Pierre Jolivet DD 21

30 System of linear elasticity I (60M unkowns) Timing relative to 1024 cores UMFPACK blowup for the factorization of E: 1min 2048 Linear speedup Fig: 3D strong scaling P 2 FE, stopping criterion ε = Number of iterations 18 / 22 Pierre Jolivet DD 21

31 System of linear elasticity II (375M unkowns) Timing relative to 1024 cores Linear speedup Fig: 2D strong scaling P 3 FE, stopping criterion ε = Number of iterations 19 / 22 Pierre Jolivet DD 21

32 The limits reached 6144-way decomposition: in R 2, 1.2G unkowns, in R 3, 300M unkowns. All systems are solved with: coarse spaces of size 100; , less than 30 iterations. 20 / 22 Pierre Jolivet DD 21

33 Outline 1 Introduction 2 Spectral coarse space 3 Implementation and numerical results 4 Conclusion 21 / 22 Pierre Jolivet DD 21

34 Final words What to remember: FreeFem++ + = easy framework to solve large systems. Two-level DDM New problems being tackled: nonlinear elasticity (Newton-Krylov-Schwarz methods), reuse of Krylov and deflation subspaces. 22 / 22 Pierre Jolivet DD 21

35 Final words What to remember: FreeFem++ + = easy framework to solve large systems. Two-level DDM New problems being tackled: nonlinear elasticity (Newton-Krylov-Schwarz methods), reuse of Krylov and deflation subspaces. Thanks for your attention. 22 / 22 Pierre Jolivet DD 21

36 References Chevalier, C. and F. Pellegrini (2008). PT-Scotch: a tool for efficient parallel graph ordering. In: Parallel Computing 34.6, pp Davis, T.A. (2004). Algorithm 832: UMFPACK an unsymmetric-pattern multifrontal method. In: ACM Transactions on Mathematical Software 30.2, pp Hernandez, V., J.E Roman, and V. Vidal (2005). SLEPc: A scalable and flexible toolkit for the solution of eigenvalue problems. In: ACM Transactions on Mathematical Software 31.3, pp Karypis, G. and V. Kumar (1998). A fast and high quality multilevel scheme for partitioning irregular graphs. In: SIAM Journal on Scientific Computing 20.1, pp / 3 (+ 22) Pierre Jolivet DD 21

37 References Knyazev, A.V., M.E. Argentati, I. Lashuk, and E. Ovtchinnikov (2007). Block Locally Optimal Preconditioned Eigenvalue Xolvers (BLOPEX) in Hypre and PETSc. In: SIAM Journal on Scientific Computing 29.5, pp Lehoucq, R.B., D.C. Sorensen, and C. Yang (1998). ARPACK users guide: solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods. Vol. 6. Society for Industrial and Applied Mathematics. Nataf, F., H. Xiang, V. Dolean, and N. Spillane (2011). A coarse space construction based on local Dirichlet to Neumann maps. In: SIAM Journal on Scientific Computing 33.4, pp Snir, M., S.W. Otto, D.W. Walker, J. Dongarra, and S. Huss-Lederman (1995). MPI: The complete reference. The MIT Press. 2 / 3 (+ 22) Pierre Jolivet DD 21

38 References Spillane, N., V. Dolean, P. Hauret, F. Nataf, C. Pechstein, and R. Scheichl (2011). A robust two-level domain decomposition preconditioner for systems of PDEs. In: Comptes Rendus Mathematique , pp Tang, J.M., R. Nabben, C. Vuik, and Y.A. Erlangga (2009). Comparison of two-level preconditioners derived from deflation, domain decomposition and multigrid methods. In: Journal of Scientific Computing 39.3, pp / 3 (+ 22) Pierre Jolivet DD 21