Scalable Domain Decomposition Preconditioners For Heterogeneous Elliptic Problems

Transcription

1 Scalable Domain Decomposition Preconditioners For Heterogeneous Elliptic Problems Pierre Jolivet, F. Hecht, F. Nataf, C. Prud homme Laboratoire Jacques-Louis Lions Laboratoire Jean Kuntzmann INRIA Rocquencourt 10th Workshop of the INRIA-Illinois-ANL Joint Laboratory November 26th, 2013

2 A short introduction to DDM Consider the linear system: Au = f R n. Ω 2 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

3 A short introduction to DDM Consider the linear system: Au = f R n. Given a decomposition of 1; n, (N 1, N 2 ), define: the restriction operator R i from 1; n into N i, R T i as the extension by 0 from N i into 1; n. Ω 1 Ω 2 2 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

4 A short introduction to DDM Consider the linear system: Au = f R n. Given a decomposition of 1; n, (N 1, N 2 ), define: the restriction operator R i from 1; n into N i, Ri T as the extension by 0 from N i into 1; n. Then solve concurrently: u m+1 1 = u m 1 + A 1 11 R 1 (f Au m ) u m+1 2 = u m 2 + A 1 22 R 2 (f Au m ) where u i = R i u and A ij := R i AR T j. Ω 1 Ω 2 2 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

5 A short introduction II We have effectively divided, but we have yet to conquer. Duplicated unknowns coupled via a partition of unity: I = N i=1 R T i D i R i / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

6 A short introduction II We have effectively divided, but we have yet to conquer. Duplicated unknowns coupled via a partition of unity: N I = Ri T D i R i. i= Then, u m+1 = N i=1 R T i D i u m+1 i. 3 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

7 A short introduction II We have effectively divided, but we have yet to conquer. Duplicated unknowns coupled via a partition of unity: N I = Ri T D i R i. i= Then, u m+1 = N i=1 R T i D i u m+1 i. M 1 = N Ri T i=1 D i A 1 ii R i. 3 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

8 Contributions and goals Based on algebraic results with the p. of u., we propose: 1 a reformulation of the global matrix-vector product eliminating the need of a global ordering, 2 a construction of a so-called coarse operator to enhance a simple preconditioner. We are interested in the solution of various SPD systems, independently of: the discretization order, the contrast in the coefficients, the number of subdomains. 4 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

9 Using the overlap to its fullest extent DDM methods are seldom used as standalone solvers. Krylov methods and overlapping Schwarz methods Au = efficient global matrix-vector product 5 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

10 Using the overlap to its fullest extent DDM methods are seldom used as standalone solvers. Krylov methods and overlapping Schwarz methods Au = N j=1 AR T j D j R j u 5 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

11 Using the overlap to its fullest extent DDM methods are seldom used as standalone solvers. Krylov methods and overlapping Schwarz methods N R i Au = R i ARj T j=1 D j R j u 5 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

12 Using the overlap to its fullest extent DDM methods are seldom used as standalone solvers. Krylov methods and overlapping Schwarz methods N R i Au = R i ARj T j=1 D j R j u = j O i A ij D j R j u O i are the neighbors of Ω i, O i = O i {i}. 5 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

13 Using the overlap to its fullest extent DDM methods are seldom used as standalone solvers. Krylov methods and overlapping Schwarz methods N R i Au = R i ARj T D j R j u = A ij D j R j u j=1 j O i = R i Rj T A jj D j R j u. local unknowns on Ω j j O i no need for the global matrix, only local to neighbors mappings. explicit point-to-point communications via R i R T j. reuse of the operators from the preconditioner, A ii. O i are the neighbors of Ω i, O i = O i {i}. 5 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

14 Limitations of one-level methods One-level methods don t require exchange of global information. This hampers numerical scalability of such preconditioners. 6 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

15 Two-level preconditioners I A common technique in the field of DDM, MG, deflation: introduce an auxiliary coarse problem. Let Z be a rectangular matrix. Define E := Z T AZ. Z has O(N) columns, hence E is much smaller than A. 7 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

16 Two-level preconditioners I A common technique in the field of DDM, MG, deflation: introduce an auxiliary coarse problem. Let Z be a rectangular matrix. Define E := Z T AZ. Z has O(N) columns, hence E is much smaller than A. Enrich the original preconditioner, e.g. additively c.f. (Tang et al. 2009). P 1 = M 1 + ZE 1 Z T, 7 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

17 Two-level preconditioners II The construction of Z and the assembly of E are challenging. Let each domain compute concurrently ν i vectors { Λ ij } νi j=1. Define local dense rectangular matrices: W i = [ D i Λ i1 D i Λ i2 D i Λ iνi ]. Then, define the global deflation matrix as: Z = [ R T 1 W 1 R T 2 W 2 R T N W N ] 8 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

18 Generalized eigenvalue problems For theoretical justification of Z, see (Spillane et al. 2011). Solved by ARPACK concurently: where A N i Λ j = λ j D i R T i,0r i,0 A N i D i Λ j A N i is the local unassembled matrix, ( R i,0 is the restriction from Ω i to Ω i j O i Ω j ). 9 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

19 Workflow during one coarse operator correction How to compute ZE 1 Z T u R n? 10 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

20 Workflow during one coarse operator correction How to compute Z T u R n? Z T u = = m n n operations & MPI_Gather 10 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

21 Workflow during one coarse operator correction How to compute E 1 Z T u R n? Z T u = = m n n (Z T AZ) 1 Z T u = \ = operations & MPI_Gather + linear solve 10 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

22 Workflow during one coarse operator correction How to compute ZE 1 Z T u R n? Z T u = = m n = n (Z T AZ) 1 Z T u = \ = = Z(Z T AZ) 1 Z T u operations & MPI_Gather + linear solve + MPI_Scatter & operations 10 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

23 Workflow during one coarse operator correction How to compute ZE 1 Z T u R n? Z T u = = m n = n (Z T AZ) 1 Z T u = \ = = Z(Z T AZ) 1 Z T u operations & MPI_Gather + linear solve + MPI_Scatter & operations Communication pattern = global reduction at the coarse level. 10 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

24 Workflow during one coarse operator correction How to compute ZE 1 Z T u R n? Z T u = = m n = n = (Z T AZ) 1 Z T u = \ = = Z(Z T AZ) 1 Z T u operations & MPI_Gather + linear solve + MPI_Scatter & operations Communication pattern = global reduction at the coarse level. 10 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

25 Distribution of the coarse operator How can one solve E 1 z = c R m? Some constraints: 1 E cannot be centralized on a single MPI process, 2 E cannot be distributed on all MPI processes, 3 the solution must be computed fast and reliably. 11 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

26 Distribution of the coarse operator How can one solve E 1 z = c R m? Some constraints: 1 E cannot be centralized on a single MPI process, 2 E cannot be distributed on all MPI processes, 3 the solution must be computed fast and reliably. = use a direct solver with a distributed matrix on few master processes (number chosen at runtime). 11 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

27 Assembly for Schwarz methods Recalling E = ZAZ T, it can be proven that the block (i, j) E ij = Wi T A ij W j = Wi T R i Rj T A jj W j 1 compute locally T i = A ii W i (csrmm), 2 send to each neighbor, S j = R j R T i T i, 3 receive from each neighbor U j = R i R T j T j, 4 compute locally E i,i = Wi T T i (gemm), 5 compute locally E i,j = Wi T U j (gemm). Note: steps 2 and 3 overlap with step 4, if j O i, R i R T j = / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

28 Example of heterogeneous coefficients (κ u) = f + BC κ(x, y) 13 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

29 2D geometry (E 1, E 2 ) = (200, 0.01) GPa (ν 1, ν 2 ) = (0.25, 0.45) σ = f + BC 14 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

30 3D geometry 15 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

31 Machine used for scaling runs Curie Thin Nodes 5,040 compute nodes. 2 eight-core Intel Sandy Bridge@2.7 GHz per node. IB QDR full fat tree. 1.7 PFLOPs peak performance. 16 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

32 Strong scaling (linear elasticity) 1 subdomain/mpi process, 2 OpenMP threads/mpi process. Runtime (seconds) Linear speedup 3D (P 2 FE) 2D (P 3 FE) #processes

33 Strong scaling (linear elasticity) 1 subdomain/mpi process, 2 OpenMP threads/mpi process. 2.1B d.o.f. in 2D (P 3 FE) 300M d.o.f. in 3D (P 2 FE) 100% 100% Ratio 80% 60% 40% 20% 80% 60% 40% 20% 0% #processes #processes Factorization Deflation vectors Coarse operator Krylov method 0%

34 Weak scaling (scalar diffusion equation) 1 subdomain/mpi process, 2 OpenMP threads/mpi process. Efficiency relative to 256 MPI processes 100 % 80 % 60 % 40 % 20 % 0 % D (P 2 FE) 2D (P 4 FE) #d.o.f. (in millions) #processes

35 Weak scaling (scalar diffusion equation) Time (seconds) 1 subdomain/mpi process, 2 OpenMP threads/mpi process M d.o.f. sbdmn in 2D (P 4 FE) k d.o.f. sbdmn in 3D (P 2 FE) #processes #processes Factorization Deflation vectors Coarse operator Krylov method 0

36 Distributed global matrix Local to global mapping = distribution of the global matrix à la PETSc (split row-wise). Comparing performance of setup and solution phases between our solver against purely algebraic (+ near null space) solvers: GASM one-level domain decomposition method (ANL), Hypre BoomerAMG algebraic multigrid (LLNL), GAMG algebraic multrigrid (ANL/LBL). 20 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

37 Relative residual error Solution of a linear system I Homogeneous 3D Poisson equation discretized by P 1 FE solved on 2,048 MPI processes, 111M d.o.f Schwarz GenEO PETSc GASM Hypre BoomerAMG PETSc GAMG #iterations Time (seconds) " % " " GASM BoomerAMG GAMG Schwarz GenEO Setup Solution 21 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

38 Relative residual error Solution of a linear system II Heterogeneous 3D linear elasticity equation discretized by P 2 FE solved on 2,048 MPI processes, 127M d.o.f. " 100 Setup Solution Schwarz GenEO PETSc GASM Hypre BoomerAMG PETSc GAMG (fail) #iterations Time (seconds) 50 0 % % % GASM BoomerAMG GAMG Schwarz GenEO 22 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

39 Final words Limitations: scaling of the coarse operator in 3D beyond 10k subdomains, deflation vectors need elementary matrices to be computed. Summary: scalable framework for building two-level preconditioners for both Schwarz or substructuring methods (FETI-1), easily interfacable (FEM, FVM) without a global ordering. Outlooks: adaptive (re)construction/recycling of the coarse operator, nonlinear and saddle point problems. 23 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

40 Final words Limitations: scaling of the coarse operator in 3D beyond 10k subdomains, deflation vectors need elementary matrices to be computed. Summary: scalable framework for building two-level preconditioners for both Schwarz or substructuring methods (FETI-1), easily interfacable (FEM, FVM) without a global ordering. Outlooks: adaptive (re)construction/recycling of the coarse operator, nonlinear and saddle point problems. Thank you! 23 / 23 Pierre Jolivet Scalable Domain Decomposition Preconditioners

41 References Backup slides Spillane, N., V. Dolean, P. Hauret, F. Nataf, C. Pechstein, and R. Scheichl (2011). A robust two-level domain decomposition preconditioner for systems of PDEs. In: Comptes Rendus Mathematique , pp Tang, J., R. Nabben, C. Vuik, and Y. Erlangga (2009). Comparison of two-level preconditioners derived from deflation, domain decomposition and multigrid methods. In: Journal of Scientific Computing 39.3, pp / 6 (+ 23) Pierre Jolivet Scalable Domain Decomposition Preconditioners

42 References Backup slides Solvers parameters Schwarz GenEO: ν i = 20, overlap = 1 (geometric). PETSc GASM: overlap = 10 (algebraic). Hyper BoomerAMG: HMIS coarsening, extended classical interpolation, no CF-relaxation, 2 levels of aggressive coarsening. PETSc GAMG: 1 smoothing step, -mg_levels_ksp_type richardson -mg_levels_pc_type sor. OpenMPI bindings for hybrid runs: --bind-to-socket --bycore. 2 / 6 (+ 23) Pierre Jolivet Scalable Domain Decomposition Preconditioners

43 References Backup slides Distribution of the coarse operator Uniform distribution Non-uniform distribution Distribution of E when built with N = 16 using 4 masters. On the right, the number of values per master is roughly the same if the values below the diagonal are dropped (symmetric coarse operator). 3 / 6 (+ 23) Pierre Jolivet Scalable Domain Decomposition Preconditioners

44 References Backup slides Timings for assembling the coarse operator N P dim(e) O i (average) Memory cost of E 1 Time 3D MB 2.78 s MB 3.35 s MB 93 MB 4.42 s s MB 138 MB 6.91 s 5.68 s MB 172 MB s 8.04 s MB 241 MB s s N P dim(e) O i (average) Memory cost of E 1 Time 2D MB 9.39 s MB 9.96 s MB 57 MB 9.92 s s MB 83 MB s 6.20 s MB 73 MB s 5.10 s MB 118 MB s 6.96 s 4 / 6 (+ 23) Pierre Jolivet Scalable Domain Decomposition Preconditioners

45 References Backup slides Strong scaling (linear elasticity) 3D 2D N Factorization Deflation Solution #it. Total #d.o.f s s s s s s s s s s 9.73 s s s s 6.05 s s s s s s s s s s s s 8.64 s s s s 4.79 s s / 6 (+ 23) Pierre Jolivet Scalable Domain Decomposition Preconditioners

46 References Backup slides Weak scaling (scalar diffusion equation) 3D 2D N Factorization Deflation Solution #it. Total #d.o.f s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s / 6 (+ 23) Pierre Jolivet Scalable Domain Decomposition Preconditioners