Une méthode parallèle hybride à deux niveaux interfacée dans un logiciel d éléments finis

Size: px

Start display at page:

Download "Une méthode parallèle hybride à deux niveaux interfacée dans un logiciel d éléments finis"

Kerry Mason
5 years ago
Views:

1 Une méthode parallèle hybride à deux niveaux interfacée dans un logiciel d éléments finis Frédéric Nataf Laboratory J.L. Lions (LJLL), CNRS, Alpines Inria and Univ. Paris VI Victorita Dolean (Univ. Nice Sophia-Antipolis) Frédéric Hecht (LJLL) Pierre Jolivet (LJLL) Nicole Spillane (LJLL) Maison de la Simulation 2014

2 Outline 1 Some Applications 2 An abstract 2-level Schwarz: the GenEO algorithm 3 FreeFem++ code 4 Numerical Results 5 Conclusion F. Nataf FF++ DDM 2 / 38

3 Outline 1 Some Applications 2 An abstract 2-level Schwarz: the GenEO algorithm 3 FreeFem++ code 4 Numerical Results 5 Conclusion F. Nataf FF++ DDM 3 / 38

Motivation Large Sparse discretized system with strongly

E.g. Darcy pressure equation, P 1 -finite elements: Applications:

4 Motivation Large Sparse discretized system with strongly heterogeneous coefficients (high contrast, nonlinear, multiscale) E.g. Darcy pressure equation, P 1 -finite elements: Applications: flow in heterogeneous / stochastic / layered media structural mechanics time dependent waves etc. AU = F cond(a) α max α min h 2 Goal: iterative solvers robust in size and heterogeneities F. Nataf FF++ DDM 4 / 38

5 Black box solvers (solve(mat,rhs,sol)) Direct Solvers Iterative Solvers Pros Robustness Naturally Cons Difficult to Robustness Domain Decomposition Methods (DDM): Hybrid solver should be naturally parallel and robust General form: Au = f, solved with PCG for a preconditioner M 1. What s Classical: Robustness with respect to problem size (scalability) What s New here: Provable Robustness for the Schwarz method in the SPD case with respect to: Coefficients jumps System of PDEs F. Nataf FF++ DDM 5 / 38

6 Black box solvers (solve(mat,rhs,sol)) Direct Solvers Iterative Solvers Pros Robustness Naturally Cons Difficult to Robustness Domain Decomposition Methods (DDM): Hybrid solver should be naturally parallel and robust General form: Au = f, solved with PCG for a preconditioner M 1. What s Classical: Robustness with respect to problem size (scalability) What s New here: Provable Robustness for the Schwarz method in the SPD case with respect to: Coefficients jumps System of PDEs F. Nataf FF++ DDM 5 / 38

7 The First Domain Decomposition Method The original Schwarz Method (H.A. Schwarz, 1870) (u) = f in Ω u = 0 on Ω. Ω 1 Ω 2 Schwarz Method : (u1 n, un 2 ) (un+1 1, u n+1 2 ) with (u n+1 1 ) = f in Ω 1 u n+1 1 = 0 on Ω 1 Ω u n+1 1 = u2 n on Ω 1 Ω 2. (u n+1 2 ) = f in Ω 2 u n+1 2 = 0 on Ω 2 Ω u n+1 2 = u n+1 1 on Ω 2 Ω 1. Parallel algorithm, converges but slowly, overlapping subdomains only. As a preconditioner in a Krylov method, convergence is acceptable. F. Nataf FF++ DDM 6 / 38

8 Strong and Weak scalability How to evaluate the efficiency of a domain decomposition? Strong scalability (Amdahl) How the solution time varies with the number of processors for a fixed total problem size Weak scalability (Gustafson) How the solution time varies with the number of processors for a fixed problem size per processor. Not achieved with the one level method Number of subdomains ASM The iteration number increases linearly with the number of subdomains in one direction. F. Nataf FF++ DDM 7 / 38

9 How to achieve scalability Stagnation corresponds to a few very low eigenvalues in the spectrum of the preconditioned problem. They are due to the lack of a global exchange of information in the preconditioner. u = f in Ω u = 0 on Ω The mean value of the solution in domain i depends on the value of f on all subdomains. A classical remedy consists in the introduction of a coarse problem that couples all subdomains. This is closely related to deflation technique classical in linear algebra (see Y. Saad, J. Erhel, Nabben and Vuik) and multigrid techniques. F. Nataf FF++ DDM 8 / 38

10 Adding a coarse space We add a coarse space correction (aka second level) Let V H be the coarse space and Z be a basis, V H = span Z, writing R 0 = Z T we define the two level preconditioner as: M 1 ASM,2 := RT 0 (R 0AR T 0 ) 1 R 0 + N i=1 R T i A 1 i R i. The Nicolaides approach (1987) is to use the kernel of the operator as a coarse space, this is the constant vectors, in local form this writes: Z := (Ri T D i R i 1) 1 i N where D i are chosen so that we have a partition of unity: N Ri T D i R i = Id. i=1 F. Nataf FF++ DDM 9 / 38

11 Theoretical convergence result Theorem (Widlund, Dryija) Let M 1 ASM,2 be the two-level additive Schwarz method: κ(m 1 ASM,2 (1 A) C + H ) δ where δ is the size of the overlap between the subdomains and H the subdomain size. This does indeed work very well Number of subdomains ASM ASM + Nicolaides F. Nataf FF++ DDM 10 / 38

12 Failure for Darcy equation with heterogeneities (α(x, y) u) = 0 in Ω R 2, u = 0 on Ω D, u n = 0 on Ω N. IsoValue Our approach Decomposition α(x, y) Jump ASM ASM + Nicolaides Fix the problem by an optimal and proven choice of a coarse space Z. F. Nataf FF++ DDM 11 / 38

13 Outline 1 Some Applications 2 An abstract 2-level Schwarz: the GenEO algorithm Choice of the coarse space Parallel implementation 3 FreeFem++ code 4 Numerical Results 5 Conclusion F. Nataf FF++ DDM 12 / 38

14 Objectives Strategy Define an appropriate coarse space V H 2 = span(z 2 ) and use the framework previously introduced, writing R 0 = Z2 T the two level preconditioner is: P 1 ASM 2 := RT 0 (R 0AR T 0 ) 1 R 0 + N i=1 R T i A 1 i R i. The coarse space must be Local (calculated on each subdomain) parallel Adaptive (calculated automatically) Easy and cheap to compute Robust (must lead to an algorithm whose convergence is proven not to depend on the partition nor the jumps in coefficients) F. Nataf FF++ DDM 13 / 38

15 Problem setting I Given f (V h ) find u V h a(u, v) = f, v A u = f v V h Assumption throughout: A symmetric positive definite (SPD) Examples: Darcy Elasticity Eddy current a(u, v) = Ω κ u v dx a(u, v) = Ω C ε(u) : ε(v) dx a(u, v) = Ω ν curl u curl v + σ u v dx Heterogeneities / high contrast in parameters F. Nataf FF++ DDM 14 / 38

16 Problem setting II 1 V h... FE space of functions in Ω based on mesh T h = {τ} 2 {φ k } n k=1 (FE) basis of V h 3 Technical assumptions fulfilled by standard FE and bilinear forms a(, ) F. Nataf FF++ DDM 15 / 38

17 Schwarz setting I Overlapping decomposition: Ω = N j=1 Ω j (Ω j union of elements) V j := span { φ k : supp(φ k ) Ω j } such that every φ k is contained in one of those spaces, i.e. N V h = j=1 Example: adding layers to non-overlapping partition (partition and adding layers based on matrix information only!) V j F. Nataf FF++ DDM 16 / 38

18 Schwarz setting II Local subspaces: V j V h j = 1,..., N Coarse space (defined later): V 0 V h Additive Schwarz preconditioner: M 1 AS,2 = N j=0 R j A 1 j R j where A j = R j AR j and R j R j : V j V h natural embedding F. Nataf FF++ DDM 17 / 38

$Overlapping zone / Choice of coarse space Overlapping zone: Ω j = {x Ω j : i j : x Ω i } Observation: partition of unity operator satisfies Ξ j Ωj \Ω j =$

19 Overlapping zone / Choice of coarse space Overlapping zone: Ω j = {x Ω j : i j : x Ω i } Observation: partition of unity operator satisfies Ξ j Ωj \Ω j = id Coarse space should be a sum of local contributions: V 0 = N j=1 V 0, j where V 0, j V j E.g. V 0, j = span{ξ j p j,k } m j k=1 F. Nataf FF++ DDM 18 / 38

20 Choice of coarse space (continued) ASM theory needs stable splitting: v = v 0 + Suppose v 0 = N j=1 Ξ jπ j v Ωj where Π j... local projector N j=1 v j 2 Ξ j (v Π j v) a,ω }{{} j v j = Ξ j (v Π j v) 2 a,ω j + Ξ j (v Π j v)) 2 a,ω j \Ω j HOW? C v 2 a,ω j Minimal requirements: Π j be a-orthogonal (a,d denotes the restriction of a to D) Stability estimate: Ξ j (v Π j v) 2 a,ω j c v 2 a,ω j F. Nataf FF++ DDM 19 / 38

21 Choice of coarse space (continued) ASM theory needs stable splitting: v = v 0 + Suppose v 0 = N j=1 Ξ jπ j v Ωj where Π j... local projector N j=1 v j 2 Ξ j (v Π j v) a,ω }{{} j v j = Ξ j (v Π j v) 2 a,ω j + Ξ j (v Π j v)) 2 a,ω j \Ω j HOW? C v 2 a,ω j Minimal requirements: Π j be a-orthogonal (a,d denotes the restriction of a to D) Stability estimate: Ξ j (v Π j v) 2 a,ω j c v 2 a,ω j F. Nataf FF++ DDM 19 / 38

22 Abstract eigenvalue problem Minimal requirements: 1 Π j be a-orthogonal 2 Stability estimate: Ξ j (v Π j v) 2 a,ω j c v 2 a,ω j Fulfillment of 2 If there exist a non zero function w such that w a,ωj = 0, it is necessary to project on Span(w). The kernel of a Darcy equation is the constant function and that of elasticity is spanned by rigid body motions. The corresponding coarse space will be refered to as ZEM (zero energy modes). For highly heterogeneous problems, we take a larger coarse space deduced from the stability estimate. F. Nataf FF++ DDM 20 / 38

23 Abstract eigenvalue problem Geneo.EVP per subdomain: Find p j,k V h Ωj and λ j,k 0: a Ωj (p j,k, v) = λ j,k a Ω j (Ξ j p j,k, Ξ j v) v V h Ωj A j p j,k = λ j,k X j A j X j p j,k (X j... diagonal) a D... restriction of a to D In the two-level ASM: Choose first m j eigenvectors per subdomain: V 0 = span { Ξ j p j,k } j=1,...,n k=1,...,m j This automatically includes Zero Energy Modes. F. Nataf FF++ DDM 21 / 38

24 Abstract eigenvalue problem Geneo.EVP per subdomain: Find p j,k V h Ωj and λ j,k 0: a Ωj (p j,k, v) = λ j,k a Ω j (Ξ j p j,k, Ξ j v) v V h Ωj A j p j,k = λ j,k X j A j X j p j,k (X j... diagonal) a D... restriction of a to D In the two-level ASM: Choose first m j eigenvectors per subdomain: V 0 = span { Ξ j p j,k } j=1,...,n k=1,...,m j This automatically includes Zero Energy Modes. F. Nataf FF++ DDM 21 / 38

25 Comparison with existing works N. & al (SIAM 2011): Ω j κ p j,k v dx = λ j,k Ω j κ p j,k v dx Efendiev, Galvis, Lazarov & Willems (2011): a Ωj (p j,k, v) = λ j,k a Ωj (ξ j ξ i p j,k, ξ j ξ i v) i neighb(j) v V h Ωj v V Ωj ξ j... partition of unity, calculated adaptively (MS) Our gen.evp: a Ωj (p j,k, v) = λ j,k a Ω j (Ξ j p j,k, Ξ j v) v V h Ωj both matrices typically singular = λ j,k [0, ] F. Nataf FF++ DDM 22 / 38

26 Theory Two technical assumptions. Theorem (Spillane, Dolean, Hauret, N., Pechstein, Scheichl (Num. Math. 2013)) If for all j: 0 < λ j,mj+1 < : [ κ(m 1 ASM,2 A) (1 + k 0) 2 + k 0 (2k 0 + 1) N max j=1 ( )] λ j,mj +1 Possible criterion for picking m j : (used in our Numerics) λ j,mj +1 < δ j H j H j... subdomain diameter, δ j... overlap F. Nataf FF++ DDM 23 / 38

27 E Eigenvalues and eigenvectors (Elasticity) Eigenvector number 1 is e-15; exageration coefficient is: Eigenvector number 2 is e-16; exageration coefficient is: Eigenvector number 3 is e-15; exageration coefficient is: Eigenvector number 4 is e-05; exageration coefficient is: Eigenvector number 5 is e-05; exageration coefficient is: Eigenvector number 6 is e-05; exageration coefficient is: Eigenvector number 7 is ; exageration coefficient is: Eigenvector number 8 is ; exageration coefficient is: Eigenvector number 9 is ; exageration coefficient is: Eigenvector number 10 is ; exageration coefficient is: Eigenvector number 11 is ; exageration coefficient is: Eigenvector number 12 is ; exageration coefficient is: Eigenvector number 13 is ; exageration coefficient is: Eigenvector number 14 is ; exageration coefficient is: Eigenvector number 15 is ; exageration coefficient is: eigenvalue (log scale) eigenvalue number Logarithmic scale F. Nataf FF++ DDM 24 / 38

28 Numerical results (Darcy) IsoValue e e e e e e+06 IsoValue Channels and inclusions: 1 α , the solution and partitionings (Metis or not) F. Nataf FF++ DDM 25 / 38

29 Convergence AS P BNN : AS + Z Nico P BNN : AS + Z D2N GMRES P BNN : AS + Z D2N Error Iteration count F. Nataf FF++ DDM 26 / 38

30 Numerical results Optimality m i is given automatically by the strategy. #Z per subd. ASM ASM+Z Nico ASM+Z Geneo max(m i 1, 1) 273 m i m i Taking one fewer eigenvalue has a huge influence on the iteration count Taking one more has only a small influence F. Nataf FF++ DDM 27 / 38

31 Numerical results via a Domain Specific Language FreeFem++ ( with: Metis Karypis and Kumar 1998 SCOTCH Chevalier and Pellegrini 2008 UMFPACK Davis 2004 ARPACK Lehoucq et al MPI Snir et al Intel MKL PARDISO Schenk et al MUMPS Amestoy et al PaStiX Hénon et al Slepc via PETSC Runs on PC (Linux, OSX, Windows) and HPC (Babel@CNRS, HPC1@LJLL, Titane@CEA via GENCI PRACE) Why use a DS(E)L instead of C/C++/Fortran/..? performances close to low-level language implementation, hard to beat something as simple as: varf a(u, v) = int3d(mesh)([dx(u), dy(u), dz(u)]' * [dx(v), dy(v), dz(v)]) + int3d(mesh)(f * v) + on(boundary mesh)(u = 0) F. Nataf FF++ DDM 28 / 38

32 Strong scalability in two dimensions heterogeneous elasticity (P. Jolivet with Frefeem ++) Elasticity problem with heterogeneous coefficients with automatic mesh partition Timing relative to processes Linear speedup 10 #iterations #processes Speed-up for a 1.2 billion unknowns 2D problem. Direct solvers in the subdomains. Peak performance wall-clock time: 26s. F. Nataf FF++ DDM 29 / 38

33 Strong scalability in three dimensions heterogeneous elasticity Elasticity problem with heterogeneous coefficients with automatic mesh partition Timing relative to processes Linear speedup 10 #iterations #processes Speed-up for a 160 million unknowns 3D problem. Direct solvers in subdomains. Peak performance wall-clock time: 36s. F. Nataf FF++ DDM 30 / 38

34 Darcy pressure equation κ(x, y) x y Figure: Two dimensional diffusivity κ F. Nataf FF++ DDM 31 / 38

35 Weak scalability in two dimensions Darcy problems with heterogeneous coefficients with automatic mesh partition Weak efficiency relative to processes 100 % 80 % 60 % 40 % 20 % 0 % #d.o.f. (in millions) #processes Efficiency for a 2D problem. Direct solvers in the subdomains. Final size: 22 billion unknowns. Wall-clock time: 200s. F. Nataf FF++ DDM 32 / 38

36 Weak scalability in three dimensions Darcy problems with heterogeneous coefficients with automatic mesh partition Weak e ciency relative to processes 100 % 80 % 60 % 40 % 20 % 0% #d.o.f. (in millions) #processes Efficiency for a 3D problem. Direct solvers in the subdomains. Final size: 2 billion unknowns. Wall-clock time: 200s. F. Nataf FF++ DDM 33 / 38

37 Outline 1 Some Applications 2 An abstract 2-level Schwarz: the GenEO algorithm 3 FreeFem++ code 4 Numerical Results 5 Conclusion F. Nataf FF++ DDM 34 / 38

38 Outline 1 Some Applications 2 An abstract 2-level Schwarz: the GenEO algorithm 3 FreeFem++ code 4 Numerical Results 5 Conclusion F. Nataf FF++ DDM 35 / 38

39 Outline 1 Some Applications 2 An abstract 2-level Schwarz: the GenEO algorithm 3 FreeFem++ code 4 Numerical Results 5 Conclusion F. Nataf FF++ DDM 36 / 38

40 Conclusion Summary Using generalized eigenvalue problems and projection preconditioning we are able to achieve a targeted convergence rate. Works for BNN and FETI methods as well (not shown here) Implementation requires only element stiffness matrices Future work Available in the next public release of FreeFem++ Purely algebraic setting Non symmetric, undefinite problems Nonlinear time dependent problem (Reuse of the coarse space) F. Nataf FF++ DDM 37 / 38

41 Bibliography Coarse Space for Schwarz Preprints available on HAL: N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, An Algebraic Local Generalized Eigenvalue in the Overlapping Zone Based Coarse Space : A first introduction, C. R. Mathématique, Vol. 349, No , pp , F. Nataf, H. Xiang, V. Dolean, N. Spillane, A coarse space construction based on local Dirichlet to Neumann maps, SIAM J. Sci Comput., Vol. 33, No.4, pp , F. Nataf, H. Xiang, V. Dolean, A two level domain decomposition preconditioner based on local Dirichlet-to-Neumann maps, C. R. Mathématique, Vol. 348, No , pp , V. Dolean, F. Nataf, R. Scheichl, N. Spillane, Analysis of a two-level Schwarz method with coarse spaces based on local Dirichlet to Neumann maps, CMAM, 2012 vol. 12, P. Jolivet, V. Dolean, F. Hecht, F. Nataf, C. Prud homme, N. Spillane, High Performance domain decomposition methods on massively parallel architectures with FreeFem++, J. of Numerical Mathematics, 2012 vol. 20. N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, Abstract Robust Coarse Spaces for Systems of PDEs via Generalized Eigenproblems in the Overlaps, to appear Numerische Mathematik, THANK YOU FOR YOUR ATTENTION! F. Nataf FF++ DDM 38 / 38

42 Bibliography Coarse Space for Schwarz Preprints available on HAL: N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, An Algebraic Local Generalized Eigenvalue in the Overlapping Zone Based Coarse Space : A first introduction, C. R. Mathématique, Vol. 349, No , pp , F. Nataf, H. Xiang, V. Dolean, N. Spillane, A coarse space construction based on local Dirichlet to Neumann maps, SIAM J. Sci Comput., Vol. 33, No.4, pp , F. Nataf, H. Xiang, V. Dolean, A two level domain decomposition preconditioner based on local Dirichlet-to-Neumann maps, C. R. Mathématique, Vol. 348, No , pp , V. Dolean, F. Nataf, R. Scheichl, N. Spillane, Analysis of a two-level Schwarz method with coarse spaces based on local Dirichlet to Neumann maps, CMAM, 2012 vol. 12, P. Jolivet, V. Dolean, F. Hecht, F. Nataf, C. Prud homme, N. Spillane, High Performance domain decomposition methods on massively parallel architectures with FreeFem++, J. of Numerical Mathematics, 2012 vol. 20. N. Spillane, V. Dolean, P. Hauret, F. Nataf, C. Pechstein, R. Scheichl, Abstract Robust Coarse Spaces for Systems of PDEs via Generalized Eigenproblems in the Overlaps, to appear Numerische Mathematik, THANK YOU FOR YOUR ATTENTION! F. Nataf FF++ DDM 38 / 38

Toward black-box adaptive domain decomposition methods

Toward black-box adaptive domain decomposition methods Frédéric Nataf Laboratory J.L. Lions (LJLL), CNRS, Alpines Inria and Univ. Paris VI joint work with Victorita Dolean (Univ. Nice Sophia-Antipolis)