From Direct to Iterative Substructuring: some Parallel Experiences in 2 and 3D

Transcription

1 From Direct to Iterative Substructuring: some Parallel Experiences in 2 and 3D Luc Giraud N7-IRIT, Toulouse MUMPS Day October 24, 2006, ENS-INRIA, Lyon, France

2 Outline 1 General Framework 2 The direct substructuring approach 3 The iterative substructuring approach Description of the preconditioners Variant of Additive Shwarz preconditioner M AS M AS v.s. Neumann-Neumann 4 Parallel numerical experiments 2D experiments in semiconductor device modelling 3D experiments on the academic cube 5 Prospectives

3 Outline 1 General Framework 2 The direct substructuring approach 3 The iterative substructuring approach Description of the preconditioners Variant of Additive Shwarz preconditioner M AS M AS v.s. Neumann-Neumann 4 Parallel numerical experiments 2D experiments in semiconductor device modelling 3D experiments on the academic cube 5 Prospectives

4 Background in 3D The model PDE 8 < : div(k. u) = f in Ω, u = 0 on Ω Dirichlet, (K u, n) = 0 on Ω Neumann. The associated linear system 0 A 11 A 1Γ u f 1 Au = A T 1Γ A (1) Γ + A (2) Γ A T 2Γ u Γ A f Γ A 0 A 2Γ A 22 u 2 f 2

5 Background in 2D The same model PDE but in 2D ( x (a(x, y) v x ) y (b(x, y) v ) y = F (x, y) in Ω, v = 0 on Ω. The associated linear system A 11 0 A 1Γ x 1 b 1 A h x = 0 A 22 A 2Γ x 2 A b 2 A A T 1Γ A T 2Γ A (1) Γ + A (2) x Γ Γ b Γ

6 Algebraic Gaussian elimination Space dimension free exposure Algebraic splitting and block Gaussian elimination 0 A 11 0 A 1Γ x b 0 A 22 A 2Γ x 2 A b 2 A A T 1Γ A T 2Γ A (1) Γ + A (2) Γ {z } x Γ b Γ A h 2X 2X Sx Γ = b Γ i=1 A T iγa 1 ii b i S = i=1 A (i) Γ AT iγa 1 ii A iγ = 2X i=1 S (i) Local unassembled Schur complement associated with Ω i ( ) A (Ω i ) Aii A iγi = S (i) = A (i) Γ i A T iγ i A 1 A T iγ i A (i) Γ i ii A iγi }{{} Contribution matrix

7 Outline 1 General Framework 2 The direct substructuring approach 3 The iterative substructuring approach Description of the preconditioners Variant of Additive Shwarz preconditioner M AS M AS v.s. Neumann-Neumann 4 Parallel numerical experiments 2D experiments in semiconductor device modelling 3D experiments on the academic cube 5 Prospectives

8 The sparse direct multifrontal view point [Amestoy, Duff, L Excellent - 98] [Amestoy, Duff, Koster, L Excellent - 01], [Amestoy, Guermouche, L Excellent, Pralet - 06] Ω 1 Ω 3 Ω 2 Ω 4 Ω 1 Ω 2 Ω 3 Ω 4 The direct substructuring approach on top of MUMPS 1: Call #domains sequential MUMPS in parallel on each A (Ω i ) to form the local unassembled Schur complement (local ordering) 2: Convert the local ordering on the interface into a global ordering (simple pre-processing) 3: Call one parallel MUMPS on the distributed matrix elemental distributed matrices S = #domains X i=1 S (i) {z } feature of MUMPS for redundant coordinates

9 The sparse direct multifrontal view point [Amestoy, Duff, L Excellent - 98] [Amestoy, Duff, Koster, L Excellent - 01], [Amestoy, Guermouche, L Excellent, Pralet - 06] Ω 1 Ω 3 Ω 2 Ω 4 Ω 1 Ω 2 Ω 3 Ω 4 The direct substructuring approach on top of MUMPS 1: Call #domains sequential MUMPS in parallel on each A (Ω i ) to form the local unassembled Schur complement (local ordering) 2: Convert the local ordering on the interface into a global ordering (simple pre-processing) 3: Call one parallel MUMPS on the distributed matrix elemental distributed matrices S = #domains X i=1 S (i) {z } feature of MUMPS for redundant coordinates

10 Outline 1 General Framework 2 The direct substructuring approach 3 The iterative substructuring approach Description of the preconditioners Variant of Additive Shwarz preconditioner M AS M AS v.s. Neumann-Neumann 4 Parallel numerical experiments 2D experiments in semiconductor device modelling 3D experiments on the academic cube 5 Prospectives

11 Elliptic PDEs properties Algebraic splitting and block Gaussian elimination: # domains = N sub-domains case A I1 I A I1 Γ 1. B C 0... A IN I N A IN Γ N A Γ1 I 1... A ΓN I N A ΓΓ Su Γ = i=1 u I1.. u IN u Γ i=1 C A = f I1.. f IN fγ C A! NX NX RΓ T i S (i) R Γi u Γ = f Γ RΓ T i A Γi I i A 1 I i I i f Ii where S (i) = A (i) Γ i Γ i A Γi I i A 1 I i I i A Ii Γ i Spectral properties for elliptic PDE s κ(a) = O(h 2 ) κ(s) = O(h 1 ) e (k) A 2 p κ(a) 1 p κ(a) + 1! k e (0) A

12 A simple mathematical framework Block preconditioners U a algebraic space of vectors associated with unknowns on Γ U l subspaces of U such that U = U U n R l : the canonical pointwise restriction from U U l M = nx l=1 Examples : Rl T M 1 l R l where M l = R l SRl T U l associated with each edge/face: block Jacobi (geometric information needed) U l associated with Ω l : additive Schwarz

13 Structure of the Local Schur Complement Non-Overlapping Domain Decomposition E m Ω i E g Ω j E l E k Ωi = Γ i = E l E k E m E g Distributed Schur Complement 0 1 S (i) mm S mg S mk S ml S (i) S gm S (i) gg S = gk S gl S km S kg S (i) C kk S kl A S lm S lg S lk S (i) ll Sgg = S(i) gg + S (j) gg If A is SPD then S is also SPD CG In a distributed memory environment: S is distributed non-assembled

14 Additive Schwarz preconditioner [Carvalho, Giraud, Meurant - 01] Preconditionner properties U i associated with the entire interface Γ i of sub-domain M AS = #domains i=1 R T i ( S(i) ) 1 Ri 0 1 S mm S mg S mk S ml S (i) = BS gm S gg S gk S gl S km S kg S kk S kl A S lm S lg S lk S ll Assembled local Schur complement S (i) and S (i) are dense matrices... Remarks M AS is SPD if S is SPD 0 1 S (i) mm S mg S mk S ml S (i) S gm S (i) gg S = gk S gl S km S kg S (i) C kk S kl A S lm S lg S lk S (i) ll local Schur complement

15 Cheaper Additive Shwarz preconditioner form Main characteristics Cheaper in memory space Flops reduction Without any additional communication cost S (i) Sparsification strategy for { skl if s ŝ kl = kl ɛ( s kk + s ll ) 0 else We end-up with a sparse variant of the preconditioner M spas = #domains i=1 R T i (Ŝ(i) ) 1 Ri

16 M AS v.s. Neumann-Neumann Neumann-Neumann preconditioner [J.F Bourgat, R. Glowinski, P. Le Tallec and M. Vidrascu - 89] [Y.H. de Roek, P. Le Tallec and M. Vidrascu - 91] S (1) = S (2) = S 2 S 1 = 1 S (1) 1 + S (2) « «««A (i) Aii A iγ I 0 Aii 0 I A 1 = A iγ A (i) = A Γ iγ A 1 ii I 0 S (i) ii A Γi 0 I S (i) 1 ` = 0 I A (i) «1 0 I M NN = #domains X i=1 R T i D i S (i) «#domains 1 X Di R i while M AS = i=1 R T i «S(i) 1 Ri

17 Outline 1 General Framework 2 The direct substructuring approach 3 The iterative substructuring approach Description of the preconditioners Variant of Additive Shwarz preconditioner M AS M AS v.s. Neumann-Neumann 4 Parallel numerical experiments 2D experiments in semiconductor device modelling 3D experiments on the academic cube 5 Prospectives

18 Application to semiconductor device modelling [Giraud, Koster, Marroco, Rioual - 01], [Giraud, Marroco, Rioual - 05] The drift diffusion equations Find (φ, φ n, φ p ) so that div(ɛ φ) + q[n(φ, φ n ) P(φ, φ p ) Dop] = 0, div(qµ n N(φ, φ n ) φ n ) + qgr(φ, φ n, φ p ) = 0, div(qµ p P(φ, φ p ) φ p ) qgr(φ, φ n, φ p ) = 0, with mixed Dirichlet-Neumann boundary conditions.

19 Heterojunction device (mixed finite elements) Mesh size # subdomains Size Schur Medium edges Large edges

20 Explicit Schur complement calculation Experiments on a SGI O2K S = N A (i) Γ i=1 A iγa 1 ii A Γia }{{} implicit = N }{{} S (i) explicit i= Facto Matvec 20 Krylov Facto Matvec 20 Krylov Implicit Explicit Krylov solvers CG on SPD systems (Poisson equations) Full-GMRES on unsymmetric systems (electrons/holes equations)

21 Iterative solvers Embeded iterative schemes: numerical effects Newton Steps ε Krylov Performance of the preconditioners Medium 16 Large 32 M bj M AS M bj M AS Newton its iter CG iter GMRES time (s) Direct v.s. Hybrid iterative solve Medium 16 Large 32 M AS Dss M AS Dss Newton time (s)

22 3D experiments [Giraud, Haidar, Watson - ongoing] 3D Poisson problem: Number of CG iterations where either: H h constant while # sub-domains is varied horizontal view Increasing mesh size H h while # sub-domains kept constant vertical view # sub-domains # processors sub-domains size M AS M SpAS AS M SpAS M AS M SpAS M M AS M SpAS The solved problem size vary from 1.1 up to 42.8 Millions of unknowns The number of iterations increases slightly when increasing # sub-domains This increase is less significant when the local mesh size H h grows With a few hundred processors those runs cannot be performed using a direct substructuring approach with current version of MUMPS

23 Numerical scalability 3D Discontinuous problem : Jumps in diffusion coefficient functions a() = b() = c():

24 Numerical scalability 3D Discontinuous problem : Jumps in diffusion coefficient functions a() = b() = c(): Number of CG iterations where either: H h constant while # sub-domains is varied horizontal view Increasing mesh size H h while # sub-domains kept constant vertical view # sub-domains # processors sub-domains size M AS M SpAS M AS M SpAS M AS M SpAS M AS M SpAS

25 Parallel performance 3D Discontinuous problem: Implementation details: Setup Schur: MUMPS Setup Precond: dense Schur (LAPACK)- sparse Schur (MUMPS) Target computer : System Xserve MAC G5 Parallel elapsed time: 10 3 processors H h vary ɛ = 10 4 Jumps in diffusion coefficient functions a() = b() = c(): Sub-domains size setup Schur setup Precond time per iter total # iter dense local Schur Precond M AS - sparse local Schur Precond M SpAS

26 Local data storage M AS vs M SpAS Memory behaviour M AS M SpAS Subdomains size ɛ = 10 5 ɛ = MB 7.5 MB (10%) 1.8 MB ( 5%) MB 12.7 MB (14%) 2.7 MB ( 3%) MB 19.4 MB (10%) 3.8 MB ( 2%) MB 28.6 MB ( 7%) 10.2 MB ( 2%)

27 Outline 1 General Framework 2 The direct substructuring approach 3 The iterative substructuring approach Description of the preconditioners Variant of Additive Shwarz preconditioner M AS M AS v.s. Neumann-Neumann 4 Parallel numerical experiments 2D experiments in semiconductor device modelling 3D experiments on the academic cube 5 Prospectives

28 Prospectives Objective Control the growth of iterations when increasing the # processors Various possibilities (future work) Numerical remedy: two-level preconditioner - Coarse space correction, ie solve a closed problem on a coarse space - Various choices for the coarse component (eg one d.o.f. per sub-domain) Computer Science remedy : several processors per sub-domain - two-level of parallelism - 2D cyclic data storage MUMPS is back!!! Final comments The advances in sparse directs techniques anables us to consider preconditioning techniques that were out of reach before

29 Prospectives Objective Control the growth of iterations when increasing the # processors Various possibilities (future work) Numerical remedy: two-level preconditioner - Coarse space correction, ie solve a closed problem on a coarse space - Various choices for the coarse component (eg one d.o.f. per sub-domain) Computer Science remedy : several processors per sub-domain - two-level of parallelism - 2D cyclic data storage MUMPS is back!!! Final comments The advances in sparse directs techniques anables us to consider preconditioning techniques that were out of reach before

30 Prospectives Objective Control the growth of iterations when increasing the # processors Various possibilities (future work) Numerical remedy: two-level preconditioner - Coarse space correction, ie solve a closed problem on a coarse space - Various choices for the coarse component (eg one d.o.f. per sub-domain) Computer Science remedy : several processors per sub-domain - two-level of parallelism - 2D cyclic data storage MUMPS is back!!! Final comments The advances in sparse directs techniques anables us to consider preconditioning techniques that were out of reach before

31 Credit to co-workers Co-workers L. M. Carvalho (UFRJ Brasil, mainly during PhD at CERFACS), A. Haidar (CERFACS), J. Koster (Norway ministery of research, mainly when at Parallab), P. Le Tallec (Ecole Polytechnique & Paris Dauphine, mainly when at Paris Dauphine & INRIA), J. C. Rioual (NEC, mainly during PhD at CERFACS), A. Marrocco (INRIA), G. Meurant (CEA), L. Watson (Virginia Tech). and of course to the MUMPS team

32 More information giraud Merci pour votre attention a final personal advert...

33 More information giraud Merci pour votre attention a final personal advert...

34