On the Robustness and Prospects of Adaptive BDDC Methods for Finite Element Discretizations of Elliptic PDEs with High-Contrast Coefficients

Transcription

1 On the Robustness and Prospects of Adaptive BDDC Methods for Finite Element Discretizations of Elliptic PDEs with High-Contrast Coefficients Stefano Zampini Computer, Electrical and Mathematical Sciences & Engineering, Extreme Computing Research Center Saudi Arabia inty Logo Lock-up SHAXC KAUST, May 10th, 2016 S. Zampini BDDC 1 / 25

2 Motivation Iterative solution of large and sparse elliptic linear systems arising from FEM Ax = b, κ(a) = λ M(A) λ m(a) h 2 Preconditioned conjugate gradient M 1 Ax = M 1 b, κ(m 1 A) = λ M(M 1 A) λ m(m 1 A) Use domain decomposition preconditioners: Couple local Cholesky solvers through a suitable coarse space Obtain convergence rates which are independent of the number of subdomains, of the discretization space, and slowly deteriorates with the size of the subdomain problems. Accomodate heterogeneities in the coefficients of PDE. [A. Toselli and O. B. Widlund, Domain Decomposition Methods: algorithm and theory, 2005] S. Zampini BDDC 2 / 25

3 Introduction: a priori estimates The characterization of the BDDC method [Dohrmann, SISC 25, 2003] relies on the selection of primal (coarse) degrees of freedom the choice of an averaging procedure. Typical preconditioned spectra for elliptic problems (if a suitable primal space is found from the theory) κ(m 1 A) C(1 + log(h/h)) 2, H/h n loc with H maximum diameter of sudomains, h the mesh size and C independent on the number of subdomains and possibly independent on the coefficients of the PDE [Mandel, Dohrmann, Tezaur, Appl. Numer. Math. 54, 2005], [Li and Widlund, IJNME, 2008]. Dual of the FETI-DP method [C. Farhat et. al, IJNME 50, 2001]. S. Zampini BDDC 3 / 25

4 Introduction: some references Contact problems [P. Avery, G. Rebel, M. Lesoinne, C. Farhat. CMAME 93, 2004] Indefinite problems [J. Li, X. Tu. NLAA 16, 2009], [C. Farhat, J. Li ANM 54, 2005], [C. Farhat, J. Li, P. Avery IJNME 63, 2005] Porous media flow [X. Tu. ETNA 20, 2005] Electromagnetic problems [Y. J. Li, J. M. Jin IEEE Trans. Antennas Propag. 54, 2006] Incompressible Stokes [J. Li, O. B. Widlund. SISC 44, 2006] Linear elasticity [A. Klawonn, O. B. Widlund CPAM 59, 2006] Stokes problem [H. H. Kim, C. O. Lee, E. H. Park SISC 47, 2010] Stokes Darcy coupling [J. Galvis, M. Sarkis. CAMCS 5, 2010] Almost Incompressible Elasticity [L. Pavarino, O. B. Widlund, S. Z. SISC 32, 2010] S. Zampini BDDC 4 / 25

5 Introduction: some references Spectral Elements [L. F. Pavarino, CMAME 196, 2007] Lowest order Nédélec elements [A. Toselli, IMAJNA 26, 2006], [C. Dohrmann, O. B.Widlund, CPAM 2015], [S. Z. submitted 2016] Discontinuous Galerkin [M. Dryja, J. Galvis, M. Sarkis. J. Complexity 23, 2007] Mortar discretizations [H. H. Kim. SINUM 46, 2008], [H. H. Kim, M. Dryja, O. B. Widlund. SINUM 47, 2009] Reissner-Mindlin plates and Tu-Falk elements [J. H. Lee, SINUM, 2015] Naghdi shells and MITC elements [L. Beirão da Veiga, C. Chinosi, C. Lovadina, L. F. Pavarino. Comp. Struct. 102, 2012] IsoGeometric Analysis [L. Beirão da Veiga, L. Pavarino, S. Scacchi, O. B. Widlund, S.Z. SISC, 36, 2014] Lowest order Raviart-Thomas elements [D.-S. Oh, O. B. Widlund, S. Z., C. Dohrmann, TR, 2015] S. Zampini BDDC 5 / 25

6 Introduction: adaptive coarse space Convergence of DD algorithms usually deteriorates when jumps in the coefficients are not aligned with Γ. S. Zampini BDDC 6 / 25

7 Introduction: adaptive coarse space Convergence of DD algorithms usually deteriorates when jumps in the coefficients are not aligned with Γ. Coarse space has to be adaptively generated using generalized eigenvalue problems [J. Mandel and B. Sousedík, DD XVI, 2007], [B. Sousedík, J. Sístek, and J. Mandel, Computing, 95, 2013], [A. Klawonn, P. Radtke and O. Rheinbach, SINUM 53, 2015] [A. Klawonn, P. Kühn and O. Rheinbach, TR 2015], [H.H. Kim and E.T. Chung, SIAM J. Multiscale Model. Simul., 13, 2015, H.H. Kim, E.T. Chung and J. Wang CMAME, 2015], [C. Pechstein, C. R. Dohrmann, Seminar talk 2013], [J. Calvo and O. B. Widlund, submitted, 2015],[L. Beirão da Veiga, L. Pavarino, S. Scacchi, O. B. Widlund, S.Z. Submitted, 2015] S. Zampini BDDC 6 / 25

8 Introduction: adaptive coarse space Convergence of DD algorithms usually deteriorates when jumps in the coefficients are not aligned with Γ. Coarse space has to be adaptively generated using generalized eigenvalue problems [J. Mandel and B. Sousedík, DD XVI, 2007], [B. Sousedík, J. Sístek, and J. Mandel, Computing, 95, 2013], [A. Klawonn, P. Radtke and O. Rheinbach, SINUM 53, 2015] [A. Klawonn, P. Kühn and O. Rheinbach, TR 2015], [H.H. Kim and E.T. Chung, SIAM J. Multiscale Model. Simul., 13, 2015, H.H. Kim, E.T. Chung and J. Wang CMAME, 2015], [C. Pechstein, C. R. Dohrmann, Seminar talk 2013], [J. Calvo and O. B. Widlund, submitted, 2015],[L. Beirão da Veiga, L. Pavarino, S. Scacchi, O. B. Widlund, S.Z. Submitted, 2015] Given a user-defined threshold λ, we can guarantee that κ Cλ S. Zampini BDDC 6 / 25

9 Introduction: adaptive coarse space Convergence of DD algorithms usually deteriorates when jumps in the coefficients are not aligned with Γ. Coarse space has to be adaptively generated using generalized eigenvalue problems [J. Mandel and B. Sousedík, DD XVI, 2007], [B. Sousedík, J. Sístek, and J. Mandel, Computing, 95, 2013], [A. Klawonn, P. Radtke and O. Rheinbach, SINUM 53, 2015] [A. Klawonn, P. Kühn and O. Rheinbach, TR 2015], [H.H. Kim and E.T. Chung, SIAM J. Multiscale Model. Simul., 13, 2015, H.H. Kim, E.T. Chung and J. Wang CMAME, 2015], [C. Pechstein, C. R. Dohrmann, Seminar talk 2013], [J. Calvo and O. B. Widlund, submitted, 2015],[L. Beirão da Veiga, L. Pavarino, S. Scacchi, O. B. Widlund, S.Z. Submitted, 2015] Given a user-defined threshold λ, we can guarantee that κ Cλ Similar techniques have been also studied for enriching the coarse space of Schwarz algorithms and FETI [J. Galvis and Y. Efendiev, Multiscale Models Simul. 8, 2010], [N. Spillane and D. J. Rixen, IJNME, 2013], [N. Spillane et. al.,c.r. Math. Acad. Sci. Paris 2013], [N. Spillane et. al., Num. Math, 126, 2014] For multigrid, BootstrapAMG [A. Brandt et. al., SISC, 33, 2011]. S. Zampini BDDC 6 / 25

10 Introduction: on the road to exascale Adaptive BDDC satisfies 3 of the pillars for exascale algorithms [J. Dongarra, et al, Int. J. High Perf. Comp. Appl. 6, 2011] Reduces the global synchronization steps and MatVecs in Krylov methods Increases arithmetic intensity of the preconditioning step Increases concurrency of the preconditioning step S. Zampini BDDC 7 / 25

11 Introduction: on the road to exascale Adaptive BDDC satisfies 3 of the pillars for exascale algorithms [J. Dongarra, et al, Int. J. High Perf. Comp. Appl. 6, 2011] Reduces the global synchronization steps and MatVecs in Krylov methods Increases arithmetic intensity of the preconditioning step Increases concurrency of the preconditioning step Other key features of the algorithm Cholesky based Local and coarse problem additively combined (overlap) Multilevel extensions with high F/C coarsening ratios O(10 2 ) O(10 4 ) (however, convergence is exponentially dependent on the number of levels) S. Zampini BDDC 7 / 25

12 Introduction: on the road to exascale Adaptive BDDC satisfies 3 of the pillars for exascale algorithms [J. Dongarra, et al, Int. J. High Perf. Comp. Appl. 6, 2011] Reduces the global synchronization steps and MatVecs in Krylov methods Increases arithmetic intensity of the preconditioning step Increases concurrency of the preconditioning step Other key features of the algorithm Cholesky based Local and coarse problem additively combined (overlap) Multilevel extensions with high F/C coarsening ratios O(10 2 ) O(10 4 ) (however, convergence is exponentially dependent on the number of levels) Can it compete with optimally designed MG? S. Zampini BDDC 7 / 25

13 Introduction: on the road to exascale Adaptive BDDC satisfies 3 of the pillars for exascale algorithms [J. Dongarra, et al, Int. J. High Perf. Comp. Appl. 6, 2011] Reduces the global synchronization steps and MatVecs in Krylov methods Increases arithmetic intensity of the preconditioning step Increases concurrency of the preconditioning step Other key features of the algorithm Cholesky based Local and coarse problem additively combined (overlap) Multilevel extensions with high F/C coarsening ratios O(10 2 ) O(10 4 ) (however, convergence is exponentially dependent on the number of levels) Can it compete with optimally designed MG? Usually not, but there are problems where MG fails. S. Zampini BDDC 7 / 25

14 Non-overlapping DD: matrix subassembling Ω subdivided in N non-overlapping open subdomains N Ω = Ω i, Ω j Ω i =, Γ = Ω j Ω i. i=1 i j S. Zampini BDDC 8 / 25

15 Non-overlapping DD: matrix subassembling Ω subdivided in N non-overlapping open subdomains N Ω = Ω i, Ω j Ω i =, Γ = Ω j Ω i. i=1 i j A is never assembled explicitly; matrix vector product as A = R T A R, A = diag( ), with the matrix of the FEM problem on Ω i. S. Zampini BDDC 8 / 25

16 Non-overlapping DD: Block factorizations Block factorization for A based on the split Ŵ = ŴΓ W I [ ] [ ] [ A 1 I A 1 = A I Γ A 1 I I ΓΓ A T I ΓA 1 S 1 Γ I ΓΓ ] with S Γ = A ΓΓ A T I ΓA 1 A I Γ. S. Zampini BDDC 9 / 25

17 Non-overlapping DD: Block factorizations Block factorization for A based on the split Ŵ = ŴΓ W I [ ] [ ] [ A 1 I A 1 = A I Γ A 1 I I ΓΓ A T I ΓA 1 S 1 Γ with S Γ = A ΓΓ A T I ΓA 1 A I Γ. Block preconditioner κ(m 1 A) = κ(m 1 Γ S Γ) ] [ A 1 M 1 = [ I A 1 I ΓΓ A I Γ M 1 Γ ] [ I A T I ΓA 1 I ΓΓ I ΓΓ ] ] S. Zampini BDDC 9 / 25

18 Non-overlapping DD: Block factorizations Block factorization for A based on the split Ŵ = ŴΓ W I [ ] [ ] [ A 1 I A 1 = A I Γ A 1 I I ΓΓ A T I ΓA 1 S 1 Γ with S Γ = A ΓΓ A T I ΓA 1 A I Γ. Block preconditioner κ(m 1 A) = κ(m 1 Γ S Γ) ] [ A 1 M 1 = [ I A 1 I ΓΓ A I Γ M 1 Γ ] [ I A T I ΓA 1 I ΓΓ I ΓΓ ] ] DD solvers: BDD [J. Mandel, 1993] Algebraic solvers: ShyLU [S. Rajamanickam et al. IPDPS, 2012], STRUMPACK [P. Ghysels et. al. 2015] and many others (search the web for Schur hybrid solvers ) S. Zampini BDDC 9 / 25

19 BDDC: primal and dual spaces Basic idea of BDDC: instead of S Γ, invert S Γ, defined on a partially assembled space Ŵ W = W I W Γ W, WΓ = ŴΠ W Discontinuous on dual dofs, continuous on primals dofs Π. Primal vertices to prevent subdomains from floating. Additional primal dofs (functionals) can be needed for edges and faces of Γ to obtain an algorithmically scalable and robust method. S. Zampini BDDC 10 / 25

20 BDDC: preconditioner application M 1 Γ = R D,Γ S T 1 Γ R D,Γ, S. Zampini BDDC 11 / 25

21 BDDC: preconditioner application Block Cholesky S 1 Γ = R T Γ ( N [ i=1 M 1 Γ 0 R (i)t = R D,Γ S T 1 Γ R D,Γ, ] [ T I I ] 1 [ ] ) 0 R (i) R Γ + ΦS 1 ΠΠ ΦT. S. Zampini BDDC 11 / 25

22 BDDC: preconditioner application Block Cholesky Primal basis S 1 Γ = R T Γ ( N [ i=1 Φ = R T ΓΠ RT Γ M 1 Γ 0 R (i)t ( N [ i=1 = R D,Γ S T 1 Γ R D,Γ, ] [ T I 0 R (i)t I ] [ T I ] 1 [ ] ) 0 R (i) R Γ + ΦS 1 ΠΠ ΦT. I ] 1 [ I Π Π ] R (i) Π ). S. Zampini BDDC 11 / 25

23 BDDC: preconditioner application Block Cholesky Primal basis S 1 Γ = R T Γ Primal coarse problem ( N [ i=1 Φ = R T ΓΠ RT Γ M 1 Γ 0 R (i)t ( N [ i=1 S ΠΠ = R T Π S ΠΠ R Π, [ S (i) ΠΠ = A(i) ΠΠ T I Π = R D,Γ S T 1 Γ R D,Γ, ] [ T I 0 R (i)t I ] [ T I S ΠΠ = diag(s(i) ΠΠ ) ] [ T Π T I ] 1 [ ] ) 0 R (i) R Γ + ΦS 1 ΠΠ ΦT. I I [C. Dohrmann, SISC, 2003], [C. Dohrmann, NLAA 2007], [J. Li and O. B. Widlund 2008] ] 1 [ ] 1 [ I Π Π ] R (i) Π ] I Π Π ). S. Zampini BDDC 11 / 25

24 BDDC: preconditioner application Block Cholesky Primal basis S 1 Γ = R T Γ Primal coarse problem ( N [ i=1 Φ = R T ΓΠ RT Γ M 1 Γ 0 R (i)t ( N [ i=1 S ΠΠ = R T Π S ΠΠ R Π, [ S (i) ΠΠ = A(i) ΠΠ T I Π = R D,Γ S T 1 Γ R D,Γ, ] [ T I 0 R (i)t I ] [ T I S ΠΠ = diag(s(i) ΠΠ ) ] [ T Π T I ] 1 [ ] ) 0 R (i) R Γ + ΦS 1 ΠΠ ΦT. I I ] 1 [ ] 1 [ I Π Π ] R (i) Π ] I Π Π [C. Dohrmann, SISC, 2003], [C. Dohrmann, NLAA 2007], [J. Li and O. B. Widlund 2008] Coarse problem obtained by subassembling, we can rercurse multilevel BDDC ). S. Zampini BDDC 11 / 25

25 Introduction: current implementation More details on BDDC method and its implementation in PETSc: [S. Z. PCBDDC : a class of robust dual-primal methods in PETSc, SIAM CS&E special issue on software, 2015] S. Zampini BDDC 12 / 25

26 Introduction: current implementation More details on BDDC method and its implementation in PETSc: [S. Z. PCBDDC : a class of robust dual-primal methods in PETSc, SIAM CS&E special issue on software, 2015] 92% efficiency on 195K cores and 12.4B dofs (hexahedra, boxes, Poisson) Setup phase Solve phase Application phase time (s) time (s) time (s) K 100K 150K 200K 0 50K 100K 150K 200K Number of subdomains/processors 0 50K 100K 150K 200K S. Zampini BDDC 12 / 25

27 Introduction: current implementation More details on BDDC method and its implementation in PETSc: [S. Z. PCBDDC : a class of robust dual-primal methods in PETSc, SIAM CS&E special issue on software, 2015] 92% efficiency on 195K cores and 12.4B dofs (hexahedra, boxes, Poisson) Setup phase Solve phase Application phase time (s) time (s) time (s) K 100K 150K 200K 0 50K 100K 150K 200K Number of subdomains/processors 0 50K 100K 150K 200K 99% efficiency on 32K cores and 1B dofs (tetrahedra, ParMetis, H(div)) Setup phase 2-levels 3-levels Solve phase 2-levels 3-levels PCG iterations time (s) K 8K 16K 32K time (s) K 8K 16K 32K Number of subdomains/processors levels 3-levels 0 4K 8K 16K 32K S. Zampini BDDC 12 / 25

28 Introduction: current implementation More details on BDDC method and its implementation in PETSc: [S. Z. PCBDDC : a class of robust dual-primal methods in PETSc, SIAM CS&E special issue on software, 2015] 92% efficiency on 195K cores and 12.4B dofs (hexahedra, boxes, Poisson) Setup phase Solve phase Application phase time (s) time (s) time (s) K 100K 150K 200K 0 50K 100K 150K 200K Number of subdomains/processors 0 50K 100K 150K 200K 99% efficiency on 32K cores and 1B dofs (tetrahedra, ParMetis, H(div)) Setup phase 2-levels 3-levels Solve phase 2-levels 3-levels PCG iterations time (s) time (s) K 8K 16K 32K 2 0 4K 8K 16K 32K Number of subdomains/processors levels 3-levels 4K 8K 16K 32K Other implementations: Klawonn and Rheinbach [A. Klawonn and O. Rheinbach, ZAMM-Z, 90, 2010], [A. Klawonn, M. Lanser, and O. Rheinbach, SISC, 2014]. Badia s group in Barcelona [S. Badia, A. F. Martin and J. Principe, SISC, 2014]. BDDCML (J. Sístek, S. Zampini BDDC 12 / 25

29 BDDC: adaptive primal space With elliptic problems: κ C R D,Γ RT Γ 2 SΓ [J. Mandel, C. Dohrmann 2003] S. Zampini BDDC 13 / 25

30 BDDC: adaptive primal space With elliptic problems: κ C R D,Γ RT Γ 2 SΓ [J. Mandel, C. Dohrmann 2003] Key for efficiency, explicit knowledge of S (i) = ΓΓ A(i)T I Γ A(i) 1 I Γ S. Zampini BDDC 13 / 25

31 BDDC: adaptive primal space With elliptic problems: κ C R D,Γ RT Γ 2 SΓ [J. Mandel, C. Dohrmann 2003] Key for efficiency, explicit knowledge of S (i) = ΓΓ A(i)T I Γ A(i) 1 I Γ Solve for each subdomain face F (dense blocks) [C. Pechstein, C. R. Dohrmann, 2013] where ( S (i) F : (j) S F )φ = µ(s (i) F : S (j) F )φ A : B = (A 1 + B 1 ) 1, S (k) F = S (k) FF S (k) FF S (k) 1 F F S (k) F F S. Zampini BDDC 13 / 25

32 BDDC: adaptive primal space With elliptic problems: κ C R D,Γ RT Γ 2 SΓ [J. Mandel, C. Dohrmann 2003] Key for efficiency, explicit knowledge of S (i) = ΓΓ A(i)T I Γ A(i) 1 I Γ Solve for each subdomain face F (dense blocks) [C. Pechstein, C. R. Dohrmann, 2013] where ( S (i) F : (j) S F )φ = µ(s (i) F : S (j) F )φ A : B = (A 1 + B 1 ) 1, S (k) F = S (k) FF S (k) FF S (k) 1 F F S (k) F F Add to the primal space Ŵ Π {(S (i) F : S (j) F )φ k 1/µ k > λ} S. Zampini BDDC 13 / 25

33 BDDC: adaptive primal space With elliptic problems: κ C R D,Γ RT Γ 2 SΓ [J. Mandel, C. Dohrmann 2003] Key for efficiency, explicit knowledge of S (i) = ΓΓ A(i)T I Γ A(i) 1 I Γ Solve for each subdomain face F (dense blocks) [C. Pechstein, C. R. Dohrmann, 2013] where ( S (i) F : (j) S F )φ = µ(s (i) F : S (j) F )φ A : B = (A 1 + B 1 ) 1, S (k) F = S (k) FF S (k) FF S (k) 1 F F S (k) F F Add to the primal space Ŵ Π {(S (i) F : S (j) F )φ k 1/µ k > λ} More complicated formulas for edges in 3D lead to fully provable condition number bounds [A. Klawonn, M. Kühn and O. Rheinbach, 2015], [H.H. Kim, E.T. Chung and J. Wang, 2015], [J. Calvo, O. B. Widlund, 2015] S. Zampini BDDC 13 / 25

34 Adaptive primal space: implementation details Current BDDC code in PETSc 3.6 considers (S (i) E (S (i) F : S (j) E : S (j) (i) F )φ = µ( S F : F )φ : S (k) (i) (j) (k) E )φ = µ( S E : S E : S E )φ S (j) MUMPS or MKL PARDISO: factorize and compute S (i) at once Factorization of could be reused. S (i) F S (i) 1 F explicitly inverted. obtained by explicitly inverting S (i) S (i) 1 is reused to solve the substructure correction Nearest neighbor communication to assemble the sum of Schurs LAPACK used to solve each local GEP S. Zampini BDDC 14 / 25

35 Computational challenges in 3D (2D is easy) S. Zampini BDDC 15 / 25

36 Computational challenges in 3D (2D is easy) Costs of the factorizations: flops O(ni 2 ), memory O(n 4/3 i ) Size of largest frontal matrix With Schur Without Schur βx α,α = 0.66±0.05,β = 2.09±1.13 βx α,α = 0.67±0.02,β = 0.57± number of unknowns 10 S. Zampini BDDC 15 / 25

37 Computational challenges in 3D (2D is easy) Costs of the factorizations: flops O(ni 2 ), memory O(n 4/3 i ) Size of largest frontal matrix With Schur Without Schur βx α,α = 0.66±0.05,β = 2.09±1.13 βx α,α = 0.67±0.02,β = 0.57± number of unknowns 10 Explicit inversion of S (i) : O(n 3 Γ i ), n Γ i could be poorly load balanced procs S S 1 GEP min max min max min max Precompute basis functions O(n 4/3 i ) S. Zampini BDDC 15 / 25

38 Future directions S. Zampini BDDC 16 / 25

39 Future directions Exploit many-/multi- cores acceleration (does not solve the complexity problem) S. Zampini BDDC 16 / 25

40 Future directions Exploit many-/multi- cores acceleration (does not solve the complexity problem) Multiple subdomains per core flops O(n 2 i ), mem O(n 4/3 i ) flops O((n i /p) 2 ), mem O((n i /p) 4/3 ) S. Zampini BDDC 16 / 25

41 Future directions Exploit many-/multi- cores acceleration (does not solve the complexity problem) Multiple subdomains per core flops O(n 2 i ), mem O(n 4/3 i ) flops O((n i /p) 2 ), mem O((n i /p) 4/3 ) H-solvers for S (i) S. Zampini BDDC 16 / 25

42 Future directions Exploit many-/multi- cores acceleration (does not solve the complexity problem) Multiple subdomains per core flops O(n 2 i ), mem O(n 4/3 i ) flops O((n i /p) 2 ), mem O((n i /p) 4/3 ) H-solvers for S (i) Exact factorizations of 1 has to be spectrally equivalent. are not strictly required. However, the solver S. Zampini BDDC 16 / 25

43 Future directions Exploit many-/multi- cores acceleration (does not solve the complexity problem) Multiple subdomains per core flops O(n 2 i ), mem O(n 4/3 i ) flops O((n i /p) 2 ), mem O((n i /p) 4/3 ) H-solvers for S (i) Exact factorizations of 1 has to be spectrally equivalent. are not strictly required. However, the solver Can we partially factorize (with a given accuracy) and obtain 1 (possibly hierarchical) S (i)? and a S. Zampini BDDC 16 / 25

44 Numerical results: experimental setting FENICS library for finite elements: ParMETIS for mesh partitioning. PETSc-dev (branch stefano zampini/feature-pcbddc-saddlepoint) One MPI process/subdomain/core. Intel MKL as BLAS/LAPACK backend. MUMPS for local problems and Schur complements. PCG with random rhs, zero initial guesses, rtol 1.e-8. S. Zampini BDDC 17 / 25

45 BDDC for porous media flows Let U = {u H(div, Ω) : u n = 0 on Ω}, P = L 2 (Ω). Find (u, p) U P s.t. (v, q) U P u K(x) 1 v dx + p div v dx = 0, v U, Ω Ω div u q dx = gq, q P, Ω where K(x) uniformly positive definite tensor (mixed BC can be handled as well). Discretized with RT 0 or BDM 1 elements for velocities and C 0 polynomials elements (discontinuous) for pressures (LBB stable) Ω [Brezzi and Fortin, Mixed and Hybrid Finite elements methods, 1993] Joint work with X. Tu. [S. Z. and X. Tu, 2016, submitted] S. Zampini BDDC 18 / 25

46 BDDC for porous media flows: SPE10 SPE10 benchmark Ω = 1200ft 2200ft 170ft mesh , each hexahedron subdivided in 6 tetrahedra 6.7M cells dofs: 20.2M with RT 0, 45M with BDM 1. Diagonal permeability tensor s coefficients from SPE10 S. Zampini BDDC 19 / 25

47 BDDC for porous media flows: control SPE10 κ Condition number and number of iterations as a function of eigenvalue threshold λ and number of subdomains N. Left RT 0, right BDM 1 N λ = 10 λ = 5 λ = 2.5 λ = / / / / / / / / / / / / / / / /6 N λ = 10 λ = 5 λ = 2.5 λ = / / / / / / / / / / / / / / / /6 Coarsening ratios (F/C,Γ/C) as a function of eigenvalue threshold λ and number of subdomains N. Left RT 0, right BDM 1 N λ = 10 λ = 5 λ = 2.5 λ = /25 481/16 281/9 172/ /17 272/11 165/7 105/ /11 153/8 98/5 64/ /8 89/5 60/4 41/2 N λ = 10 λ = 5 λ = 2.5 λ = /61 795/35 408/18 230/ /42 458/25 243/13 140/ /29 263/18 146/10 86/ /20 155/13 90/7 54/4 S. Zampini BDDC 20 / 25

48 BDDC for porous media flows: timings SPE10 Setup/Solve times for 2-levels BDDC as a function of eigenvalue threshold λ and number of subdomains N. Left RT 0, right BDM 1 N λ = 10 λ = 5 λ = 2.5 λ = / / / / / / / / / / / / / / / /3.9 N λ = 10 λ = 5 λ = 2.5 λ = / / / / / / / / / / / / / / /5.9 **/** Setup/Solve times for 3-levels BDDC (coarsening ratio 16, coarse threshold 10, 2 Chebyshev its, eigs computed) as a function of eigenvalue threshold λ and number of subdomains N. Left RT 0, right BDM 1 N λ = 10 λ = 5 λ = 2.5 λ = / / / / / / / / / / / / / / / /1.8 N λ = 10 λ = 5 λ = 2.5 λ = / / / / / / / / / / / / / / / /2.3 S. Zampini BDDC 21 / 25

49 BDDC for Controlled Source EM Let U = {u H(curl, Ω) : u n = 0 on Ω}. Consider the following problem: find u U s.t. v U α u v dx + β u v dx = Ω Ω Ω f v Arises in time-domain quasi-static approximation of Maxwell s equations: α = µ 1 0 (µ 0 the magnetic permeability) β = σ/δ t, with σ > 0 conductivity (anisotropic case could be handled as well) δ t the time step. or in block preconditioning of frequency domain problems. Discretized with Nédélec elements on tetrahedra. [Brezzi and Fortin, Mixed and Hybrid Finite elements methods, 1993] [S. Z., submitted, 2015]. S. Zampini BDDC 22 / 25

50 BDDC for Controlled Source EM: 2D, higher order elements κ κ Threshold test, fixed mesh. Dofs from 800K to 11M (depending on the order), 64 subdomains. p=1 10 p=2 p=3 p=4 κ= µ µ p=1 10 p=2 p=3 p=4 κ= µ µ iterations iterations First kind 20 p=1 p=2 15 p=3 p= µ Second kind 20 p=1 p=2 15 p=3 p= µ W Π / W Γ W Π / W Γ 0.1 p= p=2 p=3 p= µ 0.1 p= p=2 p=3 p= µ S. Zampini BDDC 23 / 25

51 BDDC for Controlled Source EM: 2D, lowest order elements Threshold 1.01, fixed mesh. Dofs from 50K to 11M, 64 subdomains PASTIX MUMPS BDDC Setup phase PASTIX MUMPS BDDC Solve phase time(s) time (s) N N Comparision against MUMPS and PASTIX parallel Cholesky S. Zampini BDDC 24 / 25

52 BDDC for CSEM: reservoir test Domain 10km 8km 4km Mesh unif. refin. 18M dofs, λ =10 δ t = Ocean σ = 0.5 Background 0.02 Reservoir Figure taken from [Carcione, Progr. Electromagnetics Res. B, 26, 2010] N t set t sol it n I n Γ mem l mem c rel c eff n I, n Γ average number (K) of subdomain and local interface dofs, mem l and mem c in GB, eff = eqs (K) /core/second S. Zampini BDDC 25 / 25