Parallel Implementation of BDDC for Mixed-Hybrid Formulation of Flow in Porous Media

Transcription

1 Parallel Implementation of BDDC for Mixed-Hybrid Formulation of Flow in Porous Media Jakub Šístek1 joint work with Jan Březina 2 and Bedřich Sousedík 3 1 Institute of Mathematics of the AS CR Nečas Center for Mathematical Modelling 2 Technical University of Liberec 3 University of Maryland, Baltimore County International Conference on Domain Decomposition Methods XXIII Jeju Island, Korea, July 7th, 215

2 Motivation Geoengineering simulations numerous examples of flow in porous media oil and gas reservoirs, pollutant transport, nuclear waste deposits,... in the, plans to build the long-term nuclear waste deposit by 265 currently seven candidate sites massive granite rock with cracks Source: Jakub Šístek BDDC for flows in porous media 2 / 31

3 Motivation Subsurface flow simulations 2+ years of development of simulation tools at TUL mixed-hybrid finite element method combined meshes of 3D, 2D and 1D elements need for robust scalable parallel solvers to handle finer models Jakub Šístek BDDC for flows in porous media 3 / 31

4 Governing equations Darcy law k 1 u + p = z in Ω u = f in Ω p = p N on Ω N u n = on Ω E Ω R 3, Ω = Ω N Ω E Ω N, Ω E... natural (Dirichlet) and essential (Neumann) b. c. u... velocity of the fluid p... pressure head k... tensor of the hydraulic conductivity (sym. pos. def.) z... third spatial coordinate p h = p + z... piezometric head for which u = k p h Jakub Šístek BDDC for flows in porous media 4 / 31

5 Mixed finite element method Raviart-Thomas (RT ) finite elements V H(Ω; div) = { v L 2 (Ω); v L 2 (Ω) and v n = on Ω E } Q L 2 (Ω) Mixed formulation Find a pair {u, p} V Q that satisfies Ω k 1 u v dx Ω p v dx = Ω N p N v n ds Ω v zdx, v V Ω q udx = fq dx, q Q Ω Jakub Šístek BDDC for flows in porous media 5 / 31

6 Mixed finite element method Raviart-Thomas (RT ) finite elements V H(Ω; div) = { v L 2 (Ω); v L 2 (Ω) and v n = on Ω E } Q L 2 (Ω) Mixed formulation Find a pair {u, p} V Q that satisfies Ω k 1 u v dx Ω p v dx = Ω N p N v n ds Ω v zdx, v V Ω q udx = fq dx, q Q Ω Jakub Šístek BDDC for flows in porous media 5 / 31

7 Mixed hybrid finite element method Space of Lagrange multipliers V i = { v H(T i ; div) : v RT (T i ) } V 1 = V 1 V N E Λ = { λ L 2 (F) : λ = v n F, v V } F... set of all faces of the elements in triangulation T Mixed hybrid formulation Find a triple {u, p, λ} V 1 Q Λ that satisfies [ k 1 T i i NE i=1 u v dx p v dx+ ] T i T i \ Ω λ(v n) T i ds = Ω N p N v n ds N E i=1 v T i z dx, v V N E [ i=1 q u dx ] T i ] = fq dx, Ω q Q =, µ Λ NE i=1 [ T i \ Ω µ(u n) T i ds Jakub Šístek BDDC for flows in porous media 6 / 31

8 Mixed hybrid finite element method Space of Lagrange multipliers V i = { v H(T i ; div) : v RT (T i ) } V 1 = V 1 V N E Λ = { λ L 2 (F) : λ = v n F, v V } F... set of all faces of the elements in triangulation T Mixed hybrid formulation Find a triple {u, p, λ} V 1 Q Λ that satisfies [ k 1 T i i NE i=1 u v dx p v dx+ ] T i T i \ Ω λ(v n) T i ds = Ω N p N v n ds N E i=1 v T i z dx, v V N E [ i=1 q u dx ] T i ] = fq dx, Ω q Q =, µ Λ NE i=1 [ T i \ Ω µ(u n) T i ds Jakub Šístek BDDC for flows in porous media 6 / 31

9 System of linear algebraic equations Saddle-point system A BT B T F B B F u p λ = g f (1) A... symmetric positive definite (s.p.d.), block-diagonal matrix with respect to elements [ ] B B =... full row rank if Ω N B F analysis e.g. in [Brezzi, Fortin (1991)], [Maryška, Rozložník, Tůma (2)], [Tu (27)],... problem (1) has a unique solution Jakub Šístek BDDC for flows in porous media 7 / 31

10 Modelling of cracks Combined meshes T 123 = T 1 T 2 T 3 T i d 1 F d d = 2, 3... spatial dimension System with fluxes k 1 u d d + p d = z δ d u d... flux volume per second per unit δ d... conversion to velocity in dimension d (δ 3 = 1, δ 2 is thickness of a fracture, δ 1 cross-section of a channel) Jakub Šístek BDDC for flows in porous media 8 / 31

11 Modelling of cracks Combined meshes T 123 = T 1 T 2 T 3 T i d 1 F d d = 2, 3... spatial dimension System with fluxes k 1 u d d + p d = z δ d u d... flux volume per second per unit δ d... conversion to velocity in dimension d (δ 3 = 1, δ 2 is thickness of a fracture, δ 1 cross-section of a channel) Jakub Šístek BDDC for flows in porous media 8 / 31

12 Coupling of mesh dimensions Introduce Robin (a.k.a. Newton) boundary conditions 3D 2D f 2 = δ 2 f 2 + u + 3 n+ + u 3 u + 3 n+ = σ 3 + (p+ 3 p 2) u 3 n = σ 3 (p 3 p 2) n 2D 1D σ +/ 3 >... transition coefficients on sides of a 2D element f 1 = δ 1 f 1 + k u k 2 n k u k 2 n k = σ k 2 (p k 2 p 1 ) σ k 2 >... transition coefficient from k-th 2D element to 1D channel Jakub Šístek BDDC for flows in porous media 9 / 31

13 Coupling of mesh dimensions Introduce Robin (a.k.a. Newton) boundary conditions 3D 2D f 2 = δ 2 f 2 + u + 3 n+ + u 3 u + 3 n+ = σ 3 + (p+ 3 p 2) u 3 n = σ 3 (p 3 p 2) n 2D 1D σ +/ 3 >... transition coefficients on sides of a 2D element f 1 = δ 1 f 1 + k u k 2 n k u k 2 n k = σ k 2 (p k 2 p 1 ) σ k 2 >... transition coefficient from k-th 2D element to 1D channel Jakub Šístek BDDC for flows in porous media 9 / 31

14 System of linear algebraic equations Saddle-point system with couplings A B T B T F B C C T F B F C F C u p λ = g f (2) A... symmetric positive definite (s.p.d.), block-diagonal matrix with respect to elements [ ] C C T C = F... symmetric positive semi-definite C F C [ ] B B =... generally no longer full row rank BF Jakub Šístek BDDC for flows in porous media 1 / 31

15 System of linear algebraic equations Theorem (Solvability of the saddle-point system) Let natural boundary conditions be prescribed at a certain part of the boundary, i.e. Ω N,d for at least one d {1, 2, 3}. Then the discrete mixed-hybrid problem (2) has a unique solution. details in [Šístek, Březina, Sousedík (215)] Jakub Šístek BDDC for flows in porous media 11 / 31

16 Iterative substructuring T 123 divided into N S substructures Ω i, i = 1,..., N S Γ... interface among substructures shared degrees of freedom Local problem on Ω i A i B it BF,I it BF,Γ it B i C i CF,I it C it F,Γ BF,I i CF,I i C II i C ΓI it BF,Γ i CF,Γ i C ΓI i C ΓΓ i u i p i λ i I λ i Γ = g i f i λ i Γ... Lagrange multipliers on Ωi Γ λ i I... Lagrange multipliers interior to Ω i u i, p i, λ i I... interior unknowns from substructuring view-point Λ Γ = Λ 1 Γ ΛN S Γ Λ Γ Λ Γ... subspace of Lagrange multipliers coinciding on Γ Jakub Šístek BDDC for flows in porous media 12 / 31

17 Iterative substructuring T 123 divided into N S substructures Ω i, i = 1,..., N S Γ... interface among substructures shared degrees of freedom Local problem on Ω i A i B it BF,I it BF,Γ it B i C i CF,I it C it F,Γ BF,I i CF,I i C II i C ΓI it BF,Γ i CF,Γ i C ΓI i C ΓΓ i u i p i λ i I λ i Γ = g i f i λ i Γ... Lagrange multipliers on Ωi Γ λ i I... Lagrange multipliers interior to Ω i u i, p i, λ i I... interior unknowns from substructuring view-point Λ Γ = Λ 1 Γ ΛN S Γ Λ Γ Λ Γ... subspace of Lagrange multipliers coinciding on Γ Jakub Šístek BDDC for flows in porous media 12 / 31

18 Implicit interface problem Substructure Schur complements S i : Λ i Γ Λ i Γ, i = 1,..., N S Action of S i on a given λ i Γ defined by A i B it BF,I it BF,Γ it B i C i CF,I it BF,I i CF,I i C II i C ΓI it BF,Γ i CF,Γ i C ΓI i C ΓΓ i C it F,Γ w i q i µ i I λ i Γ = S i λ i Γ Global Schur complement Ŝ : λ Γ Λ Γ Ŝλ Γ Λ Γ Formally assembled as N S Ŝ = R it S i R i i=1 R i mapping matrix, λ i Γ = Ri λ Γ, λ i Γ Λi Γ, λ Γ Λ Γ Jakub Šístek BDDC for flows in porous media 13 / 31

19 Implicit interface problem Substructure Schur complements S i : Λ i Γ Λ i Γ, i = 1,..., N S Action of S i on a given λ i Γ defined by A i B it BF,I it BF,Γ it B i C i CF,I it BF,I i CF,I i C II i C ΓI it BF,Γ i CF,Γ i C ΓI i C ΓΓ i C it F,Γ w i q i µ i I λ i Γ = S i λ i Γ Global Schur complement Ŝ : λ Γ Λ Γ Ŝλ Γ Λ Γ Formally assembled as N S Ŝ = R it S i R i i=1 R i mapping matrix, λ i Γ = Ri λ Γ, λ i Γ Λi Γ, λ Γ Λ Γ Jakub Šístek BDDC for flows in porous media 13 / 31

20 Implicit interface problem Interface problem Ŝλ Γ = b (3) reduced right-hand side [ b i = B i F,Γ C i F,Γ C i ΓI N S b = R it b i ] i=1 A i B it B it F,I B i C i C it F,I B i F,I C i F,I C i II 1 g i f i Jakub Šístek BDDC for flows in porous media 14 / 31

21 Implicit interface problem Interface problem Ŝλ Γ = b (3) reduced right-hand side [ b i = B i F,Γ C i F,Γ C i ΓI N S b = R it b i ] i=1 A i B it B it F,I B i C i C it F,I B i F,I C i F,I C i II 1 g i f i Jakub Šístek BDDC for flows in porous media 14 / 31

22 Implicit interface problem Theorem (Solvability of the interface problem) Let natural boundary conditions be prescribed at a certain part of the boundary, i.e. Ω N,d for at least one d {1, 2, 3}. Then the matrix Ŝ in (3) is symmetric and positive definite. using the Preconditioned Conjugate Gradient (PCG) method for solving (3) only applications of Ŝ needed performed by parallel solution of discrete Dirichlet problems on each substructure BDDC used as the preconditioner details in [Šístek, Březina, Sousedík (215)] Jakub Šístek BDDC for flows in porous media 15 / 31

23 BDDC preconditioner BDDC method for Darcy flow Balancing Domain Decomposition by Constraints [Dohrmann (23)] elasticity mixed FEM [Tu (25)], multilevel [Tu (211)], [Sousedík (213)] mixed-hybrid FEM [Tu (27)], without cracks, Lagrange multipliers introduced only on Γ different local problems define constraints enforcing continuity of functions from Λ Γ at coarse degrees of freedom among substructures space Λ Γ ΛΓ Λ Γ Λ Γ substructure faces arithmetic averages basic constraints edges may appear at intersections of 2D elements corners pointwise continuity not needed for RT elements but improve convergence for numerically difficult problems, selected by the face-based algorithm from [Šístek et al. (212)] Jakub Šístek BDDC for flows in porous media 16 / 31

24 BDDC set-up Algebraic coarse basis functions on Ω i Solve for multiple right-hand sides A i B it BF,I it BF,Γ it B i C i CF,I it CF,Γ it B F,I i CF,I i C II i C ΓI it BF,Γ i CF,Γ i C ΓI i C ΓΓ i D it D i X i Z i Φ i I Φ i Γ L i = I D i... matrix of coarse degree of freedom I... identity matrix Φ i Γ... coarse basis functions X i, Z i, Φ i I... auxiliary matrices not used further local coarse matrix SCC i = ΦiT Γ S i Φ it Γ = Li [Pultarová (212)] global coarse matrix S CC = N S i=1 RiT C S CC i Ri C RC i matrix relating local-to-global coarse degrees of freedom Jakub Šístek BDDC for flows in porous media 17 / 31

25 BDDC action Algorithm (BDDC preconditioner M BDDC : r Γ Λ Γ λ Γ Λ Γ ) 1 Solve the global coarse problem N S S CC η C = RC it ΦiT Γ W i R i r Γ i=1 2 Solve local Neumann problems A i B it BF,I it B it F,Γ B i C i CF,I it CF,Γ it B F,I i CF,I i C II i C ΓI it BF,Γ i CF,Γ i C ΓI i C ΓΓ i D it D i 3 Combine and average the corrections x i z i η i I η i Γ l i N S ( ) λ Γ = R it W i ηγ i + Φi Γ Ri C η C i=1 = W i R i r Γ W i... matrix of interface weights Jakub Šístek BDDC for flows in porous media 18 / 31

26 A note on scaling W i studied e.g. in [Klawonn, Rheinbach, Widlund (28)], [Čertíková, Šístek, Burda (213)], [Oh, Widlund, Dohrmann (TR213)],... Generalized scaling by diagonal stiffness Diagonal entry given by W i jj = C i ΓΓ,jj + 1 A i kk k(j)... the row in block A i of the element face to which the Lagrange multiplier λ i Γ,j belongs Jakub Šístek BDDC for flows in porous media 19 / 31

27 Parallel implementation Flow123d simulation of subsurface flow and pollution transport mixed-hybrid FEM open-source (GPL license) developed at TUL current version (15/3/ 15) object-oriented C++ code 1+ years of development 5 active developers lead developer J. Březina BDDCML equation solver Adaptive-Multilevel BDDC [Sousedík, Šístek, Mandel (213)] open-source (LGPL license) developed at IM AS CR current version 2.5 (8/6/ 15) Fortran 95 + MPI library 5+ years of development relies on MUMPS both serial and parallel bddcml.html Jakub Šístek BDDC for flows in porous media 2 / 31

28 Numerical results Fox Location: CTU Supercomputing Centre, Prague Architecture: SGI Altix UV Processor Type: Intel Xeon 2.67GHz Computing Cores: 72 RAM: 576 GB (8 GB/core) HECToR Location: EPCC, Edinburgh Architecture: Cray XE6 Processor Type: 16 core AMD Opteron 2.3GHz Interlagos Computing Cores: 9,112 Computing Nodes: 2816 RAM: 9 Tb TB (32 GB/node) access through PRACE-DECI graphics from Jakub Šístek BDDC for flows in porous media 21 / 31

29 Benchmark problems: Weak scaling on a square unit square domain, only 2D elements 2 64 cores of SGI Altix UV PCG tolerance r (k) / b < 1 7 pressure head with mesh velocity vectors N n n/n n Γ n f n c its. cond. time (sec) set-up PCG solve 2 27k 13k k 11k k 13k 1.2k M 111k 2.8k M 14k 5.9k M 113k 13.k Jakub Šístek BDDC for flows in porous media 22 / 31

30 Benchmark problems: Weak scaling on a cube unit cube domain, only 3D elements 2 64 cores of SGI Altix UV PCG tolerance r (k) / b < 1 7 pressure head with mesh velocity vectors N n n/n n Γ n f n c its. cond. time (sec) set-up PCG solve 2 217k 18k k 19k 2.3k k 118k 5.7k M 13k 12.8k M 16k 29.8k M 95k 59.6k Jakub Šístek BDDC for flows in porous media 23 / 31

31 Benchmark problems: Strong scaling test on a cube unit cube domain, 1D, 2D and 3D elements (k = νi, ν = 1, 1,.1) 2.1 million elements, 14.6 million degrees of freedom cores of HECToR PCG tolerance r (k) / b < 1 7 pressure head with mesh velocity vectors N n/n n Γ n f n c its. cond. time (sec) set-up PCG solve k 47k k 65k k 86k k 116k k 151k k 196k Jakub Šístek BDDC for flows in porous media 24 / 31

32 Benchmark problems: Strong scaling test on a cube time [s] set-up 1 PCG its. total optimal number of processors computational time speed-up set-up PCG its. total optimal number of processors parallel speed-up Speed-up on np processors computed as t np... time on np processors s np = 16 t 16 t np Jakub Šístek BDDC for flows in porous media 25 / 31

33 Geoengineering problem: Bedřichov tunnel experimental measurement site 2.1 km long tunnel with water pipes for the city of Liberec fractured granite rock data by courtesy of Dalibor Frydrych (TUL) 3D elements + 2D elements for cracks 1.1 million elements, 7.8 million degrees of freedom hydraulic conductivity k = νi, ν = ms 1 transition coefficient σ 3 = 1 s 1, thickness of cracks δ 2 = 1.1 m cores of HECToR PCG tolerance r (k) / b < m (length) 458 m (height) 1754 m (width) Jakub Šístek BDDC for flows in porous media 26 / 31

34 Bedřichov tunnel strong scaling test system of cracks division into 64 substructures detail of tunnel geometry enforced refinement at cracks Jakub Šístek BDDC for flows in porous media 27 / 31

35 Bedřichov tunnel strong scaling test N n/n n Γ n f n c its. cond. time (sec) set-up PCG solve k 2k k 28k k 45k k 72k k 11k k 155k time [s] set-up 1 PCG its. total optimal number of processors computational time speed-up set-up PCG its. total optimal number of processors parallel speed-up Jakub Šístek BDDC for flows in porous media 28 / 31

36 Bedřichov tunnel further experiments Comparison of different weighting options arithmetic avg. mod. ρ-scal. diagonal scal. N n Γ n c its. cond. its. cond. its. cond. 32 2k k k e k k k n/a 2.5e Effect of using corners without corners with corners N time (sec) time (sec) its. its. set-up PCG solve set-up PCG solve Jakub Šístek BDDC for flows in porous media 29 / 31

37 Bedřichov tunnel further experiments Comparison of different weighting options arithmetic avg. mod. ρ-scal. diagonal scal. N n Γ n c its. cond. its. cond. its. cond. 32 2k k k e k k k n/a 2.5e Effect of using corners without corners with corners N time (sec) time (sec) its. its. set-up PCG solve set-up PCG solve Jakub Šístek BDDC for flows in porous media 29 / 31

38 Conclusions Parallel BDDC solver for flows in porous media BDDC for Darcy flow with combined mesh dimensions connection of two existing codes Flow123d + BDDCML good scalability for single mesh dimension and 3D 2D couplings geoengineering problems challenging highly refined meshes, large hydraulic conductivities in cracks,... generalized averaging by diagonal stiffness on interface positive effect of using corners Future work analysis for 1D 2D 3D couplings application of Adaptive-Multilevel BDDC Šístek, J., Březina, J., Sousedík, B.: BDDC for mixed-hybrid formulation of flow in porous media with combined mesh dimensions. Numer. Linear Algebra Appl., 215, available online. Jakub Šístek BDDC for flows in porous media 3 / 31

39 Conclusions Parallel BDDC solver for flows in porous media BDDC for Darcy flow with combined mesh dimensions connection of two existing codes Flow123d + BDDCML good scalability for single mesh dimension and 3D 2D couplings geoengineering problems challenging highly refined meshes, large hydraulic conductivities in cracks,... generalized averaging by diagonal stiffness on interface positive effect of using corners Future work analysis for 1D 2D 3D couplings application of Adaptive-Multilevel BDDC Šístek, J., Březina, J., Sousedík, B.: BDDC for mixed-hybrid formulation of flow in porous media with combined mesh dimensions. Numer. Linear Algebra Appl., 215, available online. Jakub Šístek BDDC for flows in porous media 3 / 31

40 Contacts Thank you for your attention. Jakub Šístek Institute of Mathematics of the AS CR Nečas Center for Mathematical Modelling Prague, BDDCML library webpage Jakub Šístek BDDC for flows in porous media 31 / 31