Massively Parallel Domain Decomposition Preconditioner for the High-Order Galerkin Least Squares Finite Element Method

Size: px

Start display at page:

Download "Massively Parallel Domain Decomposition Preconditioner for the High-Order Galerkin Least Squares Finite Element Method"

Sydney Tate
5 years ago
Views:

1 Massively Parallel Domain Decomposition Preconditioner for the High-Order Galerkin Least Squares Finite Element Method Masayuki Yano Massachusetts Institute of Technology Department of Aeronautics and Astronautics January 26, 2009 M. Yano (MIT) Qualifier Examination January 26, / 67

2 Outline 1 Introduction 2 High-Order Galerkin Least Squares 3 Balancing Domain Decomposition by Constraint 4 Results 5 Conclusion and Future Work M. Yano (MIT) Qualifier Examination January 26, / 67

3 Outline 1 Introduction 2 High-Order Galerkin Least Squares 3 Balancing Domain Decomposition by Constraint 4 Results 5 Conclusion and Future Work M. Yano (MIT) Qualifier Examination January 26, / 67

4 Motivation Discretization Requirements Discretization should support High-Order: Simulation of unsteady flows with multiple scales benefits from higher-order methods Turbulence (DNS/LES) Acoustics / wave propagation Even in steady aerodynamics, high-order methods are more efficient in terms of error/dof [Barth, 1997][Venkatakrishnan, 2003] Unstructured Mesh: Handling complex geometries Selected Previous Work on High-Order Methods High-order schemes have been designed for various frameworks Finite difference [Lele, 1992] Finite volume [Barth, 1993][Wang, 2004] Finite element [Babuska, 1981][Patera, 1984] M. Yano (MIT) Qualifier Examination January 26, / 67

5 Motivation Example: High-Order Discretization Drag convergence for NACA 0012 [Fidkowski, 2005] Decay of turbulent kinetic energy in LES modeling [Kosovic, 2000] High-order schemes can significantly improve time to achieve engineering required accuracy. M. Yano (MIT) Qualifier Examination January 26, / 67

6 Motivation Massively Parallel Algorithms Modern supercomputers are massively parallel (1000+ procs). Parallelization is essential for problems of practical interests. Flow over large, complex geometries Unsteady simulations (e.g. DNS/LES) Trend of High Performance Computing in Past 15 Years FLOPS #1 #500 Average JNWT FUN3D Cart3D Processors Largest Smallest Average JNWT FUN3D Cart3D 10 8 Statistics from Top500.org Statistics from Top500.org M. Yano (MIT) Qualifier Examination January 26, / 67

7 Motivation High-Performance Computing in Aeronautics Typical large jobs remain in O(100) procs. Range of scales present in viscous flows requires implicit solvers The scalability of most of [CFD] codes tops out around 512 cpus... [Mavriplis, 2007] Exceptions: FUN3D inviscid matrix-free Newton-Krylov-Schwarz solver using 2048 procs [1999] Cart3D inviscid multigrid-accelerated Runge-Kutta solver using 2048 procs [2005] NSU3D RANS multigrid solver with line implicit solver in boundary layers using 2048 procs [2005] Need highly scalable implicit solver to take advantage of the future computers with 10,000+ procs. M. Yano (MIT) Qualifier Examination January 26, / 67

8 Objectives and Approaches Objectives 1 High-order discretization on unstructured mesh 2 Highly scalable solution algorithm for massively parallel systems Approaches 1 Galerkin Least Squares (GLS) Study behavior of high-order GLS discretization. Use h/p scaling artificial viscosity for subcell shock capturing. 2 Domain Decomposition (DD) Preconditioner Solve Schur complement system using preconditioned GMRES based on DD. Test the preconditioner for high-order discretization of advection-dominated flows. M. Yano (MIT) Qualifier Examination January 26, / 67

9 Outline 1 Introduction 2 High-Order Galerkin Least Squares 3 Balancing Domain Decomposition by Constraint 4 Results 5 Conclusion and Future Work M. Yano (MIT) Qualifier Examination January 26, / 67

10 Galerkin Least Squares Discretization Background Stabilized FEM for advection-dominated flows SUPG developed by Hughes [1982]; analyzed by Johnson [1984] SUPG extended to GLS by Hughes [1989] GLS Discretization for Advection-Diffusion Differential operator and bilinear form Lu β u (ν u) = f a(u,φ) = (β u,φ) Ω + (ν u, φ) Ω, Find u V s.t. a(u,φ) + (Lu,τLφ) Ω,Th = (f,φ) Ω + (f,τlφ) Ω,Th, where (, ) Ω,Th = K K dk and τ ( h, P e 1 h 2, P e 1 u,φ V H 1 (Ω) φ V. M. Yano (MIT) Qualifier Examination January 26, / 67

11 Stabilization Matrix Stability and A Priori Error Estimate for Hyperbolic Case (ν = 0) a(u, u) h β u 2 L 2 (Ω) h β (u uh ) L2 (Ω) + u u h L2 (Ω) Ch p+ 1 2 u H p+1 (Ω) High-Order Stabilization Matrix In order to keep cond(l stabilized) cond(l unstabilized ) = O(1) as p or h 0, τ h 2 /p 2, P e 1 One choice of τ in R d d+1 τ 1 = β ξ i p 2 d d + d i=1 i=1 j=1 ξ i x k ξ j x l ν kl M. Yano (MIT) Qualifier Examination January 26, / 67

12 Outline 1 Introduction 2 High-Order Galerkin Least Squares 3 Balancing Domain Decomposition by Constraint 4 Results 5 Conclusion and Future Work M. Yano (MIT) Qualifier Examination January 26, / 67

Domain Decomposition Preconditioners Motivation Performance of ILU degrades with number of processors even with line partitioning [Diosady, 2007].

13 Domain Decomposition Preconditioners Motivation Performance of ILU degrades with number of processors even with line partitioning [Diosady, 2007]. Data parallelism alone is not sufficient to obtain good performance in massively parallel environment (1000+ procs.) Selected Previous Work (Elliptic) Bourgat proposes Neumann-Neumann method for Schur complement system [1988]. Mandel improves scalability with Balancing DD [1993]. Dorhmann extends BDD to BDDC [2003]. M. Yano (MIT) Qualifier Examination January 26, / 67

14 Schur Complement System Discrete Harmonic Extension (Elliptic Equation) Decompose Ω into non-overlapping domains Ω i and define Γ i = Ω i \ Ω and Γ = N i=1 Γ i. Decompose V h H 1 (Ω) into V h (Ω \ Γ) = {v V h : v Γ = 0} V h (Γ) = {v V h : a(v,φ) = 0, φ V h (Ω \ Γ)} V h (Γ) is called the space of discrete harmonic extensions. Variational Problem Find u = ů + ū V h (Ω \ Γ) V h (Γ) s.t. a(ů,ψ) = (f,ψ) Ω, ψ V h (Ω \ Γ) a(ū,φ) = (f,φ) Ω, φ V h (Γ) Ω i Γ Ωk ů requires local Dirichlet solves Ω j ū requires global interface solve M. Yano (MIT) Qualifier Examination January 26, / 67

15 Schur Complement System Schur Complement Operator Schur complement operator S : V h (Γ) V h (Γ) defined by Sv,φ = a(v,φ), v,φ V h (Γ) where,, : W W R s.t. F, v F(v), F W, v W. Schur Complement Problem Find u V h (Γ) s.t. Su,φ = (f,φ) Ω, φ V h (Γ) Forming S is expensive; action of S on v V h (Γ) is computed via local Dirichlet solves and minimal communication. Problem solved using a Krylov space method (e.g. GMRES) Scalable preconditioner needed to accelerate convergence M. Yano (MIT) Qualifier Examination January 26, / 67

16 BDDC Spaces Features of BDDC Equipped with a coarse space that makes subdomain problems wellposed provides global communication Example of primal constrains Values at corners of Ω i Averages on the edges of Ω i V h (Γ) Spaces V h (Γ) = global discrete harmonic extension N i=1 V N i=1 h(γ i ) = collection of local discrete V h(γ i ) harmonic extensions (i.e. discontinuous across Γ) Ṽ h (Γ) = {v N i=1v h (Γ i ) : v continuous on primal constraints} Ṽ h (Γ) M. Yano (MIT) Qualifier Examination January 26, / 67

17 BDDC Spaces Dual and Primal Spaces Decompose Ṽh(Γ) into i V h, (Γ i ) and V h,π (Γ). Dual problems are decoupled Ṽ h (Γ) Primal problem has DOF of O(N) Dual Space ( ): V h, (Γ i ) = {v V h (Γ i ) : v = 0 on primal constraint} Primal Space (Π): V h,π (Γ) = {v Ṽh(Γ) : X Ni=1 ã i (v Ωi, φ Ωi ) = 0, φ N i=1v h, (Γ i )} M. Yano (MIT) Qualifier Examination January 26, / 67

18 BDDC Preconditioner Primal and Dual Schur Complement Primal Schur complement: S Π : V h,π (Γ) V h,π (Γ) S Π v Π,φ Π = N i=1 ã i (v Ωi,φ Ωi ), v Π,φ Π V h,π (Γ) Local dual Schur complement: S,i : V h, (Γ i ) V h, (Γ i) S,i v,i,φ,i = ã i (v,i,φ,i ) v,i,φ,i V h, (Γ i ) BDDC Preconditioner M 1 BDDC = N i=1 R T D,i (T sub,i + T coarse )R D,i where T sub,i = R T Γ,i S 1,i R Γ,i and T coarse = R T ΓΠ S 1 Π R ΓΠ Condition Number Estimate for Coercive, Symmetric a(, ) κ(m 1 BDDC S) C(1 + log2 (H/h)) M. Yano (MIT) Qualifier Examination January 26, / 67

19 Robin-Robin Interface Condition (IC) Background Adaptation of N-N condition to nonsymmetric bilinear form Proposed by Achdou for BDD [1997] Applied to FETI [Toselli, 2001] and BDDC [Tu, 2008] Robin-Robin IC Subtract Γ i 1 2 β ˆn iuφ from interfaces. Modified local bilinear form is ã i (u,φ) = (ν u, φ) Ωi (β u,φ) Ω i 1 2 (β φ, u) Ω i (u, vβ ˆn) Ω i Ω N ã i (, ) is positive in local space V h (Ω i ): ã i (u, u) ν u H 1 (Ω i ). Resulting IC on Γ i is Robin type ( ν u 1 2 β) ˆn = 0 on Γ i M. Yano (MIT) Qualifier Examination January 26, / 67

20 Outline 1 Introduction 2 High-Order Galerkin Least Squares 3 Balancing Domain Decomposition by Constraint 4 Results 5 Conclusion and Future Work M. Yano (MIT) Qualifier Examination January 26, / 67

Poisson Equation Poisson Equation Objectives Test scaling

different primal constraints Solution Partitioning

2 0.4 0.6 0.8 1 0 0 0.2 0.4 0.6 0.8 1 M.

21 Poisson Equation Poisson Equation Objectives Test scaling with : Number of subdomains Size of subdomain p for different primal constraints Solution Partitioning Examples M. Yano (MIT) Qualifier Examination January 26, / 67

22 Poisson Equation Scaling with Number of Subdomains and Subdomain Size GMRES iteration is independent of number of subdomains and size of subdomain (p = 1). Corner + state average (C+SA) primal constraint requires approx. half the iterations of just corner constraint (C). GMRES iterations x 4 Subdomain (C) 8 x 8 Subdomain (C) 4 x 4 Subdomain (C+SA) 8 x 8 Subdomain (C+SA) Number of Subdomains GMRES iterations Subdomains (C) 16 Subdomains (C) 4 Subdomains (C+SA) 16 Subdomains (C+SA) (DOF per subdomain) 1/2 M. Yano (MIT) Qualifier Examination January 26, / 67

23 Poisson Equation Scaling with p Poisson equation solved on unstructured mesh 8000 elem. Number of iterations nearly independent of p when C+SA primal constraints are used. 40 Size of Primal Problem Corners only: N subdomain Corners + Edge State Average: 3 N subdomain GMRES iterations p = 1 (C) p = 4 (C) 5 p = 1 (C+SA) p = 4 (C+SA) Number of Subdomains M. Yano (MIT) Qualifier Examination January 26, / 67

h 2 /p 2 ) Solution (ν = 10 3 ) Anisotropic Mesh and Partitioning 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.

24 Advection Diffusion: Boundary Layer Boundary Layer Equation Objectives Study effect of: interface conditions (Neumann-Neumann vs. Robin-Robin) τ scaling (h 2 vs. h 2 /p 2 ) Solution (ν = 10 3 ) Anisotropic Mesh and Partitioning M. Yano (MIT) Qualifier Examination January 26, / 67

25 Advection Diffusion: Boundary Layer Interface Conditions (IC) Robin-Robin IC performs significantly better than Neumann-Neumann IC in advection dominated cases. Interface conditions are identical as ν N N (p=1) R R (p=1) N N (p=1) R R (p=1) GMRES iterations advection diffusion ν GMRES iterations advection diffusion ν N subdomain = 16 N subdomain = 64 M. Yano (MIT) Qualifier Examination January 26, / 67

26 Advection Diffusion: Boundary Layer τ Matrix Scaling Performance of preconditioner degrades for p > 1 if τ h 2 in diffusion dominated cases. With τ h 2 /p 2, the preconditioner performs similar to p = 1 case. GMRES iterations p=1 p=3, τ v ~h 2 p=3, τ v ~h 2 /p 2 5 advection diffusion ν N subdomain = 64 M. Yano (MIT) Qualifier Examination January 26, / 67

27 Outline 1 Introduction 2 High-Order Galerkin Least Squares 3 Balancing Domain Decomposition by Constraint 4 Results 5 Conclusion and Future Work M. Yano (MIT) Qualifier Examination January 26, / 67

28 Conclusion and Future Work Conclusion BDDC preconditioner shows good scalability for: Both diffusion-dominated and advection-dominated flows with Robin-Robin interface condition All ranges of interpolation order p with proper choices of τ and primal constraints Future Work Extension to 3D Inexact solvers for subdomain problems Complex geometries and highly anisotropic mesh M. Yano (MIT) Qualifier Examination January 26, / 67

29 Questions

30 Supplemental Slides

31 Instability of the Standard Galerkin Method Discretization Advection-diffusion equation Lu β u (ν u) = f 1D Boundary Layer Galerkin: Find u V H 1 (Ω) s.t. a(u,φ) = (f,φ) Ω, φ V where a(u,φ) = Ω φβ u + ν φ u P e = βh 2ν = 1/4 1.8 Stability and A Priori Error Estimate a(u, u) ν u 2 L 2 (Ω) = ν u 2 H 1 (Ω) ν u u h H 1 (Ω) Chp u H p+1 (Ω) Not coercive in H 1 (Ω) as ν 0. P e = 5 M. Yano (MIT) Qualifier Examination January 26, / 67

32 Stabilized Finite Element Methods Stabilized Methods for Advection-Dominated Flows Streamline-Upwind Petrov-Galerkin (SUPG) [Hughes 1982] Galerkin Least-Squares (GLS) [Hughes 1989] Residual Free Bubbles (RFB) [Brezzi 1994] Variational Multiscale based on local Green s function [Hughes 1995] M. Yano (MIT) Qualifier Examination January 26, / 67

33 Finite Element vs. Finite Volume Advantages of Finite Element Method Maintains element-wise compact stencil for high-order discretization. Ease of boundary condition treatment. Reduces communication volume for DD. Straightfoward treatment of elliptic operator. Rigorous mathematical framework for a priori error estimation and DD convergence estimation. Disadvantages of Finite Element Method Requires stabilization term for H 1 (Ω) stability. DOF increases with the solution order. M. Yano (MIT) Qualifier Examination January 26, / 67

The continuity constraint must be enforced on nodal and edge basis during

34 High-Order Galerkin Basis Basis Function Types Three types of basis functions (in 2D): Node, Edge, Element. Basis type defined by its support. The continuity constraint must be enforced on nodal and edge basis during assembly. Node Edge Element M. Yano (MIT) Qualifier Examination January 26, / 67

35 High-Order Galerkin: Matrix Storage Object-based Block Storage Jacobian stored blockwise, with the varying size of blocks Matrix converted to CRS format before solved with UMFPACK Jacobian of 3 3 Mesh (18 elem, p = 5, arbitrary basis) nz = 7186 Galerkin M. Yano (MIT) nz = DG (with compact lifting) Qualifier Examination January 26, / 67

36 Comparison with Discontinuous Galerkin Features (common to DG) High-order discretization on unstructured mesh H 1 (Ω) stability for advection-dominated flows Advantages (compared to DG) Straightforward treatment of elliptic operators (no lifting required) Fewer DOF required to represent a solution in H 1 (Ω) Disadvantages Potentially expensive calculation of stabilization terms Absence of block-wise compact stencil (more complex preconditioning strategy required) M. Yano (MIT) Qualifier Examination January 26, / 67

37 High-Order SUPG vs. GLS p-refinement with Traditional τ If h ν/ β, SUPG p-refinement fails to converge. Asymptotic convergence rates as h 0 are identical ( e H 1 (Ω) hp ) p=1 p=3 0.2 p=5 p=7 exact p=1 p=3 0.2 p=5 p=7 exact SUPG GLS Solution to advection-diffusion equation (h = 0.1, ν/ β = 0.01) M. Yano (MIT) Qualifier Examination January 26, / 67

38 High-Order SUPG vs. GLS p-refinement with High-Order Modified τ With the high-order modified τ, both SUPG and GLS p-refinement converge to the exact solution. GLS converges much rapidly than SUPG with p p=1 p=3 0.2 p=5 p=7 exact p=1 p=3 0.2 p=5 p=7 exact SUPG GLS Solution to advection-diffusion equation (h = 0.1, ν/ β = 0.01) M. Yano (MIT) Qualifier Examination January 26, / 67

39 Shock Capturing Background Artificial viscosity can regularize underresolved features. Method applied to DG by Persson [2006] and Barter [2007] to achieve subcell shock capturing. Resolution Indicator and h/p-scaling Artificial Viscosity high-order resolution indicator based on orthogonal polynomial expansion [Persson, 2006] S e = (u Πp 1 u, u Π p 1 u) K (u, u) K, u P p (K) where Π p 1 is the L 2 (K) projection onto P p 1 (K). If log 10 (S e ) s 0, add piecewise constant viscosity ν h/p to the element. M. Yano (MIT) Qualifier Examination January 26, / 67

( ) 1 x 2 u2 + u y + (ν artificial(u) u) = 0 1.5 1 0.5 0 0 0.

40 Shock Capturing: Result Burger s Equation Burger s equation solved on 450 element mesh. ( ) 1 x 2 u2 + u y + (ν artificial(u) u) = Solution (p = 5) Resolution Indicator M. Yano (MIT) Qualifier Examination January 26, / 67

ÃiV,xi = 0 where A 0 is SPD and A i is symmetric (K ij is SPSD for N-S).

41 Euler Equation: Gaussian Bump Formulation Symmetric form using Hughes entropy variables [1986] ( s + γ + 1 V = ρe ) γ 1 p, ρu i p, ρ p The governing equation becomes Ã 0 V,t + ÃiV,xi = 0 where A 0 is SPD and A i is symmetric (K ij is SPSD for N-S). Gaussian Bump: Pressure and Partitioning M. Yano (MIT) Qualifier Examination January 26, / 67

42 Euler Equation: Gaussian Bump Euler Equation BDDC with R-R IC performs well for Euler equation. With N-N IC, 400+ GMRES iteration required for 32 subdomains. 60 GMRES iterations elem/subdom (p=1) elem/subdom (p=3) 80 elem/subdom (p=1) Number of Subdomains Average of GMRES iterations required for last three Newton steps M. Yano (MIT) Qualifier Examination January 26, / 67

43 Parallel Scalability of ILU Scalability of ILU for DG Discretization [Diosady 2007] Navier-Stokes equation Partitioning based on line connectivity Line-ordered ILU with p-multigrid within each subdomain 600 Linear Iterations vs. Number of Processors None Lines 16 Speed Up vs Number of Processors None Lines LinearIterations 300 Speedup Number of Processors Number of Processors Iterations vs. Processors Parallel Speed-Up M. Yano (MIT) Qualifier Examination January 26, / 67

44 Schur Complement for Non-Self-Adjoint Operator Discrete Harmonic Extension (Non-Self-Adjoint) Decompose V h H 1 (Ω) into V h (Ω \ Γ) = {v V h : v Γ = 0} V h (Γ) = {v V h : a(v,φ) = 0, φ V h (Ω \ Γ)} Vh (Γ) = {ψ V h : a(w,ψ) = 0, w V h (Ω \ Γ)} Find u = ů + ū V h (Ω \ Γ) V h (Γ) s.t. a(ů,φ) = (f,φ) Ω, φ V h (Ω \ Γ) a(ū,ψ) = (f,ψ) Ω, ψ Vh (Γ) Schur Complement Operator Schur complement S : V h (Γ) V h (Γ) s.t. Sv,ψ = a(v,ψ), v V h (Γ), ψ V h (Γ) M. Yano (MIT) Qualifier Examination January 26, / 67

45 Neumann-Neumann Preconditioner Local Spaces and Local Schur Complement Decompose V h (Ω i ) H 1 (Ω i ) of Ω i into V h (Ω i \ Γ i ) = {v i V h (Ω i ) : v Ωi = 0} V h (Γ i ) = {v i V h (Ω i ) : ã i (v i, φ) = 0, φ V h (Ω i \ Γ i )} Local Schur complement: S i : V h (Γ i ) V h (Γ i ) S i v i,φ i = ã i (v i,φ i ), v i,φ i V h (Γ i ) V h (Γ) Interpolation and Weighting Function Interpolation: R T i : V h (Γ i ) V h (Γ) s.t. N i=1 RT i V h (Γ i ) = V h (Γ) Weighting Function: D i V h (Γ i ) s.t. D i = 1/(# of Ω i sharing DOF on Γ) N i=1 V h(γ i ) Note, v = N i=1 R T i D i v Ωi, v V h (Γ) M. Yano (MIT) Qualifier Examination January 26, / 67

46 Neumann-Neumann Preconditioner Neumann-Neumann (N-N) Preconditioner Precondition S by applying S 1 i : V (Γ i ) V(Γ i ) on each Ω i and interpolate the result back to V(Γ) M 1 NN = N i=1 Ri T D i S 1 D i R i For ã i (u,φ) = ( u, φ) Ωi, application of S 1 i corresponds to solving a local problem with Neumann interface condition on Γ i ( u) ˆn = 0 on Γ i i Condition Number Estimate for Coercive, Symmetric a(, ) κ(m 1 NN S) C H2(1 + log(h/h))2 M. Yano (MIT) Qualifier Examination January 26, / 67

47 Balancing Domain Decompositions Problems of Neumann-Neumann Preconditioner Local Schur complement S i may be singular Limited scalability due to lack of coarse space Coarse Space Introduce coarse space V h,0 that makes subdomain problems well-posed provides global communication DOF coarse = O(N) Examples of DD with a Coarse Space BDD: Balancing Domain Decomposition [Mandel, 1993] BDDC: BDD by Constraints [Dohrmann, 2003] M. Yano (MIT) Qualifier Examination January 26, / 67

48 Balancing Domain Decomposition Balancing Domain Decomposition Define a coarse space: V h,0 (Ω) = span { } Ri T δ i, Kernel(S i ) V h,0 Define projection P 0 : V h (Ω) V h,0 (Ω), P 0 = R0 T S 1 0 R 0S Apply Neumann-Neumann preconditioner to balanced problem M 1 BDD = RT 0 S 1 0 R 0 + (I P 0 ) Examples of DD with a Coarse Space ( Ni=1 Ri T D i S 1 i D i R i S ) (I P 0 ) BDD: Balancing Domain Decomposition [Mandel, 1993] V h,0 = N i=1 Kernel(S i). Apply multiplicative coarse correction such that residual of subdomain problem is in range(s i ). (i.e. balanced problem) BDDC: BDD by Constraints [Dohrmann 2003] V h,0 = {Ψ i }, harmonic extensions satisfying primal constraints. Solve additive coarse and constrained subdomain problems. M. Yano (MIT) Qualifier Examination January 26, / 67

49 Comparison of Primal and Dual Substructuring Methods Primal Type Preconditions the Schur complement system by solving Neumann problems using the flux jumps. Neumann-Neumann method [Bourgat 1988] Balancing Domain Decomposition (BDD) [Mandel, 1993] BDD by Constraints (BDDC) [Dohrmann 2003] Dual Type Preconditions the flux equations by solving Dirichlet problems using function jumps. Finite Element Tearing and Interconnecting (FETI) [Farhat 1991] Dual-Primal FETI (FETI-DP) [Farhat 2001] Spectrum of BDDC and FETI-DP preconditioned operators are identical [Mandel 2005][Li 2005] M. Yano (MIT) Qualifier Examination January 26, / 67

50 BDDC Spaces Partially Assembled Spaces Collection of V h (Γ i ): W h N i=1 V h(γ i ) V h (Γ) Fully assembled FE space: Ŵ h V h (Γ) = N i=1 RT i V h (Γ i ) Partially assembled space: Wh V h,0 (Γ) + N i=1 RT i RΓ,i T V h(γ i ) Idea: BDDC preconditioner applies inverse of S : Wh W h to Shucr complement system. Ŵ h Wh W h M. Yano (MIT) Qualifier Examination January 26, / 67

51 Schur Complement System: Matrix Form Schur Complement Decompose local stiffness matrix and load vector into ( ) ( A (i) = A (i) II A (i) ΓI A (i) IΓ A (i) ΓΓ Schur complement system is given by Ŝu Γ = ĝ and f (i) = f (i) I f (i) Γ ) where Ŝ = ĝ = N R (j),t S (j) R (j), j=1 N R (j),t g (j), j=1 ( S (j) = A (j) ΓΓ A(j) ΓI g (j) = f (j) Γ ( A(j) ΓI A (j) II A (j) II ) 1 f (j) ) 1 A (j) IΓ I M. Yano (MIT) Qualifier Examination January 26, / 67

52 Application of Schur Complement Application of S (j) Calculation of u (j) Γ = S(j) v (j) corresponds to solving a local Dirichlet problem: Application of ( S (j)) 1 ( S (j) v (j) = A (j) ΓΓ v(j) A (j) ΓI Calculation of u (j) Γ = (S(j) ) 1 v (j) Γ Neumann problem: ( A (j) II A (j) ΓI A (j) IΓ A (j) ΓΓ A (j) II ) 1 A (j) IΓ v(j) corresponds to solving a local )( u (j) I u (j) Γ ) = ( 0 v (j) Γ ) M. Yano (MIT) Qualifier Examination January 26, / 67

53 BDDC: Coarse Correction Coarse Correction Coarse-level correction operator is given by T coarse = Ψ(Ψ T SΨ) 1 Ψ T where Ψ ( Ψ (1),T,,Ψ (N),T) T R DOF( V h (Γ i )) N primal Coarse Basis The local coarse basis, Ψ (i) is the harmonic extension to Ω i that satisfies the primal constraints, i.e. ( S (i) C (i),t )( ) ( ) Ψ (i) 0 C (i) 0 Λ (i) = R (i) Π where C (i) enforces primal constraints and Λ (i) is Lagrange multiplier. M. Yano (MIT) Qualifier Examination January 26, / 67

54 BDDC: Subdomain Correction Subdomain Correction Subdomain correction is given by N ( ) ( T sub = R (i),t S (i) C (i),t Γ 0 C (i) 0 i=1 ) 1 ( R (i) Γ 0 ) Provides subdomain corrections for which all coarse level, primal variables vanish. Primal constraints ensure the local problem is invertible. M. Yano (MIT) Qualifier Examination January 26, / 67

55 BDDC: Change of Basis Idea [Klawonn, 2004] Change basis functions on interface to make primal constraints explicit. Removes the need for the Lagrange multipliers as primal continuity constraints are satisfied by construction. Example: State Average Change of basis ψ k ψ l such that Dual Variables: edge ψ l(ξ)dξ = 0, l = 1,...,n 1 Primal Variable: ψ n (ξ) = 1, ξ edge Let T COB be the weighting matrix that maps from ψ to φ, then new stiffness matrix is A ψ,ij = a(ψ i,ψ j ) = (T T COB A φt COB ) i,j M. Yano (MIT) Qualifier Examination January 26, / 67

56 Abstract Additive Schwarz Preconditioner Assumption Consider symmetric, coercive a(, ) : V V and a i (, ) : V i V i. Assume C 0, ω, E = (ǫ ij ) N i,j=1 such that Stable Decomposition Local Stability v V, v i V i : N i=0 a i (v i, v i ) C 0 a(v, v) i = 1,...,N, v i V i : a(v i, v i ) ωa i (v i, v i ) Strengthened Cauchy-Shwarz Inequality i, j = 1,..., N, u i V i, v j V j : a(u i, v j ) ǫ ij a(u i, u i )a(v j, v j ) Conditioner Number Estimate [Dryja, 1995] κ C 0 ω (1 + ρ(e)) M. Yano (MIT) Qualifier Examination January 26, / 67

57 BDDC Condition Number Estimate Rayleigh Quotient Applying Rayleigh quotient formula to the inner product M BDDC,, ( ) λ min M 1 BDDC S = min v 0 λ max ( M 1 BDDC S ) = max v 0 min v=r T D RT ΓΠ v Π+ P N i=1 RT D,i RT Γ,i v,i v Π V h,π (Γ),v,i V h, (Γ i ) Sv, v ( S Π v Π, v Π + ) N i=1 S,iv,i, v i Sv, v ( min v=r T D RT ΓΠ v Π+ P N i=1 RT D,i RT Γ,i v S,i Π v Π, v Π + ) N i=1 S,iv,i, v i v Π V h,π (Γ),v,i V h, (Γ i ) M. Yano (MIT) Qualifier Examination January 26, / 67

58 Eigenvalues: Lower Bound Schur Complement on Ṽh(Γ) Sv,φ = N i=1 ã i (v Ωi,φ Ωi ), v,φ Ṽh(Ω) Derivation S Π v Π, v Π + N i=1 S,i v,i, v,i = Sv Π, v Π + N i=1 ã i (v,i, v,i ) Sv, v min v=r T D RT ΓΠ v Π+ P N i=1 RT D,i RT Γ,i v,i v Π V h,π (Γ),v,i V h, (Γ i ) = Sv Π, v Π + Sv, v = S(v Π + v ), (v Π + v ) = Sv, v ( ) N S Π v Π, v Π + S,i v,i, v i λ min ( M 1 BDDC S ) 1 i=1 M. Yano (MIT) Qualifier Examination January 26, / 67

59 Eigenvalues: Upper Bound Derivation Sv, v 2[ SRD T RT ΓΠ v Π, RD T RT ΓΠ v Π + S( N i=1 RD,iR T Γ,iv T,i ), ( N i=k RD,kR T Γ,kv T,k ) ] SRDR T ΓΠv T Π, RDR T ΓΠv T Π + N i=1 S(RD,iR T Γ,iv T,i ), (RD,iR T Γ,iv T,i ) For symmetric, coercive bilinear form with p = 1, SRD T RT ΓΠ v Π, RD T RT ΓΠ v Π (1 + log 2 (H/h)) S Π v Π, v Π S(RD,i T RT Γ,i v,i), (RD,i T RT Γ,i v,i) (1 + log 2 (H/h)) S,i v,i, v,i Thus, ««Sv, v 1 + log 2 H min h v=rd T RT ΓΠ v Π+ P N i=1 RD,i T RT Γ,i v,i v Π V h,π (Γ),v,i V h, (Γ i )! NX S Π v Π, v Π + S,i v,i, v i i=1 ( ) ( ) λ max M 1 BDDC S C 1 + log 2 (H/h) M. Yano (MIT) Qualifier Examination January 26, / 67

60 BDDC: GMRES Convergence Estimate GMRES Residual Bound [Eisenstat, 1983] Let c anc C 2 be such that c 0 u, u u, Tu Tu, Tu C0 2 u, u Then, ( ) m/2 r m r o 1 c2 0 C0 2 Advection-Diffusion Convergence Estimate [Tu, 2008] With state and flux average as primal constraints, BDDC (R-R) satisfies c 0 = 1 CH(H/h)(1 + log(h/h)) C 0 = C(1 + log(h/h)) 4 M. Yano (MIT) Qualifier Examination January 26, / 67

61 Nonlinear Equation Solution Scheme Discrete form using Galerkin Method M h du h dt + R h (U h (t)) = 0 Time Stepping Scheme U m+1 h For steady problems t. ( 1 = U m h t M h + R ) 1 h R h (U m h U ) h Linear System A h x h = b h must be solved at each time step A = 1 t M h + R h U h x = U m h b = R h (U m h ) M. Yano (MIT) Qualifier Examination January 26, / 67

62 Parallel GLS Solver GLS Solver Features Many ideas borrowed from ProjectX (Discontinuous Galerkin solver developed at ACDL since 2002) High-order GLS discretization on unstructured mesh h/p-scaling artificial viscosity for shock capturing. Parallel communication via MPI Local direct solve using UMFPACK Equation set: Poisson, advection-diffusion, Burger, Euler 70, 000 lines of C code M. Yano (MIT) Qualifier Examination January 26, / 67

63 BDDC Spaces Primal and Dual Spaces Decompose Ṽh(Γ) = (V h,π (Γ)) ( N i=1 V h, (Γ i )) Local Dual: V h, (Γ i ) = {v V h (Γ i ) : v = 0 on primal constraint} Primal: V h,π (Γ) = {v Ṽh(Γ) : N i=1 ã i (v Ωi, φ Ωi ) = 0, φ N i=1 V h, (Γ i )} Dual space V h, (Γ) = N i=1 V h, (Γ i ) is completely localized Primal space has DOF of O(N) Primal and Dual Schur Complement Primal Schur complement: S Π : V h,π (Γ) V h,π (Γ) S Π v Π,φ Π = N i=1 ã i (v Ωi,φ Ωi ), v Π,φ Π V h,π (Γ) Local dual Schur complement: S,i : V h, (Γ i ) V h, (Γ i) S,i v,i,φ,i = ã i (v,i,φ,i ) v,i,φ,i V h, (Γ i ) M. Yano (MIT) Qualifier Examination January 26, / 67

64 BDDC Preconditioner Injections and Averaging Operator R T ΓΠ : V h,π(γ) Ṽh(Γ) R T Γ : V h, (Γ i ) Ṽh(Γ) R T D,i : Ṽh(Γ) V h (Γ) s.t. v = N i=1 R T D,i v Ω i, v V h (Γ) BDDC Preconditioner BDDC preconditioner is given by M 1 BDDC = N i=1 R T D,i (T sub,i + T coarse )D i R D,i where T sub,i = R T Γ,i S 1,i R Γ,i and T coarse = R ΓΠ S 1 Π RT ΓΠ Condition Number Estimate for Coercive, Symmetric a(, ) κ(m 1 BDDC S) C(1 + log2 (H/h)) M. Yano (MIT) Qualifier Examination January 26, / 67

65 Galerkin Least Squares Discretization Background SUPG developed by Hughes [1982]; analyzed by Johnson [1984] SUPG extended to GLS by Hughes [1989] GLS Discretization Find u V H 1 (Ω) s.t. a(u,φ) + (Lu,τLφ) Ω,Th = (f,φ) Ω + (f,τlφ) Ω,Th, where (, ) Ω,Th = K K dk and τ ( h, P e 1 h 2, P e 1. φ V Stability and A Priori Error Estimate for Hyperbolic Case (ν = 0) a(u, u) h β u 2 L 2 (Ω) h β (u uh ) L2 (Ω) + u u h L2 (Ω) Ch p+ 1 2 u H p+1 (Ω) M. Yano (MIT) Qualifier Examination January 26, / 67

66 Stabilization Matrix High-Order Stabilization Matrix In order to keep cond(l stabilized) cond(l unstabilized ) = O(1) as p and h 0, τ h 2 /p 2, P e 1 One choice of τ in R d d+1 τ 1 = β ξ i p 2 d d + d i=1 Topics Studied in High-Order GLS Comparison of SUPG and GLS i=1 j=1 ξ i x k ξ j x l ν kl Subcell-shock capturing using h/p scaling artificial viscosity and resolution based shock indicator GLS for highly anisotropic mesh M. Yano (MIT) Qualifier Examination January 26, / 67

67 BDDC Preconditioner Features of BDDC Equipped with a coarse space (primal space) that provides global communication and makes subdomain problems well-posed. Operates on partially assembled space: Ṽh(Γ) Ṽ h (Γ) = {v L 2 (Ω) : v N i=1 V h(γ i ) and v is continuous on primal constraints} Examples of primal constraints Values at the corners of Ω i State averages on the edges of Ω i V h (Γ) Ṽ h (Γ) N i=1 V h(γ i ) M. Yano (MIT) Qualifier Examination January 26, / 67

68 BDDC Spaces Primal and Dual Spaces Decompose Ṽh(Γ) = (V h,π (Γ)) ( N i=1 V h, (Γ i )) Local Dual: V h, (Γ i ) = {v V h (Γ i ) : v = 0 on primal constraint} Primal: V h,π (Γ) = {v Ṽh(Γ) : N i=1 ã i (v Ωi, φ Ωi ) = 0, φ N i=1 V h, (Γ i )} Dual space V h, (Γ) = N i=1 V h, (Γ i ) is completely localized Primal space has DOF of O(N) -Space Π-Space M. Yano (MIT) Qualifier Examination January 26, / 67

Implicit Solution of Viscous Aerodynamic Flows using the Discontinuous Galerkin Method

Implicit Solution of Viscous Aerodynamic Flows using the Discontinuous Galerkin Method Per-Olof Persson and Jaime Peraire Massachusetts Institute of Technology 7th World Congress on Computational Mechanics