A Nodal High-Order Discontinuous Galerkin Dynamical Core for Climate Simulations

Transcription

1 A Nodal High-Order Discontinuous Galerkin Dynamical Core for Climate Simulations Institute for Mathematics Applied to the Geosciences (IMAGe) National Center for Atmospheric Research (NCAR) Boulder CO 80305, USA. International Conference On Spectral And High Order Methods June 18-22, 2007 Chinese Academy of Sciences, Beijing, China.

2 HOMME: High-Order Method Modeling Environment Supported by the DoE SciDAC and CCPP programs and NSF sponsorship of NCAR HOMME Developers Henry Tufo (PI, CU Boulder & NCAR ) John Dennis, Amik St-Cyr, Joe Tribbia & Ram Nair (Co-Is, NCAR) Jim Edward (IBM/NCAR), Hae-Won Choi (Postdoc, CU) Mike Levy and Theron Voran (Grad Students), Mark Taylor (SA Collaborator, Sandia National Laboratories) Project URL:

3 HOMME Motivation The DG Baroclinic Model (HOMME) 1 Vertical aspects (Lagrangian Dynamics, Remapping) 2 Horizontal Aspects (DGM, Discretization) Validation 1 2D 2 3D Parallel Performance Some Computational Issues Summary

4 Motivation HOMME Why do we need a new climate model? Because, the existing models have serious limitations to satisfy all of the following properties: 1 Local and global conservation 2 High-order accuracy 3 High parallel efficiency 4 Geometric flexibility ( Local method) 5 Monotonic (non-oscillatory) advection Discontinuous Galerkin Method (DGM) based model has the potential to address all of the above issues Recently, the Spectral Element (SE) model in HOMME shown to efficiently scale O (10, 000) processors on IBM BG/L.

5 HOMME (High-Order Method Modeling Environment) The Discontinuous Galerkin (DG) model is the conservative option in the HOMME framework HOMME Grid: The sphere is decomposed into 6 identical regions, using the equiangular projection (Sadourny, 1972) Local coordinate systems are free of singularities Creates a non-orthogonal curvilinear coordinate system Cubed Sphere Geometry: Logical cube-face orientation F 5 (Top) z F4 F 1 F 2 F 3 F 1 y F 6 (Bottom) x

6 Grid System HOMME Metric Tensor G ij, [Cubed-Sphere Sphere] Transform Central angles x 1,x 2 [ π/4,π/4] are the independent variables. G ij = R 2 ρ 4 cos 2 x 1 cos 2 x 2 [ 1 + tan 2 x 1 tanx 1 tanx 2 tanx 1 tanx tan 2 x 2 where ρ 2 = 1 + tan 2 x 1 + tan 2 x 2, i,j {1,2} ] Metric tensor in terms of longitude-latitude (λ,θ): [ G ij = A T R cos θ λ/ x 1 R cos θ λ/ x A; A = 2 R θ/ x 1 R θ/ x 2 ] The matrix A is used for transforming spherical velocity (u,v) to the covariant (u 1,u 2 ) and contravariant (u 1,u 2 )

7 3D Prognostic Equations for the DG Model Momentum Equations: No explicit vertical advection terms u 1 t + c E 1 u 2 t + c E 2 = Gu 2 (f + ζ) R T x 1(lnp) = Gu 1 (f + ζ) R T x2(ln p) c ( ) x 1, x 2, E 1 = (E,0), E 2 = (0,E), E = Φ + 1 ( u1 u 1 + u 2 u 2) 2 G = det(gij ), E is the energy term, Φ is the geopotential, and ζ is the relative vorticity.

8 3D Prognostic Equations: Flux-Form Continuity Equations Temperature field is advected with the mass variable p t ( p) + c (U i p ) = 0 t (Θ p) + c (U i Θ p ) = 0 t (q p) + c (U i q p ) = 0 where U i = ( u 1, u 2), p = Gδp, δp is the pressure thickness, and Θ is the potential temperature. Vertical layers are coupled with the hydrostatic relations: Φ = C p Θ Π, Φ = RT lnp where (p/p 0 ) κ and T Denotes the layer mean temperature.

9 Vertical Lagrangian Coordinates Vertically moving Lagrangian Surfaces The velocity fields (u 1,u 2 ), and total energy (Γ E ) are remapped onto the reference coordinates using the 1-D conservative cell-integrated semi-lagrangian (CISL) method (Nair & Machenhauer, 2002) P= Pressure thickness Lagrangian Surface t + t t P L 2 P E2 L 1 E 1 Topography Terrain following Lagrangian control volume coordinates

10 Computational Grid Structure for DG Model k 1/2 ( φ,p) k 1/2 k ( u, v, θ, q, δp ) k k+1/2 ( φ, p) k+1/2 GLL Grid box The remapping frequency is O(10) t Potential temperature Θ is retrieved from the remapped total energy Γ E = c p T + δ(pφ) δp + K E

11 The Vertical Lagrangian Dynamics The hydrostatic pressure at Lagrangian surface, Lin (MWR, 2004) p l = p top + l δp k, l = 1,2,3,...,N k=1 where p top represents the pressure at the model top, p l denotes the pressure at each Lagrangian surface. There are total N + 1 Lagrangian surfaces span N layers. The geopotential height at Lagrangian surface: Φ l = Φ s + l Φ k, l = N,N 1,...,1 k=n where Φ s represents the surface geopotential height at the model bottom and Φ l denotes the geopotential height at each Lagrangian surface.

12 DG-3D Model Horizontal Aspects: Shallow Water Model Flux-form SW equations (Vector invariant form) Nair et al. (MWR, 2005) u 1 t + x 1E = G u 2 (f + ζ) u 2 t + x 2E = G u 1 (f + ζ) t ( G h) + x 1( G u 1 h) + x 2( G u 2 h) = 0 where G = det(g ij ), h is the height, f Coriolis term; energy term and vorticity are defined as E = Φ (u 1 u 1 + u 2 u 2 ), ζ = 1 [ u2 G x 1 u ] 1 x 2.

13 The DG & SE Methods in HOMME DG SE Boundary Discontinuity FV 0 C Continuous DGM is a hybrid approach (DG SE + FV) The domain D is partitioned into non-overlapping elements Ω ij such that the element boundaries are discontinuous. Based on conservation laws but exploits the spectral expansion of SE method and treats the element boundaries using FV tricks.

14 DG Spatial Discretization for an Element Ω SW Equations can be written as 2D Scalar conservation law U t + F(U) = S(U), in Ω (0,T); (x1,x 2 ) Ω where U = U(x 1,x 2,t), ( / x 1, / x 2 ), F = (F,G) is the flux function, and S is the source term. Weak Galerkin formulation: Multiplication of the basic equation by a test function ϕ h V h and integration over an element Ω. Z Z Z Z U h ϕ h dω F(U h ) ϕ h dω + F(U h ) n ϕ h dγ = S(U h ) ϕ h dω t Ω Ω Γ Ω where U h is an approximate solution in Ω.

15 DGM: The Flux Term HOMME Element (Left) Element (Right) Element (Left) Element (Right) _ U h U + h U h U h After Num. Flux Operation Along the boundaries (Γ) of an element the solution U h is discontinuous (U h and U + h are the left and right limits). Therefore, the analytic flux F(U h ) n must be replaced by a numerical flux such as the Lax-Friedrichs Flux: F(U h ) n = 1 i h(f(u h 2 ) + F(U+ h )) n α(u+ h U h ). For the SW system, α is the upper bound on the absolute value of eigenvalues of the flux Jacobian F (U); (Nair et al., 2005) α 1 = max u 1 + Φ G 11, α 2 = max u 2 + Φ G 22

16 DG-3D Model: Computational Domain ( 1, 1) GLL Grid (8x8) (1, 1) η ( 1, 1) Cubed-Sphere (N e = 5) with 8 8 GLL points ξ (1, 1) Flux is the only communicator at the element edges Each face of the cubed-sphere is partitioned into N e N e rectangular non-overlapping elements (i.e., total 6 N 2 e ). Each element is mapped onto the Gauss-Lobatto-Legendre (GLL) grid defined by 1 ξ,η 1, for integration.

17 DGM: Spatial Discretization High-order nodal basis set The nodal basis set is constructed using a tensor-product of Lagrange-Legendre polynomials (h i (ξ)) with roots at Gauss-Lobatto quadrature points. (ξ 2 1)P N h i (ξ) = (ξ) 1 N(N + 1)P N (ξ i )(ξ ξ i ) ; h i (ξ)h j (ξ) = w i δ ij. where P N (ξ) is the N th order Legendre polynomial, and w i weights associated with the Gauss quadrature. The approximate solution (U h ) and test function (ϕ h ) are represented in terms of nodal basis set. U ij (ξ, η) = NX i=0 j=0 1 NX U ij h i (ξ)h j (η) for 1 ξ, η 1, The nodal version was shown to be more computationally efficient than the modal in (Dennis et al., 2006).

18 DG-3D Model: Explicit Time Integration Final form for the nodal discretization d dt U 4 ij(t) = xi 1 [I Grad + I Flux + I Source ], x2 j w i w j Solve the ODE system d U = L(U) dt in (0,T) Ω Time integration: Explicit third-order Runge-Kutta (SSP) scheme (Gottlieb et al., 2001) Options for explicit diffusion ( 2 or 4 ). Boyd-Vandeevan spatial Filter

19 2D results Horizontal Advection: Moving Vortices on the Sphere Initial field and DG solution after 12 days. Max error is O(10 5 ) Deformational Flow Test: Nair & Jablonowski (MWR, 2007) The vortices are located at diametrically opposite sides of the sphere, the vortices deform as they move along a prescribed trajectory. Analytical solution is known and the trajectory is chosen to be a great circle along the NE direction (α = π/4).

20 2D results SW Test-2: Geostrophic Flow (Nair et al. 2005) High-order accuracy and spectral convergence Steady state geostrophic flow (α = π/4). Max height error is O(10 6 ) m.

21 2D results SW Test-5: Flow over a Mountain (Dennis et al. 2006) No spectral ringing for the height fields Flow over a mountain ( 0.5 o ). Initial height field (left) initial and after 15 days of integration (right)

22 DG-3D: Baroclinic Instability Test (JW-Test) Jablonowski & Williamson (QJRMS, 2006) To assess the evolution of an idealized baroclinic wave in the Northern Hemisphere. The initial conditions are quasi-realistic and defined by analytic expressions. Analytic solutions do not exist. Initial Conditions

23 JW-Test: Evolution of Surface Pressure over the NH Baroclinic waves are triggered by perturbing the velocity field at (20 E, 40 N) This test case recommends up to 30 days of model simulation Ne = Nv = 8 (approx. 1.6 ) with 26 vertical levels and t = 30 Sec.

24 DG-3D Model Vs. NCAR Spectral Model The HOMME-DG dynamical core successfully simulates baroclinic instability. Simulated temperature (K) and surface pressure (hpa) at day 8 for a baroclinic instability test with the HOMME-DG model and the NCAR global spectral model (right). The horizontal resolution is approximately 1.4. Note that the DG solution is free of spectral ringing.

25 Jablonowski-Williamson Baroclinic Test (Convergence)

26 DG Model Vs. NCAR Climate Models Simulated surface pressure at day 11 for a baroclinic instability test with DG model, and NCAR global spectral model and a FV model. The models use 26 vertical levels and with approximate horizontal resolution of 0.7.

27 Parallel Performance (3D) - Frost [IBM BG/L] DG-3D parallel performance: Sustained Mflops on IBM BG/L (1024 DP nodes, 700 MHz PPC 440s): Approx. 9% peak Held-Suarez (preliminary) test: 800 days idealized climate simulation (1 resolution, 26 vertical levels, t = 10 Sec) 300 Sustaind FLOP Per Processor MFLOP elements: 1 task/node (CO) elements: 2 task/node (VN) 7776 elements: 1 tasks/node (CO) 7776 elements: 2 tasks/node (VN) Processor Parallel performance (strong scaling) results for JW-Test Held-Suarez test (800 days).

28 Some computational issues with limiting (Levy et al. 2007) Monotonic limiters do exist for relatively low-order DGM (N 4). The WENO based limiters (Qiu & Shu, 2005) are not only computationally expensive but results in a parallel communication bottleneck. Extending to high-order (N > 4) is impractical. In HOMME N > DG advection of a non-smooth Doswell vortex combined with a third-order WENO limiter (40 40) elements, limiting an element requires a 5 5 stencil!

29 Summary HOMME The DG-3D model successfully simulates the Baroclinic instability test and the results are comparable with that of the NCAR global spectral model. The preliminary scaling results are impressive and comparable to the SE version in HOMME. The R-K time integration scheme is robust for the DG-3D model, but very time-step restrictive. More efficient time integration schemes are required for practical climate simulations (semi-implicit). Efficient monotonic limiter for advection? Future Work: Time integration schemes. Coupling of the CAM/CCSM physics for the real climate simulations. Targeting for large-scale parallelism with O(100K) processors.