A parallel finite element multigrid framework for geodynamic simulations with more than ten trillion unknowns

Size: px

Start display at page:

Download "A parallel finite element multigrid framework for geodynamic simulations with more than ten trillion unknowns"

Aron White
5 years ago
Views:

A parallel finite element multigrid framework for geodynamic simulations with more than ten trillion unknowns Dominik Bartuschat, Ulrich Rüde Chair for System Simulation, University of

1 A parallel finite element multigrid framework for geodynamic simulations with more than ten trillion unknowns Dominik Bartuschat, Ulrich Rüde Chair for System Simulation, University of Erlangen-Nürnberg October 23, 2017 CSEConf2017 Oslo, Norway joint work with B. Gmeiner, D. Thoennes, N. Kohl (FAU) D. Drzisga, M. Huber, L. John, C. Waluga, B. Wohlmuth (TUM) S. Bauer, M. Mohr, P. Bunge (LMU) NEO

2 Earth mantle convection Why simulations? Driving force for plate tectonics Cause of earthquakes and mountain formation Coupled multiphysics problem Problem dimensions 109 years time scale 6400 km Earth radius 3000 km Earth mantle thickness 1012 degrees of freedom (1 km resolution) NEO 2

Simplified model r (2 µ (#, x) " (u)) + rp = (#, x) g @t # + u r# u

3 Simplified model r (2 µ (#, x) " (u)) + rp = (#, x) # + u r# u µ " = p # g 1 2 NEO ru + ru> r u =Solution 0 of the Stokes equations r ( r#) = velocity viscosity strain rate tensor dynamic pressure temperature density gravity thermal conductivity heat sources [1] [1] B. Gmeiner et al. Performance and Scalability of Hierarchical Hybrid Multigrid Solvers for Stokes Systems, SIAM J. Sci. Comput. (2015) 3

Refinement on all grids use clever data structures matrix-free implementation

4 Hierarchical Hybrid Grids (HHG) Parallel multigrid for tetrahedral finite elements partition domain into tetrahedra parallelize all operations Bey s Tetrahedral Refinement on all grids use clever data structures matrix-free implementation Elliptic problems always require global communication NEO Coarser grids for the global data transport 4

5 Tetrahedral discretization Dominik Bartuschat Chair for System Simulation A Framework for Geodynamic Simulations with More Than Unknowns 5

6 HHG concepts Structured refinement of unstructured base mesh Geometrical hierarchy: Vertex, edge, face, volume Parallel hierarchical grid traversal: NEO 6

Coupled flow and transport (const. viscosity) Iso-surfaces of temperature distribution [1] u + rp = # Ra # r r u=0 @t # + u r # = Pe 1 # Finite Elements with 6.

7 Coupled flow and transport (const. viscosity) Iso-surfaces of temperature distribution [1] u + rp = # Ra # r r # + u r # = Pe 1 # Finite Elements with DoF, time steps Run-time 7 days on mid-size cluster (LSS): 288 cores (9 nodes) NEO [1] B. Gmeiner et al. "Performance and scalability of hierarchical hybrid multigrid solvers for Stokes systems, SIAM J. Sci. Comput. (2015) 7

8 Rheology for anisothermal viscosity Viscosity model [1] (asthenosphere thickness 410 km) d a r inner relative asthenosphere thickness relative core radius Temperature distribution [2,3] recovered from seismic data [1] D. R. Davies et al. "Reconciling dynamic and seismic models of Earth's lower mantle: The dominant role of thermal heterogeneity" Earth Planet. Sci. Lett. (2012) [2] S. P. Grand et al. "High resolution global tomography: a snapshot of convection in the Earth" Geol. Soc. Am. Today (1997) [3] L. Stixrude and C. Lithgow-Bertelloni "Thermodynamics of mantle minerals I. Physical properties" Geophys. J. Int. (2005) Dominik Bartuschat Chair for System Simulation A Framework for Geodynamic Simulations with More Than Unknowns 8

9 Temperature distribution NEO 9

10 Velocity boundary conditions [1] J. A. Boyden et al. "Next-generation plate-tectonic reconstructions using GPlates" Geoinformatics: cyberinfrastructure for the solid earth sciences, Cambridge Univ. Press (2011) [2] M. Gurnis et al. "Plate tectonic reconstructions with continuously closing plates" Compt. Geosci. (2012) Dominik Bartuschat Chair for System Simulation A Framework for Geodynamic Simulations with More Than Unknowns 10

11 Stationary velocity NEO 11

Weak scaling for Stokes problem Optimized for minimal memory consumption 10 13 unknowns correspond to 80 TByte for solution vector JUQUEEN has 450 TByte memory Matrix-free implementation essential

12 Weak scaling for Stokes problem Optimized for minimal memory consumption unknowns correspond to 80 TByte for solution vector JUQUEEN has 450 TByte memory Matrix-free implementation essential nodes threads DoFs multigrid iterations time to solution parallel efficiency s 100 % s 97 % s 92 % s 95 % s 88 % Copyright: Jülich Supercomputing Centre Excellent weak scalability up to degrees of freedom (DoF) with all-at-once (Uzawa) multigrid [1] [1] B. Gmeiner et al. "A quantitative performance analysis for Stokes solvers at the extreme scale, J. Comput. Sci. (2016) Dominik Bartuschat Chair for System Simulation A Framework for Geodynamic Simulations with More Than Unknowns 12

13 Conclusion 1013 degrees of freedom for FE are possible Multigrid scales to Peta (and beyond) Excellent time to solution HHG - lean matrix-free implementation Work in progress: Re-design for higher-order finite elements NEO 13

High performance mantle convection modeling

High performance mantle convection modeling Jens Weismüller, Björn Gmeiner, Siavash Ghelichkhan, Markus Huber, Lorenz John, Barbara Wohlmuth, Ulrich Rüde and Hans-Peter Bunge Geophysics Department of Earth-