Large-scale multi-physics earthquake scenarios with the ADER-DG method on modern supercomputers

Transcription

1 Largescale multiphysics earthquake scenarios with the ADERDG method on modern supercomputers Stephanie Wollherr, Dr. AliceAgnes Gabriel, Dr. Betsy Madden, Thomas Ulrich LudwigsMaximilians Universität (LMU) München in collaboration with

2 Overview Challenges in simulating earthquakes The ADERDG software package SeisSol Optimization for modern supercomputers New physical extensions 2004 Sumatra Andaman earthquake

3 Introduction What is an earthquake? frictional failure of brittle solids under compression on a zone of weakness (= fault) fundamental understanding of earthquake source processes coupled to seismic wave propagation using numerical models which physical processes lead to failure of rocks? under which conditions does an earthquake result in a devastating event? physicsbased seismic hazard assessment Community fault model for California including seismicity. Plesch et al. (2007)

4 Earthquake rupture simulations multiphysics approach: coupling of frictional failure on a predefined fault surface to seismic wave propagation What do we need to model realistic earthquakes? threedimensional properties of the subsurface Earth structure including potentially complex fault geometry as well as surface structure (topography, bathymetry) frictional constitutive relationship, initial fault strength and tectonic background stress 3D structure of the material density Landers fault system discretized by triangular elements Topography of the Sumatra region (GEBCO)

5 Challenges Why do we need HPC? resolving the engineering relevant frequency content of the seismic wave field (020 Hz) and its interaction with Earth structure, nonlinear rheologies and frictional failure on the fault highorder accurate numerical methods mutliscales: from a couple of 100 kilometers to meter scales on the fault large meshes with over 100 of millions of elements I/O requirements: efficient initialization of initial conditions highresolution output of the 3D wavefield and source dynamics Visualization of the abs.particle velocity around the Landers fault system

6 SeisSol an ADERDG based software package Solving the elastic waveequation... with an Arbitrary HighOrder DERivative (ADER) in time and a modal Discontinuous Galerkin (DG) method in space FE approach but discontinuities at element interfaces communication between elements by upwind fluxes (exact Riemann solver) purely local scheme highorder accuracy in space and time based on unstructured tetrahedral meshes originally developed at the LMU (Käser&Dumbser, 2006), optimized in collaboration with TU Munich Open source (github.com/seissol)

7 SeisSol an ADERDG based software package with dynamic rupture internal boundary condition handles coupling to frictional failure causes no spurious oscillations in the slip rate SCEC benchmark: particle velocity at a 60 dipping fault. Refined tetrahedral mesh. (Pelties et al. (2014),Geosci.Model Dev.)

8 SeisSol an ADERDG based software package with dynamic rupture internal boundary condition handles coupling to frictional failure causes no spurious oscillations in the slip rate But computational expensive! SCEC benchmark: particle velocity at a 60 dipping fault. Refined tetrahedral mesh. (Pelties et al. (2014),Geosci.Model Dev.) many relatively small matrix operations complicated load balancing between elements which have to solve for source processes and pure wave propagation ones

9 Optimization Wave propagation on petascale supercomputers... highly optimized kernel operations for sparse/dense matrix products generated by offline code generator strongscalability due to a hybrid OpenMP and MPI parallelization speed up by a factor of 510, 90% parallel efficiency and 45% of peak performance on SuperMUC Phase 1 parallel I/0 scheme customized mesh format and reader A Parallel Server for Adaptive GeoInformation (ASAGI) Simulation of propagating waves on a volcano, Mount Merapi. Model consisting in more than 99 mio. elements Peak performance of the hardware FLOPS and nonzero operations for strong scaling tests of order O=6 References: 1) Breuer et al., PRACE Award, ISC ) Rettenberg & Bader (2015), IEEE International Conference on Cluster Computing 3) Rettenberger et al. (2016), EASC2016

10 Optimization...including dynamic rupture optimization for heterogeneous supercomputers offload scheme: schedules subtask of the complex multiphysics process heterogeneously to either the host or the XeonPhi platform Reference: Heinecke et al.(2014), Gordon Bell Finalist '14

11 Optimization...including dynamic rupture optimization for heterogeneous supercomputers offload scheme: schedules subtask of the complex multiphysics process heterogeneously to either the host or the XeonPhi platform Dynamic earthquake simulation based on the Mw Landers event Petascale (10 ¹⁵Flops/s) simulation on Tianhe2, SuperMuc Phase 1 and Stampede Illustration of the Landers fault system embedded in a realistic geological structure including topography 200 x 10⁶ element tetrahedral mesh (5 x 10¹⁰DoFs) ran on ~ cores of SuperMuc Phase 1 for 200,000 time steps over 7h of computation 1.25 PFLOPS sustained performance Reference: Heinecke et al.(2014), Gordon Bell Finalist '14

12 Petascale simulation Landers scenario highdetailed rupture evolution Jumps, branching, reverse slip, multiple rupture fronts highfrequency ground motion caused by rupture complexity (no stochastic model ingredients) Evolution of slip rate over time Reference: Heinecke et al.(2014), Gordon Bell Finalist '14

13 Petascale simulation Landers scenario highdetailed rupture evolution Jumps, branching, reverse slip, multiple rupture fronts highfrequency ground motion caused by rupture complexity (no stochastic model ingredients) HPC hardware is required to resolve engineering frequency band 0 >10 Hz Evolution of slip rate over time Reference: Heinecke et al.(2014), Gordon Bell Finalist '14

14 New physics crucial but computational expensive physical model extensions such as... attenuation of seismic waves = energy loss as they propagate through the Earth important secondorder effect when resolving highfrequencies and access possible strong ground shaking Sparsity patterns of a typical matrix chain product in the discretisation of SeisSol Reference: Uphoff & Bader (2016), to be published in proceedings of HPCS 2016

15 New physics offfault plasticity high stresses around the fault need to be accommodated by inelastic processes such as plastic deformation of the host rock computational intensive: checking of the yield criterion for each element in each timestep (overhead factor of ~45) Accumulated plastic strain around the fault Reference: Wollherr & Gabriel (2016), in prep.

16 New physics offfault plasticity high stresses around the fault need to be accommodated by inelastic processes such as plastic deformation of the host rock computational intensive: checking of the yield criterion for each element in each timestep (overhead factor of ~45) Impact: reduction of the peak slip rate influence on rupture propagation and jumping dimension of possible earthquakes Accumulated plastic strain around the fault Reference: Wollherr & Gabriel (2016), in prep.

17 Recent challenges 2004 Sumatra Andaman earthquake Mw fatalities from the resulting tsunami Challenges: very slow and long rupture over a large region km rupture length 8 to 10 min of rupture rupture speed of around 23 km/s SumatraAdaman Sea region, Shearer & Bürgmann (2004)

18 Recent challenges 2004 Sumatra Andaman earthquake complex plateboundary interface CAD and mesh generation is a bottleneck Limited mesh coarsening and refinement ( 1% sizing difference between elements) meshes with ~100 million of elements ust nic cr ocea ction subdu tal ntinen crust co zone city w velo zone lo Meshes created with SimModeler customized meshing suite by Simmetrix ( com/) Reference: Model by Thomas Ulrich

19 Recent challenges Good results for the direction of the displacement, but still to high! adding more complexities such as stochastic small scale heterogeneities and 3D velocity structure Synthetics vs observations (compiled by Jade et al. (2005), Gahalaut et al. (2006) and Subaraya et al. (2005))

20 Recent challenges Good results for the direction of the displacement, but still to high! adding more complexities such as stochastic small scale heterogeneities and 3D velocity structure The ASCETE project (www. ascete.de) coupling of seafloor displacement resulting from earthquake rupture simulations as spatiotemporal input for tsunami models Which conditions are responsible for the generation of devastating tsunamis? Synthetics vs observations (compiled by Jade et al. (2005), Gahalaut et al. (2006) and Subaraya et al. (2005))

21 Summary & Outlook SeisSol enables the study of different representations of complexity which possibly influence the highfrequency content of the seismic wave field with efficient execution on modern HPC systems Modern numerical methods further our understanding of earthquake source physics, support physicsbased ground motion research for seismic hazard analysis ExaHype An Exascale Hyperbolic PDE Engine Opensource software for hyperbolic conservation laws applied to seismology & astrophysics addressing the needs of tomorrow s exascale supercomputers energy efficiency, high scalability and memory efficiency

22 Thank you very much for your attention! The SeisSol core team Dr. AliceAgnes Gabriel (LMU) Dr. Betsy Madden (LMU) Thomas Ulrich (LMU) Stephanie Wollherr (LMU) Dr. Alexander Heinecke (Intel) Sebastian Rettenberger (TUM) Carsten Uphoff (TUM) Prof. Dr. Michael Bader (TUM)