Sustained Petascale Performance of Seismic Simulations with SeisSol

Transcription

1 SIAM EX Workshop on Exascale Applied Mathematics Challenges and Opportunities Sustained Petascale Performance of Seismic Simulations with SeisSol M. Bader, A. Breuer, A. Heinecke, S. Rettenberger C. Pelties, A.-A. Gabriel Technische Universität München, Ludwig-Maximilians-Universität München SIAM EX, July,

2 HPC Meets Geoscience Alexander Alice-Agnes Alexander Christian Sebastian Breuer Gabriel Heinecke Pelties Rettenberger SIAM EX, July,

3 Overview and Agenda SeisSol: dynamic rupture and seismic wave propagation unstructured tetrahedral meshes high-order ADER-DG discretisation Optimisation for Heterogeneous Petascale Platforms: code generation to optimize element-local matrix kernels hybrid MPI/OpenMP parallelisation offload scheme to address multiphysics Performance on Tianhe-, Stampede and SuperMUC: weak scaling of wave propagation component strong scaling for 99 Landers M. earthquake SIAM EX, July,

4 Dynamic Rupture and Earthquake Simulation Tohoku subduction zone: CAD model and tetrahedral mesh (C. Pelties) Use of Adaptive Tetrahedral Meshes: curved subduction zones that meet surface at shallow angles high impact on uplift for tsunamigenic earthquakes complicated fault systems with multiple branches non-linear multiphysics dynamic rupture simulation goal: automated meshing process (incl. CAD generation) SIAM EX, July,

5 Dynamic Rupture and Earthquake Simulation Landers fault system: simulated ground motion and tetrahedral mesh Use of Adaptive Tetrahedral Meshes: curved subduction zones that meet surface at shallow angles high impact on uplift for tsunamigenic earthquakes complicated fault systems with multiple branches non-linear multiphysics dynamic rupture simulation goal: automated meshing process (incl. CAD generation) SIAM EX, July,

6 + Aq Technische Universität München : Governing x + Bq eq s y + Cq z = Seismic Wave Propagation with SeisSol B C Elastic Wave Equations: B C A formulation) q y + Cq q t + Aq z x + Bq y + A Cq = z. = B C with q = (σ, σ, σ, σ B, σ, σ, u, v, w) T - - µ - - -µ -µ A µ µ - - -µ A = -µ B = -µ -µ - - B - - C A µ - B u high order discontinuous Galerkin discretisation C - - µ -µ = ADER-DG: -µ v A high approximationc order = in space andtime: -µ -µ B additional - features: - local time C stepping, B high accuracy - - w - of C - - A earthquake - faulting - (full frictional sliding) Dumbser, Käser et al. [,] µ M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol SIAMC EX=, B July, -µ C = inecke, S. Rettenberger, Optimization of SeisSol

7 Department of Informatics V Technische Universität München Technische Universität München SeisSol in a Nutshell ADER-DG SeisSol in a nutshell: ADER-DG Update scheme Q n+ k = Q k - S X k J k M- F -,i I(t n, t n+, Q n k )N k,i A + k N- k,i + i= X i= +M - K I(t n, t n+, Q n k )A k +M - K I(t n, t n+, Q n k )B k +M - K I(t n, t n+, Q n k )C k F +,i,j,h I(t n, t n+, Q n k(i) )N k,ia - k(i) N- k,i Cauchy Kovalewski I(t n, t n+, Q n k )= JX (t n+ - t n ) j (j + j Q k(t n ) j= (Q k ) t = -M - (K ) T Q k A k +(K ) T Q k B k +(K ) T Q k C k A. M. Breuer, Bader eta. al.: Heinecke, Sustained Petascale S. Rettenberger, Performance of Seismic Optimization Simulations of with SeisSol SeisSol M. SIAM Bader, EX, C. July Pelties,

8 Technische Universität München tiplication with the sparse Jacobians A k, B k and ure shows all involved sparsity patterns of the A Optimisation of(b Sparse = ) scheme. Matrix Operations Apply sparse matrices to multiple DOF-vectors Q k Code Generator for Sparse Kernels: (Breuer et al. []) Figure. Sparsity patterns appearing in the ADER tim avoid overhead of CSR (or similar) data structures; store CSR elements vector, only Volume integration: In the volume integration, g we multiply the sparse stiffness matrices K ξ, K η unknowns I(t n,t n+,q n k ) previously obtained by the left. The intermediate results are again multip full unrolling of all element operations using a code generator use intrinsics and apply blocking to improve vectorisation SIAM EX, July,

9 Technische Universität München Optimisation of Sparse Matrix Operations Computation Apply sparse of the matrices first time to multiple derivative DOF-vectors / t Q k Q. k e) derivative Computation / t of the Q k. second derivative / t Q k. Dense vs. Sparse Kernels: (Breuer et al. []) recursions switch of thetoader densetime kernels integration dependingfor onaachieved fifth order timemethod. to solution erate zero-blocks for sparse in and the derivatives, dense kernels: ivory blocks hit zero blocks. exploit zero-blocks generated during recursive CK computation Evaluation of the ADER Time Integration SIAM EX, July,

10 Mesh Generation and Partitioning Mesh Generation: high-quality meshes required (shallow subduction zones, complicated fault structures) with 9 grid cells using SimModeler by Simmetrix ( Two-stage approach to provide parallel mesh partitions: graph-based partitioning (ParMETIS) create customised parallel format (based on netcdf) for mesh partitions highly scalable mesh input via netcdf/mpi-io in SeisSol SIAM EX, July,

11 Optimization for Intel Xeon Phi Platforms Offload Scheme: Host PCIe Xeon Phi to address load imbalances of multiphysics simulation hides communication with Xeon Phi and between nodes OpenMP parallelisation: MPI comm., receiver output dynamic rupture fluxes, fault output download cells for receivers, DR, MPI upload MPIreceived cells upload dynamic rupture updates time integration of MPI boundary cells time integration of non-mpi cells, volume integration wave propagation fluxes to address manycore parallelism with coprocessors download all data (if required) apply dynamic rupture updates, pack transfer data careful parallelisation of all loops plot wave field (if required) SIAM EX, July, 9

12 Supercomputing Platforms LRZ, Munich 9 compute nodes ( thin node islands), Intel SNB-EP cores (. GHz) Infiniband FDR interconnect (fat tree) # in Top :.9 PFlop/s TACC, Austin compute nodes,, cores SNB-EP (c) + Xeon Phi SEP per node Mellanox FDR interconnect (fat tree) # in Top :. PFlop/s NSCC, Guangzhou compute nodes used,. Mio cores SNB-EP (c) + Xeon Phi SP per node TH-Express custom interconnect # in Top :. PFlop/s SIAM EX, July,

13 Weak Scaling of Wave Propagation SuperMUC, classic Stampede SuperMUC, gr. buff. Tianhe-, card Tianhe-, cards Tianhe-, cards % parallel efficiency 9 9 # nodes goal: test scalability towards large problem sizes cubic domain, uniformly refined tetrahedral cells weak scaling:, elements per card/node SIAM EX, July,

14 Weak Scaling of Wave Propagation SuperMUC, classic Stampede SuperMUC, gr. buff. Tianhe-, card Tianhe-, cards Tianhe-, cards % parallel efficiency # nodes more than 9 % parallel efficiency on Tianhe- and Stampede % on full SuperMUC (no overlapping) SIAM EX, July,

15 Weak Scaling of Wave Propagation SuperMUC, classic Stampede SuperMUC, gr. buff. Tianhe-, card Tianhe-, cards Tianhe-, cards % parallel efficiency 9 9 # nodes. PFlop/s on Tianhe- ( nodes). PFlop/s on Stampede ( nodes). PFlop/s on SuperMUC (9 nodes) SIAM EX, July,

16 Weak Scaling Peak Efficiency SuperMUC, gr. buff. Stampede Tianhe- SuperMUC, classic % peak: hardware % peak: non-zero 9 9 # nodes SIAM EX, July,

17 99 Landers M. Earthquake multiphysics simulation of dynamic rupture and resulting ground motion of a M. earthquake fault inferred from measured data, regional topography from satellite data, physically consistent stress and friction parameters D velocity structure, low velocity near surface SIAM EX, July,

18 Strong Scaling of Landers Scenario Stampede SuperMUC, classic SuperMUC, gr. buff. Tianhe-, card Tianhe-, cards Tianhe-, cards % parallel efficiency # nodes 9 million tetrahedrons;,9 element faces on fault th order, 9 billion degrees of freedom SIAM EX, July,

19 Strong Scaling of Landers Scenario Stampede SuperMUC, classic SuperMUC, gr. buff. Tianhe-, card Tianhe-, cards Tianhe-, cards % parallel efficiency # nodes more than % parallel efficiency on Stampede and Tianhe- (when using only one Xeon Phi per node) multiple-xeon-phi performance suffers from MPI communication SIAM EX, July,

20 Strong Scaling of Landers Scenario Stampede SuperMUC, classic SuperMUC, gr. buff. Tianhe-, card Tianhe-, cards Tianhe-, cards 9 9 % parallel efficiency 9 # nodes. PFlop/s on Tianhe- ( nodes). PFlop/s on Stampede ( nodes). PFlop/s on SuperMUC (9 nodes) SIAM EX, July,

21 Landers Strong Scaling Peak Efficiency SuperMUC, gr. buff. Stampede Tianhe- SuperMUC, classic % peak: hardware % peak: non-zero 9 9 # nodes SIAM EX, July,

22 Landers Earthquake Production Run On SuperMUC: h min computing time fault output and receivers. PFLOPS sustained performance Observations: complex rupture dynamics (fault branching, etc.) high-frequency signals (up to Hz) from rupture propagate directly into wave field ground shaking in the engineering frequency band s simulated time SIAM EX, July,

23 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

24 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

25 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

26 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

27 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

28 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

29 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

30 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

31 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

32 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

33 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

34 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

35 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

36 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

37 Multiphysics Dynamic Rupture Simulation spontaneous rupturing due to exceeded stress limits including rupture jumps, fault branching, etc. tackles fundamental questions on the dynamics of earthquakes in natural fault zone systems SIAM EX, July,

38 SeisSol: Earthquake Petascale Multiphysics Dynamic Rupture Simulations with SeisSol: high-order ADER-DG on unstructured adaptive meshes focus on complicated geometries and rupture physics non-linear interaction of rupture process and seismic waves Petascale Performance on Heterogeneous Platforms: exploits high computational intensity of ADER-DG requires careful tuning of the entire simulation pipeline code generation to accelerate element kernels scalable mesh input for M cells on k cores offload-scheme for multiphysics with Xeon Phi SIAM EX, July,

39 I Would Like to Thank... Intel: Mikhail Smelyanskiy, Karthikeyan Vaidyanathan Pradeep Dubey all colleagues from the three supercomputing centers, esp.: Arndt Bode (Leibniz Supercomputing Centre) William Barth (Texas Advanced Computing Centre) Xiang-Ke Liao (NCSS and NUDT) for financial support: Volkswagen Foundation (project ASCETE) KONWIHR the entire SeisSol team and all contributors You! SIAM EX, July, 9

40 References [] A. Breuer, A. Heinecke, M. Bader, C. Pelties: Accelerating SeisSol by generating vectorized code for sparse matrix operators. In: Advances in Parallel Computing, IOS Press,. Proceedings of ParCo [] A. Breuer, A. Heinecke, S. Rettenberger, M. Bader, A.-A. Gabriel, C. Pelties: Sustained Petascale Performance of Seismic Simulations with SeisSol on SuperMUC. In: Supercomputing, LNCS, p.. PRACE ISC Award. [] M. Dumbser, M. Käser: An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes II. The three-dimensional isotropic case. Geophys. J. Int. (),. [] A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A.-A. Gabriel, C. Pelties, A. Bode, W. Barth, X.-K. Liao, K. Vaidyanathan, M. Smelyanskiy, P. Dubey: Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers. Gordon Bell Prize Finalist. [] M. Käser, M. Dumbser, J. de la Puente, H. Igel: An arbitrary high-order Discontinuous Galerkin method for elastic waves on unstructured meshes III. Viscoelastic attenuation. Geophys. J. Int. (),. [] C. Pelties, J. de la Puente, J.-P. Ampuero, G. B. Brietzke, M. Käser: Three-dimensional dynamic rupture simulation with a high-order discontinuous Galerkin method on unstructured tetrahedral meshes. J. Geophys. Res.: Solid Earth, (B),. SIAM EX, July,