A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries

Transcription

1 A Framework for Hybrid Parallel Flow Simulations with a Trillion Cells in Complex Geometries SC13, November 21 st 2013 Christian Godenschwager, Florian Schornbaum, Martin Bauer, Harald Köstler, Ulrich Rüde Chair for System Simulation Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany

2 Outline walberla Framework Lattice Boltzmann Method Benchmarked Test Cases Benchmark Results Conclusion & Future Work 2

3 The walberla Framework

4 walberla an HPC Framework Focus on lattice Boltzmann method Written in C++ Contains hand-crafted, machine-specific high-performance compute kernels Also generic, easily adaptable compute kernels for prototyping Modules for handling complex geometries Particulate flow simulations by coupling with our physics engine pe Models for multiphase and free surface flows 4

5 walberla an HPC Framework Hybridly parallelized (MPI + OpenMP) No data structures growing with number of processes involved Scales from laptop to recent petascale machines Parallel output Portable (Compiler/OS) Automated tests / CI servers Open Source release early 2014 llvm/clang 5

6 Examples Turbulent flow (Re=11000) around a sphere (Ehsan Fattahi, Daniel Weingaertner) Study of hemodynamical impact of stenoses in coronary arteries. 6

7 Examples Liquid-Gas-Solid Flow Simulation: Stable Floating Positions of Box-Shaped Particles (Simon Bogner) Constructing a hollow cylinder by electron beam melting (Matthias Markl, Regina Ammer) Rigid bodies simulated with pe 7

8 Lattice Boltzmann Method LBM

9 Lattice Boltzmann Method Explicit, mesoscopic method for solving fluid flow problems (or heat, arbitrary advectiondiffusion equations ) Discretization of Boltzmann equation Provides solution for Navier-Stokes equations at low Mach numbers Based on uniformly structured, Cartesian grid of cells 9

10 Lattice Boltzmann Method Lattice Boltzmann equation (single-relaxation time, SRT) f i (x + e i δ t, t + δ t ) = f i x, t f i x, t f i eq (u x, t, ρ x, t ) τ Lattice Boltzmann equation (two-relaxation time, TRT) f i x + e i δ t, t + δ t = f i x, t f i + x, t f i eq,+ u x, t, ρ x, t λ 0 f i x, t f eq, i (u x, t, ρ x, t ) λ 1 Macroscopic quantities (density, momentum density) ρ = f i ρu = e i f i Equilibrium distribution function TRT model can improve accuracy and stability of LBM f i eq (u, ρ) = ω i ρ 1 + e i u c2 + (e i u) 2 4 3u2 s 2c s 2c2 s 10

11 LBM computationally Streaming Step Collision Step D2Q9 11

12 LBM computationally Streaming Step Collision Step D3Q19: 19 Loads 198 Flops (TRT) 19 Stores (+19 Loads) 305 Byte 12

13 LBM Data Structures Domain partitioning into blocks containing uniform grid of cells Ghostlayer (halo) exchange of outer layer(s) uniform block decomposition 13

14 Benchmarked Testcases

15 Testcases Lid Driven Cavity (LDC) Flow Flow through Coronary Arteries Dense One block per process No load balancing Sparse, but coherent Volume fraction 0.3% Multiple blocks per process Load balancing required 15

16 Complex Geometry Initialization Complex geometry given by surface Add regular block partitioning Load balancing Discard empty blocks Allocate block data 16

17 Complex Geometry Initialization Complex geometry given by surface Add regular block partitioning Load balancing Discard empty blocks Separate domain partitioning from simulation phase File size 500,000 blocks: ~40MB Allocate block data 17

18 Domain Partitioning dx = 0.2mm target: 200 blocks block size #blocks

19 Coronary Artery Testcase Initialization Domain partitioning of coronary tree dataset One block per process 512 processes 485 blocks 458,752 processes 458,184 blocks 19

20 Hardware JUQUEEN SuperMUC Forschungszentrum Jülich, Germany LRZ, Garching (Munich), Germany IBM system IBM system Blue Gene/Q Intel Sandy Bridge-EP 28,672 nodes 9,216 nodes 458,752 cores 147,456 cores 5.9 Petaflops peak 3.2 Petaflops peak 448 TB main memory 288 TB main memory 5D Torus Network Non-blocking tree / 4:1 pruned tree 20

21 Benchmark Results Lid Driven Cavity

22 MLUP/s SuperMUC - LDC - Weak SuperMUC single socket already quite optimized! Cores SRT1 naïve, straightforward implementation 22

23 MLUP/s SuperMUC - LDC - Weak SuperMUC single socket already quite optimized! Cores SRT2 SRT1 naïve, straightforward implementation 23

24 MLUP/s SuperMUC - LDC - Weak SuperMUC single socket already quite optimized! Cores SRT SRT2 SRT1 naïve, straightforward implementation 24

25 MLUP/s SuperMUC - LDC - Weak SuperMUC single socket already quite optimized! Cores SRT TRT SRT2 SRT1 naïve, straightforward implementation 25

26 MLUP/s SuperMUC - LDC - Weak SuperMUC single socket already quite optimized! Cores limited by memory bandwidth SRT TRT SRT2 SRT1 Bandwidth limit naïve, straightforward implementation 26

27 MLUP/s JUQUEEN - LDC - Weak JUQUEEN single node limited by memory bandwidth SRT hybrid version TRT (4 threads per core) Bandwidth limit Cores 27

28 MLUP/s per core SuperMUC - LDC - Weak SuperMUC TRT kernel Cores #processes per node 16P 1T 4P 4T 2P 8T #threads per process 28

29 MLUP/s per core SuperMUC - LDC - Weak SuperMUC TRT kernel islands Cores #processes per node 16P 1T 4P 4T 2P 8T #threads per process 29

30 MLUP/s per core SuperMUC - LDC - Weak SuperMUC TRT kernel islands Cores Communication share (%) 16P 1T 4P 4T 2P 8T Comm 30

31 MLUP/s per core JUQUEEN - LDC - weak JUQUEEN TRT kernel 5 4,5 4 3,5 3 2,5 2 1,5 1 0, x cells updated per second (19 values per cell) 383 TFlop/s (6.5% peak) 800 TB/s (67% peak) Cores #processes per node 64P 1T 16P 4T 8P 8T #threads per process 31

32 Benchmark Results Coronary Artery Tree

33 MFLUP/s / Core JUQUEEN - COR - weak JUQUEEN TRT kernel 3 2,5 2 1,5 1 0,5 Efficiency Fluid Fraction 1.03 trillion load balanced lattice cells dx = 1.3μm Cores 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 Fluid Fraction 33

34 MFLUP/s / Core JUQUEEN - COR - strong JUQUEEN - TRT kernel - dx = 0.05 Efficiency Peformance , , , , , , , , , Cores Time Steps / s 34

35 MFLUP/s / Core SuperMUC - COR - strong SuperMUC - TRT kernel - dx = 0.1 mm 1,80 1,60 1,40 1,20 1,00 0,80 0,60 0,40 0,20 0,00 Efficiency Performance Cores Time Steps / s 35

36 Conclusion & Future Work

37 Conclusion & Future Work walberla runs efficiently on current petascale supercomputers Excellent scaling properties Execution rates up to 6638 LBM time steps / s in strong scaling settings Discretization of coronary artery tree into 1,033,660,569,847 load balanced lattice cells Future: Grid refinement and dynamic load balancing Useful for particulate flows with fully resolved particles 37

38 Thank you!