ZFS - Flow Solver AIA NVIDIA Workshop

Transcription

1 ZFS - Flow Solver AIA NVIDIA Workshop Andreas Lintermann, Matthias Meinke, Wolfgang Schröder Institute of Aerodynamics and Chair of Fluid Mechanics RWTH Aachen University

2 Outline Softwareproject ZFS (applications at the AIA) Structure of ZFS Computational hotspots in ZFS Scalability The future of ZFS Summary, Conclusion, and Questions

3 Softwareproject ZFS ZFS (C++) Two major flow solvers Finite-Volume solver Lattice-Boltzmann solver

4 3D-Engine Flows within the Cluster of Excellence: TMFB Unsteady 3D simulation of engine flow full DNS requires 3.0 x 10⁹ cells simulations are performed on HPC systems valves combustion chamber refinement based on Level-Set

5 Level-Set Method Representation of surfaces by zero contour 0 of a signed distance function (x, t) Level-Set representation of an engine valve Solve Level-Set equation: t +f =0 body velocity Hartmann et al., J. Comput Phys., 227: , 2008; Hartmann et al., J. Comput. Phys., 229: , 2010; Hartmann et al., Combust. Flame, 158: , 2010

6 Level-Sets and Solution-Adaptive Refinement Schneiders et al., An accurate moving boundary formulation in cut-cell, method, submitted to J. Comput Phys. Sensor: Level-Set Sensors: Entropy, Vorticity Oscillating 2D-Cylinder flow, Re=180 Solution-Adaptive Refinement

7 Simulation of Particle Transport Atmospheric Research: droplet formation in clouds particle transport, collision and coagulation Combustion: heat conduction transport and collision of coal particles particle distribution important for combustion efficiency Aims: Simulation of O(1000) particles in a turbulent flow (full DNS)

8 SFB Combustion Flame Front Propagation:

9 SFB Combustion Flame Front Propagation: Level-Set and G₀-Band

10 COupled PArallel Simulation of Gas Turbines (COPA-GT) Full simulation of all components of a jet engine Development of fully coupled tools Aeroacoustic field prediction of an engine jet our aim: noise reduction Velocity gradients of a jet at Re=3600

11 Nasal cavity flows Major goals: analysis of the respiratory capability (pressure loss) analysis of the olfactory capability (streamline analysis) analysis of the heating capability (temperature difference) analysis of the humidification capability (particle retention times) warped septum left paranasal sinus left nostril swollen center turbinate warped septum right nostril swollen inferior turbinate right paranasal sinus swollen inferior turbinate CT-data of the human head pharynx Surface of the human nasal cavity

12 Nasal cavity flows velocity [m/s] septum bending and channel convergence cause acceleration flow is split by inferior and center turbinate converging channel swollen inferior and center turbinate bended septum channel mixing zone left nostril right nostril cavity mixing zone swollen inferior and center turbinate cavity mixing zone acceleration in pharynx right nostril flow is split by inferior turbinate acceleration in pharynx small opening below inferior turbinate acceleration of flow left nostril pharynx pharynx

13 Flow in the human airways Major goals: Basic understanding of the flow in the human lungs: analysis of the respiration process analysis of particle transport Understanding COPD (Chronic Obstructive Lung Disease) particle deposition during exhalation Analysis of Diesel aerosol deposition particle deposition during inhalation

14 Flow in the human lungs

15 Particle transport in the upper human airways First approach: analyze particle transport and deposition in pipes Particle simulation in a bended pipe, Re=1200

16 Structure of ZFS ZFS (C++) Cartesian Grid generator grid refinement BCs Initialization BCs Moving Boundaries BGK-model Finite-Volume solver Lattice-Boltzmann solver Combustion model Particle-Tracing Thermal model Turbulence model Level-Set solver Output Postprocessing

17 Computational hotspots in LBM Solve the Lattice-BGK equation: f i ( x + i t, t + t) =f i ( x, t)+ t (f eq i ( x, t) f i ( x, t)) Loop over all cells: collision step propagation step Lattice-Boltzmann solver iteration loop Collision step 1st BC Loop over BC cells: 1st and 2nd BC Communication Propagation step 2nd BC

18 Inner loop structure LBM-collision (example) //Run over all cells for (ZFSId i=0; i < m_currentmaxnocells; i++){... // Calculate macroscopic variables rho = 0.0; for ( ZFSId j = 0; j < m_nodists; j ++) rho += f_old[i][j];... // Recalculate new distribution functions for (ZFSId j=0; j < nodists; j ++) // Perform relaxation towards equilibrium f[i][j] = f_old[i][j] + omega * (f_eq[i][j] - f_old[i][j]);... }

19 Computational hotspots in FV Solve the NS-equations in integral form: Z ~ dv + I S ~F ~nds =0, ~ Q =(, ~v, e) T Loop over all cells: Finite-Volume solver iteration loop Flux calculation: cell-centered gradients (LS) surface variables (MUSCL) RHS: convective fluxes (AUSM) viscous fluxes Runge-Kutta time-integration Boundary Conditions Flux-calculation Time-integration (BCs) (BCs) Communication Sponge

20 Scalabilty Scalability of ZFS (Lattice-BGK D3Q19) on CRAY Hermit at HLRS Stuttgart 5 ideal speedup x 10 9 cells x 10 9 cells 1.4 ideal speedup x 10 6 cells/core speedup 3 2 speedup number of cores number of cores Strong scaling Weak scaling total number of cells on 4096 cores: x 10⁹

21 The future of ZFS Planned and ongoing implementations: Zonal coupling of: LBM and FV part Structured FV solver FSI-code (DEAL - Uni Siegen) CAA coupling (Discontinuous Galerkin) Implementation of a distributed geometry Load-balancing and dynamic remeshing

22 Summary, Conclusions, and Questions ZFS is a scalable C++ multitool: parallel grid-generator flow solvers (LBM, FV, DG, structured FV) particle tracer post-processing tool ZFS is used in various simulation projects Mechanical engineering applications: 3D-engine flow, Combustion in combustion chambers, Noise prediction in jets of gas turbines Geophysical applications: droplet formation in clouds Bio-fluidmechanical applications: nasal cavity and lung flows ZFS has several hotspots possibly suited for GPU-calculations LBM: collision, propagation, BCs FV: flux computation, BCs

23 Summary, Conclusions, and Questions How expensive is the GPU memory exchange? how do we decrease the memory transfer? e.g. copy all data only once e.g. do exchange only if I/O is required What about the communication? how efficient is Cuda-aware MPI? is it necessary to copy data back to the CPU? What is best-practice GPU programming model? Cuda, OpenACC How big is the required code reconstruction effort? what about compatibility to other systems?