nek5000 massively parallel spectral element simulations PRACE Scientific Seminar HPC Boosts Science, 22th February 2011 P. Schlatter & D. S. Henningson Linné Flow Centre, KTH Mechanics
Fluid flows Tornado, USA (www.fas.org) Waterfall, Sweden ( 2007) Airplane wing (Wikipedia) 2
Turbulence Laminar flow Transitional flow Re = UL/ Turbulent flow > 10 4 flow likely to be turbulent L=1 m U=1 m/s Re = 1*1/10-6 =10 6 Water: = 10-6 m 2 /s 3
Outline 1. Numerics and large-scale computations Flow simulations and spectral elements Performance 2. Flow application: Turbulent flow in a 3D diffuser Comparison to real experiments 4
1. Numerics and large-scale computations 5
Flow simulations Smoke visualization (http://www.flickr.com/photos/ weeping-willow/2728907738/) Measurements Computer simulation (Stefan Kerkemeier, ETH) Navier-Stokes equations 6
Flow simulations Nonlinear, chaotic, singularly perturbed, mixed hyperbolicparabolic... Direct numerical simulation (DNS) Challenge I: n ~ O(Re 3/4 ) in each spatial direction, O(Re 1/2 ) in time resolution ~ O(Re 11/4 ) Challenge II: Incompressibility Elliptic problem for pressure (global coupling in every time step) Choose a high order method! 7
Spectral elements (Patera 84) Finite element (FE) Spectral Spectral element (SE) + (Zhang et al. 2004) 8
Spectral elements Similarities to FEM: Variational formulation. Strong form Weak form Galerkin approx. Matrix form Departure from FEM: Orthogonal Legendre polynomials, Gauss- Lobatto quadrature. Domain partitioned into E high-order quadrilateral (or hexahedral) elements. Nodal basis: Solution represented as N th-order Lagrange interpolants based on Gauss-Lobatto-Legendre points (N ~ 4 15). Converges exponentially fast with N for smooth solutions. Part of diffuser geometry with E = 127750, N = 11 9
Spectral elements Local tensor-product form (2D), h i (r) allows derivatives to be evaluated as matrixmatrix products: 90% of ops, 40% of time is in the form of small matrix-matrix products mxm-routines written in assembly by IBM 10
nek5000 SEM code by Paul F. Fischer, Argonne National Laboratory 80,000 lines of f77 (some C for I/O), MPI (no hybrid MPI-OMP) Keep it simple world s most powerful computers have very weak operating systems Much effort spent on the coarse grid solvers (AMG and XX T ) Argonne National Lab, 2008 11
nek5000 Turbulent diffuser 65 536 cores 1. Real case 2. Strong scaling 32 768 cores 12
2. Flow application: Turbulent flow in a 3D diffuser 13
Diffuser experiment at Stanford Cherry et al. (2008) Turbulent water u b = 1 m/s, h = 1cm Re b = 10,000 Fully 3D Complex separation 14
Setup in Ohlsson et al. (2010) First DNS Re b = 10,000 High order SEM code 220 million grid points 15
Complex flow 16
Complex flow 17
Observed Instantaneous large-scale visualization oscillations 18
Instantaneous visualization x=12 19
Mean flow Skin-friction Pressure Reynolds recovery number Ohlsson et al., JFM 2010 20
Mean flow <u> in streamwise planes at 2, 5, 8 cm from the diffuser throat Numerical simulation by nek5000 (left) Experiment by Cherry et al. (right) Ohlsson et al., JFM 2010 21
Observed large-scale oscillations 22
Thank you! 23
Time probes x=12 Powerspectrum T ~ 80 T ~ 20 24
Proper Orthogonal Decomposition (POD) Principal component analysis (Pearson 1901, Lumley 1967) Finds the directions of most variance in random dataset Eigenvectors of the covariance matrix N 25
POD (structures) log(e) mean Modes 26
POD (frequencies) T ~ 80 a 1,a 2,a 3 Powerspectrum Time series 27
Phase-portraits Signs of a quasi-periodic motion 28
Why dominant frequencies? Friday evening... Villermaux & Hopfinger (1994) Lawson & Davidson (2001) Confinement!
Low-order reconstruction log(e) mean Modes 30
Low-order reconstruction Fluctuations from POD modes 1,2 Fluctuations from real snapshots Fluctuations from experiments 31
Conclusions, part 1 Fluid flows nonlinear, chaotic, sensitive calls for correct numerical treatment. Resolution ~ O(Re 11/4 ) Parallel computations unavoidable Incompressibility introduces heavy communication. Spectral elements offer accuracy, scalability and flexibility. Stabilization needed for turbulence simulations 32
Conclusions, part 2 Dynamics in diffuser governed by jet- flapping, due to the confinement of the flow Most energetic POD modes are pairs of streamwise streaks shedding. Typical frequencies found through time probes & POD: Large-scale motion with T ~ 80. Large-scale motion give explanation for double peak r.m.s. 33
Navier-Stokes Eq. (3D) K-type Transition Ohlsson et al. LNCSE 76 34
Navier-Stokes Eq. (3D) K-type Transition No overintegration blow up just before skin-friction peak Resolution 1: 91 3 Resolution 2: 127 3 Overintegration M=N+1 M=N+2 M=N+3 M=N+4! Ohlsson et al. LNCSE 76 35
Stabilization of SEM Proposed methods: Overintegration [Kirby et al., Maday et al., Canuto et al.] N M M = 3/2 N N M N N Filtering [Fischer & Mullen] Spectral Vanishing Viscosity (SVV) [Pasquetti et al.] 36
Navier-Stokes Eq. (3D) Turbulent Channel Flow at Re = 590 M = N+4 M = 3/2 N Ohlsson et al. LNCSE 76 37
Setup in Ohlsson et al. (2010) laminar inflow development section straight + converging section trip-forcing region diffuser fully turbulent flow outflow with damping region Re b = 10,000 Re_tau = 320 (970) 38
Flow simulations High-order discretization in space + Small dispersion errors (grow linearly with time) + Less grid points for given accuracy Complex geometries cumbersome Time discretization Nonlinear terms explicitly, viscous terms implicitly (3rd order) 39
Time probes x=-13 Powerspectrum T=33 T + =3100 x + ~1000 t + = x + / u c 3300 40
Mean flow Green: U/u b = 0 Red: U/u b = -0.1 41
Why this bump? (Hypothesis 2) Present at all times! 42
Motivation 2D (Kuwahara et al., icfd, Tokyo) Steady point of separation Unsteady reattachment point 3D Pressure induced separation Unsteady separation point (Ohlsson et al. 2008) (Ohlsson et al. 2010) No homogeneous direction except time Increased unsteadiness Influence of walls and secondary flows..? 43
Conclusions Indications that dynamics in diffuser governed by jet instabilities. Walls contribute to complex secondary flow in diffuser. Largest pancake structures of found right after diffuser opening, due to spanwise vorticity and dw/dz dominating. 44
Results fluctuations (u rms /u b )x100 in streamwise planes at 2, 5, 12 cm from the diffuser throat Numerical simulation by nek5000 (left) Experiment by Cherry et al. (right) 45
Diffuser experiment at Stanford 46