nek5000 massively parallel spectral element simulations

Transcription

1 nek5000 massively parallel spectral element simulations PRACE Scientific Seminar HPC Boosts Science, 22th February 2011 P. Schlatter & D. S. Henningson Linné Flow Centre, KTH Mechanics

2 Fluid flows Tornado, USA ( Waterfall, Sweden ( 2007) Airplane wing (Wikipedia) 2

3 Turbulence Laminar flow Transitional flow Re = UL/ Turbulent flow > 10 4 flow likely to be turbulent L=1 m U=1 m/s Re = 1*1/10-6 =10 6 Water: = 10-6 m 2 /s 3

4 Outline 1. Numerics and large-scale computations Flow simulations and spectral elements Performance 2. Flow application: Turbulent flow in a 3D diffuser Comparison to real experiments 4

5 1. Numerics and large-scale computations 5

6 Flow simulations Smoke visualization ( weeping-willow/ /) Measurements Computer simulation (Stefan Kerkemeier, ETH) Navier-Stokes equations 6

7 Flow simulations Nonlinear, chaotic, singularly perturbed, mixed hyperbolicparabolic... Direct numerical simulation (DNS) Challenge I: n ~ O(Re 3/4 ) in each spatial direction, O(Re 1/2 ) in time resolution ~ O(Re 11/4 ) Challenge II: Incompressibility Elliptic problem for pressure (global coupling in every time step) Choose a high order method! 7

8 Spectral elements (Patera 84) Finite element (FE) Spectral Spectral element (SE) + (Zhang et al. 2004) 8

9 Spectral elements Similarities to FEM: Variational formulation. Strong form Weak form Galerkin approx. Matrix form Departure from FEM: Orthogonal Legendre polynomials, Gauss- Lobatto quadrature. Domain partitioned into E high-order quadrilateral (or hexahedral) elements. Nodal basis: Solution represented as N th-order Lagrange interpolants based on Gauss-Lobatto-Legendre points (N ~ 4 15). Converges exponentially fast with N for smooth solutions. Part of diffuser geometry with E = , N = 11 9

10 Spectral elements Local tensor-product form (2D), h i (r) allows derivatives to be evaluated as matrixmatrix products: 90% of ops, 40% of time is in the form of small matrix-matrix products mxm-routines written in assembly by IBM 10

11 nek5000 SEM code by Paul F. Fischer, Argonne National Laboratory 80,000 lines of f77 (some C for I/O), MPI (no hybrid MPI-OMP) Keep it simple world s most powerful computers have very weak operating systems Much effort spent on the coarse grid solvers (AMG and XX T ) Argonne National Lab,

12 nek5000 Turbulent diffuser cores 1. Real case 2. Strong scaling cores 12

13 2. Flow application: Turbulent flow in a 3D diffuser 13

14 Diffuser experiment at Stanford Cherry et al. (2008) Turbulent water u b = 1 m/s, h = 1cm Re b = 10,000 Fully 3D Complex separation 14

15 Setup in Ohlsson et al. (2010) First DNS Re b = 10,000 High order SEM code 220 million grid points 15

16 Complex flow 16

17 Complex flow 17

18 Observed Instantaneous large-scale visualization oscillations 18

19 Instantaneous visualization x=12 19

20 Mean flow Skin-friction Pressure Reynolds recovery number Ohlsson et al., JFM

21 Mean flow <u> in streamwise planes at 2, 5, 8 cm from the diffuser throat Numerical simulation by nek5000 (left) Experiment by Cherry et al. (right) Ohlsson et al., JFM

22 Observed large-scale oscillations 22

23 Thank you! 23

24 Time probes x=12 Powerspectrum T ~ 80 T ~ 20 24

25 Proper Orthogonal Decomposition (POD) Principal component analysis (Pearson 1901, Lumley 1967) Finds the directions of most variance in random dataset Eigenvectors of the covariance matrix N 25

26 POD (structures) log(e) mean Modes 26

27 POD (frequencies) T ~ 80 a 1,a 2,a 3 Powerspectrum Time series 27

28 Phase-portraits Signs of a quasi-periodic motion 28

29 Why dominant frequencies? Friday evening... Villermaux & Hopfinger (1994) Lawson & Davidson (2001) Confinement!

30 Low-order reconstruction log(e) mean Modes 30

31 Low-order reconstruction Fluctuations from POD modes 1,2 Fluctuations from real snapshots Fluctuations from experiments 31

32 Conclusions, part 1 Fluid flows nonlinear, chaotic, sensitive calls for correct numerical treatment. Resolution ~ O(Re 11/4 ) Parallel computations unavoidable Incompressibility introduces heavy communication. Spectral elements offer accuracy, scalability and flexibility. Stabilization needed for turbulence simulations 32

33 Conclusions, part 2 Dynamics in diffuser governed by jet- flapping, due to the confinement of the flow Most energetic POD modes are pairs of streamwise streaks shedding. Typical frequencies found through time probes & POD: Large-scale motion with T ~ 80. Large-scale motion give explanation for double peak r.m.s. 33

34 Navier-Stokes Eq. (3D) K-type Transition Ohlsson et al. LNCSE 76 34

35 Navier-Stokes Eq. (3D) K-type Transition No overintegration blow up just before skin-friction peak Resolution 1: 91 3 Resolution 2: Overintegration M=N+1 M=N+2 M=N+3 M=N+4! Ohlsson et al. LNCSE 76 35

36 Stabilization of SEM Proposed methods: Overintegration [Kirby et al., Maday et al., Canuto et al.] N M M = 3/2 N N M N N Filtering [Fischer & Mullen] Spectral Vanishing Viscosity (SVV) [Pasquetti et al.] 36

37 Navier-Stokes Eq. (3D) Turbulent Channel Flow at Re = 590 M = N+4 M = 3/2 N Ohlsson et al. LNCSE 76 37

38 Setup in Ohlsson et al. (2010) laminar inflow development section straight + converging section trip-forcing region diffuser fully turbulent flow outflow with damping region Re b = 10,000 Re_tau = 320 (970) 38

39 Flow simulations High-order discretization in space + Small dispersion errors (grow linearly with time) + Less grid points for given accuracy Complex geometries cumbersome Time discretization Nonlinear terms explicitly, viscous terms implicitly (3rd order) 39

40 Time probes x=-13 Powerspectrum T=33 T + =3100 x + ~1000 t + = x + / u c

41 Mean flow Green: U/u b = 0 Red: U/u b =

42 Why this bump? (Hypothesis 2) Present at all times! 42

43 Motivation 2D (Kuwahara et al., icfd, Tokyo) Steady point of separation Unsteady reattachment point 3D Pressure induced separation Unsteady separation point (Ohlsson et al. 2008) (Ohlsson et al. 2010) No homogeneous direction except time Increased unsteadiness Influence of walls and secondary flows..? 43

44 Conclusions Indications that dynamics in diffuser governed by jet instabilities. Walls contribute to complex secondary flow in diffuser. Largest pancake structures of found right after diffuser opening, due to spanwise vorticity and dw/dz dominating. 44

45 Results fluctuations (u rms /u b )x100 in streamwise planes at 2, 5, 12 cm from the diffuser throat Numerical simulation by nek5000 (left) Experiment by Cherry et al. (right) 45

46 Diffuser experiment at Stanford 46