Turbulence in geodynamo simulations

Transcription

1 Turbulence in geodynamo simulations Nathanaël Schaeffer, A. Fournier, D. Jault, J. Aubert, H-C. Nataf,... ISTerre / CNRS / Université Grenoble Alpes Journée des utilisateurs CIMENT, Grenoble, 23 June 2016 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

2 Outline 1 Introduction 2 The XSHELLS code Equations The SHTns library Parallelization strategy and tricks 3 Turbulence in geodynamo simulations nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

3 Cool facts about the Earth s core A broad range of time-scales from months (SV) to million years (reversals) Viscosity of water very low Ekman number E Large scale motions at the top of the core have speeds around 10 km/year (0.3 mm/sec, turnover time is about 200 years) Turbulent motion (very high Reynolds number Re 10 8 ). Magnetic Reynolds number Rm 1000 (Pm 10 5 ) Very low Rossby number Ro Magnetic field is dominated by a tilted dipole. Magnetic energy dominates kinetic energy by a factor 10 4 (4 mt estimated in the core, 0.5 mt or 5 gauss at the surface). Very low Lehnert number (Rossby based on Alfvén speed) Le Heat flux extracted by the mantle ( 10TW, < 100mW/m 2 ). Strong convection (very high Rayleigh number Ra 10 20? Probably many times critical). nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

4 Broad questions Turbulence: does it matter? How? What do the flow and magnetic field look like? What are the basic equilibriums? Can we build a reduced model? nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

5 Broad questions Turbulence: does it matter? How? What do the flow and magnetic field look like? What are the basic equilibriums? Can we build a reduced model? Observations: short time-scales (a few years) Importance of waves? Length-of-day variations? Effect of a stable ocean at the top of the core? nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

6 Basic rotating MHD in planetary cores Navier-Stokes equation t u + (2Ω e z + u) u = p + ν u+ ( b) b αg T r Induction equation Temperature equation t b = (u b) +η b t T + u. T = κ T E = ν/d 2 Ω Ra = T αgd 3 /κν 1 (?) Pm = νµ 0 σ 10 5 Pr = ν/κ 1 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

7 Outline 1 Introduction 2 The XSHELLS code Equations The SHTns library Parallelization strategy and tricks 3 Turbulence in geodynamo simulations nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

8 MHD forced by Boussinesq convection XSHELLS time-steps the following equations: t u + (2Ω 0 + u) u = p + ν u + ( b) b + c Φ 0 t b = (u b) + η b t c + u. (c + C 0 ) = κ c u = 0.b = 0 Spherical shell geometry Fields expanded using u = (T r) + (Pr) Spherical Harmonic expansion, with transforms done by SHTns Finite differences in radius Semi-implicit scheme: diffusive terms are implicit, while other terms are treated explicitly. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

9 Why Spherical Harmonics transform? The goal was to use spherical harmonics to time-step Navier-Stokes and related equations in spherical geometry. Pros Advantage of spectral methods Cons Don t need to solve for magnetic field in the insulator. Strongly reduces the number of variables to solve for. No working fast transform: O(N 3 ) for N 2 degrees of freedom. Non-local (more difficult to parallelize) nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

10 Why Spherical Harmonics transform? The goal was to use spherical harmonics to time-step Navier-Stokes and related equations in spherical geometry. Pros Advantage of spectral methods Cons Don t need to solve for magnetic field in the insulator. Strongly reduces the number of variables to solve for. No working fast transform: O(N 3 ) for N 2 degrees of freedom. Non-local (more difficult to parallelize) Spherical Harmonics Transform (SHT) is the bottleneck nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

11 SHTns: outstanding features Matrix-free algorithm: computing recurrence on-the-fly is faster than reading from memory [unless matrix-matrix product is used], and you save a lot of memory too. Hand vectorized, yet easily portable (currently supports Intel SSE2, AVX, AVX2+FMA, MIC ; IBM Blue Gene/Q) SHTns perf scales with microarchitecture: x2 w/sse2, x4 w/avx, x8 w/avx2+fma... x16 w/avx512? It also means the gap between classic implementations and SHTns will increase. Efficient OpenMP parallelization (not yet used by XSHELLS) Reach 80% to 90% of peak performance on Intel SandyBridge. N. Schaeffer, Efficient Spherical Harmonic Transforms aimed at pseudo-spectral numerical simulations, Gcubed, nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

12 SHTns: other interesting features both scalar and vector transforms. accurate up to spherical harmonic degree l = (at least). SHT at fixed m (without fft, aka Legendre transform). spatial data can be stored in latitude-major or longitude-major arrays. various normalization conventions. can be used from Fortran, c/c++, and Python programs. free software : nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

13 Parallelization strategy hybrid MPI/OpenMP parallelization Increase Data Locality: work shell by shell for spatial terms (do not compute the whole spatial fields at once: lower memory required and faster). Distribute shells to MPI processes (at least 1 shell/process). No transposition required (no MPI Alltoall()), communication with nearest neighbor only. Use OpenMP within these processes (tested up to 32 threads/shell) good scaling (e.g x 1344 x 6912 cores : 0.8 sec/step on Occigen) nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

14 Strategy non-linear terms Design choice: optimize the computation of non-linear terms first. Trick 1: Use SHTns J. Aubert plugged SHTns into Parody and observed a performance increase by a factor of about 2 for the whole code. lower memory required, faster, ready for future architectures. Trick 2: Increase Data Locality Work shell by shell when computing spatial terms (do not compute the whole spatial fields at once). lower memory required and significantly faster. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

15 Strategy linear solver The linear solver is computationally very cheap. It should deal with the data as it is arranged for the non-linear terms to be most efficient. Trick 3: Avoid transpositions and large copies Transposition is always slow (with and without MPI communications) Copying large amounts of data can also be quite slow. Problem: the forward (backward) substitutions depend on the data computed by previous (next) processes. (ISTerre) Earth s core simulations CIMENT 23 June / 29

16 Strategy blocked linear solver Solution to data dependency in the linear solver: split shells into independent blocks. rank k compute rank k+1 rank k+2 xfer compute xfer compute t rank k rank k+1 rank k+2 t Trick 4: Use self-tuning The optimal number of blocks is system dependent: trade off between wait time and transfer overhead. Find the best number of blocks at startup. 30 folds observed performance increase compared to naive solver. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

17 Summary: spherical MHD simulations with XSHELLS Spherical harmonics, finite differences (radial), versatile, very efficient even on your laptop, OpenMP and/or MPI parallelization with good scaling, open source, with user manual. seconds per iteration Performance scaling of xshells (NR=512, Lmax=511) Froggy (SandyBridge) radial OpenMP Froggy (SandyBridge) in-shell OpenMP Turing (Blue Gene/Q) in-shell OpenMP number of cores Figure 1: Strong scaling of XSHELLS with spatial resolution N r = 512, L max = nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

18 Outline 1 Introduction 2 The XSHELLS code 3 Turbulence in geodynamo simulations nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

19 The model with: Earth s core geometry (Sphere) thermochemical convection (codensity, 75% chemical driving, Aubert et al 2009); no-slip, and fixed flux homogeneous boundary conditions high rotation rate, low viscosity strong forcing (more than 4000 times critical) 1 Ekman number E = ν/d 2 Ω 2 Rayleigh number Ra = T αgd 3 /κν 3 Magnetic prandtl number Pm = νµ 0 σ 4 (Thermal) Prandtl number Pr = 1. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

20 The simulations The idea Keep super-criticality and Rm = UD/η 650 fixed. go to more Earth-like A = U/B and Pm = ν/η Earth Soderlund et. al Christensen & Aubert 2006 Andrey Sheyko 2014 Other simulations Highway initial: E = 10 5, Pm = 0.4, Ra = A = 1.5 F ν = 47% A Pm nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

21 The simulations The idea Keep super-criticality and Rm = UD/η 650 fixed. go to more Earth-like A = U/B and Pm = ν/η. A Earth Soderlund et. al Christensen & Aubert 2006 Andrey Sheyko 2014 Other simulations Highway initial: E = 10 5, Pm = 0.4, Ra = A = 1.5 F ν = 47% jump 1: E = 10 6, Pm = 0.2, Ra = A = 0.61 F ν = 24% Pm nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

22 The simulations The idea Keep super-criticality and Rm = UD/η 650 fixed. go to more Earth-like A = U/B and Pm = ν/η. A Earth Soderlund et. al Christensen & Aubert 2006 Andrey Sheyko 2014 Other simulations Highway Pm initial: E = 10 5, Pm = 0.4, Ra = A = 1.5 F ν = 47% jump 1: E = 10 6, Pm = 0.2, Ra = A = 0.61 F ν = 24% jump 2: E = 10 7, Pm = 0.1, Ra = A = 0.45 F ν = 17% Extreme parameters, require 2688 x 1344 x 1024 points (3.7 billions) 7 months computation on 512 cores (10.5 sec/step) nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

23 Energy vs Time 10 9 S2: E = 10 7 Pm = 0: 1 S1*: E = 10 6 Pm = 0 energy 10 8 jump 2 jump 1 S1: E = 10 6 Pm = 0: S0: E = 10 5 Pm = 0: t =2¼ Jump 2 ran for 1.5% of a magnetic diffusion time, and it took about 7 months to compute on 512 cores, spending 2.5 million core hours. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

24 Snapshot: initial U φ (E = 10 5, Pm = 0.4, A = 1.5) NR = 224, L max = 191 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

25 Snapshot: jump 1 U φ (E = 10 6, Pm = 0.2, A = 0.62) NR = 512, L max = 479 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

26 Snapshot: jump 2 U φ (E = 10 7, Pm = 0.1, A = 0.43) NR = 1024, L max = 893 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

27 Jump 2: Temperature field Mean temperature of each shell has been removed. (ISTerre) Earth s core simulations CIMENT 23 June / 29

28 Averages of U and B U B nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

29 Jump 2: Non-zonal mean flow Homegeneous heat flux start to produce large scale flows at E = 10 7 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

30 Jump 2: Space-time Fourier analysis Fourier Transform in the two homogeneous directions: t and φ. Two different regions inside and outside the tangent cylinder. E = 10 7, Pm = 0.1 nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

31 Jump 2: Space-time Fourier analysis Main balance of terms in vorticity equation: (ISTerre) Earth s core simulations CIMENT 23 June / 29

32 Jump 1: Influence of the magnetic field 71e8 6 5 Ek, step1 Em, step1 Ek, no B energy time (viscous) Figure 2: Energies as a function of time (normalized by the viscous diffusion time) for case jump 1 (E = 10 6 ) and for the same parameters as jump 1, but without magnetic field. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

33 Jump 1: Influence of the magnetic field With a strong dynamo magnetic field: Zonal jets are suppressed, plumes extend further. Larger plumes but convection still starts at small length-scale! (E 1/3 ). nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

34 Some numbers definition initial jump 1 jump 2 Earth s core N r L max Ek ν/d 2 Ω Ra T αgd 3 /κν ? Pm ν/η Pr ν/κ Rm UD/η ? A µρu/b Re UD/ν Ro U/DΩ Le B/ µρdω Λ B 2 /ηω F ν D ν /(D η + D ν ) 47% 24% 17%? F η D η /(D η + D ν ) 53% 76% 83%? Table 1: Various input and output parameters of our simulations, where D is the shell thickness, U the rms velocity and B the rms magnetic field. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

35 Jump 2: spectra Ek deep Ek surface Em deep Em surface l/r E = 10 7 Pm = 0.1 Ra = Rm = 600 A = 0.45 Λ = 1.2 F ν = 17% NR = 1024 L max = 893 Magnetic field dominates deep in the core but not near the surface. Velocity spectrum nearly flat at the surface but increasing deep down. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

36 Jump 2: z-averaged energy densities z-averaged equatorial energy densities, left: < U 2 eq >, right: < B 2 eq >. E = 10 7, Pm = 0.1, Rm = 600, A nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

37 20 years of geodynamo simulations 1995 : Glatzmaier & Roberts Chebychev, 64 x 32 x 49 hyperviscosity Earth-like, reversals, and all the hype. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

38 20 years of geodynamo simulations 1995 : Glatzmaier & Roberts Chebychev, 64 x 32 x 49 hyperviscosity Earth-like, reversals, and all the hype : Christensen & Aubert Chebychev, 168 x 336 x 97 E = , Pm=0.06 Extensive parameter study, scaling laws. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

39 20 years of geodynamo simulations 1995 : Glatzmaier & Roberts Chebychev, 64 x 32 x 49 hyperviscosity Earth-like, reversals, and all the hype : Christensen & Aubert Chebychev, 168 x 336 x 97 E = , Pm=0.06 Extensive parameter study, scaling laws : Kageyama et. al. Yin-Yang grid, 2048 x 1024 x 511 E = 10 6, Re=700, Pm=1 convection sheets, zonal jets. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

40 20 years of geodynamo simulations 1995 : Glatzmaier & Roberts Chebychev, 64 x 32 x 49 hyperviscosity Earth-like, reversals, and all the hype : Christensen & Aubert Chebychev, 168 x 336 x 97 E = , Pm=0.06 Extensive parameter study, scaling laws : Kageyama et. al. Yin-Yang grid, 2048 x 1024 x 511 E = 10 6, Re=700, Pm=1 convection sheets, zonal jets : Sakuraba & Roberts Chebychev, 768 x 384 x 160 E = , Re=650, Pm=0.2 Fixed heat flux leads to stronger magnetic field. nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29

41 20 years of geodynamo simulations 1995 : Glatzmaier & Roberts Chebychev, 64 x 32 x 49 hyperviscosity Earth-like, reversals, and all the hype : Christensen & Aubert Chebychev, 168 x 336 x 97 E = , Pm=0.06 Extensive parameter study, scaling laws : Kageyama et. al. Yin-Yang grid, 2048 x 1024 x 511 E = 10 6, Re=700, Pm=1 convection sheets, zonal jets : Sakuraba & Roberts Chebychev, 768 x 384 x 160 E = , Re=650, Pm=0.2 Fixed heat flux leads to stronger magnetic field. Many others nathanael.schaeffer@ujf-grenoble.fr (ISTerre) Earth s core simulations CIMENT 23 June / 29