Case Study: Numerical Simulations of the Process by which Planets generate their Magnetic Fields. Stephan Stellmach UCSC

Transcription

1 Case Study: Numerical Simulations of the Process by which Planets generate their Magnetic Fields Stephan Stellmach UCSC 1

2 The Scientific Problem 2

3 The Magnetic Field of the Earth 3

4 Contributions to the Earth s magnetic field radial component of the Earth s magnetic field at satellite altitude. Courtesy of Nils Olsen 4

5 The Magnetic Field of the Earth Deviations from a purely dipolar field: declination

6 The magnetic field of the Earth: Secular Variation Jackson et al, 2000

7 Changes of the Earth Magnetic Field: The magnetic Poles b

8 The magnetic field of the Earth: Reversals

9 The magnetic field of the Earth: Reversals

10 The magnetic field of the Earth: Reversals

11 The geodynamo hypothesis fluid flow In du ct io n electric field current density O hm s La w Ampère s Law magnetic field Geoforschungszentrum Potsdam 11

12 The Mathematical Problem: momentum equation induction equation solenoidal conditions energy equation Par. Definition Meaning Planetary core Ra = α Τ g d / (2 Ω κ) ~ buoyancy forces/ rotational forces? q = κ / η ~ magn. diff. time / therm. diff. time O(10-6) Pr =ν / κ ~ therm. diff. time / visc. diff. time O(10-2) E = ν / (2 Ω L2) ~ rot. time scale / visc. diffusion time O(10-15) 12

13 Convection in Rotating Spherical Shells Busse, 1970 Kageyama & Sato, 1997

14 Cartridge belt dynamo α Effekt Kageyama & Sato, 1997 Ω Effekt

15 Numerical Simulations of Reversals Gary Glatzmaier

16 Parameter ranges for numerical simulations 10? The problem today Earth We still can't simulate the dynamo process under realistic conditions:? Ra num mo erical del s E flows are very turbulent We can't resolve all length scales We can't resolve all time scales

17 Local models to study the planetary dynamo problem spherical shell models realistic geometry might be compared to measured data simple local plane layer model more efficient numerics larger parameter ranges more small scale structure plane boundaries conducting fluid use periodic BCs in x,y-direction Investigate simple plane layer model to isolate and understand relevant physical processes 17

18 Goal: Develop simulation software for Cartesian dynamos which runs efficiently on massively parallel computers 18

19 The numerical method we dream of: Wish List: optimal accuracy for a given spatial and temporal resolution temporal (and spatial) resolution dictated by accuracy considerations, not by stability issues optimal operation count as small as possible per time step floating point operations per time step ~ #(unknowns) per time step should be able to handle all types of boundary conditions should run efficiently on all types of machines, including the fastest massively parallel machines available strong scaling: For a given resolution, computing time ~ #(CPUs)-1 weak scaling: If the resolution is increased along with #(CPUs) such that the data volume per CPU remains constant, the computing time should also remain constant. 19

20 High Performance Computers Numerical Simulations at extreme parameter values are necessary to understand natural dynamos. Use fastest machines available today! fastest today 20

21 The fastest machine available today: IBM Blue Gene IBM Blue Gene/L scalable to at least nodes Power PC 440 CPUs, 700 MHz relatively slow good flops/mw ratio 512 MB per node relatively small Networks: 3D torus collective Petascale computing by the end of the decade? 21

22 Which numerical approach is the way to go??? Question: Can we really exploit such machines with the current models? Possible choices: global spectral methods local low order methods spectral transform method global coupling FD, FV, FE,... only local coupling 22

23 First Option: Use a Local Discretization Method Example: Finite Volume Method Basic idea of finite volume methods Divide the domain into control volumes Integrate over control volume Use numerical Approximations for surface integrals gradients volume integrals If 2nd order approx. are used, only data from the six neighboring control volumes is needed 23

24 First Option: Use a local method How to parallelize a local method? Only data from a few grid levels (marked green) has to be exchanged! 24

25 Second Option: Use a global spectral method Basic Idea of Spectral Methods: Assume that we want to solve the differential equation: Expand all unknowns u(x) into a finite series of basis functions φn Substitute finite series into differential equation Minimize Residual function R Spectral methods mainly differ in minimization techniques and chosen basis functions 25

26 Second Option: Use a global spectral method How to parallelize a spectral method? Different algorithms have been proposed (we discuss one in particular later on...) But one thing is clear: To compute a spectral transform, global communication will be necessary! 26

27 Here, we had to make a first decision: We decided to use a global spectral method - mainly because of its superior accuracy 27

28 How it works in detail: Spectral Method I/V Eight scalar PDFs Reduce to five scalar PDEs Horizontal expansion Apply time stepping vertical discretization We have eight scalar PDEs for the eight unkowns 28

29 How it works in detail: Spectral Method II/V Eight scalar PDFs Reduce to five scalar PDEs Horizontal expansion Apply time stepping vertical discretization Result from potential theory: Solenoidal vector fields can be described by to potential function, called toroidal and poloidal potentials: Horizontal averages < > are added here to guarantee that e,f,g,h are horizontally periodic. Substitution into the governing equation leads to: five PDEs for the toroidal potentials (e, g), poloidal potentials (f, h) and for the temperature T. (+ equations for the so called mean fields <ux>, <uy >, <Bx>, <By > which only depend only on z,t and are therefore inexpensive to solve.) 29

30 How it works in detail: Spectral Method II/V Eight scalar PDFs Reduce to five scalar PDEs to be more specific: apply and Horizontal expansion Apply time stepping vertical discretization to the governing equations: with 30

31 How it works in detail: Spectral Method III/V Eight scalar PDFs Reduce to five scalar PDEs Horizontal expansion Apply time stepping vertical discretization We have periodic horizontal boundary conditions -> use Fourier basis functions Φ with Гx and Гy are the length and depth of the computational domain, Lmax and Mmax are the number of Fourier modes to use Substitute into governing equations and use orthogonality of basis functions This leads to: Nx Χ Ny PDEs for the coefficients 31

32 How it works in detail: Spectral Method III/V Eight scalar PDFs Reduce to five scalar PDEs Horizontal expansion Apply time stepping vertical discretization Again, to be more specific: with 32

33 How it works in detail: Spectral Method IV/V Eight scalar PDFs Reduce to five scalar PDEs Apply time stepping vertical discretization Use semi implicit time stepping scheme (explicit for non-linear terms F, implicit for remaining linear terms) Backward-DifferencingFormula scheme for linear terms (2nd order) Horizontal expansion BDF2/AB2 scheme Adams-Bashforth scheme for nonlinear terms (2nd order) This leads to: Nx Χ Ny ODEs for the Coefficients where Ф=(e,f,g,h,T) and the index (n+1) denotes the new time level. 33

34 How it works in detail: Spectral Method V/V Eight scalar PDFs Reduce to five scalar PDEs Horizontal expansion Apply time stepping vertical discretization Boundary conditions on top and bottom boundary -> Use expansion into Chebyshev-Polynomials graphical visualization of Chebyshev polynomials z d e re g e 34

35 How it works in detail: Spectral Method V/V Eight scalar PDFs Reduce to five scalar PDEs Horizontal expansion Use the Tau method to minimize the residual Leads to: Apply time stepping vertical discretization Nx Ny quasi-tridiagonal systems of equations for the Coefficients Nonlinear terms on right hand side are evaluated in physical (x,y,z) space and are then transformed to spectral space Use Gauss-Lobatto points in physical space -> fast transforms are available for conversion between physical and transform space 35

36 Back to our wish list What is possible with our spectral code? optimal accuracy for a given spatial and temporal resolution temporal (and spatial) resolution dictated by accuracy considerations, not by stability issues optimal operation count as small as possible per time step floating point operations per time step ~ #(unknowns) per time step should run efficiently on all types of machines, including the fastest massively parallel machines available Well, we have not discussed the parallelization yet... 36

37 Question: How do we parallelize a spectral method like this? 37

38 Biggest problem: How to parallelize the transforms??? 38

39 First observation: We have chosen a tensor-product basis, i.e. our 3D basis functions φ(x,y,z) are products of 1D basis functions: φ(x,y,z) = exp(iαx) exp(iβy) Tn(z) We can thus do the transform in three steps! 1) Perform a real-to-complex FFT of the x-coordinate 2) Then, perform a complex-to-complex FFT of the y-coordinate 3) Then, perform a Chebyshev transform along z For Backward Transform, do reverse transforms in reverse order. Idea: Do each step in-processor, each CPU for a subset of data. Communicate data between the steps. 39

40 Parallel Transforms: slabs for small Linux cluster Physical space: each processor holds xy-slab Fourier/Chebyshev Space: each process holds zy-slab xy-slab zy-slab 2d real to complex FFT transpose from xy-slabs to yz-slabs Chebyshev transform along z 40

41 Parallel Transforms: pencils for a massively parallel machine Physical space: each processor holds x-pencil Fourier/Chebyshev Space: each process holds z-pencil x-pencils z-pencils Intermediate state: each process holds y-pencil y-pencils FFT along x transpose FFT along y transpose Chebyshev transf. along z 41

42 Second observation: With this data layout, all other steps in our algorithm don't require any communication! Transforms are the only operations which need explicit message passing! So, after parallelizing the transforms, we are almost done... 42

43 How efficient is our method on O(104) CPUs? open symbols: Virtual Node Mode filled symbols: Communication Coprocessor Mode 43

44 How efficient is our method on O(104) CPUs? With FPUs, the efficiency on a 2563 grid is still > 40% 44

45 Weak scaling: What happens if we increase resolution? 45

46 Weak scaling: What happens if we increase resolution? If we have more than a few ten thousand grid points per task, we achieve good weak scaling (up to the logarithmic factor log2(n)). 46

47 First conclusions It is possible to achieve good (weak and strong) scaling on massively parallel machines even with global spectral methods. But: Detailed timing shows that the code is clearly bandwidth-limited (i.e. parallel transposes scale, but consume a large fraction of the computing time) So, was it wise to choose the spectral approach? 47

48 Was is wise to choose the spectral approach? Actually, we tried the finite volume approach as well... No simple yes/no answer Apart from parallelization issues, each method has pros & cons (stability, accuracy, cache coherence...) As expected, finite volume code also scales reasonably Main difference: Spectral code exchanges huge amounts of data once in a while Finite volume code exchanges small amounts of data all the time Much will depend on detailed network structure 48

49 Application Example: Turbulent rotating Convection 49

50 Convection in a Rotating Rayleigh Benard Layers 50

51 Thank you!!! 51