Frank Jenko. Session on Plasma Physics: Petascale Computations of Turbulence in Laboratory and Astrophysical Plasmas

Transcription

1 Frank Jenko Session on Plasma Physics: Petascale Computations of Turbulence in Laboratory and Astrophysical Plasmas Max-Planck-Institut für Plasmaphysik, Germany 3 rd EU-US HPC Summer School Dublin, June 2012

2 The plasma universe

3 More than 99% of the visible universe is in a plasma state

4 Aurora borealis

5 High-k Alfvén wave turbulence: Observations versus simulations AW turbulence in the solar wind Bale et al., PRL 2005 GK simulation of AW turbulence Howes et al., PRL 2008

6 Plasmas in fusion research: ITER Idea: New source of CO 2 free energy for centuries to come Magnetic confinement in a large tokamak Goal: 500 MW of fusion power The 7 ITER parties

7 The resources for fusion energy are practically unlimited Deuterium in a bath tub full of water and Lithium in a used laptop battery suffice for a family over 50 years

8

9 What is turbulence? Turbulence is a nonlinear phenomenon Leonardo da Vinci (1529) occurs (only) in open systems involves many degrees of freedom is highly irregular (chaotic) in space and time often leads to a (statistically) quasi-stationary state far from thermodynamic equilibrium There might be some hope to 'break the deadlock' by extensive, but well-planned, computational efforts... John von Neumann

10 Interdisciplinary aspects This research involves aspects of: plasma physics turbulence physics space and astrophysics applied mathematics and computer science This research is done in a collaborative way: GENE development team (gene.rzg.mpg.de) ERC project on plasma turbulence & exascale computing ( Additional projects with experts from applied mathematics and computer science as well as from neighboring areas of physics

11 Minimizing the effort

12 Plasma turbulence: Kinetic theory! Dilute and/or hot plasmas are almost collisionless. Thus, if kinetic effects (finite Larmor radius, Landau damping, magnetic trapping etc.) play a role, magnetohydrodynamics is not applicable, and one has to use a kinetic description! Vlasov-Maxwell equations Removing the fast gyromotion leads to a dramatic speed-up Charged rings as quasiparticles; gyrocenter coordinates; keep kinetic effects Brizard & Hahm, Rev. Mod. Phys. 79, 421 (2007)

13 The nonlinear gyrokinetic equations Advection/Conservation equation X = gyrocenter position װV = parallel velocity µ = magnetic moment Appropriate field equations Nonlinear 5D equations; removal of irrelevant space-time scales

14 Turbulent fluctuations are quasi-2d Reason: Strong background magnetic field Use field-aligned coordinates and minimize the simulation volume

15 Reduction of simulation volume: full torus annulus flux tube Non-negligible contributions from sub-ion scales: ETG modes and microtearing modes

16 Major theoretical speedups G.Hammett relative to original Vlasov/pre-Maxwell system on a naïve grid, for ITER 1/ρ * = a/ρ ~ 1000 n Nonlinear gyrokinetic equations eliminate plasma frequency: ω pe /Ω i ~ m i /m e x10 3 eliminate Debye length scale: (ρ i /λ De ) 3 ~ (m i /m e ) 3/2 x10 5 average over fast ion gyration: Ω i /ω ~ 1/ρ * x10 3 n Field-aligned coordinates adapt to elongated structure of turbulent eddies: Δ /Δ ~ 1/ρ * x10 3 n Reduced simulation volume reduce toroidal mode numbers (i.e., 1/15 of toroidal direction) x15 L r ~ a/6 ~ 160 ρ ~ 10 correlation lengths x6 n Total speedup x10 16 n For comparison: Massively parallel computers ( ) x10 7

17 Extreme computing in support of ITER: The GENE code

18 The simulation code GENE GENE is physically comprehensive and computationally efficient with applications in fusion research and astrophysics Two main goals: deeper understanding of fundamental physics issues and direct comparisons with experiments/observations The differential operators are discretized via a combination of spectral, finite difference, finite element, and finite volume methods; the emphasis is on high scalability GENE is developed cooperatively by an international team, and it is publicly available (gene.rzg.mpg.de) GENE-based proposals have been selected as one of the first PRACE projects and again recently in the 4 th Call

19 The linear GK eigenvalue problem Standard approach: Initial value computation direct EV solver Christoph Kowitz, Master s Thesis

20 Iterative EV solvers via SLEPc Available methods: An extension of PETSc J.E. Roman et al., Valencia Recent result: Christoph Kowitz, Master s Thesis

21 GENE parallelization Parallelization/optimization strategy: - high-dimensional domain decomposition - either pure MPI or mixed MPI/OpenMP paradigm - optimal subroutines and processor layout determined during initialization phase (à la FFTW) - time step is chosen in an optimal way

22 GENE on BlueGene/P (strong scaling) JUGENE at Jülich

23 Some computational challenges GENE runs are compute intensive; large individual runs may require up to several million core-hours Large runs use more than grid points and require up to about 1 TB of short-term storage Long-term storage on local tapes (data transfer via simple scp or rsync commands) Many different HPC platforms are used in parallel Currently, GENE is being ported to GPGPU & Intel MIC systems

24 Building an international user base Developing a cutting-edge code like GENE requires many man-years and a lot of endurance Much work went into making the code user-friendly Release versions of the code are distributed via a website The code comes with extensive documentation (50-page manual and 50-page tutorial) A lot of time and energy also went into direct user support This is a service to the community, with costs and benefits

25 Input: GENE Launcher

26 Output: GENE Diagnostics Tool

27 Visualization of GENE results

28 A few words on code development

29 The need for version control (I) Keep track of changes(!) even in single developer cases, e.g., for more efficient bug hunting Avoid destructive interference between developers/developer groups: taken from: [Version Control with Subversion,

30 The need for version control (II) OpenSource solution provided by subversion.apache.org: taken from: [Version Control with Subversion, existing source code can easily be integrated into a subversion repository modifications can be monitored via web interfaces

31 In-Source-Documentation Even for single developers: Try to add meaningful comments in the source code! There s always the possibility that your code shall be used/extended by other people OpenSource software tools like help to create a HTML/PDF documentation:

32 Test Suite Even the most careful developer introduces bugs and often at places which you would never guess Hence, define a certain set of tests which is representative for your typical simulations checks the parallelization in different ways can be run in a reasonable amount of time and which should be checked before every major commit and after each port to a new machine/compiler (compiler bugs)!! The GENE-Test-Suite is a perl-based script: calling the code subsequently with either a default set of input files or some special purpose set (like high-resolved simulations) comparing the output up to a certain accuracy (the machine accuracy depends on the architecture!) monitoring the wall clock time to check the efficiency of the installation (the parallelization is checked in GENE itself)

33 Cache optimization Running your code efficiently on different architectures probably requires some clever memory alignment as the cache sizes/ communication buffers vary from machine to machine. In GENE, we use the strip mining or loop sectioning technique do n=ln1,ln2 do m=lm1,lm2 do l=ll1,ll2 do k=lk1,lk2! some Vlasov eq. terms rhs = rhs + a * f(:,:,k,l,m,n)! some other Vlasov eq. terms rhs = rhs + b * f(:,:,k,l,m,n) enddo enddo enddo enddo do iblock=0,nblocks-1! some Vlasov eq. terms do klmn = iblock*lklmn0,(iblock+1)*lkmn0 rhs = rhs + a * f(:,:,klmn) enddo! some other Vlasov eq. terms do klmn = iblock*lklmn0,(iblock+1)*lkmn0 rhs = rhs + b * f(:,:,klmn) enddo enddo with block size (lklmn0) < cache size such the subarrays fit ( measured during initialization) The same applies to communication buffers

34 MPI mapping The optimum MPI mapping (distribution of processes in the different dimensions) is non-trivial in general! It depends on the hardware (latency, buffer sizes) the grid sizes and numerical schemes (think of ghost cell to real cell ratio for finite differences schemes) GENE is designed as a restartable program (i.e., it has proper initialization/finalization calls) and can hence check automatically for the best mapping. Since not only the run time but also the results are being compared, it directly detects possible parallelization errors.

35 Final remarks To tackle various open problems in the area of plasma turbulence, we are required and prepared to use the next generation(s) of HPC platforms. The ultimate goal? A virtual fusion plasma! n n Fusion research: A place where fascinating physics, extreme computing, and global challenges meet More information: gene.rzg.mpg.de