HPC Simulations of Magnetic Fusion Plasmas

HPC Simulations of Magnetic Fusion Plasmas L. Villard S. Brunner, A. Casati, W.A. Cooper, J.P. Graves, M. Jucker, S. Khoshagdam, X. Lapillonne, B.F. McMillan, P. Popovich, O. Sauter, T.M. Tran, T. Vernay Centre de Recherches en Physique des Plasmas Association Euratom Suisse, EPFL, Lausanne, Switzerland A. Bottino, T. Dannert, T. Görler, R. Hatzky, F. Jenko, A. Mishchenko, B. Scott Max-Planck Gesellschaft, IPP Garching and Greifswald, Rechenzentrum, Germany S. Jolliet, Y. Idomura, Japan Atomic Energy Agency, Taitou, Tokyo, Japan P. Angelino, G.L. Falchetto, X. Garbet, V. Grandgirard, N. Mellet, Y. Sarazin CEA/DSM/IRFM, Cadarache, France T.H. Watanabe, National Institute for Fusion Science, Toki, Japan T. Stitt, Swiss Center for Supercomputing, CSCS, Manno, Switzerland

Outline 1. Introduction Complexity of the physical systems we are studying What is «special» about plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization 5. Conclusion DEISA PRACE 2010 2

Outline 1. Introduction Complexity of the physical systems we are studying What is «special» in plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization 5. Conclusion DEISA PRACE 2010 3

ITER: the way to fusion The plasma will be there (see next p) You are here Source:ITER EU+CH, Japan, USA, China, India, South Korea, Russian Federation > 5.3G construction cost Virtually inexhaustible, environmentally benign source of energy: from Deuterium and Lithium; gives Helium. (Tritium is recycled) DEISA PRACE 2010 4

Magnetic confinement: tokamak B r field line helical torsion (rotational transform) of B v field lines is essential to confine particles magneticsurface B and curvature drifts Larmor radius ρ L particle trajectory Trapped particle Passing particle DEISA PRACE 2010 5

Particle motion in a tokamak Passing (top) and trapped (bottom) particle orbits: the guiding centres almost follow the equilibrium magnetic field. 3D view (left) and 2D cross-section (right) In the plasma core, particles are very weakly collisional mean free path along the magnetic field >> system size importance of the geometry of the magnetic field DEISA PRACE 2010 6 Y. Idomura et al., C.R. Physique 7 (2006) 650-669

Complexity: many nonlinearities coils transformer induction plasma RF current drive current r j, I p operational limits equilibrium configuration heat heat bootstrap macroscopic stability r B( x r ) n T heat heat Neutral Beams fusion α particles transport turbulence microscopic stability Geometry of magnetic configuration is an essential feature of fusion plasma physics DEISA PRACE 2010 7

Timescales in the ITER plasma machine lifetime 1 shot energy confinement turbulence ion cyclotron electron cyclotron t [s] 8 10 3 10 10 0 10 5 10 8 10 12 Physics spans several orders of magnitude Direct Numerical Simulation (DNS) of everything is unthinkable Need to separate timescales using approximations How to integrate all phenomena in a consistent manner? DEISA PRACE 2010 8

Kinetic effects: wave-particle interaction f ω / k Surfers with velocity too different from the phase velocity of the wave will not ride the wave Surfers with velocity just below the phase velocity of the wave will be accelerated -> momentum and energy transfer Net energy transfer from the wave to the particles if f / v < 0 Collisionless Landau damping General: distribution function in 6D phase space To be solved with consistent electromagnetic fields v r r f ( x, v; t) r r r r E( x, t), B( x, t) DEISA PRACE 2010 9

Turbulence and Transport Hot plasma core, colder edge: => grad T grad T > (grad T) crit => instabilities Small scales, i.e. spatial scales ~ρ s (Larmor radius) large scales, i.e. spatial scales ~ system size a low frequency, i.e. ω << Ω i (cyclotron frequency) Growth and nonlinear saturation of unstable modes => turbulent state => transport (heat, particles, momentum) anomalous, essentially by collisionless processes Turbulent transport >> collisional transport In spite of this, typical grad T ~ 10 6 K / cm in tokamaks Need a theory and numerical tools! DEISA PRACE 2010 11

Gyrokinetic model Assume ω << turbulence ω ion cyclotron Average out the fast motion of the particle around the guiding center Fast parallel motion, slow perpendicular motion (drifts) Strong anisotropy of turbulent perturbations (// vs perp to B) ξ ξ phase space dimension reduction 6D ---> 5D r r r fs( q, p; t) fs( R, u, μ; t) r R = q r r ρ s k// Zonal Flows ion-driven k electron-driven 1 10 10 0 1 10 2 10 3 10 4 10 k [m 1 ] DEISA PRACE 2010 12

Gyrokinetic-Maxwell: summary Assuming frequencies << cyclotron frequencies, analytical phase space reduction from 6D to 5D A time evolution partial differential equation (PDE) describing the advection-diffusion in 5D phase space of the distribution function f A set of 5 nonlinear, coupled ordinary differential equations (ODEs) for the characteristics A set of integral partial differential equations for the perturbed fields (potentials φ, A ) in 3D real space DEISA PRACE 2010 13

Gyrokinetic equations r f s ( R, v//, μ) r f s dr fs dv// f s + r + r = C( fs, fs' ) t dt R dt R r dr r dv r // =... fct( φ, A), =...fct( φ, A) dt dt ( φ, A r ) distribution function of species s in 5D phase space solution of Maxwell s equations, with ρ, j obtained as moments of f s advection-diffusion equations of motion (characteristics) (orbits) DEISA PRACE 2010 14

Solving GK: numerical approaches Lagrangian: Particle In Cell (PIC), Monte Carlo Sample phase space with markers and follow their orbits Noise accumulation is difficult to control in long simulations Semi-Lagrangian Fixed grid in phase space, trace orbits back in time Multi-dimensional interpolation is difficult Eulerian Fixed grid in phase space, finite differences for operators CFL condition; overshoot versus dissipation Common to all three: Poisson (+Ampere) field solver Various methods: finite differences, finite elements, FFT, ORB5 (Lagrange-PIC) code: in-house (CRPP) + Max-Planck-IPP GENE (Euler) code: Max-Planck-IPP + CRPP GYSELA (Semi-Lagrange) code: CEA (+CRPP) DEISA PRACE 2010 15

Global linear simulation: unstable mode Linearised δf Contours of perturbed potential CYCLONE base case 1/ρ*=180. GYGLES code [M. Fivaz et al, CPC 111 (1998) 27] DEISA PRACE 2010 16

Nonlinear : Zonal Flows regulate turbulence Color contours of perturbed electrostatic potential hot core turbulent eddies => radial transport cold edge large scale Zonal Flows => breakup of turbulent eddies => self-regulation of turbulence Direct Numerical Simulation of Ion Temperature Gradient driven turbulence ORB5 code, resolution: 128x448x320, N = 83 million, Ω i Δt=40 ~6 h on 1024 processors IBM BG/L@EPFL No sources: final state has (undamped) Zonal Flows and decaying turbulence [S. Jolliet, et al., CPC 2007] DEISA PRACE 2010 18

Nonlinear Direct Numerical Simulation Global gyrokinetic ORB5 code. ITG-driven turbulence. Contours of φ - <φ> (~ perturbed density) Zonal flows breakup eddies and regulate turbulence DEISA PRACE 2010 19

Noise control is essential for Lagrange-PIC (with noise control) (without noise control) McMillan, et al., PoP 2008 When S/N is lost, there is a decrease in the computed transport level Loss of the large bursts DEISA PRACE 2010 21

Noise control is essential to preserve the perturbation structures without noise control with noise control McMillan, et al., PoP 2008 Contours of φ-<φ> (~ perturbed electron density) DEISA PRACE 2010 22

Turbulent processes have a chaotic nature Convergence studies with increasing N Signature of chaos: finite time after which two neighbouring initial conditions diverge from each other How to define a good (converged???) estimate of the transport? B.F. McMillan, et al, PoP 15 (2008) 062306 DEISA PRACE 2010 24

Scaling with physical system size TCV DIII-D JET ITER GENE-local GENE ORB5 GENE-local s-α inconsistent geometry [J. Candy, et al., Phys. Plasma, 2004] [X. Lapillonne, PhD Thesis, 2010] Discrepancy between the GYRO and GTC codes: now resolved Importance of finite size effects in existing tokamaks for extrapolation to ITER DEISA PRACE 2010 25

Semi-Lagrange approach Main motivation: get the better of two worlds avoid the statistical noise problem inherent to Lagrange PIC Monte Carlo methods avoid the CFL condition inherent to Euler methods t t k + Δt ( x i, v j ) f=const along orbit GYSELA code: 5D gyrokinetic t k Orbit, backward integrated from grid point Interpolation of f @ foot of orbit Operator splitting Leapfrog scheme, with averaging of integer and half-integer timesteps [V. Grandgirard, et al., J. Comput. Phys. 217 (2006) 395] DEISA PRACE 2010 26

Lagrange (ORB5) vs ½-Lagrange (GYSELA) Heat diffusivity Ion temperature gradient Heat diffusivity Alternative approaches bring useful cross-checks of numerical and physical results Fit from other codes (Dimits) Ion temperature gradient N s N χ Nϕ = 256 512 256, a / ρ = 184 [V. Grandgirard, et al., PPCF 49 (2007) B173] ORB5: N p = 256M, GYSELA : N v L // N μ = 32 8, half - torus DEISA PRACE 2010 27

The plasma edge A. Kendl, B.D. Scott, PoP 13, 012504 (2006) The plasma crosssection is shaped: elongation, triangularity As we approach the edge, the geometry becomes more complex: separatrix X-point Occurrence of transport barrier : pedestal -like n and T profiles with steep gradients, leading to overall improved confinement. An indispensable ingredient for the success of ITER DEISA PRACE 2010 29

Alternative to the tokamak: stellarator Inside the largest stellarator : LHD (NIFS, Japan) Unlike the tokamak, the stellarator is an intrisically steady-state magnetic confinement concept (does not need a net plasma current, does not need a transformer), but DEISA PRACE 2010 30

Alfvén Eigenmode in W7-X Toroidicity-induced Alfvén Eigenmode Could be destabilised by fast ions Global wave code LEMAN: DNS of Maxwell s equations (finite elements, Fourier) Solves 4-potential (φ,a) in order to avoid spectral pollution f=65.3khz <B mag >=4.2T DEISA PRACE 2010 31

Parallelization scheme ORB5 Aim: solve very large problems Domain decomposition (toroidal direction) => huge number of processors MPI move particles Domain Cloning: field quantities (density, potential) are replicated MPI global sum fields N c = 1 ~ 100 N d = 100 ~ 1000 N pes = N d *N c DEISA PRACE 2010 33

Massive parallelism - Scalability 2x10 9 particles grid 128x512x256=16.8x10 6 BG/P (Babel, Idris) (Deisa Extreme Computing Initiative) 4096 to 32768 cores XT5 (Monte Rosa, CSCS) (thanks to Tim Stitt) 1024 to 8192 cores 256k particles/core grid 256x1024x512=134x10 6 DEISA PRACE 2010 34

ITER global gyrokinetic turbulence simulation: computational requirements Turbulence spatial scales down to gyroradius ρ s = c s /Ω i c s =(T e /m i ) 1/2 Normalized size = system size / gyration radius : 1 / ρ* = a / ρ s ITER: 1 / ρ* = ~ 1000 3D field solver: min 4 pts/wavelength 7x10 9 grid cells (3D) Velocity space resolution: ~ 300 / cell 2 000x10 9 phase space cells or markers (5D) Time until statistical steady-state: ~ 2000 a / c s ~10 6 time steps This is clearly unfeasible! Cost of algorithm ~ (1 / ρ*) 4 ~ (system size) 4 DEISA PRACE 2010 36

Scalability Large Physical System Size ~ Half ITER size => 5.7x10 9 particles =>grid 512x1024x1024 (537x10 6 grid points) BG/P (JUGENE, Jülich) 16384 to 65536 cores (EFDA-HPC-FF) A parallel data transpose operation scales very badly (left) and starts to dominate the time (right) when using several racks DEISA PRACE 2010 37

Scalability Large Physical System Size speedup 4 3.5 3 2.5 2 1.5 1 0.5 ORB5 BG/P JUGENE strong scaling ideal particle push total pptransp 0 1 2 3 4 5 6 7 ncores x 10 4 ~ Half ITER size => 5.7x10 9 particles =>grid 512x1024x1024 (537x10 6 grid points) BG/P (JUGENE, Jülich) 16384 to 65536 cores (EFDA-HPC-FF) A parallel data transpose operation scales very badly (left) and starts to dominate the time (right) when using several racks DEISA PRACE 2010 38

Global large system size linear simulation ITER-size cyclone case 1/ρ* = 1120 m~170 wavelengths, solved with N θ =64 grid points: how is it possible? DEISA PRACE 2010 39

Turbulence in magnetized plasmas hot core is very anisotropic [S. Jolliet et al., Comput. Phys. Commun. 177, 409 (2007)] DEISA PRACE 2010 40

At the heart of the HP2C proposal: Field-aligned coordinates reduction of grid by a factor ~ 1/ρ* (1000 for ITER) requires major refactoring of existing codes (ORB5, GENE) and/or new code development (semi-lagrangian) DEISA PRACE 2010 41

ITER global gyrokinetic turbulence simulation: computational requirements Turbulence spatial scales down to gyroradius ρ s = c s /Ω i c s =(T e /m i ) 1/2 Normalized size = system size / gyration radius : 1 / ρ* = a / ρ s ITER: 1 / ρ* = ~ 1000 3D field solver: min 4 pts/wavelength 7x10 9 grid cells (3D) 60x10 6 Velocity space resolution: ~ 300 / cell 2 000x10 9 phase space cells or markers (5D) Time until statistical steady-state: ~ 2000 a / c s 18x10 9 ~10 6 time steps 2x10 4 This is clearly unfeasible! Feasible (*) Cost of algorithm ~ (1 / ρ*) 4 ~ (system size) 4 (1 / ρ*) 2 (*) This is one of the goals of the HP2C project. Requires major work! DEISA PRACE 2010 42

Conclusion and outlook Fusion plasma physics is an extraordinary rich and diverse field This diversity is reflected in the various theoretical and numerical approaches Direct Numerical Simulation of Turbulence remains very challenging Future developments will include more physical effects and require further numerical advances Numerical codes for the simulation of fusion plasmas are mature for the current top-end HPC platforms Further developments are required to adapt them to the future generation of HPC mainly a human resource issue! Special thanks to Prof. Thomas Schulthess for the HP2C project Thank you for your attention DEISA PRACE 2010 43