HPC Simulations of Magnetic Fusion Plasmas

Transcription

1 HPC Simulations of Magnetic Fusion Plasmas L. Villard S. Brunner, A. Casati, W.A. Cooper, J.P. Graves, M. Jucker, S. Khoshagdam, X. Lapillonne, B.F. McMillan, P. Popovich, O. Sauter, T.M. Tran, T. Vernay Centre de Recherches en Physique des Plasmas Association Euratom Suisse, EPFL, Lausanne, Switzerland A. Bottino, T. Dannert, T. Görler, R. Hatzky, F. Jenko, A. Mishchenko, B. Scott Max-Planck Gesellschaft, IPP Garching and Greifswald, Rechenzentrum, Germany S. Jolliet, Y. Idomura, Japan Atomic Energy Agency, Taitou, Tokyo, Japan P. Angelino, G.L. Falchetto, X. Garbet, V. Grandgirard, N. Mellet, Y. Sarazin CEA/DSM/IRFM, Cadarache, France T.H. Watanabe, National Institute for Fusion Science, Toki, Japan T. Stitt, Swiss Center for Supercomputing, CSCS, Manno, Switzerland

2 Outline 1. Introduction Complexity of the physical systems we are studying What is «special» about plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization 5. Conclusion DEISA PRACE

3 Outline 1. Introduction Complexity of the physical systems we are studying What is «special» in plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization 5. Conclusion DEISA PRACE

4 ITER: the way to fusion The plasma will be there (see next p) You are here Source:ITER EU+CH, Japan, USA, China, India, South Korea, Russian Federation > 5.3G construction cost Virtually inexhaustible, environmentally benign source of energy: from Deuterium and Lithium; gives Helium. (Tritium is recycled) DEISA PRACE

5 Magnetic confinement: tokamak B r field line helical torsion (rotational transform) of B v field lines is essential to confine particles magneticsurface B and curvature drifts Larmor radius ρ L particle trajectory Trapped particle Passing particle DEISA PRACE

6 Particle motion in a tokamak Passing (top) and trapped (bottom) particle orbits: the guiding centres almost follow the equilibrium magnetic field. 3D view (left) and 2D cross-section (right) In the plasma core, particles are very weakly collisional mean free path along the magnetic field >> system size importance of the geometry of the magnetic field DEISA PRACE Y. Idomura et al., C.R. Physique 7 (2006)

7 Complexity: many nonlinearities coils transformer induction plasma RF current drive current r j, I p operational limits equilibrium configuration heat heat bootstrap macroscopic stability r B( x r ) n T heat heat Neutral Beams fusion α particles transport turbulence microscopic stability Geometry of magnetic configuration is an essential feature of fusion plasma physics DEISA PRACE

8 Timescales in the ITER plasma machine lifetime 1 shot energy confinement turbulence ion cyclotron electron cyclotron t [s] Physics spans several orders of magnitude Direct Numerical Simulation (DNS) of everything is unthinkable Need to separate timescales using approximations How to integrate all phenomena in a consistent manner? DEISA PRACE

9 Kinetic effects: wave-particle interaction f ω / k Surfers with velocity too different from the phase velocity of the wave will not ride the wave Surfers with velocity just below the phase velocity of the wave will be accelerated -> momentum and energy transfer Net energy transfer from the wave to the particles if f / v < 0 Collisionless Landau damping General: distribution function in 6D phase space To be solved with consistent electromagnetic fields v r r f ( x, v; t) r r r r E( x, t), B( x, t) DEISA PRACE

10 Outline 1. Introduction Complexity of the physical systems we are studying What is «special» in plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization 5. Conclusion DEISA PRACE

11 Turbulence and Transport Hot plasma core, colder edge: => grad T grad T > (grad T) crit => instabilities Small scales, i.e. spatial scales ~ρ s (Larmor radius) large scales, i.e. spatial scales ~ system size a low frequency, i.e. ω << Ω i (cyclotron frequency) Growth and nonlinear saturation of unstable modes => turbulent state => transport (heat, particles, momentum) anomalous, essentially by collisionless processes Turbulent transport >> collisional transport In spite of this, typical grad T ~ 10 6 K / cm in tokamaks Need a theory and numerical tools! DEISA PRACE

12 Gyrokinetic model Assume ω << turbulence ω ion cyclotron Average out the fast motion of the particle around the guiding center Fast parallel motion, slow perpendicular motion (drifts) Strong anisotropy of turbulent perturbations (// vs perp to B) ξ ξ phase space dimension reduction 6D ---> 5D r r r fs( q, p; t) fs( R, u, μ; t) r R = q r r ρ s k// Zonal Flows ion-driven k electron-driven k [m 1 ] DEISA PRACE

13 Gyrokinetic-Maxwell: summary Assuming frequencies << cyclotron frequencies, analytical phase space reduction from 6D to 5D A time evolution partial differential equation (PDE) describing the advection-diffusion in 5D phase space of the distribution function f A set of 5 nonlinear, coupled ordinary differential equations (ODEs) for the characteristics A set of integral partial differential equations for the perturbed fields (potentials φ, A ) in 3D real space DEISA PRACE

14 Gyrokinetic equations r f s ( R, v//, μ) r f s dr fs dv// f s + r + r = C( fs, fs' ) t dt R dt R r dr r dv r // =... fct( φ, A), =...fct( φ, A) dt dt ( φ, A r ) distribution function of species s in 5D phase space solution of Maxwell s equations, with ρ, j obtained as moments of f s advection-diffusion equations of motion (characteristics) (orbits) DEISA PRACE

15 Solving GK: numerical approaches Lagrangian: Particle In Cell (PIC), Monte Carlo Sample phase space with markers and follow their orbits Noise accumulation is difficult to control in long simulations Semi-Lagrangian Fixed grid in phase space, trace orbits back in time Multi-dimensional interpolation is difficult Eulerian Fixed grid in phase space, finite differences for operators CFL condition; overshoot versus dissipation Common to all three: Poisson (+Ampere) field solver Various methods: finite differences, finite elements, FFT, ORB5 (Lagrange-PIC) code: in-house (CRPP) + Max-Planck-IPP GENE (Euler) code: Max-Planck-IPP + CRPP GYSELA (Semi-Lagrange) code: CEA (+CRPP) DEISA PRACE

16 Global linear simulation: unstable mode Linearised δf Contours of perturbed potential CYCLONE base case 1/ρ*=180. GYGLES code [M. Fivaz et al, CPC 111 (1998) 27] DEISA PRACE

17 Nonlinear : Zonal Flows regulate turbulence Color contours of perturbed electrostatic potential hot core turbulent eddies => radial transport cold edge large scale Zonal Flows => breakup of turbulent eddies => self-regulation of turbulence Direct Numerical Simulation of Ion Temperature Gradient driven turbulence ORB5 code, resolution: 128x448x320, N = 83 million, Ω i Δt=40 ~6 h on 1024 processors IBM BG/L@EPFL No sources: final state has (undamped) Zonal Flows and decaying turbulence [S. Jolliet, et al., CPC 2007] DEISA PRACE

18 Nonlinear Direct Numerical Simulation Global gyrokinetic ORB5 code. ITG-driven turbulence. Contours of φ - <φ> (~ perturbed density) Zonal flows breakup eddies and regulate turbulence DEISA PRACE

19 Noise control is essential for Lagrange-PIC (with noise control) (without noise control) McMillan, et al., PoP 2008 When S/N is lost, there is a decrease in the computed transport level Loss of the large bursts DEISA PRACE

20 Noise control is essential to preserve the perturbation structures without noise control with noise control McMillan, et al., PoP 2008 Contours of φ-<φ> (~ perturbed electron density) DEISA PRACE

21 Turbulent processes have a chaotic nature Convergence studies with increasing N Signature of chaos: finite time after which two neighbouring initial conditions diverge from each other How to define a good (converged???) estimate of the transport? B.F. McMillan, et al, PoP 15 (2008) DEISA PRACE

22 Scaling with physical system size TCV DIII-D JET ITER GENE-local GENE ORB5 GENE-local s-α inconsistent geometry [J. Candy, et al., Phys. Plasma, 2004] [X. Lapillonne, PhD Thesis, 2010] Discrepancy between the GYRO and GTC codes: now resolved Importance of finite size effects in existing tokamaks for extrapolation to ITER DEISA PRACE

23 Semi-Lagrange approach Main motivation: get the better of two worlds avoid the statistical noise problem inherent to Lagrange PIC Monte Carlo methods avoid the CFL condition inherent to Euler methods t t k + Δt ( x i, v j ) f=const along orbit GYSELA code: 5D gyrokinetic t k Orbit, backward integrated from grid point Interpolation of foot of orbit Operator splitting Leapfrog scheme, with averaging of integer and half-integer timesteps [V. Grandgirard, et al., J. Comput. Phys. 217 (2006) 395] DEISA PRACE

24 Lagrange (ORB5) vs ½-Lagrange (GYSELA) Heat diffusivity Ion temperature gradient Heat diffusivity Alternative approaches bring useful cross-checks of numerical and physical results Fit from other codes (Dimits) Ion temperature gradient N s N χ Nϕ = , a / ρ = 184 [V. Grandgirard, et al., PPCF 49 (2007) B173] ORB5: N p = 256M, GYSELA : N v L // N μ = 32 8, half - torus DEISA PRACE

25 Outline 1. Introduction Complexity of the physical systems we are studying What is «special» in plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization 5. Conclusion DEISA PRACE

26 The plasma edge A. Kendl, B.D. Scott, PoP 13, (2006) The plasma crosssection is shaped: elongation, triangularity As we approach the edge, the geometry becomes more complex: separatrix X-point Occurrence of transport barrier : pedestal -like n and T profiles with steep gradients, leading to overall improved confinement. An indispensable ingredient for the success of ITER DEISA PRACE

27 Alternative to the tokamak: stellarator Inside the largest stellarator : LHD (NIFS, Japan) Unlike the tokamak, the stellarator is an intrisically steady-state magnetic confinement concept (does not need a net plasma current, does not need a transformer), but DEISA PRACE

28 Alfvén Eigenmode in W7-X Toroidicity-induced Alfvén Eigenmode Could be destabilised by fast ions Global wave code LEMAN: DNS of Maxwell s equations (finite elements, Fourier) Solves 4-potential (φ,a) in order to avoid spectral pollution f=65.3khz <B mag >=4.2T DEISA PRACE

29 Outline 1. Introduction Complexity of the physical systems we are studying What is «special» in plasma physics? 2. Direct Numerical Simulations of Turbulence Challenges and progresses 3. Some other physics problems 3D configurations, fast particles, 4. High Performance Computing Massive parallelization and scaling with system size 5. Conclusion DEISA PRACE

30 Parallelization scheme ORB5 Aim: solve very large problems Domain decomposition (toroidal direction) => huge number of processors MPI move particles Domain Cloning: field quantities (density, potential) are replicated MPI global sum fields N c = 1 ~ 100 N d = 100 ~ 1000 N pes = N d *N c DEISA PRACE

31 Massive parallelism - Scalability 2x10 9 particles grid 128x512x256=16.8x10 6 BG/P (Babel, Idris) (Deisa Extreme Computing Initiative) 4096 to cores XT5 (Monte Rosa, CSCS) (thanks to Tim Stitt) 1024 to 8192 cores 256k particles/core grid 256x1024x512=134x10 6 DEISA PRACE

32 Massive parallelism - Scalability 2x10 9 particles grid 128x512x256=16.8x10 6 BG/P (Babel, Idris) (Deisa Extreme Computing Initiative) 4096 to cores XT5 (Monte Rosa, CSCS) (thanks to Tim Stitt) 1024 to 8192 cores 256k particles/core grid 256x1024x512=134x10 6 DEISA PRACE

33 ITER global gyrokinetic turbulence simulation: computational requirements Turbulence spatial scales down to gyroradius ρ s = c s /Ω i c s =(T e /m i ) 1/2 Normalized size = system size / gyration radius : 1 / ρ* = a / ρ s ITER: 1 / ρ* = ~ D field solver: min 4 pts/wavelength 7x10 9 grid cells (3D) Velocity space resolution: ~ 300 / cell 2 000x10 9 phase space cells or markers (5D) Time until statistical steady-state: ~ 2000 a / c s ~10 6 time steps This is clearly unfeasible! Cost of algorithm ~ (1 / ρ*) 4 ~ (system size) 4 DEISA PRACE

34 Scalability Large Physical System Size ~ Half ITER size => 5.7x10 9 particles =>grid 512x1024x1024 (537x10 6 grid points) BG/P (JUGENE, Jülich) to cores (EFDA-HPC-FF) A parallel data transpose operation scales very badly (left) and starts to dominate the time (right) when using several racks DEISA PRACE

35 Scalability Large Physical System Size speedup ORB5 BG/P JUGENE strong scaling ideal particle push total pptransp ncores x 10 4 ~ Half ITER size => 5.7x10 9 particles =>grid 512x1024x1024 (537x10 6 grid points) BG/P (JUGENE, Jülich) to cores (EFDA-HPC-FF) A parallel data transpose operation scales very badly (left) and starts to dominate the time (right) when using several racks DEISA PRACE

36 Global large system size linear simulation ITER-size cyclone case 1/ρ* = 1120 m~170 wavelengths, solved with N θ =64 grid points: how is it possible? DEISA PRACE

37 Turbulence in magnetized plasmas hot core is very anisotropic [S. Jolliet et al., Comput. Phys. Commun. 177, 409 (2007)] DEISA PRACE

38 At the heart of the HP2C proposal: Field-aligned coordinates reduction of grid by a factor ~ 1/ρ* (1000 for ITER) requires major refactoring of existing codes (ORB5, GENE) and/or new code development (semi-lagrangian) DEISA PRACE

39 ITER global gyrokinetic turbulence simulation: computational requirements Turbulence spatial scales down to gyroradius ρ s = c s /Ω i c s =(T e /m i ) 1/2 Normalized size = system size / gyration radius : 1 / ρ* = a / ρ s ITER: 1 / ρ* = ~ D field solver: min 4 pts/wavelength 7x10 9 grid cells (3D) 60x10 6 Velocity space resolution: ~ 300 / cell 2 000x10 9 phase space cells or markers (5D) Time until statistical steady-state: ~ 2000 a / c s 18x10 9 ~10 6 time steps 2x10 4 This is clearly unfeasible! Feasible (*) Cost of algorithm ~ (1 / ρ*) 4 ~ (system size) 4 (1 / ρ*) 2 (*) This is one of the goals of the HP2C project. Requires major work! DEISA PRACE

40 Conclusion and outlook Fusion plasma physics is an extraordinary rich and diverse field This diversity is reflected in the various theoretical and numerical approaches Direct Numerical Simulation of Turbulence remains very challenging Future developments will include more physical effects and require further numerical advances Numerical codes for the simulation of fusion plasmas are mature for the current top-end HPC platforms Further developments are required to adapt them to the future generation of HPC mainly a human resource issue! Special thanks to Prof. Thomas Schulthess for the HP2C project Thank you for your attention DEISA PRACE