School and Conference on Analytical and Computational Astrophysics November, 2011

Transcription

1 School and Conference on Analytical and Computational Astrophysics November, 2011 Multiscale and Multiphysics Challenge in Modeling of Space Weather - 2 Giovanni Lapenta Katholic University of Leuven, Centre for Plasma Astrophysics Belgium

2 Centrum voor Plasma-Astrofysica Funding from: BOF & GOA (KU Leuven), EC (Swiff, Soteria), NASA (MMS Mission), Intel Exascience Lab. Multiscale and Multiphysics Challenge in Modeling of Space Weather Giovanni Lapenta

3 Outline Meeting the multiscale challenge PIC Method Implicit PIC Moment method Application to space weather Local simulations Global simulations Meeting the multiphysics challenge Multi level simulation AMR Meeting the exascale challenge Need for exascale Intel Exascience Lab

4 CHALLENGES IN MODELING SPACE WEATHER: MULTIPLE PHYSICS ionosphere solar environment magnetotail interplanetary plasmas, dust... sun-to-earth sun-effect propagation polar cusps magnetosphere TRACE September 2005 LASCO September 2002 IMAGE July

5 FLUID to KINETIC Maxwell Fields E,B Lagrange Fluid (MHD) ρ,j Particles X p,v p Newton Plasma: Electrons, ions, fields

6 Kinetic model: Fundamental Equations Boltzmann-Maxwell model - Boltzmann equation - Maxwell equations J E B B E B E t t ) ( ) ( f St f q f t f v B v E x v coupling

7 Explicit and implicit T=0 T=t T=2t EXPLICIT Operations: 1. Solve Newton equations in previous electromagnetic fields 2. Solve Maxwell equations with previous particle positions T=0 T=t IMPLICIT Operations: Over each time step, iteratively solve the two coupled equations until convergence

8 Stability of a numerical scheme Example: A cantilever (springboard) A perturbation makes it vibrate, but vibration amplitude does not grow in time Stable system: when perturbed its vibration amplitude does not grow Unstable system: when perturbed its vibration do grow

9 Stability of the explicit scheme: analogy with pendulum Linear harmonic oscillator Analytical solution / 2 dx dt v m dv dt kx x Asin(t) Bcos(t) k / M N t real Numerical solution imaginary x v n1 n x t 1 v t n n v n x n t 2 Exact Numerical δ x Von Neumann Analysis i t i t Ae N Be N

10 Limits of the explicit kinetic models Summary of the Explicit stability constraints x t

11 Stability of the implicit scheme n n n n n n n n x x k t v v v v t x x 2 tan 2 t t N Analytical Implicit Numerical Physical Explicit 2 / N t real imaginary M k t B t A x / ) cos( ) sin( t i t i N N Be Ae x δ

12 Summary of the complete Stability Analysis Explicit stability constraints t x / fastest smallest 1 x t explicit

13 Numerical Stability Analysis Explicit stability constraints fastest t 2 x smallest millions km System scales FLUID hours ct x Implicit stability constraints ion scales L=10000 km ρ i =d i =1000 km 100 km ρ e =10 km 1 m 1 s 10 2 s 10 3 s t x / fastest smallest 1 electron scales x λ e =100 m KINETIC 10 5 s t

14 Implicit formulation of Vlasov-Maxwell Particle mover Maxwell equations: implicit second order formulation for the field E Newton equations: implicit form Field Solver n n 1 (1 ) Solvers: Coupled Non-linear n

15 Example: Electrostatic Implicit Solver Coupled Equations Lapenta, Markidis, PoP, 18, (2011); Markidis, Lapenta, JCP, 230, 7037 (2011) Implicit Computational Cycle

16

17 Exact Energy Conservation Lapenta, Markidis, PoP, 18, (2011) 2 stream instability Thermal plasma ECPIC= ECPIC=

18 Effects of error in energy conservation Lapenta, Markidis, PoP, 18, (2011) Implicit run: 498 s Explicit run: 3164 s

19 IMPLICIT MOMENT METHOD T=0 T=t J.U. Brackbill et al., JCP, 46, 271, 1982; G. Lapenta, et al, Phys. Plasmas, 13, , IMPLICIT IMPLICIT MOMENT E B 0 B E 0 t E B 00 t J 0 M 0 J M T=t 1 Moment equations from Taylor expansion to make solver feasible M n q p v n p W(x x p ) p dx dt v m dv dt q(e v B)

20 Operations needed Local Operations Hundreds of particles per cell Data locality Global Operations

21 Implicit moment method J E B B E B E t t ) ( ) ( f St f q f t f v B v E x v M n (x,t) f (x,v,t) v n dv Moment equations to make solver feasible

22 Outline Meeting the multiscale challenge PIC Method Implicit PIC Moment method Application to space weather Local simulations Global simulations Meeting the multiphysics challenge Multi level simulation AMR Meeting the exascale challenge Need for exascale Intel Exascience Lab

23 Wide Applicability Plasma Particles Moments

24 Beyond the State of the ART Brackbill, Forslund (LANL) Lapenta, Brackbill (LANL) Markidis, Lapenta (Leuven) New work of the SWIFF And Exascience Lab Venus Celeste Parsek/iPIC ipic AMR 80 s 90 s now 2005 now NEW Venere (Sandro Botticelli) Starry Night (Vincent Van Gogh) Several Circles (Vasily Kandinsky) The Voice of Space (René Magritte)

25 Performances of ipic3d KU Leuven Scaling Study ipic3d on Pleiades 20,000 15,000 Pleiades 10,000 5, Ideal ipic3d Simulations at fixed initial load per core, done increasing the system size at constant resolution

26 An example: kinetic simulation of a 3D current layer

27 3D micro-macro coupling: a typical space weather problem Large scale processes Small scale processes small/large scale coupling captured

28 The 3D Electron Flow

29 Timing considerations for 3D fully kinetic simulations - Implicit vs Explicit Explicit Implicit Gain Dx λ De =100 m d e =10 Km 100 Dy λ De =100 m d e =10 Km 100 Dz λ De =100 m d e =10 Km 100 Dt ω pe Δt=0.1 or 10 5 s ω pe Δt=100 or 10 3 s 1000 Tot 10 9 millions km System scales ion scales L=10000 km ρ i =d i =1000 km 100 km FLUID hours 1 m 1 s 10 2 s An implicit run that takes 1 day would take: 2,800,000 years with an explicit code electron scales x ρ e =10 km λ e =100 m KINETIC 10 3 s 10 5 s t

30 Outline Meeting the multiscale challenge PIC Method Implicit PIC Moment method Application to space weather Local simulations Global simulations Meeting the multiphysics challenge Multi level simulation AMR Meeting the exascale challenge Need for exascale Intel Exascience Lab

31 Mercury Magnetosphere ipic3d

32 Reference case: Earth Environment State of the art now - CCMC Typical needs Box: 100 R E x 100 R E x 100 R E Max load per processor: 16x16x16 cells particles per cell(electron populations) particles per cell (ion populations) Coupled with heliospheric models and observations

33 Scales to be Resolved millions km System scales MACRO hours L=10000 km 1 m ion scales electron scales x ρ i =d i =1000 km 100 km ρ e =10 km λ e =100 m MICRO 1 s 10 2 s 10 3 s 10 5 s t

34 State of the Art Explicit Formulation Resolution needed: electrostatic processes at electron scales: 100m (Debye length) Cells per dimension: 6,353,000 Total processors needed: e million billions Moment Implicit Formulation Resolution needed: electromagnetic processes at electron scales: 10Km (inertial length) Cells per dimension: 63,530 Total processors needed: e billions

35 Adaptive Multiphysics approach

36 Multilevel and Multidomain Each domain operates at its physics and its needed level of resolution FLUID millions km System scales L=10000 km FLUID 1 m hours ion scales ρ i =d i =1000 km 100 km ρ e =10 km 1 s 10 2 s 10 3 s KINETIC electron scales λ e =100 m KINETIC 10 5 s x t

37 Overlapping Multidomain Interdomain exchanges Each domain is an exact replica of the others, but scaled. The operations are identical for the same physics module Algorithm designed to minimize exchange of information No global operation, all solvers operate on each domain 1D and 2D versions implemented on massively parallel computers Local Operations Global Domain Operations

38 Resolution needed only in small areas

39 Multidomain approach Coarse Level Resolution needed: electromagnetic processes at ion scales: 1000Km Cells per dimension: 635 Total processors needed: e thousand petascale Finer levels Resolution needed: electromagnetic processes at electron scales: 10Km Needed only on a thin crust on small surfaces. Estimated processors: e+06 Total processors needed: 2 millions - exascale needed but sufficient

40 Outline Meeting the multiscale challenge PIC Method Implicit PIC Moment method Application to space weather Local simulations Global simulations Meeting the multiphysics challenge Multi level simulation AMR Meeting the exascale challenge Need for exascale Intel Exascience Lab

41 Support at KU Leuven First ever EC funded project on space weather Involving 16 centers in 13 countries Focus on space weather and Earth impact Observation (some simulation) Soteria Multiphysics modelling of space science applied to space weather Focus on simulation and theory To run till 2014 Involving 7 centers in 5 countries Swiff Being negotiated. Scheduled to start in march 2012 Continuation of Soteria with emphasis on space exploration instead of Earth impact. Possibility of interaction with Boulder Lunar Science. eheroes Intel based hybrid architectures Co Design of space simulation on exascale

42 SWIFF: Space Weather Integrated Modelling Framework Space Weather Integrated Forecasting Framework Coordinator: Giovanni Lapenta Katholieke Universiteit Leuven Science Lead Coordinator: G. Lapenta Participant organisation name Katholieke Universiteit Leuven Country Belgium Collaborative Project FP7 Space Create a mathematical physical framework to integrate multiple physics (fluid with kinetic) Focus on method and software development, rather than reuse of existing codes Physics based forecasting Focus on coupling small large scales Based on implicit methods and AMR V. Pierrard Belgian Institute for Space Aeronomy Belgium F. Califano Università di Pisa Italy A. Nordlund Københavns Universitet Denmark A. Bemporad Astronomical Observatory Turin - Istituto Nazionale di Astrofisica Italy Astronomical P. Travnicek Institute, Academy of Czech Sciences of the Czech Republic Republic C. Parnell University of St Andrews UK

43 Space Weather and High Performance Computing ExaScience Lab to Develop Space Weather Prediction as Driver for Intel s Future Exascale Supercomputers Intel Imec Five Flemish Universities

44 Space Weather and High Performance Computing Dynamical Exascale Entry Platform Coordinator: Thomas Lippert, Forschungszentrum Juelich GmbH Large scale integrating project (IP) Towards Exascale with application to: Detailed brain simulation Space Weather Climate simulation Computational fluid engineering High temperature superconductivity Seismic imaging Job: postdoc to develop and test implicit PIC on GPUs

45 Goals of SWIFF and Exascale Lab Petascale 10 6 km hours Exascale SYSTEM scales MACRO 10 5 km 1 m 10 3 km 1 s ION scales 10 2 km 10-3 s 10 0 km ELECTRON scales 10-1 km MICRO 10-4 s 10-5 s EXASCALE allows to bridge the micro-macro gap by increasing size and resolution by the needed 3 orders of magnitude

46 Codes and support material ec/