WARPX - Washington Approximate Riemann Plasma Solver

Transcription

1 WARPX - Washington Approximate Riemann Plasma Solver Current and Past Developers: S.Gilliam, A.Hakim, R.Lilly, N.Reddell, B.Srinivasan, A.Susanto Computational Plasma Dynamics Lab University of Washington, Seattle August 30, 2011

2 Outline Washington Approximate Riemann Plasma Solver (WARPX) Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations

3 WARPX solves primarily hyperbolic equations. Primarily developed to solve hyperbolic conservation laws (two-fluid plasma, Hall MHD, Ideal MHD, Euler,...) 3D high resolution Finite Volume (FV) and Discontinuous Galerkin (DG) spatial discretizations Explicit time integration for FV and DG Semi-Implicit and Implicit time integration for DG General Geometry - arbitrarily shaped domains.

4 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model WARPX Physical Model Washington Approximate Riemann Plasma Solver (WARPX) Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations

5 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Context for the Two-Fluid Plasma Model. Equations described are the five-moment equations Zeroth, first, and second moments of the Boltzmann Eqn. f s t + v f s x + q ( ) s E + v B f ( ) s m s v = fs t for each species s = electron, ion (additional species for multi-fluid model) Fluid eqns are obtained by multiplying the Boltzmann eqn by chosen moments and integrating over velocity space Often an isotropic pressure, P, is assumed with an EOS used for closure Resulting in 5 eqns and 5 conserved variables per species Maxwell eqns evolve B and E coll

6 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Continuity, Momentum and Energy eqns. for electrons, ions, neutrals... ρ s t + (ρ s u s ) = 0 ρ s u s + (ρ s u s u s + p s I) = ρ sq s ( E+ u t m s B) s ɛ s t + ((ɛ s + p s ) u s ) = ρ sq s m s u s E ɛ s = p s γ ρ su 2 s Different species are coupled through source terms In this case, Lorentz forces do the coupling Evolution of the fields must be calculated

7 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Maxwell eqns. couple the different species through the source terms. B t + E = 0 1 E c 2 t B q s = µ o ρ s u s m s ɛ o E = s B = 0 s q s m s ρ s Maxwell Equations are overdetermined Violation of the divergence constraint equations can lead to non-physical effects (i.e. charge generation) Casting the equations in purely hyperbolic form can be used to enforce these constraints

8 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Two-Fluid Plasma Model has disparate time scales. Characteristic speeds: species speed of sound, speed of light Numerical method needs to resolve the physics of speed and frequency disparities L c, L L v Te, v Ti, L v A, ω pe, ω ce, ω pi, ω ci ω ps = n s qs 2 q s B, ω cs = ɛ o m s m s In a typical lab plasma (FRC), characteristic time scales in seconds are: 1 ω pe = 10 14, L c = 10 9, 1 ω ci = 10 8, L = 10 5, L = 10 5 v Ti v A

9 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Equation System for the Two-Fluid Plasma Model (TFPM) Due to finite electron mass and pressure the frozen-in flux condition is broken Single-fluid MHD assumes zero Larmor radius, in the two-fluid that assumption is not made To include effects of anisotropic pressure, Larmor radius effects and higher moments of the fluid equations need to be included TFPM is dispersive and not dissipative (contrary to the Euler and Navier-Stokes) Dispersion is physical effect of multiple waves and is not a numerical artifact

10 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Applying asymptotic approximations reduces to Hall-MHD model. Ignores high frequency oscillations (c, ɛ o 0) Ignores electron inertia (m e 0) ρ i t + (ρ i u i ) = 0 ρ i u i t + (ρ i u i u i + p i I) = n i q i E + Ji B ɛ s t + ((ɛ s + p s ) u s ) = J s E Assumes n i = n e, there is no need for separate electron continuity electron momentum reduces to n e q e E = pe J e B allows separate T i and T e

11 Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model Further approximations reduces to Ideal-MHD model. Hall and diamagnetic drift terms are neglected ρ u t ɛ t + ρ t + (ρ u i) = 0 ( B + ρ u u + p i I B ) + B2 I = 0 µ o 2µ o ( ( ) ɛ + p + B2 u ) B u B = 0 2µ o µ o B ( t + u B B u ) = 0 The generalized Ohm s law reduces to E = u B since J ne B r Li L 0 u B

12 Advantages of the TFPM Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model TFPM retains all two-fluid physics by including electron inertia, finite light speed, and the Hall and diamagnetic drift terms It includes finite electron mass, finite speed of light and non-neutral effects It allows for displacement current effects to be captured Does not inherently assume charge neutrality as in MHD Neutrality does not allow local charge separation, large local electric fields Non-neutral effects are important in some plasmas

13 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations Washington Approximate Riemann Plasma Solver (WARPX) Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations

14 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations High Resolution Wave Propagation Method The wave propagation method can be applied to balance laws of the form Q t + F = S If flux Jacobian has all real eigenvalues and a complete set of right eigenvectors The homogeneous part is hyperbolic TFPM has hyperbolic homogeneous parts Hyperbolic equations allow discontinuous solutions even if the initial conditions are smooth Wave propagation method belongs to the class of Godunov methods Riemann problems

15 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations First Order Scheme Consider a 1D, homogeneous case: Q t + F x = 0 Integrating over time from t n to t n+1 : Q n+1 i = Qi n t x Qi n 1 x ( ) F n+1/2 i+1/2 F n+1/2 i 1/2 xi+1/2 x i 1/2 Q(x, t n )dx F n+1/2 i 1/2 1 tn+1 F ( ) Q(x t i 1/2, t), x i 1/2 dt t n The flux can be calculated by several different approaches

16 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations First Order Scheme (continued) Riemann problem is an initial value problem The hyperbolic equation set can be recast as Q t + A Q x = 0 where A = F Q is the flux Jacobian which is constant for linear systems The wave propagation method involves solving a Riemann problem at each cell interface Q n+1 i = Qi n t ( A + Q x i 1/2 + A ) Q i+1/2 where A + Q i 1/2 = Z p i 1/ Z i 1/2 p:s p 1 1/2 >0

17 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations First Order Scheme (continued) Z p i 1/2 = l p i 1/2 ( F i F i 1 )r p i 1/2 at each cell interface the right, r p p i 1/2, and left, li 1/2, eigenvectors and eigenvalues, s p i 1/2, are calculated from the flux Jacobian Eigensystems in linear systems are constant Nonlinear systems require special treatment, i.e. Roe averaging at the interfaces

18 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations High Resolution Corrections increase the solution accuracy. Retain the second-order terms of the Taylor series expansion of the conserved variables Q n+1 i = Qi n t x t x p ( A + Q i 1/2 + A Q i+1/2 ) ( F i+1/2 F i 1/2 ) Correction flux is defined as F i 1/2 = 1 ( sgn(s p 2 i 1/2 ) 1 t ) x sp i 1/2 Z p i 1/2 Correction makes the wave propagation method equivalent to the standard Lax-Wendroff algorithm

19 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations Sources are applied using a source splitting method. Solve the homogeneous equation separately Q t + F x = 0 Use a 4th order Runge-Kutta to solve the ODE resulting from the sources Q t = S Source terms of the two-fluid plasma model produce undamped oscillations (purely imaginary eigenvalues) Sources add physical dispersion to the system

20 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations Runge-Kutta discontinuous Galerkin (RKDG) method provides higher order spatial resolution and better flux coupling to sources. Finite Element Method provides arbitrarily high spatial order by expanding the solution in polynomial basis functions Riemann problems are solved at each interface to compute surface fluxes Solutions are allowed to be discontinuous at the cell interfaces 3 rd order Runge-Kutta is often used for the time integration Explicit RKDG is limited to CFL 1/(2p 1)

21 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations The Base RKDG Scheme Conservation Laws are given by Q t + F x = S Multiply by basis functions and integrate over the volume v hqdv + v hf da F v h dv = v h SdV t Ω Ω This is an ODE of the form dq dt = L r (Q), solved using the Runge-Kutta Ω Ω Spatial order of 2 nd 16 th is common.

22 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations Runge-Kutta Time Integration 2 nd order total variation bound (TVB) Runge-Kutta method Q 1 = Q n + tl h (Q n ) Q n+1 = 1 2 Qn Q tl r (Q 1 ) 3 rd order total variation diminishing (TVD) Runge-Kutta method Q 1 = Q n + tl r (Q n ) Q 2 = 3 4 Qn (Q1 + tl r (Q 1 )) Q n+1 = 1 3 Qn (Q2 + tl r (Q 2 ))

23 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations Purely Hyperbolic Maxwell Equations are used to enforce the divergence constraints. B t + E + γ Ψ = 0 1 E c 2 t B q s + χ Φ = µ o ρ s u s m s s 1 Φ χ t + E = s q s m s ρ s 1 Ψ γc 2 t + B = 0 Numerical divergence errors can develop and grow over time Ψ and Φ are scalar error correction potentials for the divergence of E and divergence of B γ and χ set the speed at which errors propagate out of the domain relative to the speed of light Setting γ and χ larger than 1 may be necessary but further restricts the explicit timestep

24 High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations Multiple plasma problems have been simulated with WARPX. Finite Volume and RKDG methods as well as ideal and non-ideal multi-fluids models have been used: Collisionless magnetic reconnection Z-pinch drift turbulence instabilities 3D FRC dynamics 3-fluid self-consistent formation of plasma sheaths

25 Building, Running and Implementing your own Applications Washington Approximate Riemann Plasma Solver (WARPX) Two-Fluid Plasma Model Asymptotic Approximations Advantages of the Two-Fluid Plasma Model High Resolution Wave Propagation Method Runge-Kutta Discontinuous Galerkin Method(RKDG) Purely Hyperbolic Maxwell Eqns. Plasma Simulations

26 WARPX input files are simple XML files. Preamble warpx block Input file should have a preamble and a warpx block warpx block should have a combosolver block with the function tags exactly one tag each for grid, variables, SubSolverSequence at least one tag for SubSolvers, SubSolverSteps Input files are preprocessed combosolver block grid variables Hyperbolic Solver Initialization Spacial Scheme Equations Sources SubSolvers SubSolver Step Solver Sequence

27 Shock tube example: Preamble ## python import math Anything outside the first XML tag can include Python expressions # Define constants # ratio of specific heats GAMMA = 1.4 # final time TEND = 0.3

28 Shock tube example: First block must be < warpx > Specify the run name and simulation precision Verbosity options: debug, info, warning, error, critical, disabled. < warpx > Simulation = shocktube Verbosity = debug Real = double < /warpx >

29 Shock tube example: combosolver block Time: time integration interval end-time Dt: initial time-step UseFixedDt: use constant time step value (0:No, 1:yes) Out: number of output files. < shocktube > Type = WxSolver Kind = combosolver Time = [0.0, TEND] Dt = 0.1 TEND UseFixedDt = 0 Out = 20 < /shocktube >

30 Shock tube example: grid and variables Grid must be specified before the variables block Periodic BC s can be specified in grid block PeriodicDirs = [0, 1] pararrays... < grid > Type = WxGridBox Kind = cartgrid Lower = [0.0] Upper = [1.0] Cells = [150] < /grid > < q > Type = WxVariable Kind = ParArray OnGrid = grid NumComponents = 5 GhostCells = [2, 2] < /q >

31 Shock tube example: Hyperbolic solver preamble preamble commands are only executed once specify the block containing the spatial scheme parameters specify the block(s) where hyperbolic equations kind and parameters are found specify variables to be initialized < hyperbolic > Type = WxSubSolver Kind = hypersubsolver OnGrid = grid ReadVars = [q] WriteVars = [qnew] Scheme = wave Equations = [euler] Initialize = [q, qnew]

32 Shock tube example: Spatial scheme Finite Volume and discontinuous Galerkin are implemented spacial order: 1: First Order, 2: Second Order limiter options are: minmod, superbee, van-leer, monotonized-centered or beam-warming < wave > Type = WxHyperbolicScheme Kind = wave1d Cfl = 0.9 Cflm = 1.0 spatialorder = 2 sourcesplitting = 0 limiter = van-leer < /wave >

33 Shock tube example: hyperbolic equation Various hyperbolic equations are implemented Advection, PH Maxwell, Hall-MHD, Ideal MHD, Burger s, Strain,... see folder warpx dir/src/hypperapps/ for additional applications Each has its own parameter requirements < euler > Type = WxHyperbolicEqn Kind = eulereqn gas gamma = GAMMA < /euler >

34 Shock tube example: Initial Conditions < InitialCondition > Type = WxFunction Kind = exprfunc gamma = GAMMA progn = [ rho = if(x > 0.5, 0.125, 1.0), \ pr = if(x > 0.5, 0.1, 1.0), \ Er = pr/(gamma-1) ] exprlist = [ rho, 0.0, 0.0, 0.0, Er ] < /InitialCondition > Mathematical functions can be entered for the initial conditions (or IC solvers) Python expressions can also be used (e.g. numpy.sqrt(),numpy.abs(), numpy.sin()) < /hyperbolic >

35 Shock tube example: Boundary Conditions BC s are implemented as subsolvers Right boundary is also a wall (edge = upper) dir 0: right, left dir 1: bottom, top dir 2: back, forward < leftbc > Type = WxSubSolver Kind = bcwalleuler OnGrid = grid WriteVars = [q, qnew] direction = 0 edge = lower < /leftbc >

36 Shock tube example: SubSolvers < copier > Type = WxSubSolver Kind = linearcombiner OnGrid = grid ReadVars = [qnew] coeffs = [1.0] WriteVars = [q] < /copier > Copies qnew into q < homgeneous > Type = WxSubSolverStep DtFrac = 1.0 SubSolvers = [hyperbolic] < /homogeneous > advances the hyperbolic system one time step Similar step for the BC s (applybc block) and the copier (copy block)

37 Shock tube example: SubSolvers < SolverSequence > Type = WxSolverSequence PerStep = [applybc, homogeneous, copy] < /SolverSequence > define subsolver sequence of steps to be applied all simulations must have this block

38 WARPX dependencies To get the code you will need the Git version control client C++ compiler (gcc4.0 or later), the Message Passing Library(MPI) and the HDF5 serial and parallel libraries Scons is the WARPX build system (Python) Instructions on getting and builidng the dependencies and WARPX can be found at Wiki page:

39 Scons is used to build WARPX. Scons is a python version of make Go to : warpx dir/src create a file named config.py import os mpi = os.environ.get( HOME ) + /software/mpi hdf5s = os.environ.get( HOME ) + /software/hdf5 hdf5p = os.environ.get( HOME ) + /software/hdf5mpi build warpx in serial: scons parallel=no debug=no -j 4 after the build executable can be found at: warpx dir/src/build/xwarpx/xwarpx executable for parallel bluid: warpx dir/src/build-par/xwarpx/xwarpx

40 Running WARPX is relatively easy. Before running warpx the input file must be preprocessed run script: warpx dir/scripts/wxinpparse.py -i simulation.pin run warpx executable: $ XWARPX -i simulation.inp simulations can be restarted from HDF5 output files: $ XWARPX -r simulation 5.h5 -i simulation.inp -o simulation r for parallel runs: mpirun -np 8 $ PARXWARPX -i mysim.inp

41 1D and 2D Visualization using Python WARPX produces HDF5 output files To visualize HDF5 in python you will need PyTables module warpx dir/scripts/wxdata.py module is used to read.h5 data dh.variables(): shows all the variables (qnew,qold) q.fullshape: will return the dimensions of the array import numpy as np from pylab import * import wxdata as wxdata2 dh = wxdata2.wxdata(location, n) q = dh.read( qnew ) xl = q.grid.lowerbounds[0] xu = q.grid.upperbounds[0] xcell = q.grid.numphyscells[0] x = np.linspace(xl,xu,xcell) yl = q.grid.lowerbounds[1] yu = q.grid.upperbounds[1] ycell = q.grid.numphyscells[1] y = np.linspace(yl,yu,ycell) X, Y = np.meshgrid(y,x) by = q[:, :, 14] pcolor(y,x,by) show()

42 3D Visualization using VisIt requires creating.xmf files. Get the data ready for visualization add both exprwriter and gridwriter subsolvers to your input file. Run: wxxdmf.py -i advectionwave3d wv.inp -w wavewriter -d range(0,10).xmf files are created < wavewriter > Type = WxSubSolver Kind = exprwriter OnGrid = grid ReadVars = [qnew] # list of variables be written writenames = [ wv ] # list of variables indvars = [ wave ] < wv > exprlist = [ wave ] < /wv > < /wavewriter >

43 VisIt interprets the HDF5 (.h5) files though the.xmf files Install Visit codes/visit/ codes/visit/manuals.html Move to the directory containing the.h5 and.xmf files Open the.xmf database (not the.h5 files) plot Volume wv click Draw

44 Code Structure build: different build configurations buildconf: configuration files used during the scons build hyper: files common to all hyperbolic system implementations hyperapps: implementations specific to each equation system Euler, HallMHD, IdealMHD, PhMaxwell, TwoFluid... solvers: Poisson, Runge-Kutta, Curl, Divergence, linearcombiner...

45 Code Flowchart

46 Only eigenvalues and eigenvectors are needed for your hyperbolic system implementation. Update procedure is Q n+1 i = Qi n t ( A + Q x i 1/2 + A ) Q i+1/2 where A + Q i 1/2 = p:s p 1 1/2 >0 Z p i 1/ Z i 1/2 Only need the flux Jacobian eigenvalues and eigenvectors to calculate A + Q i 1/2 = s + 1 1/2 >0 s + W +

47 Implementing your own hyperbolic equation system. Eigenvalues and eigenvectors must be set in the Riemann problem solver, rp(), and the transverse Riemann problem solver, rpt() // wave 2 with speed +c wave[0][1] = a2; wave[mu][1] = -a2*z; s[1] = +c0; // // Compute waves // // compute fluctuations for (unsigned m=0; m<2; ++m) { amdq[m] = 0.0; apdq[m] = 0.0; for (unsigned mw=0; mw<2; ++mw) { if (s[mw] < 0.0) // left going wave amdq[m] += s[mw]*wave[m][mw]; else // right going wave apdq[m] += s[mw]*wave[m][mw]; }}

48 Applying Boundary Conditions Several BC s are already implemented: Periodic: defined at variable declaration Two-Fluid: Conducting Wall BC, Copy BC, Axis BC Euler: Inflow BC, Wall BC, Axis BC wxeulergridbc register.cc has a list of the Euler BC s. example from: hyperapps/euler/wxgridwallbceuler.cc WxGridName < REAL >:: applylowerbc(dir, ParArray) {. dp[0] = d[0]; // rho dp[mu] = -d[mu]; // rho*ur dp[mv] = d[mv]; // rho*up dp[mw] = d[mw]; // rho*uz dp[4] = d[4]; // E. } Do the equivalent to the UpperBC

49 Implementing Source terms Sources advanced using 4 th order Runge-Kutta Index of input variables and output sources must be specified (InpRange, OutRange) Several sources are already implemented: Lorentz forces, charge density, current density, Braginskii viscosity, friction, radial,... wxtwofluidsrc register.cc has a list of the two-fluid sources example from: hyperapps/euler/wxgravsource.h Q t = gρ bool src(unsigned n, REAL tx, REAL q, REAL qaux, REAL s){ // assumes that q is [rho, rho*u]. } s[0] = - accelgrav*q[0]; s[1] = - accelgrav*q[1];

50 Implementing other SubSolvers Used to implement sources or routine run-time checks (i.e. frequency) 3 sections, setup(), declaretypes() and step() Subsolver are all WxStepperStatus class (SubSolverStep and SolverSequence)

51 New files need to be added to the SConscript and register.cc files for Scons to build them.. There are different registers used for each application: wxeulereqn register.cc, wxeulerfunction register.cc, wxeulergridbc register.cc, wxeulerinit register.cc, wxeulersrc register.cc, wxeulersubsolver register.cc The new classes must be added in the respective register #include wxeulereqn.h WxCreator < WxEulerEqn < float >, WxHyperbolicEqn < float >> euf ( eulereqn ); WxCreator < WxEulerEqn < double >, WxHyperbolicEqn < double >> eud( eulereqn ); the corresponding.cc file needs to be added to the SConscript file # source files in lib sources = [ wxeulereqn.cc, ]