Particle in cell simulations

Transcription

1 Particle in cell simulations Part II: Implementation of Zeltron Benoît Cerutti IPAG, CNRS, Université Grenoble Alpes, Grenoble, France. 1 Astrosim, Lyon, June 6 July 7, 017.

2 Plan of the lectures Monday: Morning: The PIC method, numerical schemes and main algorithms. Afternoon: Coding practice of the Boris push and the Yee algorithm. Tuesday: Morning: Implementation of Zeltron, structure and methods. Afternoon: Zeltron hands on relativistic reconnection simulations Evening: Seminar about application of PIC to pulsar magnetospheres. Wednesday: Morning: Boundary conditions and parallelization in Zeltron. Afternoon: Zeltron Hands on relativistic collisionless shocks simulations

3 General presentation Zeltron is an explicit, relativistic 3D PIC code created from scratch in 01. Originally designed to study particle acceleration in relativistic magnetic reconnection sites applied to astrophysics. Main developers: Benoît Cerutti (CNRS/Univ. Grenoble Alpes) Greg Werner (University of Colorado) Some general features Written in Fortran 90 Yee FDTD algorithm for the fields Boris push for the particles Efficiently parallelized with MPI (3D domain decomposition) Includes synchrotron and inverse Compton radiation reaction forces Non Cartesian mesh: spherical, cylindrical, Schwarzschild (not public) Large set of tools for data reduction and data analysis on the fly Set of boundary conditions (absorption, creation, open, reflective, ) No need for external libraries 3

4 Global structure PROGRAM: main.f90: Contains the main body of the code, initialization, main loop in time. MODULES: mod_input.f90: Input file where all the numerical and physical parameters can be set. This is the only file to modify for a given setup. mod_initial.f90: Initialize the fields and the particles (problem dependent). mod_fields.f90: Solve Maxwell equations and other operations related to fields. mod_motion.f90: Contains the particle push, applies boundary conditions and exchange particles at the boundaries between processors mod_rhoj.f90: For depositing particle charge and current on the grid. mod_analysis.f90: Data reduction and analysis on the fly and output to the disk Others: More technical modules related to I/O, interpolation, and parallelization. 4

5 Computation procedure per timestep in PIC Step 1 Solve Newton's equation B = c E t Δt E =c B 4 π J t Step 3 dp v B =q E+ dt c ( ) Deposit Solve Maxwell s Charge and equations (E,B) current densities Step (ρ,j) 5

6 General structure of the main.f90 file INITIALISATION 1. Initialize the MPI environment for parallel computing (see lecture III). Initialize the spatial grid, and Yee mesh (this lecture) 3. Set the initial conditions (Particles and fields) at t=0 (this lecture) 4. Write initial data to disk 5. Leap-frog initialization: evolve u0 u0-1/ (see lecture I) MAIN LOOP IN TIME - For each particle species 1. Boris push: un-1/ un+1/ (see lecture I). Deposit currents: J0=0.5ρnvn+1/ (see lecture I) 3. Evolve particle positions: rn rn+1 4. Apply boundary conditions and MPI communications (see lecture III) 5. Deposit currents: J=0.5ρn+1vn+1/+J0 (see lecture I) 6

7 Sample from the main.f90 file (particles)! Update u from t-dt/ to t+dt/, particle positions unchanged CALL BORIS_PUSH(-1d0,me,pcl_ed,pcl_data_ed,Bxg,Byg,Bzg,Exg,Eyg,Ezg,& Uph,xgp,ygp,Esyned,Eicsed,NED)! Computation of rho at t and half of the current density CALL RHOJ(-1d0,pcl_ed,rhoed,Jx0,Jy0,Jz0,xgp,ygp,NED,id,ngh,TOPO_COMM,ierr)! Push the particles from t to t+dt CALL PUSH_PARTICLES(pcl_ed,NED)! Applying boundary conditions to the particles CALL BOUNDARIES_PARTICLES(pcl_ed,pcl_data_ed,taged,NED)! Counting the particles leaving each subdomain CALL COUNT_ESCAPE(pcl_ed,xminp,xmaxp,yminp,ymaxp,NED,NESC)! Exchange of particles at the boundaries between processes CALL COM_PARTICLES(pcl_ed,pcl_data_ed,taged,xminp,xmaxp,yminp,ymaxp,& NED,NESC,id,ngh,TOPO_COMM,ierr)! Computation of rho at t+dt and the other half of the current density CALL RHOJ(-1d0,pcl_ed,rhoed,Jxed,Jyed,Jzed,xgp,ygp,NED,id,ngh,TOPO_COMM,ierr)! Total current density at t+dt/ Jxed=Jxed+Jx0 Jyed=Jyed+Jy0 Jzed=Jzed+Jz0 7

8 General structure of the main.f90 file MAIN LOOP IN TIME (continues) - For the fields 1. Collect currents from all species, and put current on the Yee mesh.. Push B field half time step: Bn Bn+1/ (see lecture I) 3. Push E field full time step: En En+1 (see lecture I) 4. Correct the E field (charge conservation) with Poisson solver (this lecture) 5. Push B field half time step: Bn+1/ Bn+1 (see lecture I) - Analyze and write data to disk every FDUMP time steps (this lecture) - Create a checkpoint every FSAVE time steps 8

9 Sample from the main.f90 file (fields)!=======================================================================! B FIELD at t=t+dt/!======================================================================= CALL PUSH_BHALF(Bx,By,Bz,Ex,Ey,Ez,xgp,ygp,id,ngh,TOPO_COMM,ierr)!=======================================================================! SOLVE MAXWELL'S EQUATIONS at t=t+dt!=======================================================================! E FIELD at t=t+dt CALL PUSH_EFIELD(Bx,By,Bz,Ex,Ey,Ez,Jx,Jy,Jz,xgp,ygp,id,ngh,TOPO_COMM,ierr) IF (MOD(it,FREQ_POISSON).EQ.0) THEN! Solve Poisson equation to ensure that div(e)=4*pi*rho CALL CORRECT_EFIELD(Ex,Ey,xgp,ygp,rho,id,ngh,TOPO_COMM,ierr) END IF! B FIELD at t=t+dt CALL PUSH_BHALF(Bx,By,Bz,Ex,Ey,Ez,xgp,ygp,id,ngh,TOPO_COMM,ierr) CALL FIELDS_NODES(Bx,By,Bz,Ex,Ey,Ez,Bxg,Byg,Bzg,Exg,Eyg,Ezg,xgp,ygp,& id,ngh,topo_comm,ierr) 9

10 The grid and the Yee mesh Define number of cells and spatial boundaries from the input file (mod_input.f90)! Number of cells in X INTEGER*8, PARAMETER, PUBLIC :: NCX=18! Number of cells in Y INTEGER*8, PARAMETER, PUBLIC :: NCY=18! Spatial boundaries in the X-direction DOUBLE PRECISION, PARAMETER, PUBLIC :: xmin=0d0,xmax=100.0! Spatial boundaries in the Y-direction DOUBLE PRECISION, PARAMETER, PUBLIC :: ymin=0d0,ymax=100.0! Spatial step DOUBLE PRECISION, PARAMETER, PUBLIC :: dx=(xmax-xmin)/ncx DOUBLE PRECISION, PARAMETER, PUBLIC :: dy=(ymax-ymin)/ncy Build the spatial arrays (main.f90)! Nodal lattice DO ix=1,nx xg(ix)=(ix-1)*1d0/((nx-1)*1d0)*(xmax-xmin)+xmin ENDDO DO iy=1,ny yg(iy)=(iy-1)*1d0/((ny-1)*1d0)*(ymax-ymin)+ymin ENDDO! Yee lattice xyee=xg+dx/.0 yyee=yg+dy/.0 10

11 Particle data structure The particle distribution function is characterized by 3 arrays (mod_input.f90):! BACKGROUND ELECTRONS distribution function components DOUBLE PRECISION, ALLOCATABLE, PUBLIC :: pcl_eb(:,:) DOUBLE PRECISION, ALLOCATABLE, PUBLIC :: pcl_data_eb(:,:) INTEGER*8, ALLOCATABLE, PUBLIC :: tageb(:) ALLOCATE(pcl_eb(1:7,1:NP)): Actual particle distribution function (pb independent) x List of particles y z ux uy uz wgt Part 1 Part... Part NP ALLOCATE(pcl_data_eb(1:4,1:NP)): Additional particle data information (pb dependent) Data 1 List of particles Data 3 Data 4 Part 1 Part... Data Part NP 11

12 Particle data structure The particle distribution function is characterized by 3 arrays (mod_input.f90):! BACKGROUND ELECTRONS distribution function components DOUBLE PRECISION, ALLOCATABLE, PUBLIC :: pcl_eb(:,:) DOUBLE PRECISION, ALLOCATABLE, PUBLIC :: pcl_data_eb(:,:) INTEGER*8, ALLOCATABLE, PUBLIC :: tageb(:) ALLOCATE(tag_eb(1:NP)): Each particle is identified by a unique integer number => Useful for particle tracking tag List of particles Part 1 tag1 Part tag Part NP tag3 1

13 Fields/Currents data structure Fields defined on the Yee lattice (main.f90): Used to evolve the fields (see Lecture I)!**************************************************************! Magnetic and Electric fields components Yee lattice DOUBLE PRECISION, DIMENSION(1:NX,1:NY) :: Bx,By,Bz DOUBLE PRECISION, DIMENSION(1:NX,1:NY) :: Ex,Ey,Ez The current density is needed on the Yee mesh (main.f90)!**************************************************************! Current density components Yee lattice DOUBLE PRECISION, DIMENSION(1:NX,1:NY) :: Jx,Jy,Jz Fields defined on the nodes of the grid (main.f90): Used to evolve the particles (see Lecture I)!**************************************************************! Magnetic and Electric fields components at nodes DOUBLE PRECISION, DIMENSION(1:NX,1:NY) :: Bxg,Byg,Bzg DOUBLE PRECISION, DIMENSION(1:NX,1:NY) :: Exg,Eyg,Ezg 13

14 From Yee to nodes D y Electric field j+1 Ey Bx Bz j+1/ By j Ez i Ex i +1/, j + Exi 1/, j Exgi, j = Ex i+1/ x i+1 Ey i, j +1/ + Ey i, j 1 / Eygi, j = Ezgi, j =Ez i, j Magnetic field Bx i, j +1 / + Bx i, j 1/ Bxgi, j= By i+1 /, j + By i 1 /, j Bygi, j= Bz i+1 /, j +1/ + Bzi 1/, j +1 / + Bz i+1 /, j 1 / + Bz i 1 /, j 1 / Bzgi, j = 4 14

15 Correction of the electric field Zeltron does not use a charge conserving deposition scheme => Poisson equation must be solved! Consider that: E '= E+ δ E Small correction Known field Correct δ E = δ ϕ δ ϕ= ( 4 π ρ E) Solve this equation using a Gauss Seidel iterative method Empirically, Zeltron uses 500 iterations, every 5 time steps. Relevant parameters in mod_input.f90! Poisson solver calling frequency in terms of timesteps INTEGER, PARAMETER, PUBLIC :: FREQ_POISSON=5! Number of iterations to solve Poisson's equation INTEGER, PARAMETER, PUBLIC :: NIT=500 15

16 Poisson solver δ ϕ= ( 4 π ρ E) (i,j+1) D Example: 5 points stencil (i+1,j) (i,j) (i+1,j) (i,j 1) δ ϕ δ ϕ δ ϕi+ 1, j δ ϕi, j + δ ϕi 1, j δ ϕi, j+1 δ ϕi, j +δ ϕi, j 1 δ ϕ= + + x y Δx Δ y Injecting this into Poisson and after some rearrangements yields (and if x= y): δ ϕi, j = 1 δ ϕ + δ ϕ +δ ϕ +δ ϕ +( 4 π ρ E ) Δ x ( i+1, j ) i 1, j i, j +1 i, j 1 4 Calling the subroutine in main.f90 IF (MOD(it,FREQ_POISSON).EQ.0) THEN! Solve Poisson equation to ensure that div(e)=4*pi*rho CALL CORRECT_EFIELD(Ex,Ey,xgp,ygp,rho,id,ngh,TOPO_COMM,ierr) END IF 16

17 Particle initialization: Spatial distribution Let's consider a uniform distribution in D (mod_initial.f90) Cell (ix,iy) x (ix,iy+1) (ix+1,iy+1) y (ix,iy) xmin xmin+(ix 1) x (ix+1,iy) xmin+ix x!**************************************************************! Definition of the initial position (x0,y0) as a uniform random number defined between 0 and 1 x0c=0.0 y0c=0.0 CALL RANDOM_NUMBER(x0c) CALL RANDOM_NUMBER(y0c) x0c=xmin+(ix-1)*dx+x0c*dx y0c=ymin+(iy-1)*dy+y0c*dy!************************************************************** 17

18 Particle weight A macroparticle represents a large number of physical particles following the exact same trajectory in phase space. => The particle weight gives the normalization factor to connect between numerical and physical plasma densities. For a uniform physical and numerical plasma density, all the particles have the same weight given by: N phys n ( L x L y L z ) weight = = N num N num Physical density Sample code in D from main.f90:! Weight of background particles pcl_eb(7,:)=density_ratio*nd0*(xmax-xmin)*(ymax-ymin)/np pcl_pb(7,:)=density_ratio*nd0*(xmax-xmin)*(ymax-ymin)/np 18

19 Variable particle weight Large density contrast or sharp density profiles are hard to model with a constant particle weight. => Result in a bad sampling of low density regions Example: Relativistic reconnection studies where factor >10 between the background and the sheet plasmas. Solution: Variable particle weighting: Lx L y Lz weight ( y i)=n( y i) N num 19

20 Macroscopic quantities reconstruction One can reconstruct global and fluid quantities from the particles: (mod_analysis.f90) N cell Particle spectrum: dn 1 wk Δ γ dγ k =1 N dn 1 wk dv Δ V k=1 SUBROUTINE SPECTRUM_ANGULAR cell Plasma density: Stress energy tensor: μν P μ ν μν V V P η c ( ) T = ρ+ 1 ΔV Energy density: T 00 U e Momentum density: 1 T 0 i U p ΔV Pressure tensor: ij SUBROUTINE MAP_XY ij T P 1 ΔV SUBROUTINE MAP_FLUID N cell w γ mc k k k=1 N cell u k mc wk k=1 N cell k =1 ( ) wk m c i j uk u k γk 0

21 Particle initialization: Angular distribution Let's consider an isotropic angular distribution in 3D In spherical coordinates: θ0 u0 P(x,y) φ0 cosθ0 uniform random number between 1 and 1. - φ0 uniform random number between 0 and π. 1

22 Particle initialization: Angular distribution Let's consider an isotropic angular distribution in 3D (mod_initial.f90)!**************************************************************! Definition of the initial phi0 as a uniform random value between 0 and 1 phi0=0.0 CALL RANDOM_NUMBER(phi0) phi0=phi0*.0*pi! Definition of the initial cth0=cos(theta0) as a uniform random value between 0 and 1 cth0=0.0 CALL RANDOM_NUMBER(cth0) cth0=cth0*.0-1.0! Initial 4-velocity components ux0c=u0*sqrt(1.0-cth0*cth0)*cos(phi0) uy0c=u0*sqrt(1.0-cth0*cth0)*sin(phi0) uz0c=u0*cth0

23 Particle initialization: Energy distribution Let's consider a Power law distribution: f ( γ)= dn p γ dγ with γ [γ min, γ max ] The trick is to use the cumulative distribution: γ γ F ( γ) γ γ f ( γ ' )d γ ' min max f (γ')d γ ' min = γ p+1 γ minp+1 p+1 γ max γ minp+1 Next, define a uniform random number R between 0 and 1, and sample F to invert the distribution: F(γ) 1 R p +1 p +1 1 p +1 p+1 min γ=( R(γ max γ min )+ γ 0 ) 3 γmin γ γmax

24 Particle initialization: Energy distribution However, the cumulative distribution is often a tabulated (no analytical expression) F(γ) 1 R 0 F F1 γ1 γ γmin γ γmax We need to interpolate, a linear interpolation usually suffices: R ( 1 Γ) F 1 +Γ F γ γ 1 Γ= γ γ 1 R( γ γ 1) ( γ F1 γ1 F ) γ F F1 4

25 Particle initialization: Energy distribution In Zeltron, it looks like this (mod_initial.f90): Ru=0.0 CALL RANDOM_NUMBER(Ru) Ru=Ru*0.9999! Sum over all particles in a the cell DO ic=1,ppc! Localize closest known values of cumulative distribution gfu minu=minloc(abs(gfu-ru(ic))) iu=minu(1)! F1 and F gfu1=gfu(iu) gfu=gfu(iu+1)! u1 and u u1=ud(iu) u=ud(iu+1)! Linear interpolartion in the log-linear plane (numerically more accurate) u0(ic)=exp((ru(ic)*(log(u)-log(u1))log(u)*gfu1+log(u1)*gfu)/(gfu-gfu1)) ENDDO 5

26 Hands on II: Relativistic reconnection +B0 B0 Outflow Vout Erec Inflow Magnetic energy =? Plasma kinetic energy (heating+non thermal particles) How fast does reconnection proceed? How efficient at accelerating particles? What are the main acceleration mechanisms? 6

27 A common phenomenon ESA/C. T. Russel Man made plasma devices (e.g., Tokamaks, MRX at PPPL) Planet's magnetosphere (e.g., Earth, Jupiter, Saturn) ESO/L.Calçada NASA Pulsars/Magnetars Solar corona This evening's talk! BU Astrophysical jets Gamma ray burst7

28 Usual PIC setup: The Harris sheet 4π B= J c +B0 Current layer on skin depth scale δ J B0 8

29 Usual PIC setup: The Harris sheet Background particles +B0 Drifting particles initially carrying the current, in pressure balance δ B0 9

30 The relativistic Harris solution [See Kirk & Skjæraasen 003] The Harris solution is a kinetic equilibrium between the upstream magnetic pressure and a hot plasma concentrated inside the current sheet. The solution: The reconnecting field: B=B 0 tanh The drifting plasma density: y e δ x ( ) n d =n0 cosh y δ ( ) n 0= kt 4 π e Γd β d δ Pressure balance condition across the sheet: The background plasma: B0 =nd kt 8π n b =ϵ n0, (ϵ 1 for σ 1) σ= B0 4 π nb m e c 1 30

31 Particle initialization: Drifting Maxwellian The Harris solution assumes a relativistic drifting Maxwellian distribution, i.e., the plasma has a relativistic motion of Lorentz factor Γd. ( γ ' 1) f ( u ) exp θ kt ' θ= γ ' =Γd ( γ βd u ) mc In the lab frame: Where Following Swisdak (013): ( ) u =u + u Generation of ukvia the cumulative distribution of F(uk) u is generated, but cannot be chosen independently of uk The full procedure is in mod_initial.f90 Subroutines: SET_DRIFT_MAXWELLIAN, INIT_DRIFT_MAXWELLIAN, GEN_UP, GEN_PS 31

32 Initial numerical setup e+/e pairs +B0 Periodic δ B0 δ +B0 Periodic 3