Flux Limited Diffusion in Enzo

Flux Limited Diffusion in Enzo Daniel R. Reynolds reynolds@smu.edu Department of Mathematics Southern Methodist University Enzo User Workshop, 29 June 2010

Reading The slides (Enzo2010 talk1 reynolds.pdf) are on the workshop page, http://lca.ucsd.edu/workshops/enzo2010/slides These give a subset of the information from the FLD solver module user guides, located in the enzo-2.0 directories: doc/implicit fld gfldproblem class user guide doc/split fld gfldsplit class user guide In either directory, type make to build the PDF. The implicit solver is described in some detail in the paper D.R. Reynolds, J.C. Hayes, P. Paschos and M.L. Norman, J. Comput. Phys., 228:6833 6854, 2009.

Flux-limited diffusion radiation model We consider cosmological flux-limited diffusion radiative transfer, t E + 1 a (Ev b) = (D E) ȧ a E cκe + η, E(x, t) is the comoving radiation energy density [erg cm 3 ], η(x, t) is the emissivity [erg cm 3 s 1 ], κ(x, t) is the opacity [cm 1 ], The flux-limiter D transitions between streaming and opaque limits, D(E) = diag (D 1 (E), D 2 (E), D 3 (E)), where the directional D i (E) are given by the Larsen n=2 limiter, { } c i E D i (E) =, R (3κ)2 + Ri 2 i (E) = max E, 10 20. [see Morel 2000; Hayes & Norman 2003; Reynolds, Hayes, Paschos & Norman, 2009]

Assumed Radiation Spectrum Frequency-dependence of the radiation field has been integrated out, using an assumed radiation spectrum, χ E (ν): E ν (ν, x, t) = Ẽ(x, t)χ E (ν), E(x, t) ν HI E ν (ν, x, t) dν = Ẽ(x, t) ν HI χ E (ν) dν, Ẽ is an intermediate quantity (for analysis) that is never computed, Choices of χ E (ν) include [monochromatic, power law, blackbody]: ( ) 1.5 ν 8πh ( ν χ E (ν) = δ νhi (ν), χ E (ν) = ν HI, c χe (ν) = ( ). hν exp k b 10 1 5 η(x, t) relates to the true emissivity η ν (ν, x, t) via η(x, t) = ν HI η ν (ν, x, t) dν. ) 3

Local Thermodynamic Equilibrium Coupling (model 10) For problems without chemistry, we couple E to the specific gas energy, t e + 1 a v b e = 2ȧ a e 1 aρ b (pv b ) 1 a v b φ + G Λ. (1) All but the final two terms are already handled by Enzo s hydrodynamics solvers, so we consider only a correction equation G = cκ ρ b E is the local heating rate, Λ = η ρ b t e c = 2ȧ a e c + G Λ, (2) is the local cooling rate, κ = C 0 ( ρb C 1 ) C2 is the opacity (C0 C 2 are inputs), η = 4κ σ SB T 4 is a black-body emissivity, σ SB is the Stefan-Boltzmann constant, T is the gas temperature.

Chemistry-Dependent Coupling (models 1 & 4) Enzo s FLD solvers currently allow hydrogen chemistry, using the model t e c = 2ȧ a e c + G Λ, t n HI + 1 a (n HI v b ) = α rec n e n HII n HI Γ ph HI. n HI is the comoving HI number density [cm 3 ], α rec is the case-b recombination rate, Heating and cooling are computed as G = c n HI ρ b ν HI ( E ν σ ν νhi ) HI ν dν, Λ = ne ρ b [ cehi n HI + ci HI n HI + re HII n HII + brem n HII + m h ρ units a 3 (comp 1 (T comp 2 ) + comp X (T comp T )) ]. The temperature-dependent cooling rates ce HI, ci HI, re HII, brem and comp are all taken from Enzo s built-in rate tables.

Chemistry-Dependent Coupling (models 1 & 4) The photo-ionization rate is computed as Γ ph HI = c ν HI E ν σ HI hν dν. The frequency-integrated opacity is now chemistry-dependent, [ ] / [ ] κ = n HI E ν σ HI dν E ν dν, ν HI where all simplify to integrals with χ E (computed at startup). The difference between models 1 and 4 is: Model 1 implements all of the above terms, Model 4 assumes an isothermal gas. ν HI

Enzo Operator-Split Numerics We discretize space with a finite volume approach, and evolve the coupled system in an operator-split framework, solving one component at a time: (i) FLD system, (ii) Gravity, (iii) Hydrodynamics (also advects n HI and E), (iv) Dark-matter, star particles, etc. The FLD system is evolved using either: gfldproblem fully-implicit reaction-diffusion PDE system to evolve (E, e c, n HI ), or gfldsplit separate implicit steps evolve E and then {e c, n HI }.

Split Solver Radiation Subsystem We discretize t E in time using a two-level θ-method: E n E n 1 θ t ( (D E n ) ȧ a E n cκe n + η ) (1 θ) t ( (D E n 1) ȧ a E n 1 cκe n 1 + η ) = 0, The input parameter 0 θ 1 defines the time-discretization: θ = 1 gives implicit Euler [O( t)-accurate], θ = 0.5 gives trapezoidal [O( t 2 )-accurate]. For θ > 0 this equation is linearly implicit in s = E n E n 1. We approximately solve this linear system Js = b to a tolerance δ, Js b 2 δ, using a scalable, multigrid-preconditioned, CG iteration [HYPRE].

Split Solver Chemistry & Gas Subsystem We solve {ec n, n n HI } using an implicit version of the quasi-steady-state approximation, where we freeze inter-variable couplings and solve each ODE exactly. Writing these analytical solvers as e n c = sol e ( E, e n 1 c, n HI, t ), n n HI = sol HI ( E, e, n n 1 HI, t ), we couple these implicitly through defining the nonlinear equations ( ) f e (e c, n HI ) = ec n E sol n 1 +E n e 2, ec n 1, nn 1 HI +n n HI 2, t = 0, ( ) f HI (e c, n HI ) = n n E HI sol n 1 +E n HI 2, en 1 +e n 2, n n 1 HI, t = 0, and solve this nonlinear system using a robust fixed-point iteration.

Adaptive Time Step Selection We estimate the time error in a variable u as ( 1 N ( u n err = i u n 1 )p ) 1/p i N ω i i=1 where ω i = { u n i un 1 i + 10 3, e i + 10 3. The input p defines the norm. An input tolerance τ i,tol then defines t n i = (τ i,tol /err) t n 1, i {E, e, HI }. Since evolved separately, {e c, n HI } may subcycle faster than E. Users may override t E & t e,hi with the inputs t max & t min. All physics modules will limit their step to t min{ t E, t CFL }.

Variable Rescaling The solvers and time step controls work best when the internal values of {E, e c, n HI } are properly normalized. If Enzo s DensityUnits, LengthUnits and TimeUnits are insufficient, the user may input additional variable scaling factors for use in the solver: RadiationScaling sets the value s E, EnergyCorrectionScaling sets the value s e, ChemistryScaling sets the value s n. These are used to rescale {E, e c, n HI } to solver variables {Ẽ, ẽ c, ñ HI }: Ẽ = E s E, ẽ c = e c s e, ñ HI = n HI s n.

Radiation Boundary Conditions We currently allow 3 boundary condition types for E: 0. Periodic, 1. Dirichlet, i.e. E(x, t) = g(x), x Ω, and 2. Neumann, i.e. E(x, t) n = g(x), x Ω. Notes: In most cases, the BC types (and g) are problem-dependent. If left unspecified, these default to periodic. If a non-periodic type is set but values are not, we use g = 0. When adding new FLD problems, custom conditions should be set in gfldsplit Initialize.C or gfldproblem Initialize.C.

Build Configuration To use any FLD solver module, Enzo must be configured with: gmake photon-yes gmake hypre-yes gmake use-mpi-yes [enables all radiation solvers] [enables HYPRE solver interface] [enables MPI] Moreover, the machine Makefile must specify how to include and link with an available HYPRE library (version 2.4.0b). If you must compile HYPRE yourself, use the configuration option --with-no-global-partition for runs using over 1000 tasks. Optional/recommended Enzo configuration options include: gmake emissivity-yes [enables coupling with star-maker] gmake precision-64 [the solvers prefer double precision]

Startup Parameters The main problem parameter file must have the following parameters: RadiativeTransferFLD [0] this must be 2. ImplicitProblem [0] use 3 for gfldsplit, 1 for gfldproblem. ProblemType [0] FLD-based solvers use values in the 400 s. RadHydroParamfile [NULL] the filename containing all FLD-specific solver parameters (next slide). RadiativeTransferOpticallyThinH2 [1] this must be 0. RadiationFieldType [0] can be any value except 10 or 11. RadiativeTransferFLDCallOnLevel [0] must currently be 0. RadiativeTransfer [0] this must be be 0. RadiativeCooling [0] must currently be 0.

FLD Solver Parameters (separate input file) The following parameters control various aspects of gfldsplit: RadHydroESpectrum [1] form for radiation spectrum χ E (ν): -1. is monochromatic at hν = 13.6 ev, 0. is power law, 1. is T = 10 5 blackbody. RadHydroChemistry [1] controls whether to use n HI (1 yes, 0 no) RadHydroHFraction [1] controls the fraction of baryonic matter comprised of Hydrogen (RadHydroHFraction [0, 1]). RadHydroModel [1] determines which model for radiation-matter coupling we wish to use: 1. Chemistry-dependent model with case B recombination coeff. 4. Same as model 1, with an isothermal gas energy. 10. Local thermodynamic equilibrium model (no n HI ).

FLD Solver Parameters continued RadHydroMaxDt [10 20 ] sets t max in scaled time units. RadHydroMinDt [0] sets t min in scaled time units. RadHydroInitDt [10 20 ] sets the initial t E in scaled time units. RadHydroDtNorm [2] sets p in computing the time error estimate. RadHydroDtRadFac, RadHydroDtGasFac, RadHydroDtChemFac [10 20 ] the values of τ i,tol in computing t E, t e and t HI. RadiationScaling, EnergyCorrectionScaling, ChemistryScaling [1.0] the scaling factors s E, s e and s n. RadHydroTheta [1.0] the t E discretization parameter, θ. RadHydroSolTolerance [10 8 ] linear solver tolerance δ.

FLD Solver Parameters continued RadiationBoundaryX0Faces, RadiationBoundaryX1Faces, RadiationBoundaryX2Faces [0 0] BC types at each face: 0. periodic (must match on both faces in a given direction) 1. Dirichlet 2. Neumann EnergyOpacityC0-EnergyOpacityC2 [1, 1, 0] the opacity-defining constants C 0 -C 2 for the LTE model. RadHydroMaxMGIters [50] max number of MG-CG iterations. RadHydroMGRelaxType [1] - the MG relaxation method: 0. Jacobi 1. Weighted Jacobi 2. Red/Black Gauss-Seidel (symmetric) 3. Red/Black Gauss-Seidel (nonsymmetric) RadHydroMGPreRelax [1] number of pre-relaxation MG sweeps. RadHydroMGPostRelax [1] number of post-relaxation MG sweeps.

Customization To set up a new FLD problem: Allocate a baryon field with FieldType set to RadiationFreq0. Set η(x, t) by either: Edit gfldsplit RadiationSource.src90 or gfldproblem RadiationSource.src90, Fill in the baryon field Emissivity0, and edit logic in gfldsplit Evolve.C or gfldproblem Evolve.C to use that field (emulate logic for StarMakerEmissivityField). Edit gfldsplit Initialize.C or gfldproblem Initialize.C to call the problem initializer and set BCs. All other requirements for setting up a new ProblemType in Enzo are like normal (InitializeNew.C, problem initialization files, etc.).

Iliev Test 5 Example Dynamic I-front Expansion [on Triton: /home/enzo-1/ilievetal5] Dynamic ionization test of an initially-neutral hydrogen region: Box size L = 15 kpc; Run time T f = 500 Myr. T = 10 5 blackbody spectrum, at rate Ṅγ = 5 10 48 photon/s. Initial conditions: n = 10 3 cm 3, T = 100 K, E = 10 30 No available analytical solution, but: erg cm 3. Front transitions from R- to D-type as it reaches Strömgren radius, r R I = r S [ 1 e tα B (T i )n H ] 1/3, r D I = r S [1 + (7c s t)/(4r S )] 4/7, Eventually stalls at r f = r S ( 2Ti T e ) 2/3, where Ti and T e are the temperatures behind and ahead of the I-front. [Whalen & Norman, ApJS, 2006; Iliev et al., MNRAS, 2009]

Hydrodynamic Radiative Ionization Results 2.5 Convergence in I front Position 16 3 mesh 4.5 Temperature profile, t = 175 Myr 32 3 mesh 2 1.5 64 3 mesh 128 3 mesh NW: I-front position history 4 3.5 r I /r S log(t) 3 1 0.5 NE: T profile (175 Myr) 2.5 2 r I r I * /rs 0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 t/t rec 0.045 0.04 0.035 0.03 0.025 0.02 0.015 0.01 0.005 Error in I front Position 16 3 mesh 32 3 mesh 64 3 mesh 0 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 t/t rec SW: r I conv. wrt x SE: T convergence wrt t relative error 1.5 0 0.2 0.4 0.6 0.8 1 r/l box 10 1 10 2 10 3 10 4 10 5 10 6 10 7 10 8 5e 4 2e 4 1e 4 5e 5 2e 5 1e 5 Temperature error profile, t = 175 Myr 0 0.1 0.2 0.3 0.4 0.5 r/l box

Shapiro & Giroux [Isothermal, Static Cosmic Ionization] [in enzo-2.0: run/radiationtransportfld/sg q5z4 sp] Repeat of previous test, but in a cosmologically expanding universe, with a static, isothermal gas, using a monochromatic radiation spectrum. Four tests: q 0 z i L i [kpc] ρ b,i [g cm 3 ] H 0 Ω m Ω Λ Ω b 0.5 4 80 1.18e-28 0.5 1.0 0 0.2 0.05 4 60 2.35e-28 1.0 0.1 0 0.1 0.5 10 36 1.18e-28 0.5 1.0 0 0.2 0.05 10 27 2.35e-28 1.0 0.1 0 0.1 Analytical solution given by r I (t) = r S,i ( 1/3 a(t) λe τ(t) e τ(b) [1 2q 0 + 2q 0 (1 + z i )/b] db) 1/2, τ(a) = λ [F (a) F (1)] [ 6q 2 0(1 + z i ) 2] 1, F (a) = [ 2 4q 0 2q 0 1+z i a 1 ] [ 1 2q0 + 2q 0 1+z i a α λ = B n H,i ] 1/2. H 0(1+z i ), [Shapiro & Giroux, ApJ, 1987]

Cosmological Ionization Results 1 0.9 0.8 0.7 r i (t)/r s (t) vs redshift q0=0.5, z0=4 q0=0.05, z0=4 q0=0.5, z0=10 q0=0.05, z0=10 NW: I-front radii vs scaled z Error in r 11 x i (t)/r s (t) vs scaled redshift, q 0 = 0.5, z 0 = 4 10 3 16 3 mesh 10 32 3 mesh 64 3 mesh 9 8 r/r s 0.6 0.5 0.4 0.3 NE: I-front error vs scaled z (r true r)/r s 7 6 5 4 0.2 3 0.1 2 r/r s 0 0 0.5 1 1.5 2 2.5 3 log[(1+z)/(1+z i )] 0.6 0.5 0.4 0.3 0.2 0.1 0 r i (t)/r s (t) vs redshift, z i =4 q0=0.5 (computed) q0=0.5 (analytic) q0=0.05 (computed) q0=0.05 (analytic) 0.1 0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 log[(1+z)/(1+z )] i SW: I-front radii for z i = 4. SE: Weak CPU scaling (N src N CPU ) Kraken @ NICS: O(N log N) scaling Average wall time / step (s) 25 20 15 10 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 log[(1+z)/(1+z 0 )] Cosmology Weak Scaling (Kraken): Tavg vs Cores 5 1 8 64 512 4096 32768 Cores

Reionization Simulations (FLD + StarMaker) [in enzo-2.0: run/radiationtransportfld/cosmologyfld RT] Geoffrey So has constructed an interface to StarMaker to seed η(x, t): Requires Enzo configuration with EMISSIVITY enabled, Adds StarMakerEmissivityField=1 to main parameter file, The interface fills the Emissivity0 baryon field based on emission from star particles; the FLD modules copy this field into η(x, t). Utilizes identical startup machinery as typical cosmology runs, via CosmologySimulationInitialize.C [ProblemType 30], with additional input file options to enable the FLD solver module.

Model Solution Approach Configuration Non-Cosmological Problems Cosmological Problems Reionization Simulations (FLD + StarMaker) ρ E xhi End

Summary of Current Results The gfldsplit and gfldproblem solver modules implement a grey, field-based, flux-limited diffusion radiation approximation for unigrid runs: Implicit MG-CG solvers enable scalable solution on many thousands of cores, independently of the number of ionization sources. Accurately solves couplings between radiation, ionization and gas energy, due to implicit formulation and coupled solvers. Split and implicit formulations allow for tradeoffs between robustness/efficiency and accuracy. However, this approximation has its shortcomings: Single radiation field allows full absorption by hydrogen, even though higher-frequency radiation should pass through. Though better than simpler approximations, grey approach cannot accurately handle multi-species problems (hence H-only restriction). Currently limited to unigrid Enzo simulations.

Continuing Work Extending radiation approximation to multi-frequency case, E E ν : New interpolation of ν-space, based off of [Gnedin & Abel, 2001; Ricotti et al., 2002], in collaboration with Pascal Paschos. Approach is designed to characterize underlying radiation spectrum for cosmology problems, with a minimal amount of extra overhead. Extending grey FLD solvers to AMR grids: HYPRE-FAC gravity solver already completed by James Bordner. Radiation solver will piggyback off of self-gravity developments. Extending gfldsplit to allow He, molecular gases, metal chemistry.

Acknowledgements We gratefully acknowledge support by NSF AAG program and point out contributions by collaborators Michael Norman, UCSD John Hayes, LLNL (B-Division) Pascal Paschos, UCSD Geoffrey So, UCSD