Comparison of ODE Solver for Chemical Kinetics and Reactive CFD Applications

Transcription

1 See discussions, stats, and author profiles for this publication at: Comparison of ODE Solver for Chemical Kinetics and Reactive CFD Applications CONFERENCE PAPER JANUARY 2014 DOI: / READS 58 2 AUTHORS, INCLUDING: Christopher P. Stone Computational Science and Engineering, LLC 32 PUBLICATIONS 180 CITATIONS SEE PROFILE Available from: Christopher P. Stone Retrieved on: 29 February 2016

2 AIAA SciTech January 2014, National Harbor, Maryland 52nd Aerospace Sciences Meeting AIAA Comparison of ODE Solvers for Chemical Kinetics and Reactive CFD Applications Christopher P. Stone Computational Science and Engineering, LLC, Chicago, IL, Fabrizio Bisetti Clean Combustion Research Center King Abdullah University of Science and Technology, Thuwal, Saudi Arabia, Accurate and efficient methods for solving stiff initial value problems (IVPs) are a critical component of turbulent combustion simulations with finite-rate chemistry. The optimal algorithm depends upon the size and stiffness of the chemical mechanism as well as the time intervals over which the ordinary differential equations (ODEs) are integrated. We survey a collection of methods for stiff IVPs. We demonstrate and assess the performance of selected solvers for two test problems using the GRI Mech 3.0 mechanism: a 0-d H 2-air ignition and a 1-d CH 4-air laminar premixed flame propagation simulation. High-order one-step methods such as implicit Runge-Kutta and Rosenbrock methods are found to be efficient for realistic CFD time-steps (e.g., δt = O(1) µs) due to low start-up costs and large possible step sizes. The widely used multi-step VODE solver is also efficient for both large and small CFD time-step sizes of interest. The one-step methods are easy to implement, are vectorizable and have low storage requirements. All of these characteristics make them viable alternatives to VODE for applications on newer parallel computing devices such as graphics processing units. Nomenclature h ODE integration time-step δt CFD time-step NS number of chemical species NR number of elemental reactions O(... ) on-the-order-of ( ) time rate-of-change () k quantity for species k () mass-averaged mixture quantity p thermodynamic pressure R universal gas constant (J/mol-K) R mixture specific gas constant (J/Kg-K) R k gas constant for species k (J/mol-K) ρ density (kg/m 3 ) Owner, Computational Science and Engineering LLC, Chicago, IL, AIAA member, chris.stone@computational-science.com Asst. Professor, Clean Combustion Research Center, King Abdullah University of Science and Technology, fabrizio.bisetti@kaust.edu.sa 1 of 19 Copyright 2014 by Christopher P Stone and Fabrizio Bisetti. Published by the, Inc., with permission.

3 I. Introduction Predicting turbulent combustion phenomena such as extinction and re-ignition with reactive CFD simulations requires finite-rate chemical kinetics with detailed or reduced mechanisms. However, the computational costs can overwhelm the available computer resources. High fidelity combustion simulations with finite-rate kinetics must solve NS additional species equations in addition to the ones for mass, momentum, and energy. Detailed chemical mechanisms consist of hundreds of chemical species with thousands of elemental reactions leading to intractable storage and computational costs. The computational cost is further increased by the numerical stiffness of the chemical kinetics. For example, in H 2 -air combustion, the induction (µs) and NO formation (ms) time-scales differ by a factor of Operator splitting (e.g., Strang splitting 2 ) is commonly used to decouple the stiff chemical kinetics and non-stiff convection-diffusion components of the conservation equations, as well as to reduce the size of the system of equations to be solved. In this approach, the contribution of chemistry at each grid point is treated as an independent system of ordinary differential equations (ODEs) and integrated over the specified CFD time-step (δt). That is, a large partial differential equation (PDE) system is broken into a sequence of smaller ODE systems, one for each grid point. The species and the temperature equations are integrated in time using a constant-pressure or constant-volume assumption. The CFD time-step must be relatively small to avoid large splitting errors caused by thermal expansion, diffusion, and convection. As a whole, solving individual ODEs is far less expensive than the fully coupled PDE system. Even with this simplification, the computational cost of solving finite-rate kinetics via local IVPs can consume in excess of 90% of the run-time in CFD simulations. Efficient numerical methods are required to solve stiff ODEs, such as those encountered in combustion kinetics. Past studies 1 on ODE chemistry solver efficiency and accuracy have focused on 0-d ignition problems and integrate until chemical equilibrium is reached after O(10) ms, which is the upper bound on the ignition delay times of most hydrocarbons. Solvers based on high-order, backward differentiation formulas (BDFs), such as VODE, 3 have been found to be efficient and robust for these types of problems and are widely used throughout the combustion community for mechanism development, reduced-order analysis, as well as CFD simulations. Time-accurate turbulent combustion simulations require time-steps between 0.1 and 10 µs. The evolution of the mixture state due to chemical reactions over a finite time interval may be computed more efficiently by methods other than BDFs. Operator-splitting provides a vast amount of parallelism since each ODE system can be solved concurrently. Algorithms other than high-order multi-step methods may be more suitable for this type of fine-grain parallel processing, especially on modern multi-core CPUs and graphics processing units (GPU). Firstly, we present the mathematical model describing the evolution of the chemical state vector due to reactions. Several classes of ODE solvers are then discussed along with implementation details. Finally, we present performance metrics from two case studies with a wide range of ODE algorithms. II. Mathematical Model The ODE system governing the species and energy conservation equations for gas-phase combustion at each grid point are ẏ k = ω kw k ρ T = 1 N s h k ω k, (2) c p where y k, W k, h k, and ω k are the mass fraction, molar mass, enthalpy, and molar production rate for species k, and T, c p, and ρ are the mixture temperature, specific heat at constant pressure, and density. Equations 1 and 2 are closed by the equation of state for an ideal gas, p = ρrt, where p is the thermodynamic pressure and R is the mixture gas constant. A constant-pressure or -volume process is obtained by keeping either p or ρ constant. a a c p and h k are replaced by c v and u k for a constant-volume process. k (1) 2 of 19

4 The net molar production rate terms in Eqs. 1 and 2 are non-linear functions of pressure p, T, and the species molar concentrations [X k ]. They are also the source of the stiffness in the ODE system and their calculation is the most computationally costly component of the integration. They are expressed as follows: NR k NS i NS i ω k = C i [ν ki ν ki] k f,i [X j ] ν ji kr,i [X j ] ν ji, (3) i where NR k is the number of reactions involving species k, C i is a parameter accounting for any thirdbody and/or pressure effects, ν ki and ν ki are the reactant and product stoichiometric matrices defining the coefficients for species k in reaction i, NS i is the number of reactants and products in reaction i, and k f and k r are the forward and reverse reaction rate constants. Details regarding the third-body and pressure fall-off effects embodied in C i can be found in Safta et al. 4 The forward rate constant is given in Arrhenius form as ( k f,i = A i T β i exp E ) i. (4) RT If reaction i is irreversible, k r,i is zero. Otherwise, it is defined as where the equilibrium constant K c,i is given by K c,i = = ( po j j k r,i = k f,i K c,i, (5) ) NS i j ν ji Kp,i (6) RT ( po ) NS i j νji exp RT j ν ji ( sj R j h ) j. (7) R j T p o is the standard pressure at one atmosphere (in the appropriate units) and s j is the entropy for species j. Temperature-dependent thermodynamic properties per unit mass (e.g., c p,k, h k, s k ) are computed from polynomial fits using the following formulas c p,k R k = a k1 + a k2 T + a k3 T 2 + a k4 T 4 h k = a k1 T + a k2 R k 2 T 2 + a k3 3 T 3 + a k4 4 T 4 + a k5 5 T 5 + a k6 T s k = a k1 log(t ) + a k2 T + a k3 T 2 + a k4 T 3 + a k5 T 4 a k6 R k T + a k,7. The polynomial coefficients a ki are taken from the NASA 7-term polynomial database. 5 The equations above may be combined linearly to provide an efficient polynomial expression for the Gibbs free energy in Eq. 7. The mass fraction and temperature equations in vector form are (8) u(t) = f(u(t), q) u(t = t 0 ) = [y 1 (t 0 ),..., y NS (t 0 ), T (t 0 )] T, where f is the right-hand-side (RHS) function, u(t) is the vector of unknowns at some time t, q is a set of non-integrated parameters (e.g., p or ρ), and t is the independent time variable with initial conditions given at t 0. The length of u is N = NS+1 unknowns. Note that Eqn. 9 is in autonomous form, neglecting any explicit dependence of f on t. (9) 3 of 19

5 III. Numerical Methods We now present the various ODE solvers, which will be investigated in this study. All methods seek to advance the solution from u(t) to u(t+h), where h is the adjustable integration step size. The different solvers are classified into two main categories: multi-step and one-step methods. Multi-step methods use past time-steps (e.g., u(t-h),..., u(t-4h)), while one-step methods use u(t) only. That is, one-step methods treat each integration step as a new problem. Fully-implicit solvers are generally used to solve stiff kinetics problems. Several semi-implicit and linearlyimplicit methods have also been considered, as well as a fully-explicit method for the sake of comparison. For details on the taxonomy of ODE solvers, the reader is referred to the book on numerical methods for stiff systems of ODEs by Hairer and Wanner. 6 III.A. Multi-step methods Multi-step backward differentiation formula (BDF) methods are widely used to solve stiff ODE systems. The BDF method is given by j α i u n i = hβ 0 f(u n, q), (10) i=0 where α i and β 0 are coefficients depending upon the order p of the solver and j is the number of past steps used (p j). Note that j = 1 gives the backwards Euler method and that α 0 = 1 by convention. BDF methods are A-stable for p 2 and A(α) stable for p 6. The practical limit is p 5 (α = o ) and this is the maximum order available in VODE, 3 a widely used BDF implementation. See Hairer and Wanner 6 for a complete list of stability conditions and coefficients. Equation 10 is a non-linear equation and is solved iteratively. Defining the iteration function as G(u n ) u n hβ 0 f(u n, q) j α i u n i, (11) we seek a solution such that G = 0. VODE and other modern fully-implicit methods solve Eqn. 11 with a Newton-Raphson iteration of the form [ ] I hβ 0 J(u (i) n ) δu (i) n = G(u (i) n ) (12) u (i+1) n = u (i) n + δu (i) n, where J is the Jacobian of the source vector in Eqn. 9. A linear system must be solved at each iteration. Note that for linear or locally near-linear systems, we can evaluate J at u (0) n and reuse the Jacobian matrix throughout the iterative process. This reduces the nominal 2nd-order convergence rate, but is computationally advantageous for expensive Jacobians. The iterations continue until either the correction δu (i) n or the root function G drops below a user-specified tolerance. BDF methods such as those implemented in VODE are quite efficient and may adapt both h and p depending on a local error estimate. VODE estimates the truncation error by the difference between the polynomial predictor (i.e., order p extrapolation to u n ) and the solution to Eqn. 10. BDF methods suffer from order reduction during start-up, since they have no past history to use. Thus, they require very small start-up step sizes to enforce the user-specified tolerance. They also require storage for the history data, but the requirements are usually small relative to the storage of the Jacobian matrix itself. Note that the memory required for the matrix storage is doubled if the Jacobian is recycled. These matrix storage requirements are minimal for modern computer architectures if a single problem is solved: O(160) kb for NS = 100 and O(15) MB for NS = 1,000. However, memory requirements may become prohibitive if one attempts to solve 100 s or 1000 s of ODEs concurrently in parallel computing environments characterized by low memory per-core, such as high-performance GPUs and other hardware accelerators. i=1 III.B. One-step methods One-step methods do not use solutions at previous time steps to advance the solution to u(t+h). Unlike multi-steps methods, which take several steps to reach their maximum order, one-step methods have a fixed 4 of 19

6 order at each step. The most common one-step methods are the Runge-Kutta (RK) family of implicit and explicit algorithms. A s-stage RK method can be written as u n+1 = u n + s ( i=1 b ik i k i = hf u n + ) s j=1 α ijk j, where a ij and b i are the constant parameters that identify the algorithm. Several categories of RK methods exist, depending on the values of the parameters a ij. Explicit methods (ERK) are obtained when a ij is strictly lower-triangular (i.e., a ii = 0); Fully-implicit methods (FIRKs) are obtained when a ij has a full structure; and singly diagonally implicit methods (SDIRKs) are a special case of a ij lower triangular with a ii = γ (i.e., a constant diagonal). ERK methods are efficient for non-stiff problems, but are only conditionally stable and perform poorly for stiff problems. In stiff combustion applications, the step-size h is limited by stability instead of accuracy. Since they are fully explicit, ERK methods do not require the calculation of a Jacobian matrix (and the associated matrix-vector multiplications and inversions). This reduces the storage requirements and computational costs at each step considerably compared to implicit methods. We use the DOPRI5 ERK solver of Dormand and Prince. 6 DOPRI5 is a 7-stage, 5th-order embedded RK method. Embedded methods solve for both order p and p + 1 solutions simultaneously. Error control is based on the difference between the p and p + 1 solutions. The FIRK methods studied here are all L-stable and stiffly-accurate and are available in the FATODE 7 package. b We have evaluated the 3-stage, 5th-order Radua 1A (R1A) and 2A (R2A) algorithms and the 3- stage, 4th-order Lobatto-3C (L3C) algorithm. Like BDF methods, FIRKs require the solution of a non-linear system of equations, but the linear system size increases from N to sn with s RHS function evaluations per iteration. Embedded solutions are used for error control. SDIRK methods are a subclass of FIRKs with restrictions on the a ij coefficient matrix. Due to the structure of a ij, each stage is computed in sequence instead of solving the large sn non-linear system. Each stage is implicit and solved similarly to Eqn. 12. The Jacobian matrix may be reused for all stages, thereby reducing the cost. Like the FIRKs, we have chosen L-stable and stiffly-accurate SDIRK methods from the FATODE 7 package. We have tested two 2nd-order schemes and one 4th-order scheme. SDIRK-2a is a 2-stage, 2nd-order method; 3b is 2nd-order order with 3-stages; and 4b is a 5-stage, 4th-order method. The last multi-stage one-step methods considered are Rosenbrock (ROS) methods. ROS can be described as solving a linearized version of Eq. 13. This leads to the following s-stage ROS scheme 6 k i = ( hf u n + ) i 1 j=1 α ijk j + hj i j=1 γ ijk j u n+1 = u n + s i=1 b ik i, where α ij, γ ij, and b i are the unique method coefficients. ROS methods are usually designed so that α ij is lower triangular and each stage can be solved sequentially. As a result, ROS methods can be implemented similarly to SDIRK methods. However, ROS methods are a distinct class of methods with unique stability and efficiency properties. We have investigated three stiffly-accurate and L-stable ROS schemes available in the FATODE 7 package. ROS2, ROS3 and ROS4 are 2nd, 3rd and 4th-order ROS schemes with embedded schemes of order p 1 and p stages. The major distinction between ROS and the implicit RK methods lies in the role of the Jacobian matrix. J is used to converge the non-linear system via Eq. 12 in the fully-implicit schemes and is not part of the solution. Conversely, J appears explicitly in the ROS formula (Eq. 14). As a result, the J matrix must be computed and updated at each time-step in ROS methods. This requirement increases the computational cost of ROS methods if the calculation of the Jacobian is costly. Recall that J can be reused across multiple steps in fully implicit methods and is updated only when the convergence rate of the Newton-Raphson iterative solver deteriorates. There exist a class of methods (W-methods) whereby an inexact Jacobian is employed, rather than the exact J. These methods, proposed by Steinhaug and Wolfbrandt, 8 may be significantly more efficient than the Rosenbrock methods. However, they are beyond the scope of the present investigation and shall be considered at a later time. The non-iterative nature of the ROS methods has several advantages from a parallel processing point of view. Much work as been done lately on using GPUs for stiff chemical kinetics. Stone and Davis 9 and (13) (14) b asandu/software/fatode/index.html (April, 2013). 5 of 19

7 Niemeyer and Sung 10 have demonstrated dramatic run-time acceleration for non-stiff or mildly stiff ODEs using GPUs with ERK or stabilized Runge-Kutta methods. The authors ascribe the success of the methods to the ease of implementation in single-instruction/multiple-data (SIMD) environments. Unlike ERK methods, ROS methods are L-stable and can handle stiff ODEs. Since they do not require any iterative solution, they can be implemented efficiently in a SIMD environment much like ERK methods. To the best of our knowledge, the efficiency and stability of the ROS methods for realistic combustion applications has not been assessed. We have also implemented an implicit Trapezoidal (TRAP) solver for comparison. The scheme is shown below: u n+1 = u n + h 2 [f(u n) + f(u n+1 )]. (15) This method is 2nd-order accurate, L-stable, and fully-implicit. It is possible to embed an Euler method within the scheme; however, this would lead to 1st-order error control. Instead, we estimate the truncation error with Richardson extrapolation. 11 We first take one step of size h. If that converges, we take 2 equal steps of size h/2. The difference between the h and h/2 solutions gives a 4th-order estimation of the truncation error. Note that the extrapolated 4th-order solution (i.e., Simpson s rule) is not unconditionally stable, 6 so we use the Richardson extrapolation for error estimation and step-size control only. III.B.1. Extrapolation Methods The last class of solvers considered are extrapolation methods. Extrapolation methods combine low-order integration algorithms (e.g., modified-midpoint rule) with polynomial extrapolation (i.e., Romberg integration 11 ) to achieve arbitrarily high orders. As noted earlier, the extrapolated implicit trapezoidal method is not unconditionally stable and, therefore, is ill-suited for stiff ODE integration. However, the implicit Euler and, with some modifications, the implicit mid-point rule are A(α)-stable with extrapolation. We have tested the EulSim (semi-implicit Euler) and SIMPR (semi-implicit mid-point rule) algorithms from Deuflhard et al. 12 Extrapolated methods integrate Eq. 9 over a large step-size H using a sequence of smaller, uniform step-sizes h i := H/n i where n i is some pre-defined sequence of sub-steps (e.g., 1, 2, 4,..., 8, 16). The sequence of lower-order solutions are extrapolated to generate high-order ones. Like the BDF algorithms, both of these methods are h and p-adaptive. See Deuflhard 12 for an overview of the optimal strategy for the selection of h and p. Note that p never changes by more than 1 per step as in VODE. As a result, EulSim and SIMPR require several steps to reach high-order and this limits their efficiency for small integration time intervals. Conversely, they may prove to be very efficient and accurate for long integration intervals and often require far fewer time-steps compared to other variable and fixed-order methods. The ability to reuse the J matrix is an important feature of the EulSim and SIMPR algorithms. Note that these are semi-implicit algorithms, like the ROS methods, and the solutions rely explicitly on an up-todate J. The two extrapolation codes estimate the contractivity of the solution and reuse the J matrix over many sub-steps if the system is found to be locally linear. 12 The contractivity is cheaply computed from the sequence of fixed h i sub-steps within each global H time-step. This feature is not generally available in the ROS methods, but will be investigated in future studies. III.C. Implementations We use the TChem thermochemical library 4 to compute the RHS function, which includes the molar reaction rates and thermodynamic properties. All of the implicit and semi-implicit algorithms described above require the system Jacobian matrix J. The TChem library is used to compute the analytical Jacobians of the source vector in Eq. 9 for the constant-pressure system. In addition, we have implemented an algorithm to compute an approximation to J using numerical finite-differences as in the VODE solver. Columns of J are approximated as J j := f u j f(u + δ j) f(u) δ j, (16) 6 of 19

8 where δ j is a finite increment of the j th -component of u. That is, N RHS function evaluations are required for each J approximation. The column increment δ j is based on the current h, u, and f values, as well as the machine precision ϵ. We assume J is dense and ignore any sparsity. LAPACK routines are used for all matrix operations such as the LU factorization. The algorithm of Brown et al. 13 is used to estimate a suitable initial step-size h 0. This method attempts to find h 0 such that a 1st-order (Euler) step is accurate to the specified tolerance. This strategy is well suited for the p-adaptive methods (e.g., VODE, EulSim), since they start with only a 1st-order accurate solution, but is overly conservative for fixed, high-order, one-step methods. All methods have some manner of step-size control. The RK, ROS and TRAP algorithms adapt h with the following algorithm: h new = β ( e WRMS ) 1/p h, (17) where β is a safety factor ( 0.9), e is the estimated truncation error of u n+1,. WRMS is the root-meansquare weighted by the absolute and relative tolerances, and p is the order of the algorithm. IV. Results Two different model problems are considered to assess the performance of the various ODE integration methods. The first test problem is a 0-d ignition case. The second test problem is a 1-d laminar premixed flame propagation simulation. Both cases use the GRI-Mech 3.0 c, a widely used mechanism for methane combustion. The GRI mechanism has 53 species and 325 reactions. IV.A. Zero-dimensional ignition model problem We have simulated the ignition of an H 2 -air mixture at atmospheric pressure with initial H 2 /O 2 /N 2 mole fractions of 2/1/4, respectively, and an initial T = 1001 K. The time evolution for these conditions from t = 0 to 1 ms is shown in Fig. 1. Mass Fractions H 2 H O O 2 OH H 2 O HO 2 NO N 2 O HNO N Temperature (K) Time (ms) Figure 1. Selected species mass fractions and temperature during the ignition of an H 2 -air mixture with GRI-Mech 3.0. Initial H 2 /O 2 /N 2 mole concentrations are 2/1/4 at atmospheric pressure and 1001 K. We are interested in measuring the performance of the ODE solvers under conditions similar to those in a CFD solver with operator-splitting. The most notable difference between the ignition problem and a CFD setting is the size of the integration time interval. Instead of integrating the problem continuously from t = 0 to t = 1 ms, we have divided the problem into 1, 10,... 10,000 sub-intervals. This strategy mimics a CFD time-step δt ranging from 1 ms to 0.1 µs. Each sub-interval is treated as a new integration problem and the c Smith et al, mech. 7 of 19

9 ODE solver restarts from the new solution. Thus, all existing parametric (e.g., h and p) and history data (e.g., J, u(t jh)) are discarded. The benchmarks presented below were computed on a Linux workstation with 2 quad-core Intel Xeon E5606 CPUs operating at 2.13 GHz and with 8 MB of L2 cache. The workstation has 12 GB of DDR main memory and is running the Centos 5.4 Linux OS. All run-time measurements were averaged over a minimum of 5 runs. The ODE solvers, TChem library, and driver software were compiled using the Intel C/C++ and Fortran compilers (v ). We used full compiler optimization (i.e., -O3), which includes automatic loop unrolling and vectorization. All solvers used the same (scalar) relative and absolute tolerances of and 10 9, respectively. Figures 2 and 3 show the run-time in seconds for the various solvers using finite-difference and analytical Jacobians, respectively. The differences in run-times are quite significant, but diminish as the number of sub-intervals increases, i.e., as δt decreases. VODE performs best across all δt values with or without analytical Jacobians. The IRKs and Extrapolation methods are competitive with VODE when δt is large due to their high order (p 5). d However, both EulSim and SIMPR perform very poorly for small δt due to their high start-up costs. Only fixed, high-order methods (e.g.,, ROS3/4, and all the IRKs) come within a factor of 2-3 of the performance of VODE for small δt. In general, the higher-order methods outperform the low-order methods. The notable exceptions are the 2nd-order TRAP and 5th-order DOPRI solvers. The TRAP solver run-time is roughly 10x faster than any other low-order method (i.e., p 2). This is likely due to the more accurate error estimator based on Richardson extrapolation. As expected, the explicit and conditionally stable DOPRI5 solver is poorly suited for stiff integration over long time-scales. Surprisingly, the 2nd-order ROS method is slower than DOPRI5 and the 2nd-order SDIRK solvers are only 5x faster. It is likely that the efficiency of these two low-order methods is limited by the rather stringent error control tolerances used in this study. We observe that the performance of the ROS methods are impacted by the usage of analytical Jacobians. Recall that the ROS methods evaluate J at each time-step, while all other implicit methods reuse the Jacobian matrix. When using the finite-difference approximation to the Jacobian, the overall number of RHS function evaluations increases dramatically for the ROS methods, leading to poor performance. The number of time-steps and RHS function evaluations using analytical Jacobians are shown in Figures 4 and 5, respectively. Note that the ROS3 solver requires the fewest number of RHS function evaluations when using the analytical J for small δt, leading to a low overall run-time. The fully-implicit and extrapolation-based solvers require approximately one J evaluation per integration for δt 1 µs (i.e., they can reuse J for most internal time-steps). The ROS methods require at least twice the number of J evaluations. This leads to a large number of additional RHS functional evaluations when using finite-difference approximate Jacobians. As such, the ROS methods are competitive only if the analytical Jacobian is available. The run-time with VODE and the other high-order solvers increases considerably with small δt. This is due to various factors. Firstly, the solvers are forced to recompute the J matrix at least once per integration interval and secondly, they must start with a conservative h 0 approximation. For δt = 1 ms, VODE only required 24 Jacobian evaluations over 1247 internal steps. The number of Jacobian evaluations increases to 10,001 over 66,970 steps for δt = 0.1 µs. The drastic change in the number of steps is due to the slow rate at which h is allowed to change within VODE, e as well as the fact that the maximum h is limited to δt. VODE begins with p = 1, so that a conservative value for h 0 is appropriate. However, this feature affects the performance of the higher-order one-step (p 1) methods adversely. The rate of change in h as in Eq. 17 is restricted to 10x to avoid spurious overshoots. Therefore, an overly conservative and roughly 1st-order h 0 approximation requires several internal time-steps to reach an optimal h for the case of p 1. We can assume that values at a given grid-point will change only slightly due to transport (i.e., convection and diffusion) for time-accurate simulations. Therefore, the previous h used is a reasonable guess for the restarted integration. We have tested this hypothesis by reusing the final h value as the new h 0 value. We have imposed a ± 10% random variation on h 0 to mimic the impact of convective-diffusive effects on the new integration step. The resulting run-times, using analytical Jacobians, are shown in Figure 6 for the high-order algorithms. The VODE run-times from Figure 3 are also shown for comparison. The change is significant for all solvers and all are now faster than the baseline VODE for δt = 0.1 µs. The IRK solvers are also faster than VODE d The IRK-R1a solver fails for the δt = 1 ms case for unknown reasons. e VODE attempts p-adaption before h-adaption. 8 of 19

10 Run-time (sec) CVODE TRAP DOPRI5 EulSim SIMPR SDIRK-2a IRK-R1a Ros2 Ros3 Ros4 for δt 1 µs. IV.B. Figure External time-step (µs) Run-time (sec) for the ODE solvers using numerical (finite-difference) Jacobians. Premixed flame model problem We turn our attention to the performance of selected ODE solvers on a convection-diffusion-reaction CFD problem. We simulate a freely propagating 1-d laminar premixed flame using the laminarsmoke f CFD application. This application was used by Cuoci et al 14 to model laminar co-flow diffusion flames with detailed kinetics and transport properties. LaminarSMOKE is based on the open-source OpenFOAM CFD library and uses an implicit pressure-correction method to solve the momentum equation. Chemical reactions are modeling using an operator-splitting approach. Further details of the splitting algorithm and validation can be found in the original reference. 14 We modeled a stoichiometric methane-air premixed flame at atmospheric conditions on a uniform 1- d mesh. The GRI-Mech methane-air mechanism was used with the TChem kinetics library. The initial conditions were specified by interpolating the steady-state premixed flame profile generated by Cantera 15 onto our 3 cm uniform mesh. The laminar flame speed (S L ) computed by Cantera is 38.4 cm/s and the thermal flame thickness (δ f ), defined as (T b T u ) / dt/dx max, is approximately 0.5 mm. The profiles of temperature and selected species mass fractions across the flame are shown in Fig. 7. Note that the region occupied by the reaction zone of the flame constitutes only a negligible fraction of the simulation domain, a scenario often encountered in combustion simulations involving flame propagation. We simulated the premixed flame on three grid resolutions (100, 50, and 25 µm) in order to assess the robustness and accuracy of the laminarsmoke solver. The premixed flame was allowed to propagate for 10 ms and the flame-front displacement was measured by tracking the location of dt/dx max. The modeled S L was computed and compared to the Cantera solution. These three resolutions predicted S L within 1.3, 1.8 and 0.8% of the Cantera solution, indicating that the laminarsmoke application with TChem is capable of accurately simulating the unsteady propagation of a laminar premixed flame. The laminarsmoke benchmark tests were conducted on the Stampede cluster at the Texas Advanced Computing Center (TACC). The compute nodes have two 8-core Intel Xeon E CPUs operating at 2.7 GHz and with 20 MB of L2 cache. The nodes have 32 GB of DDR main memory. The ODE solvers, TChem library, OpenFOAM, and laminarsmoke software components were compiled using the Intel C/C++ and Fortran compilers (v13.1.0) with full optimization enabled (i.e., -O3 and vectorization). Note that, as before, all benchmarks were executed serially. ODE performance measurements were conducted on the 50 µm mesh (i.e., 600 points). The absolute and relative (scalar) tolerances for the ODE solvers for the integration of the chemical kinetics are f Available at 9 of 19

11 Run-time (sec) CVODE TRAP DOPRI5 EulSim SIMPR SDIRK-2a IRK-R1a Ros2 Ros3 Ros External time-step (µs) Figure 3. Run-time (sec) for the ODE solvers using analytical Jacobians. and 10 7, the default settings in laminarsmoke. ODE performance metrics such as the total chemistry integration time, the number of ODE solver steps, and the number of RHS function/jacobian evaluations were averaged over 100 time-steps. Only the 3rd-order and higher SDIRK, IRK and ROS solvers were assessed as these performed well for small δt in the 0-d ignition problem. The explicit DOPRI5 was discarded because it is ill-suited for all relevant time-step sizes. Similarly, the extrapolation methods, while efficient for long-duration integrations, perform poorly for small time-steps relevant to reactive CFD applications. The CFD time-step was varied between 0.5 to 4 µs. These values correspond to a maximum Courant numbers of 0.03 to 0.26, which is small for convection problems. It is important to note that the diffusion time-step limit (i.e., δt d = x 2 /max(d k )) is typically far more restrictive (especially for intermediate species such as atomic hydrogen, H) and forces us to use implicit or semi-implicit methods for the scalar transport. We use small δt values for time-accurate simulations even in low Mach conditions to limit operator splitting errors. Cuoci et al. 14 restricted the maximum Courant number to 0.1 for their laminarsmoke simulations. Based on our initial investigation regarding the impact of h 0 on the fixed-order methods and on the small CFD time-steps to be used, we have chosen to initialize h as the CFD integration time-step δt. This was found to be more efficient for the fixed-order methods. However, VODE benefits from using the h 0 recycling strategy discussed previously. The average chemistry integration wallclock time per-cell as a function of δt is shown in Figure 8 for the case of analytical Jacobians. We observe that the high-order methods (i.e., p 4) scale more efficiently with increasing δt. VODE, and increased in cost by less than a factor of 2x even though δt increased by 8x. The Rosenbrock and solvers scale less efficiently, with run-time increasing by more than 3x from δt = 0.5 to 4 µs. As seen previously, there is little performance difference between the lower and higher-order solvers for small δt, but the higher-order methods become more efficient as δt increases. Figure 9 shows the average integration times for the ODE solvers with approximated finite-difference Jacobians. As observed previously, analytical Jacobians have a substantial impact on computational cost. Numerical Jacobians increase the run-time cost most significantly for small δt since all solvers must compute at least one Jacobian matrix per CFD time-step. At δt = 0.5 µs, the run-time is approximately 2.6x higher with the numerical Jacobians for all the ODE solvers. This penalty decreases to less than 2x for δt = 4 µs for the fully implicit solves since they can reuse these costly Jacobians over more steps. The Rosenbrock solvers are the most strongly impacted since they must compute a new Jacobian each time-step. The Rosenbrock solvers are 2.9x slower with the numerical Jacobians and this difference is consistent from δt = 0.5 to 4 µs. As noted, there is less difference between the solvers at δt = 0.5 µs due to the short integration time. The difference between the fastest solver,, and slowest,, is only 25%. At δt = 4 µs, 10 of 19

12 10 6 Total number of ODE integration steps CVODE TRAP DOPRI5 EulSim SIMPR SDIRK-2a IRK-R1a Ros2 Ros3 Ros4 Figure External time-step (µs) Total number of solver time-steps for the ODE solvers using analytical Jacobians. this difference has increased to over 100%. Overall we observe that is the fastest solver for this application and is consistently faster than VODE by 10-17%. is competitive with VODE at larger δt and is faster than VODE for small δt. Figures show the average number of steps, RHS function evaluations and (analytical) Jacobian matrix evaluations during each CFD time-step. We observe that fixed high-order methods require only one step for small δt and less than four steps for the largest integration intervals. This is in sharp contrast to VODE, which needs roughly 5x more steps than the fixed-order solvers. Even though VODE requires many more steps, it requires only one function evaluation per step, which balances the cost. Note that the other fixed-order solvers require a function evaluation each stage, not just each step. The solver requires substantially more steps than any other fixed-order method and, as a result, requires many more RHS function evaluations. The large number of steps indicates that is inefficient for the current error tolerances and higher-order solvers are preferred. The number of Jacobian matrix evaluations for the Rosenbrock methods are significantly higher than any of the other iterative method that recycle their Jacobians over many steps. In fact, we observe that the iterative methods are able to recycle their Jacobians over nearly all steps. This explains the lower impact of the numerical Jacobians on the IRK and SDIRK methods for larger δt. That is, the added cost of the numerical Jacobian is amortized over a larger number of time steps. The performance measures reported above were averaged over all grid-points giving a per-cell cost basis. We shall now investigate the performance of the ODE solvers at a finer scale. The wallclock time, number of steps, number of function evaluations, and number of Jacobian evaluations for δt = 1 µs are shown in Figures at each grid point on the 50 µm mesh. The flame front, as shown in Fig. 7, is located at x = 10 mm. The cost for all solvers is seen to rise rapidly in the vicinity of the reaction zone. The wallclock time for the ROS3 and solvers increases by a factor of 50 and 60, respectively, over this narrow region. The higher-order methods experience a lower peak cost. The cost of all methods quickly decreases until x > 11 mm after which the cost reduces more slowly depending upon the order of the method. The high-order IRK and SDIRK methods decreases more quickly and the lower-order methods more slowly. The costs in the pre- and post-flame regions are nearly constant for all solvers. These non-reacting regions constitute a large portion of the CFD domain. Therefore, even a low per-cell cost in these regions can have a significant impact on the total chemistry time. In these regions, the lower-order ROS3 and methods are most efficient since, like all the fixed-order methods, they can integrate over δt = 1 µs in only 1 step and require fewer function evaluations. It is interesting to note that the Rosenbrock methods have the same pre and post-flame integration costs yet the SDIRK and IRK methods are more expensive in the postflame region. Even though these methods require only 1 time-step, they are taking more Newton-Raphson iterations. This is evident from the higher number of RHS function evaluations in the post-flame region. 11 of 19

13 10 7 Total number of RHS function evaluations Figure 5. Run-time (sec) CVODE TRAP DOPRI5 EulSim SIMPR SDIRK-2a IRK-R1a Ros2 Ros3 Ros External time-step (µs) Total number of function calls the ODE solvers using analytical Jacobians. CVODE TRAP IRK-R1a Ros3 Ros External time-step (µs) Figure 6. Run-time (sec) for select high-order ODE solvers using analytical Jacobians and reusing h of 19

14 Figure 7. Mass Fraction Average run-time (ms) per CFD time-step H2 O2 OH H2O CH4 CO CO2 N2 Temp X (mm) Profiles of major species and temperature through laminar premixed CH4-Air flame CVODE ROS3 ROS Temperature (K) CFD Time-step (µs) Figure 8. Average chemistry integration time per-cell (ms) for select stiff ODE solvers for using analytical Jacobians in laminar premixed flame test. 13 of 19

15 9 Average run-time (ms) per CFD time-step CVODE ROS3 ROS CFD Time-step (µs) Figure 9. Average chemistry integration time per-cell (ms) for select stiff ODE solvers for using numerical Jacobians in laminar premixed flame test. Average number of ODE steps per CFD time-step CVODE ROS3 ROS CFD Time-step (µs) Figure 10. Average number of ODE solver steps per-cell for select stiff ODE solvers for using analytical Jacobians in laminar premixed flame test. 14 of 19

16 Average number of function evaluations per CFD time-step CVODE ROS3 ROS CFD Time-step (µs) Figure 11. Average number of RHS function evaluations per-cell for select stiff ODE solvers for using analytical Jacobians in laminar premixed flame test. Average number of Jacobian evaluations per CFD time-step CVODE ROS3 ROS CFD Time-step (µs) Figure 12. Average number of Jacobian matrix evaluations per-cell for select stiff ODE solvers for using analytical Jacobians in laminar premixed flame test. 15 of 19

17 Wallclock time (ms) VODE ROS3 ROS4 Figure 13. Number of Steps X (mm) Wall-clock time (ms) at each cell for δt = 1 µs for laminar premixed flame simulation VODE ROS3 ROS X (mm) Figure 14. Number of time-steps at each cell for δt = 1 µs for laminar premixed flame simulation. 16 of 19

18 Number of Function Evaluations VODE ROS3 ROS4 Figure X (mm) Number of function evaluations at each cell for δt = 1 µs for laminar premixed flame simulation. Number of Jacobian Evaluations VODE ROS3 ROS4 Figure X (mm) Number of Jacobian evaluations at each cell for δt = 1 µs for laminar premixed flame simulation. 17 of 19

19 V. Conclusions We have presented two studies on chemical kinetic integration methods. The first study examined the performance of various stiff ODE solvers for a canonical chemical kinetics test case, the 0-d ignition problem. We measured the performance of the ODE solvers for the ignition of a H 2 -air atmospheric mixture with the GRI-Mech 3.0 mechanism. The VODE solver was shown to be the most efficient for large integration intervals when using either the finite-difference or analytical Jacobian. High-order methods are significantly faster than low-order ones, but the difference decreases rapidly as the CFD time-step δt is reduced. The fixed-order, one-step solvers (e.g., IRK and SDIRK) performed better than VODE for short intervals when the previous h was used to initialize h 0. This allowed the high-order solvers to take significantly larger time-steps over the short integration intervals. Analytical Jacobians were found to have the single most significant effect on the total run-time. Rosenbrock methods benefitted the most from analytical Jacobians since they must recompute the Jacobian matrix each time-step whereas the iterative solvers (e.g., VODE, IRK and SDIRK) can recycle previous Jacobian matrices. The second case studied was a convection-diffusion-reaction problem using operator splitting. The laminarsmoke CFD solver 14 was used to model a freely propagating laminar premixed methane-air flame at stoichiometric and atmospheric conditions. The performance of a selected set of one-step IRK, SDIRK and ROS methods were compared to the VODE solver for the integration of the chemical state of the mixture at each grid point. As found in the 0-d ignition problem, analytical Jacobians have a significant impact on the overall performance and improve the chemistry integration time by more than a factor of two. The solver was found to be faster than VODE over a range of practical CFD time-steps. The solver was also found to be roughly as efficient as VODE. The low-order SDIRK and Rosenbrock solvers ( and ROS3) were significantly slower than the other one-step methods: required twice the number of time-steps and ROS3 required 2-4x the number of Jacobian evaluations. The ability to recycle the Jacobian matrix is clearly an advantage for the iterative methods. Based on these studies, we have found that the high-order one-step IRK and SDIRK solvers are at least as efficient as VODE and should be considered for detailed chemical kinetics. The Rosenbrock class of solvers were also studied and found to be within 2-3x the cost of these other solvers. Rosenbrock solvers, while slower, have great potential in fine-grained parallel processing environments such as GPUs due to their similarity to explicit Runge-Kutta methods. Strategies for reusing Jacobian matrices should be investigated as well as the inexact Jacobian W-methods. These may significantly improve the ROS efficiency. We also investigated the spatial variations of the performance metrics in the computational domain. We observed a marked rise in computational cost across the narrow flame front. The cost with the fixed loworder methods (ROS3 and ) was lower in the non-reacting pre- and post-flame zones but much higher in the narrow combustion region. This suggests the possibility of using solvers of different orders in different regions of the domain depending on the local conditions. Acknowledgements This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number OCI Computations were executed, in part, on the Stampede supercomputering system at the Texas Advanced Computing Center (TACC), which is funded by NSF award OCI References 1 Radhakrishnan, K., Comparison of Numerical Techniques for Integration of Stiff Ordinary Differential Equations Arising in Combustion Chemistry, NASA Technical Paper 2372, October Singer, M., Pope, S., and Najm, H., Operator-splitting with ISAT to Model Reacting Flow with Detailed Chemistry, Combustion Theory and Modelling, Vol. 10, No. 2, 2006, pp Hindmarsh, A., Brown, P., Grant, K., Serban, R., Shumaker, D., and Woodward, C., SUNDIALS: Suite of Nonlinear and Differential/Algebraic Equation Solvers, ACM Transactions on Mathematical Software, Vol. 31, No. 3, 2005, pp Safta, C., Najm, H., and Knio, O., TChem - A Software Toolkit for the Analysis of Complex Kinetic Models, Report SAND , May McBride, B., Gordon, S., and Reno, M., Coefficients for Calculating Thermodynamic and Transport Properties of Individual Species, TM 4513, NASA, October of 19

20 6 Hairer, E. and Wanner, G., Solving Ordinary Differential Equations II: Stiff and Differential-Algebrain Problems, Springer, 2nd ed., Zhang, H. and Sandu, A., FATODE: A Library for Forward, Adjoint, and Tangent Linear Integration of Stiff Systems, High Performance Computing Symposium at SpringSim 2011, Steihaug, T. and Wolfbrandt, A., An Attempt to Avoid Exact Jacobians and Nonlinear Equations in the Numerical Solution of Stiff differential Equations, Math. Comp., Vol. 33, 1979, pp Stone, C. and Davis, R., Techniques for Solving Stiff Chemical Kinetics on Graphical Processing Units, J. of Propulsion and Power, Vol. 29, No. 4, 2013, pp Niemeyer, K. E. and Sung, C.-J., Accelerating moderately stiff chemical kinetics in reactive-flow simulations using GPUs, J. Computational Physics, Vol. 256, 2014, pp Press, W., Teukolsky, S., Vetterling, W., and Flannery, B., Numerical Recipes in Fortran77: The Art of Scientific Computing, 2nd ed., Cambridge University Press, Deuflhard, P., Recent Progress in Extrapolation Methods for Ordinary Differential Equations, SIAM Review, Vol. 27, No. 4, December Brown, P., Bryne, G., and Hindmarsh, A., VODE: A Variable-Coefficient ODE Solver, SIAM Journal on Scientific and Statistical Computing, Vol. 10, No. 5, 1989, pp Cuoci, A., Frassoldati, A., Faravelli, T., and Ranzi, E., A computational tool for the detailed kinetic modeling of laminar flames: Application to C2H4/CH4 coflow flames, Combustion and Flame, Vol. 160, 2013, pp code.google.com/p/cantera. 19 of 19