Chapter 22 Examples of Computation of Exact Moment Dynamics for Chemical Reaction Networks

Chapter 22 Examples of Computation of Exact Moment Dynamics for Chemical Reaction Networks Eduardo D. Sontag Abstract The study of stochastic biomolecular networks is a key part of systems biology, as such networks play a central role in engineered synthetic biology constructs as well as in naturally occurring cells. This expository paper reviews in a unified way a pair of recent approaches to the finite computation of statistics for chemical reaction networks. 22. Introduction The study of biochemical networks is of great interest not only for the understanding of natural biological systems, but also in the engineering design of biological control systems, and specifically in the field of synthetic biology. Chemical systems are inherently stochastic, as reactions depend on thermally induced random effects. For large systems, deterministic mean-field models are appropriate, but such models cannot account for random fluctuations, and stochastic models, and specifically the Chemical Master Equation CME, a discrete-space continuous-time Markov process that describes stochastic chemical kinetics, are required for a more accurate description. Tools from dynamical systems and from control theory play key roles in the analysis of the CME. The CME is typically an infinite-dimensional linear differential equation, and even its steady-state solutions are very difficult to compute in closed form. Various techniques, typically moment closure tools based on the mass fluctuation kinetics and fluctuation-dissipation ideas are used to approximate solutions or moments [5, 0,, 4]. In this expository paper, we first introduce the setup, and then review in a unified way results for two types of stochastic chemical reaction systems for which moments can be effectively computed: feedforward networks FFN, treated in [2], and complex balanced networks CBN, treated in [3], and provide several worked examples. E. D. Sontag B Northeastern University, Boston, MA, USA e-mail: eduardo.sontag@gmail.com Springer International Publishing AG, part of Springer Nature 208 R. Tempo et al. eds., Emerging Applications of Control and Systems Theory, Lecture Notes in Control and Information Sciences - Proceedings, https://doi.org/0.007/978-3-39-67068-3_22 295

296 E. D. Sontag 22.2 Preliminaries We start by reviewing standard concepts regarding master equations for biochemical networks, see for instance []. Chemical Reaction Networks. Chemical reaction networks involve interactions among a finite set of species S {S i, i, 2,...n} where one thinks of the S i s as counting the numbers of molecules of a certain type or individuals in an ecological model, or cells in a cell population model: S i t k i number of units of species i at time t. In stochastic models, one thinks of these as random variables, which interact with each other. The complete vector S S,..., S n is called the state of the system at time t, and it is probabilistically described as a Markov stochastic process which is indexed by time t 0 and takes values in Z n 0. Thus, St is a Z n 0-valued random variable, for each t 0. Abusing notation, we also write St to represent an outcome of this random variable on a realization of the process. We will denote p k t P [St k] for each k Z n 0. Then pt p k k Z n 0 is the discrete probability density also called the probability mass function of St. To describe the Markov process, one needs to formally introduce chemical reaction networks. A chemical reaction network is a finite set R {R j, j, 2,..., m} of formal transformations or reactions R j : n a i j S i i n b i j S i, j {, 2,..., m} 22. i among species, together with a set of m functions ρ j : Z n 0 R 0, j,..., m, with ρ j 0 0, the propensity functions for the respective reactions R j. The coefficients a i j and b i j are nonnegative integers, the stoichiometry coefficients, and the sums are understood informally, indicating combinations of elements. The intuitive interpretation is that ρ j S,..., S n dt is the probability that reaction R j takes place, in a short interval of length dt, provided that the complete state was S S,..., S n at the beginning of the interval. In principle, the propensities can be quite arbitrary functions, but we will focus on mass-action kinetics, for which the functions ρ j are polynomials whose degree is the sum of the a i j s in the respective reaction. Before discussing propensities, we introduce some more notations and terminology. The linear combinations n i a i j S i and n i b i j S i appearing in the m reactions are the complexes involved in the reactions. For each reaction R j, we collect the coefficients appearing on its left-hand side and on its right-hand side into two vectors, respectively: SR j a j : a j,..., a nj and TR j b j : b j,..., b nj prime indicates transpose. We call S, T : R C the source and target functions, where C Z n 0 is the set of all vectors { a j, b j, j...m }. We identify complexes with elements of C. The reactants S i of the reaction R j are those species appearing with a nonzero coefficient, a i j 0 in its left-hand side and the products S i of reaction R j are those species appearing with a nonzero coefficient b i j 0 in its right-hand side.

22 Examples of Computation of Exact Moment Dynamics 297 For every vector of nonnegative integers v v,..., v n Z n 0, let us write the sum of its entries as v : v + + v n. In particular, for each j {,..., m}, we define the order of the reaction R j as a j : n i a i j, which is the total number of units of all species participating in the reaction R j. The n m stoichiometry matrix Γ {γ i j } is defined as the matrix whose entries are defined as follows: γ i j : b i j a i j, i,..., n, j,..., m. The integer γ i j counts the net change positive or negative in the number of units of species S i each time that the reaction R j takes place. We will denote by γ j the jth column of Γ. With these notations, γ j b j a j, j,..., m. We will assume that γ j 0 for all j each reaction changes at least some species. For example, suppose that n 4, m 2, and the reactions are R : S + S 2 S 3 + S 4, R 2 : 2S + S 3 S 2 which have orders + 2 and 2 + 3, respectively. The set C has four elements, which list the coefficients of the species participating in the reactions:c {,, 0, 0,0, 0,,,2, 0,, 0,0,, 0, 0 } with SR a,, 0, 0, SR 2 a 2 2, 0,, 0, TR b 0, 0,,, TR b 2 0,, 0, 0 andγ,,,,γ 2 2,,, 0. The reactants of R are S and S 2, the reactants of R 2 are S and S 3, the products of R are S 3 and S 4, the only product of R 2 is S 2, and the stoichiometry matrix is using MATLAB-like notation, listing row by row: Γ [, 2;, ;, ;, 0]. It is sometimes convenient to write n i a ρ j S i j S i n i b i j S i to show that the propensity ρ j is associated to the reaction j, and to combine two reactions R j and R k that are the reverse of each other complexes are transposed: SR j TR k and SR k TR j, using double arrows: n i a ρ j S i j S i n i b i j S i. When propensities are given by mass-action kinetics, as discussed below, one simply writes on the arrows the kinetic constants instead of the full form of the kinetics. Chemical Master Equation. A Chemical Master Equation CME, which is the differential form of the Chapman Kolmogorov forward equation, is a system of linear differential equations that describes the time evolution of the joint probability distribution of the S i t s: ρ k S dp k dt m ρ j k γ j p k γ j m ρ j k p k, k Z n 0 22.2 where, for notational simplicity, we omitted the time argument t from p, and the function ρ j has the property that ρ j k γ j 0 unless k γ j coordinatewise inequality. There is one equation for each k Z n 0, so this is an infinite system of linked equations. When discussing the CME, we will assume that an initial probability vector p0 has been specified, and that there is a unique solution of 22.2 defined for all t 0. See [9] for existence and uniqueness results. A different CME results for each choice of propensity functions, a choice that is dictated by physical chemistry considerations. The most commonly used propensity functions, and the

298 E. D. Sontag ones best-justified from elementary physical principles, are ideal mass-action kinetics propensities, defined as follows see [4], proportional to the number of ways in which species can combine to form the jth source complex: ρ j k κ j n i ki a i j H k a j j,..., m. 22.3 where, for any scalar or vector, we denote H u if u 0 coordinatewise and H u 0 otherwise. In other words, the expression can only be nonzero provided that k i a i j for all i,..., n and thus the combinatorial coefficients are welldefined. Observe that the expression in the right-hand side makes sense even if k 0, in the following sense. In that case, k i < 0 for some index i, so the factorial is not welldefined, but on the other hand, k i a i j k i < 0 implies that H k a j 0. So ρ j k can be thought of as defined by this formula for all k Z n, even if some entries of k are negative, but is zero unless k 0, and the combinatorial coefficients can be arbitrarily defined for k 0. In particular, ρ j k γ j 0 unless k γ j in 22.2. The m nonnegative kinetic constants are arbitrary, and they represent quantities related to the volume, shapes of the reactants, chemical, and physical information, and temperature. The model described here assumes that temperature and volume are constant, and that the system is well-mixed no spatial heterogeneity. Derivatives of Moments Expressed as Linear Combinations of Moments. Notice that ρ j k can be expanded into a polynomial in which each variable k i has an exponent less or equal to a i j. In other words, ρ j k c j a j κ c j k c j is understood coordinatewise, and by definition k c j k c j...k c nj n and r 0 for all integers, for suitably redefined coefficients κ c j s. Suppose given a function M : Z n 0 R to be taken as a monomial when computing moments. The expectation of the random variable MS is by definition E [MSt] k Z p kt Mk, since n 0 p k t P [St k]. Let us define, for any γ Z n, the new function γ M given by γ Mk : Mk + γ Mk. With these notations, d m dt E [MSt] E [ ρ j St γ j MSt ] 22.4 see [] for more details. We next specialize to a monomial function: Mk k u k u ku 2 2...ku n n where u Zn 0. There results γ j Mk ν I u, j d νk ν for appropriate coefficients d ν, where { } I u, j : ν Z n ν u µ, u µ 0 0 µ i 0 for each i such that γ i j 0

22 Examples of Computation of Exact Moment Dynamics 299 inequalities in Z n 0 are understood coordinatewise. Thus, for 22.3: d dt E[ St u] m c j a j ν I u, j d ν κ c j E [ St ν+c j ]. 22.5 In other words, we can recursively express the derivative of the moment of order u as a linear combination of other moments. This results in an infinite set of coupled linear ordinary differential equations, so it is natural to ask whether, for given a particular moment or order u of interest, there is a finite set of moments, including the desired one, that satisfies a finite set of differential equations. This question can be reformulated combinatorially, as follows. For each multi-index u Z n 0, let us define R 0 u {u}, R u : { ν + c j, j m, c j a j, ν I u, j }, and, more generally, for any l, R l+ u : R R l u where, for any set U, R l U : u U Rl u. Finally, we set Ru : i0 Ri u. Each set R l u is finite, but the cardinality #Ru may be infinite. It is finite if and only if there is some L 0 such that Ru L i0 Ri u, or equivalently R L+ u L i0 Ri u. Equation 22.5 says that the derivative of the u-th moment can be expressed as a linear combination of the moments in the set R u. The derivatives of these moments, in turn, can be expressed in terms of the moments in the set R u, for each u R u, i.e. in terms of moments in the set R 2 u. Iterating, we have the following: Finite reachability implies linear moment closure observation: Lemma. Suppose N : #Ru <, and Ru {u u,..., u N }. Then, with xt : E [S u t],...,e[s u N t], there is an A R N N so that ẋt Axt, t 0. A classical case is when all reactions have order 0 or, i.e., a j {0, }. Since µ 0 in the definition of I u, j, it follows that a j µ for every index j. Therefore, ν + a j u + a j µ u for all u, and the same holds for ν + c j if c j a j. So all elements in Ru have degree u, and thus #Ru <. A more general case is as follows. 22.3 Feedforward Networks A chemical network is of feedforward type FFN if one can partition its n species S i, i {, 2,..., n} into p layers S,..., S p and there are a total of m m + d reactions, where d of the reactions are pure degradation or dilution reactions D j : S i j 0, j {,..., d} and the additional m reactions R j, j {, 2,..., m} can be partitioned into p layers R,..., R p in such a manner that, in the each reaction layer R π there may be any number of order-zero or order-one reactions involving species in layer π, but every higher order reaction at a layer π > must have the form: a i j S i + a iq j S iq a i j S i + a iq j S iq + b iq+ j S iq+ + b iq+q j S iq+q, where all the species S i,..., S iq belong to layers having indices < π, and the species S iq+,..., S iq+q are in layer π. In other words, multimers of species in previous

300 E. D. Sontag layers can catalyze the production of species in the given layer, but are not affected by these reactions. This can be summarized by saying that for reactions at any given layer π, the only species that appear as reactants in nonlinear reactions are those in layers < π and the only ones that can change are those in layer π. A more formal way to state the requirements is as follows. The reactions R j that belong to the first layer R are all of order-zero or one, i.e. they have a j {0, } this first layer might model several independent separate chemical subnetworks; we collect them all as one larger network, and if R j R π : { ai j 0 and a j > S i s<π S π γ i j 0 S i S π. 22.6 FFN s have the finite reachability property [2]: given any desired moment u, there is a linear differential equation ẋt Axt for a suitable set of N moments xt : E [S u t],..., E [S u N t], which contains the moment u of interest. Notice that steady-state moments can then be computed by solving Ax 0. The proof uses a Lyapunov-like construction. In practice, we simply compute 22.5 starting from the desired moment, then recursively apply the same rule to the moments appearing on the right-hand side, and so forth until no new moments appear. The integer N at which the system closes might be very large, but the procedure is guaranteed to stop. The last section of the paper [2] explains how certain non-feedforward networks also lead to moment closure, provided that conservation laws ensure that variables appearing in nonlinear reactions take only a finite set of possible values. Steady States of CME. Often, the interest is in long-time behavior, after a transient, that is to say in the probabilistic steady state of the system: the joint distribution of the random variables S i S i that result in the limit as t provided that such a limit exists in an appropriate technical sense. This joint distribution is a solution of the steady-state CME sscme, the infinite set of linear equations obtained by setting the right-hand side of the CME to zero, that is: m ρ j k γ j p k γ j m ρ j k p k, k Z n 0 22.7 with the convention that ρ j k γ j 0 unless k γ j. When substituting massaction propensitiesρ j k κ n ki j i a i j H k a j the steady-state equation 22.7 becomes: m κ j n ki γ i j H k b j p k γ j i a i j m κ j n i ki a i j H k a j p k 22.8

22 Examples of Computation of Exact Moment Dynamics 30 for all k Z n 0. Equivalently, for all k Zn 0 : m n i ki γ i j! ki b i j! H k b j p k γ j m n i k i! ki a i j! H k a j p k 22.9 when introducing new constants : κ j / n i ai j!. Writing λ k : λ k... λk n n and k! : k!...k n! for each k k,..., k n Z n 0 and λ λ,...,λ n R n >0, 22.9 is: m k γ j! k b j! H k b j p k γ j m k! k a j! H k a j p k, k Z n 0 22.0 Since 22.0 is a linear equation on the { p k, k Z n 0}, any rescaling pk s will satisfy the same equation; for probability densities, one normalizes to a unit sum. If there are conservation laws satisfied by the system then steady-state solutions will not be unique, and the equation Ax 0 must be supplemented by a set of linear constraints that uniquely specify the solution. For example, consider a reversible reaction S κ κ2 S 2 propensities are mass-action, ρ i S, S 2 κ i S i. The first moments means satisfy ẋ κ 2 x 2 κ x and ẋ 2 κ x κ 2 x 2. Any vector ξ, ξ 2 with κ ξ κ 2 ξ 2 is a steady state of these equations. However, the sum of the numbers of molecules S and S 2 is conserved in the reactions. Given a particular total number, β, the differential equations can be reduced to just one equation, say for x : ẋ κ 2 β x κ x κ + κ 2 x + κ 2 β, which has the affine form ẋ Ax + b. At steady state, we have the unique solution ξ βκ 2 /κ + κ 2, ξ 2 βκ /κ + κ 2 obtained by imposing the constraint ξ + ξ 2 β. It can easily be proved see e.g. [3] that at steady state, S is a binomial random variable Bβ, p with p +µ, where µ κ /κ 2. We later discuss further conservation laws. A Worked Example. For networks with only zero and first-order reactions, which are feedforward, it is well known that one may compute all moments in closed form. For example, start with a reversible reaction S κ δ S 2 with mass-action propensities, thinking of S as the active form of a certain gene and S 2 as the inactive form of this gene. Transcription and translation are summarized, for simplicity, as one reaction ρ S S + S 3 and degradation or dilution of the gene product S 3 is a linear reaction η S 3. The stoichiometry matrix is Γ [,, 0, 0;,, 0, 0; 0, 0,, ]. Suppose, we are interested in the mean and variance of S 3 subject to the conservation law S + S 2 β, for some fixed positive integer β. A linear differential equation for these second-order moments: M E[S ], E[S 2], E[S S 3 ], E[S 3 ], E[S3 2] is M AM + b, where A [ δ κ, 0, 0, 0, 0;κ δ + 2 δ β, 2 δ 2 κ, 0, 0, 0; 0,ρ, δ η κ,δ β, 0;ρ, 0, 0, η, 0;ρ, 0, 2 ρ,η, 2 η] and b [δ β;δ β; 0; 0; 0]. One can then solve AM + b 0 to obtain steady state moments.

302 E. D. Sontag A Simple Nonlinear Example. We consider a feedforward system with three species; S catalyzes production S 2, and S and S 2 are both needed to produce S 3 : 0 κ δ S κ 2 δ 2 κ 3 δ 3 0, S S + S 2, S 2 0, S + S 2 S + S 2 + S 3, S 3 0. Computing E[S3 ], the mean of S 3, requires a minimal differential equation of order 5, for the momentsm E[S 3 ], E[S S 2 ], E[S 2 ], E[S 2], E[S ] and has form M AM + b, where A [ δ 2,κ 3, 0, 0, 0; 0, δ δ 2,κ,κ 2, 0; 0, 0, δ 2, 0,κ 2 ; 0, 0, 0, 2 δ, 2 κ + δ ; 0, 0, 0, 0, δ ] and b [0; 0; 0;κ ;κ ], 22.4 Poisson-Like Solutions and Complex Balanced Networks We observe that for any given positive vector λ R n >0, the set of numbers Π { p k λ k /k!, k Z n } 0 22. satisfies the sscme equations 22.0 if and only if m Rewriting this as: λ k γ j k b j! H k b j m λ k k a j! H k a j, k Z n 0, 22.2 c C { j b j c} λ k γ j k b j! H k b j c C { j a j c} a sufficient condition for 22. to be a solution is that λ k k c! H k a j, k Z n 0, 22.3 { j b j c} λ k γ j k c! H k b j { j a j c} λ k k c! H k a j, k Z n 0 for each individual complex c C, or, equivalently, H k c k c! { j b j c} λ k γ j H k c k c! { j a j c} λ k, k Z n 0. A sufficient condition for this to hold is that, for all complexes: λ a j λ a j, k Z n 0 22.4 { j b j c} { j a j c}

22 Examples of Computation of Exact Moment Dynamics 303 Fig. 22. Complex balancing: outflows and inflows must balance at each complex c. The left-hand side of 22.4 is κ 3 λ a3 + κ 4 λ a4 and the right-hand side is κ λ a + κ 2 λ a2 conversely, this last condition is necessary for all complexes for which k c. One can write λ c and bring this term outside of the sum, in the right-hand side. When property 22.4 holds for every complex, one says that λ is a complex balanced steady state of the associated deterministic chemical reaction network. That is, the system of differential equations ẋ Γ Qx, where Qx is a column vector of size m whose jth entry isρ j x and xt R n 0 for all t. Complex balancing means that, for each complex, outflows and inflows balance out. This is a Kirschoff current law in-flux out-flux, at each node. See Fig. 22.. Foundational results in deterministic chemical network theory were obtained by Horn, Jackson, and Feinberg [2, 3]. One of the key theorems is that a sufficient condition for the existence of a complex balanced steady state is that the network be weakly reversible and have deficiency zero. The deficiency is computed as n c l r, where n c is the number of complexes, r is the rank of the matrix Γ, and l is the number of linkage classes connected components of the reaction graph. Weak reversibility means that each connected component of the reaction graph must be strongly connected. One of the most interesting features of this theorem is that no assumptions need to be made about the kinetic constants. Of course, the choice of the vector λ will depend on the kinetic constants. We refer the reader to the citations for details on deficiency theory, as well as, of interest in the present context, several examples discussed in [3]. The theorems for weakly reversible deficiency zero networks are actually far stronger, and they show that every possible steady state of the corresponding deterministic network is complex balanced, and that they are asymptotically stable relative to stoichiometry classes. The connection with sscme solutions was a beautiful observation made in [], but can be traced to the nonlinear traffic equations from queuing theory, described in Kelly s textbook [7], Chap. 8 see also [8] for a discussion, The elements of Π given by formula 22. add up to: k Z n 0 p k... k 0 k n 0 λ k k!... λ k n k n! Z : e λ...e λ n Thus, normalizing by the total, {p k /Z, k Z n 0 } is a probability distribution. However, because of stoichiometric constraints, solutions are typically not unique, and general solutions appear as convex combinations of solutions corresponding to invariant subsets of states. A solution with only a finite number of nonzero p k s will then have a different normalization factor Z.

304 E. D. Sontag Conservation Laws, Complex Balanced Case. When steady states do not form an irreducible Markov chain, the solutions of the form 22. are not the only solutions in the complex balanced case. Restrictions to each component of the Markov chain are also solutions, as are convex combinations of such restrictions. To formalize this idea, suppose that there is some subset Z 0 Z n with the following stoichiometric invariance property: k Z 0 k ± γ j Z 0 for all j,..., m. The same property is then true for the complement of Z 0. Consider, the set Z : Z 0 Z n 0. For each k Z, the left-hand side term in Eq. 22.2 either involves an index k γ j > 0, and hence, in Z, or it is zero because k b j 0 implies k γ j 0 and so it does not matter that k γ j / Z. Thus, p k λ k k! if k Z, 0 if k Zn 0 \ Z 22.5 is also a solution, in the complex balanced case observe that, for indices in Z n 0 \ Z, Eq. 22.2 is trivially satisfied, since both sides vanish. So we need to divide by the sum Z of the elements in 22.5 in order to normalize to a probability distribution. The restriction to Z will the unique steady-state distribution provided that the restricted Markov chain has appropriate irreducibility properties. In particular, suppose that the nullspace of A α i j R m n includes C for example, A could be the orthogonal complement of the stoichiometric subspace spanned by C, and pick any vector β β,...,β q R q. Then Z 0 {k A k β} has the invariance property, and the sum of the elements in 22.5 is: Zβ,..., β q k,...,kn 0 A kβ λ k k! λ k 2 2 k 2!... λkn n k n! zero if sum empty. The normalized form of 22.5 has p k 0 for k Z n 0 \ Z, and λ k λ k 2 2 p k Zβ,..., β q k! k 2!... λkn n 22.6 k n! for k Z. A probabilistic interpretation is as follows. Suppose given n independent Poisson random variables, S i, i,..., n, with parameters λ i respectively, so P [S k, S 2 k 2,..., S n k n ] e λ +...+λ n λk k! λ k 2 2 k 2!... λkn n k n! 22.7 for k 0 and zero otherwise. Let us introduce the following new random variables: Y j : n i α ji S i, j,..., q. Observe that P [ ] Y β,..., Y m β q equals k,...,kn 0 α k +...+α n knβ,..., α q k +...+αqn knβq P [S k, S 2 k 2,..., S n k n ]

22 Examples of Computation of Exact Moment Dynamics 305 which is e λ +...+λ n Zβ,..., β q. Therefore, for each k Z, p k in 22.6 equals the conditional probability P [S k,s 2 k 2,...,S n k n ], which is the same as P[Y β,...,y q β q] P [ S k, S 2 k 2,..., S n k n Y β,..., Y q β q ]. If our interest is in computing this conditional probability, the main effort goes into computing Zβ,...,β q. The main contribution of the paper [3] was to provide effective algorithms for the computation of Zβ,...,β q recursively on the β i s. A package for that purpose, called MVPoisson, was included with that paper. Conditional moments E[S r j Y β,..., Y m β q ], r, including the conditional expectation when r, as well as centered moments such as the conditional variance, can be computed once that these conditional probabilities are known. It is convenient for that purpose to first compute the factorial moments. Recall that, the rth factorial moment E[W r ] of a random variable W is defined as the expectation of W!/W r!. For example, when r, E[W r ] E[W], and for r 2, E[W r ] E[W 2 ] E[W], and thus, the mean and variance of W can be obtained from these. We denote the conditional factorial moment of S i given Y β, as E[S r j Y]. It is not difficult to see Theorem 2 in [3] that: E[S r j Y] λ r j Zβ rα j,β 2 rα 2 j,...,β q rα q j Zβ,...,β q when all β i rα i j 0 and zero otherwise. The paper [3] discusses mixed moments such as covariances too. For example, for r we have the conditional mean: E[S j Y] λ j Zβ α j,β 2 α 2 j,...,β q α q j Zβ,...,β q 22.8 when all β i α i j, and zero otherwise, and for r 2 the conditional second moment: E[S 2 j Y] λ 2 j Zβ 2α j,β 2 2α 2 j,..., β q 2α q j Zβ,...,β q + E[S j Y] when all β i 2α i j, and zero otherwise. We next work out a concrete example. Worked Example: Simple Binding. Suppose that two molecules of species S and S 2 can reversibly combine through a bimolecular reaction to produce a molecule of species S 3 : S + S 2 κ κ2 S 3. Since the deficiency of this network is n c l r 2 0 and it is reversible and hence weakly reversible as well, we know that there is a complex balanced equilibrium and every equilibrium is complex balanced. We may pick, for example, λ,, K, where K : κ /κ 2. The count of S molecules goes down by one every time that a reaction takes place, at which time the count of S 3 molecules goes up by one. Thus, the sum of the number of S molecules plus the number of S 3 molecules remains constant in time, equal to

306 E. D. Sontag their starting value, which we denote as p. Similarly, the sum of the number of S 2 molecules plus the number of S 3 molecules remains constant, equal to some number n. In the general notations, we have a a 3, a 22 a 23, a 2 a 2 0, β p,β 2 n. In the steady-state limit as t, these constraints persist. In other words, all p k should vanish except those corresponding to vectors k k, k 2, k 3 such that k + k 3 p and k 2 + k 3 n. The set consisting of all such vectors is invariant, so λ k λ k 2 2 λ k 3 3 p k k! k 2! k 3! if k + k 3 p and k 2 + k 3 n 0 otherwise is a solution of the sscme. In order to obtain a probability density, we must normalize by the sum Zp, n of these p k s. Because of the two constraints, the sum can be expressed in terms of just one of the indices, let us say k. Observe that, since k + k 3 p and k 3 0, necessarily k p. Since k 2 n k 3 n + k p must be nonnegative, we also have the constraint k max{0, p n}. So the only nonzero terms are for k {max{0, p n},..., p}. With k 3 p k, k 2 n k 3 n + k p, we have: Zp, n p lmax{0,p n} K p l l! n + l p! p l! min{p,n} l0 K l p l! n l! l! 22.9 The second form if the summation makes it obvious that Zp, n Zn, p. When n p, we can also write Zp, n n!p! p l0 n! p K p l 22.20 n p + l! l which shows the expression as a rational function in which the numerator is a polynomial of degree p on n. This was derived assuming that n p, and the factorials in the denominator do not make sense otherwise. However, let us think of each term n! as the product nn... n p + l +, which may include zero as well n p+l! as negative numbers. With this understanding, the formula in 22.20 makes sense even when n < p. Observe that such a term vanishes for any index l < p n. Thus, for n < p, 22.20 reduces to: p p p! lp n n p+l! l K p l or equivalently, with a change of indices l p l and then using p p l p l : p! n l0 p K l n l! p l p! n l0 p K l n l! l n l0 K l n l!p l!l!. In this last form, we have the same expression as the last one in 22.9. In conclusion, provided that we interpret the quotient of combinatorial numbers in 22.20

22 Examples of Computation of Exact Moment Dynamics 307 as a product that may be zero, formula 22.20 is valid for all n and p, not just for n p. In particular, we have; Z0, n n!, Z, n K n +, n! Z2, n 2n! K 2 n 2 + K 2 + 2 K n +, etc. In terms of the Gauss s hypergeometric function 2 F 0, we can also write: Zp, n p! n! 2 F 0 n, p; ; K.The recursion on n obtained by using the package MVPoisson from [3] is as follows by symmetry, a recursion on p can be found by exchanging n and p: Zp, n + 2 K K n + K p K + Zp, n + Zp, n +. n + 2 n + 2 Now 22.8 gives the conditional mean of the first species, S j for this index, r for the first moment, and λ as zero if p < or n < 0 and otherwise ϕp, n : E[S S + S 3 p, S 2 + S 3 n] Zp, n Zp, n. For example, ϕ, n 2K, ϕ2, n K n+ n+ K 2 n 2 + K 2 +2 Kn+. Worked Example: Synthesis and Degradation, and Binding. Suppose molecules of species S can be randomly created and degraded, and they can also reversibly combine with molecules of S 2 through a bimolecular reaction to produce molecules of species S 3 : κ κ 2 κ 3 S, S + S 2 S 3. There are n c 4 complexes:, S, S + κ4 S 2, and S 3, and l 2 linkage classes. The stoichiometry matrix Γ [,,, ; 0, 0,, ; 0, 0,, ] has rank r 2, so the deficiency of this weakly reversible network is n c l r 4 2 2 0. Thus, there is a complex balanced equilibrium and every equilibrium is complex balanced. We may pick, for example, λ λ,,µ, where λ : κ κ 2 and µ : κ κ 3 κ 2 κ 4. Notice that, there is only one nontrivial conserved quantity, S 2 + S 3 n, since S is not conserved. We have: Zn k,k 2,k 3 0 k 2 +k 3 n λ k k! λ k 2 2 k 2! λ k 3 3 k 3! k 0 λ k k! n k 2 0 µ n k 2 k 2!n k 2! eλ n! + µn. The normalized probability 22.5, for k k, k 2, k 3 0 with k 2 + k 3 n, is: p k λ k µ k 3 n! λ Zn k! k 2! k 3 k µ k 3! e λ +µ n k!k 2!k 3 and as discussed earlier, this is the conditional probability P [! S k, S 2 k 2, S 3 k 3 S2 + S 3 n ]. Using this expression, we may compute, for example, the conditional marginal distribution of S 2 : P [ S 2 r S2 + S 3 n ] k 0 n! λ k µ n r e λ + µ n k!r!n r! n p r p n r r

308 E. D. Sontag where we use p : / + µ, so µ p, which shows this conditional marginal p distribution is a binomial random variable with parameters n and p κ 2κ 4 κ 2 κ 4 +κ κ 3. Worked Example: Competitive Binding. We consider the following example using now A, B,... for species to simplify notations: A + B κ B,+ D A + C κc,+ E κb, κc, so for the associated deterministic system, the steady states satisfy κ B,+ AB κ B, D and κ C,+ AC κ C, E, so one such equilibrium is,,,λ,µ where λ : κ B,+ κ B,, µ : κ C,+ κ C,. The following quantities are conserved: A + D + E n A, B + D n B, C + E n C and subject to these constraints, one may pick the partition function: Zn A, n B, n C k A,k B,k C,k D,k E S λ k D µ k E k A! k B! k C! k D! k E! S {k A, k B, k C, k D, k E 0 k A + k D + k E n A, k B + k D n B, k C + k E n C }. In order to rewrite this function as a double sum, we first show that S is equal to the following set: S {k A, k B, k C, k D, k E 0 k D n B, 0 k E min{n A k D, n C }, k A n A k D + k E, k B n B k D, k C n C k E }. Indeed, suppose that k A, k B, k C, k D, k E S. Then k D 0 and from k B + k D n B we have that k D n B k B n B. Also, k E 0, and from k C + k E n C we have that k E n C k C n C and from k A + k D + k E n A we have that k D + k E n A k A n A and hence, k E n A k D, so k E min{n A k D, n C }. Thus, k A, k B, k C, k D, k E S. Conversely, suppose that k A, k B, k C, k D, k E S. We have that k D and k E are nonnegative. From k E n A k D, it follows that k A n A k D + k E 0, from k D n B, it follows k B n B k D 0, and from k E n C, we have k C n C k E > 0. Therefore, we may rewrite the partition function as follows using i, j instead of k D, k E as indices: Zn A, n B, n C n B i0 n B i0 λ i min{n A i,n C } n B i! i! j0 λ i n B i! i! Qn A i, n C n A! µ j n A i j! n C j! j! n B i0 n A λ i i n B i! Qn A i, n C.

22 Examples of Computation of Exact Moment Dynamics 309 where min{p,n} Qp, n : l0 µ l p l!n l!l!, Qp, n : p!qp, n min{p,n} l0 p l µ l n l!. The sum in Q is numerically better behaved than that in Q when p is large and n is small. We find that Q is itself the partition function Zp, n given by formula 22.9 for the simpler binding example S + S 2 S 3 and can also be written as p! n! 2F 0 p, n; ; µ, in terms of 2 F 0, Gauss s hypergeometric function. For example, when n B 0 or, the formula specializes to: Zn A, 0, n C Qn A, n C, Zn A,, n C Qn A, n C + λ Qn A, n C the first of these is not surprising, as when n B 0 the species B can only be zero, so the system reduces to the previous example, with S A, S 2 C, and S 3 E, and the mean of species D given the constraints n A,, n C is by Eq. 22.8: E[D n A,, n C ] λ Zn A, 0, n C Zn A,, n C λ Qn A, n C Qn A, n C + λ Qn A, n C. Using Q, we may write, alternatively, Zn A, 0, n C n A Qn! A, n C, Zn A,, n C n A Qn! A, n C + λ n A Qn A, n C and thus, cancelling the n A! terms, and using that Zn A, 0, n C n A Qn A, n C, E[D n A,, n C ] n A! n λ A Qn A,n C Qn A,n C +λn A Qn, which is far better behaved numerically when n A,n C A is large. We also remark that there is a third-order recursion for Z, obtained by the algorithm MVPoisson from [3]. In order to conveniently display the recurrences, let us use the following notations. We will write Z instead of Zb, b 2, b 3, and a notation like Z + + i means a shift of the ith argument by the indicated number of plus signs. For example, Z ++ 3 means Zb, b 2, b 3 + 2. There are three recurrences of order three, as follows, for each of the three arguments: 3 + b Z +++ λµz λµb λµb 2 λµb 3 + λµ λ µz + λb λb 2 + µb µb 3 + 2 λ + 2 µ Z ++, M3 + b 3 b 2 + 2Z +++ 2 λ 2 λµz + λ 2 b λ 2 b 2 λµb + 2 λµb 2 + λµb 3 λ 2 + 3λµ + λ µ Z + 2 + λµb b 2 λµb 2 2 λµb 2 b 3 + 2 λµb 4 λµb 2 2 λµb 3 4 λµ λb 2 + 2 µb 2 2 λ + 4 µz ++ 2, L3 + b 3 b 3 + 2Z +++ 3 λµ + µ 2 Z + λµb + λµb 2 + 2 λµb 3 + µ 2 b µ 2 b 3 + 3λµ µ 2 λ + µz + 3 + λµb b 3 λµb 2 b 3 λµb 3 2 + 2 λµb 2 λµb 2 4 λµb 3 4 λµ + 2 λb 3 µb 3 + 4 λ 2 µz ++ 3. The algorithm provides 27 initial conditions, the values of Z for the triples,,,,, 2,,, 3, 3, 3, 3 listed in that order. We display them as three matrices, respectively shown below. The first matrix lists the elements of the form,,, the next one 2,,, and the last one 3,,. In each matrix, elements are listed in the usual matrix order:, i, j is the i, jth entry of the matrix.

30 E. D. Sontag λ + µ + λ 2 + µ + 2 λ + µ 2 + λ 2 2 + µ 2 + 4 λ 2 + µ 6 + λ 6 4 + µ 6 + 2 λ 6 + µ 2 + 6 λ 6 + µ 4 + 2 λ 2 + µ 2 + 36 µ + λ + µ + µ + 2 2 λ + 2 µ2 + µ + κ 4 2 λ2 + µ + λ + µ 2 + 4 4 λ2 + µ + 2 λ + 4 µ2 + µ 2 + κ 8 2 2 λ2 + 2 µ + λ + µ 6 + 2 4 λ2 + 2 µ + 2 λ + 2 µ2 + µ 6 + 24 κ 3 2 2µ + λ + µ 2 + γ 6 γ 2 2 µ + λ2 + 2 2µ + λ + µ 4 + β 2 β 2 6 λ3 + 2 µ + λ2 + 4 2µ + λ + µ 2 + 36 α α 2 where we are using these notations: κ µ 2 + 6 λ + 2 µ2 + µ 2 + 2 κ 2 2 λ2 + µ 2 + 6 λ + 4 µ2 + µ 4 + 24 κ 3 2 λ2 + 2 µ 2 + 6 λ + 2 µ2 + µ 2 + 72 γ 2 µ2 + 2µ + 2 λ + 2 µ2 + µ 2 + 2 γ 2 2 µ2 + µ + 6 λ + 6 µ3 + 2 µ2 + µ 4 + 36 β 2 µ + 2 λ2 + 2 µ2 + 2µ + 2 λ + 4 µ2 + µ 4 + 24 β 2 2 µ 2 + 6 λ2 + 2 µ2 + µ + 6 λ + 2 µ3 + 4 µ2 + µ 8 + 72 α 2 λ3 + 2 µ + 2 λ2 + 4 µ2 + 2µ + 2 λ + 2 µ2 + µ 2 + 72 α 2 36 λ3 + 2 µ 2 + 6 λ2 + 4 µ2 + µ + 6 λ + 36 µ3 + 2 µ2 + µ 24 + 26 so, reading-out entries from the matrices above we have, for example: Z,, λ + µ +, Z2, 2, 2 λ 2 /4 + µ + /2λ + µ 2 /4 + µ/2 + /8, Z3, 2, 3 β 2.

22 Examples of Computation of Exact Moment Dynamics 3 We remark that the reduced indices for the sums defining the partition function can be obtained in a more systematic form, through the use of Smith canonical forms. Suppose that P is a matrix in Z q n that represents q conservation laws on n species. For instance, P [, 0, 0,, ; 0,, 0,, 0; 0, 0,, 0, ] in the competitive binding example. We assume, as in this and other examples, that q n and that the matrix P has full row rank q. Under this assumption, the integer matrix P can be represented in Smith canonical form see, for example, [6], meaning that there exist two unimodular that is to say, invertible over the ring of integers matrices U Z q q and V Z n n so that U PV [ 0], where diag δ,...,δ q, 0 is a q n q matrix of zeroes, and the δ i s are the elementary divisors of the matrix P. The elementary divisors are unique up to sign change, there are formulas that express then in terms of the minors of P see [6] for details. For example, for the above example, we have U I 3 3 identity matrix, V [, 0, 0,, ; 0,, 0,, 0; 0, 0,, 0, ; 0, 0, 0,, 0; 0, 0, 0, 0, ] and δ δ 2 δ 3, so U PV [I 0]. In general, if we wish to find nonnegative integer solutions of Ak b, for a given nonnegative integer vector b, we use that U PV V k Ub, so, using the indices l V k, [ 0]l Ub, which means that the last n q indices l are free, and the constraint Vl 0 is imposed to insure nonnegativity of k. For instance, in the competitive binding example, and recalling that U I and I, the equation [ 0]l Ub gives that l b, l 2 b 2, l 3 b 3, and l 4 i, l 5 j are arbitrary. Thus, we can express the sum as a sum over the two indices k 4 i and k 5 j, with k b i + j, k 2 b 2 i, and k 3 b 3 j. The nonnegativity condition Vl 0, applied with the above matrix V, says that these expressions must be nonnegative: which means that the sum can be reexpressed as a sum over i 0, j 0, subject to i b 2, j b 3, and i + j b. This is exactly the same as the set S computed by hand. References. Anderson, D.F., Craciun, G., Kurtz, T.G.: Product-form stationary distributions for deficiency zero chemical reaction networks. Bull. Math. Biol. 728, 947 970 200 2. Feinberg, M.: Chemical reaction network structure and the stability of complex isothermal reactors I. The deficiency zero and deficiency one theorems. Chem. Eng. Sci. 42, 2229 2268 987 3. Feinberg, M.: The existence and uniqueness of steady states for a class of chemical reaction networks. Arch. Ration. Mech. Anal. 32, 3 370 995 4. Gillespie, D.T.: The chemical Langevin equation. J. Chem. Phys. 3, 297 306 2000 5. Gomez-Uribe, C.A., Verghese, G.C.: Mass fluctuation kinetics: capturing stochastic effects in systems of chemical reactions through coupled mean-variance computations. J. Chem. Phys. 26, 02409 2007 6. Jacobson, N.: Basic Algebra I, 2 edn. Freeman and Company 995 7. Kelly, F.: Reversibility and Stochastic Networks. Wiley, New York 979 8. Mairesse, J., Nguyen, H.-T.: Deficiency zero Petri nets and product form. In: Franceschinis, G., Wolf, K. eds. Applications and Theory of Petri Nets, pp. 03 22. Springer 2009 9. Meyn, S.P., Tweedie, R.L.: Stability of Markovian Processes III: Foster-Lyapunov criteria for continuous-time processes. Adv. Appl. Prob. 25, 58548 993

32 E. D. Sontag 0. Singh, A., Hespanha, J.P.: A derivative matching approach to moment closure for the stochastic logistic model. Bull. Math. Biol. 69, 909 925 2007. Sontag, E.D.: Lecture notes on mathematical systems biology, Rutgers University, 2002 205. http://www.math.rutgers.edu/sontag/ftp_dir/systems_biology_notes.pdf 2. Sontag, E.D., Singh, A.: Exact moment dynamics for feedforward nonlinear chemical reaction networks. IEEE Life Sci. Lett., 26 29 205 3. Sontag, E.D., Zeilberger, D.: A symbolic computation approach to a problem involving multivariate Poisson distributions. Adv. Appl. Math. 44, 359 377 200 4. Van Kampen, N.G.: Stochastic Processes in Physics and Chemistry. Elsevier Science 200