Exact Simulation of Continuous Time Markov Jump Processes with Anticorrelated Variance Reduced Monte Carlo Estimation

Transcription

1 53rd I onference on Decision and ontrol December 5-7,. Los Angeles, alifornia, USA xact Simulation of ontinuous Time Markov Jump Processes with Anticorrelated Variance Reduced Monte arlo stimation Peter A. Maginnis, Matthew West and Geir. Dullerud Abstract We provide an exact, continuous extension to previous work in anticorrelated stochastic process simulation that was performed in an approximate, discrete setting. These methods reduce the variance of continuous Monte arlo for Markov jump process systems. We rigorously construct antithetic Poisson processes and analytically prove the negative correlation between pairs. We then show how these anticorrelated Poisson processes can be used to drive Markov jump processes via a random change representation. Finally, we provide a sufficient condition for variance reduction in the jump process context as well as demonstrate a simple example. I. INTRODUTION In the broad study of stochastic systems, simulation is an increasing necessity. With the proliferation of nonlinear, non- Gaussian applications of interest in filtering and estimation, for example, often stochastic simulation is the only tractable option. The cost of such computations grows dramatically with the scale, complexity or desired accuracy of the simulation, and practical, provable methods to increase efficiency of simulation are often indispensable. A common problem is the reduction of variance of Monte arlo estimates of system features. In previous work ], 7], we have defined and proved several anticorrelated variance reduction techniques for the simulation and estimation of Markov jump processes using a discrete-au leaping approximation. In this work, we extend those efforts to an exact, continuous- domain. Specifically, we rigorously construct principle-based simulation schemes that produce anticorrelated ensembles of continuous Markov jump process paths that can be used to construct reduced variance mean estimators. The crux of the approach is the random -change RT representation of a Markov jump process due to urtz 5]: Xt = x + Y i ρ i s, Xs ds ζ i, driven by independent, unit-rate Poisson processes Y i. This model description is extremely general, and can be used to construct virtually any discrete-space Markov process. Inspired by classical anticorrelated variance reduction techniques primarily here pairs of antithetic variates, we carefully construct pairs of realizations Y i,, Y i, of the Poisson process that are pairwise negatively correlated. Using I independent realizations of these anticorrelated pairs of Poisson processes, we may construct anticorrelated realizations X, X of the Markov jump process of interest, producing an ensemble from which to construct variancereduced estimates of the mean behavior of the process. The variance reduction approach of producing anticorrelated realizations of complicated stochastic processes by manipulating the elemental processes that drive them is already in use in many contexts. The most common is probably the antithetic simulation of rownian motion to produce continuous Markov processes, where the construction of perfect antithetic pairs is trivial due to symmetry. Producing valid realizations of the Poisson process that are significantly negatively correlated requires somewhat more effort, and the inherent asymmetry of the process does not readily demonstrate optimality of any particular approach. The Markov jump processes that they drive, however, are of tremendous theoretical and practical interest, for example in the study of chemical reaction networks 9], gene regulation networks ], and atmospheric aerosol simulation ]. In each of these areas, variance reduction techniques are of great benefit to reduce the cost of stochastic simulation. The manipulation of the random process inputs of the RT as a variance reduction approach has precedents as well, for example the common reaction path method of Rathinam, et al. 9] and the strong coupling of Poisson process inputs by Anderson ], both in service of parameter sensitivity analysis. Furthermore, the use of related anticorrelated ensemble techniques in the context of the discrete-au-leaping approximation of the Markov jump process is the subject of ]. The primary contribution of this paper is the construction of antithetic Poisson processes and their use in the RT for the purpose of exact simulation and variance reduced mean estimation of continuous processes. The sequel is organized as follows. First, we propose several techniques to produce anticorrelated unit rate Poisson processes using any existing anticorrelated technique, though here only explicitly considering antithetic pairs. For each method, we provide analytical and numerical support, and illustrate the usefulness and challenges of each. Next we show how to use these methods in support of simulation of Markov jump processes, as well as prove a sufficient condition for variance reduction in this setting. Finally, we provide a simple numerical example illustrating the potential effectiveness of the method. II. ANTIORRLATD ONTINUOUS-TIM POISSON PROSSS We begin by defining several algorithms to exactly simulate Poisson processes, while ensuring that they be as negatively correlated as possible. Such an anticorrelated ensemble of realizations could then be used to construct reduced variance mean estimates of the process, and, moreover, be U.S. Government work not protected by U.S. copyright 3

2 used to produce anticorrelated ensembles of general Markov jump process paths via the RT. onsider a unit-rate Poisson process Y t over a finite interval, T ]. We define sampling algorithms to produce anticorrelated sample paths of this process in order to create variance reduced pathwise mean estimators. In order to produce pathwise anticorrelated processes, we require in addition a method of random variable simulation to produce Poisson variables {X j } M j=, X j Poisλ, an ensemble of pairwise negatively correlated Poisson random variables such that ovx j, X j. Without loss of generality, we restrict our analysis to an antithetic method producing pairwise antithetic variables X := F λ U and X := F λ U, where U Unif, and Fλ is the formal inverse of the DF of a Poisson random variable with parameter λ. This is a well-known, classical technique ], but its use in this specific context is covered in detail in ] and ]. Our objective is to produce pairs of Poisson process sample paths Y, Y such that, for every t, T ], t, Y t. The motivation for this is that an estimator of the mean behavior Y t] of the process, say Ψt = Y t + Y t, will have variance given by VarΨt = VarY t + t, Y t VarY t, 3 where the last term would be the variance of an estimator constructed using independent Poisson process paths. The first approach to producing such ensembles is the endpoint approach. One way to simulate a Poisson process is to sample the value of the process a Poisson random variable denoting the number of jumps occurring by that at some endpoint T of and then sample when all of those jumps occurred in,t] which will each have Unif, T distribution. The idea of the endpoint method is to instead simulate two Poisson processes simultaneously by using an antithetic pair of Poisson random variates as their values at T. Thus by construction, these two values, Y and Y, will be negatively correlated. We simulate the jump s as in the naive i.i.d. case. The endpoint method is summarized in Algorithm. A pair of antithetic sample Algorithm ndpoint Method Using method, simulate anticorrelated Poisson random variables Y j Pois Simulate the corresponding jump s t j i.i.d. i Unif, T, i {,..., Y j } for each path j {,..., M} paths simulated using the endpoint method are shown in Figure. Now, while the two paths are explicitly negatively correlated at T, they are also negatively correlated throughout the interval, and thus, a pathwise mean Y t Y t Fig.. Two anticorrelated sample paths of a unit rate Poisson process simulated using the endpoint antithetic pathwise sampling technique. In this case Y anti., Y Pois are antithetically sampled. estimator constructed using these samples will have reduced variance relative to a naive estimator throughout the interval, as shown in Lemma. To prove this result, we first prove a very useful Lemma that computes the covariance at interior points in terms of covariances at endpoints of certain intervals. Lemma : Let Y t and Y t, t, T ] be two possibly correlated unit rate Poisson processes. For any T < T, T ], define G, T := σ{y, Y, Y, Y }, the σ-algebra defined by these four random variables. If, for any t, T, conditioned on G, T, Y t and Y t are independent, then for every t T, T ]: t, Y t =, Y + t T T ovn, T, N, T where N j, T := Y j Y j is the increment of the process. Alternatively: t, Y t = T t T + t T, Y + t T T, Y 5 Proof: For the sake of brevity, we prove only. The proof of 5 is similar but more algebraically tedious. ] ]] Y ty t = Y ty t G ] ]] = Y t G Y t G = Y + t T ] N, T T T Y + t T ] ] N, T T T ] = Y Y + T t T + t T ] T N, T N, T. 3

3 Note that t = T + T t T + t T T T, so that h i t, Y t = Y Y t 7 h i = Y Y T t h i T + N, T N, T T =, Y + T t T T ovn, T, N, T, 9 and the claim holds. This Lemma may now be immediately used to prove that there is negative correlation between endpoint antithetic Poisson process pairs at every t, T ], not just the endpoint. Lemma : Suppose Y t, Y t are simulated using Algorithm for t, T ] and some anticorrelated sampling method. Then t, Y t, for every t, T ]. Proof: We proceed by application of Lemma. First, note that, by construction, Y since these are precisely the antithetically sampled Poisson random variables. Further, note that, conditioned on Y and Y, Y t and Y t are independent for every t, T. Thus, by, = t t, Y t t = ovn T, T, N, T T, Y, for every t, T ]. Thus, at T, the paths will have the same negative correlation as is produced by the random variable method, and this negative correlation will be felt by the interior points with a quadratic relationship. As a direct consequence of this result, unbiased pathwise mean estimators constructed using these anticorrelated samples will have reduced variance. orollary : Suppose Ψt := Y t + Y t and Ψ t := Y t + Y t are two unbiased mean estimators of Y t] constructed using sample paths drawn i.i.d. and via the antithetic endpoint technique, respectively. Then VarΨt VarΨ t = VarΨt + t, Y t. Figure illustrates this fact by plotting the variance of two -sample mean estimators, one using the traditional Monte arlo estimation scheme i.i.d. sampe paths and one using two pairs of endpoint antithetically sampled Poisson process paths versus. Here, we observe a dramatic reduction of variance at =, and some gains for smaller. We may improve performance in the interior of the interval via an obvious extension, the oncatenation technique. y exploiting the independent increments property of ndpoint Fig.. omparison variances of a -sample unit-rate Poisson process endpoint antithetic mean estimator Ψ t to the naive i.i.d. -point mean estimator Ψ t. Poisson processes, we split the interval, T ] into smaller sub-intervals and simulate each Poisson process increment using the endpoint technique, as shown in Alg.. A pair Algorithm oncatenation Method Divide the interval, T ] into equal length subintervals Using method, simulate anticorrelated Poisson random variable increments N j k, T k := Y j k Y j k Pois T for k {,..., } 3 Simulate the corresponding jump s t j i.i.d. i Unif k, T k j k, i {Y +,..., Y j k }, k {,..., } for each path j {,..., M} of sample paths simulated using the concatenated antithetic sampling technique for = and T = are shown in Fig. 3. As shown in Fig., by splitting the interval in half, we achieve another large reduction in the variance of a mean estimator at the concatenation point where an antithetic pair of Poisson random variables are used. Note also that, in this case there is a small sacrifice in performance at the endpoint t =. This is due to the independent increments property implying that the covariances of sums of increments are sums of covariances of increments, resulting in an upward drift of the estimator variance at the endpoint. As in orollary, the following Lemma is equivalent to variance reduction of a mean estimator using these sample paths. Lemma 3: Suppose Y t, Y t are simulated using Algorithm for t, T ] and some anticorrelated sampling method. Then t, Y t, for every t, T ]. Proof: First, note that each increment N j k, T k = Y j k Y j k, k 33

4 Y t Y t Fig. 3. Pair of anticorrelated unit-rate Poisson process sample paths simulated using concatenated antithetic pathwise sampling. Note here that Y anti. 5, Y 5 Pois5 are independent of Y Y 5 and Y anti. Y 5 Pois ndpoint oncatenate Fig.. omparison of variance of a -sample mean estimator Ψ t of a unit-rate Poisson process constructed using = step concatenated antithetic sample paths to variances of a -sample endpoint antithetic mean estimator Ψ t and the naive -sample mean estimator Ψ t. {,..., } satisfies ov N T k, T k, N T k, T k since these are each antithetically sampled random variables. Next, it is easy to see that the conditional independence requirement of Lemma is satisfied by the process within each small interval, since Y t, Y t are independent conditioned on G k, T k k for each t, T k by construction. Thus for the leftmost interval, by, we have that t, Y t N = t T ov, T, N, T for every t, T ]. y the same reasoning, for t T, T ], ov Y t, Y t = ov Y T + t T T ov, Y T N T, T = ov N, T, N, T + t T T ov N T, T, N T, T 3, N T, T. y continuing this process from left to right, it is easy to ndpoint oncatenate Fig. 5. omparison of variances of a -sample mean estimator Ψ t of a unit-rate Poisson process constructed using = step concatenated antithetic sample paths to a -sample endpoint antithetic mean estimator Ψ t and the naive -sample mean estimator Ψ t. see that, for any t, T ]. ov Y t, Y t Xk = ov N k T, kt k= T T + t k ov N «, N k T, kt «««««k T, k + T, N k T, k + T, 5 where k := max{k N : kt t} = t T. Since each of these covariances is negative, the claim holds. A direct consequence of this fact is that concatenation may not perform better than the endpoint technique as the number of concatenation steps increases. Fig. 5 illustrates 3

5 Y t Y t Fig.. An anticorrelated pair of sample paths produced using L = comparable to a -step concatenated path. Note here that Y j anti. Pois is sampled first, then Y j j 5 Y anti. inomy j, /, and finally Y j j.5 Y 5 anti. inomy j 5, / and Y j j j j anti. 7.5 Y 5] {Y 5, Y } inom5, /. this problem by showing the drift in variance that occurs when concatenation steps are used. A more ideal mean estimator is one who s endpoint performance matches that of the endpoint algorithm but that further improves performance away from the endpoint T. We achieve such an algorithm by again exploiting the fact that, conditioned on its future value, a Poisson process has binomial distribution. This algorithm, referred to here as binomial midpoint simulation method, is defined in Algorithm 3. The key discovery is that, once we sample the values of the process at the endpoints of an interval, we may conditionally sample the midpoint of that interval using a binomial distribution. When we perform this sampling, we use the anticorrelated method, reducing the variance of an estimator at this midpoint. We may then recursively sample the midpoints of these new intervals, repeatedly bijecting until we have L steps. We then finish the simulation of the Poisson process by sampling i.i.d. uniform jump s, similar to the concatenation method. Algorithm 3 inomial Midpoint Method Divide the interval, T ] into L subintervals Simulate Y j Pois as in Alg. 3 Simulate Y j T j Y inomy j, Fix l =,..., L. Then, for each k {, 3, 5,..., l } simulate N j k, T k {Y j k+ l l, Y j k l } l inomy j k+ Y j k l, l 5 Simulate jump s t j i.i.d. i Unif k, T k, i L L {Y j k +,..., Y j L k }, k {,..., L } L for each path j {,..., M} A pair of binomial antithetic sample paths for L = is shown in Fig.. While dividing the interval by powers of is not strictly necessary, dyadic intervals are chosen to produce binomial distributions with p =, since antithetic sampling is more effective for symmetric distributions. The algorithm performs better as the number of midpoints increases, until reaching a saturation point when midpoints become nearly constant, i.e. when the expected number of transitions in an interval is small. As in orollary, the following Lemma is equivalent to variance reduction of a mean estimator using these sample paths. Lemma : Suppose Y t, Y t are simulated using Algorithm 3 for t, T ] and some anticorrelated sampling method. Then t, Y t, for every t, T ]. Proof: As in the endpoint method case,, Y as this is an antithetic pair of Poisson samples. For any interior interval boundary point t = k, k {, 3,..., l }, Y l t and Y t are a conditionally sampled antithetic pair, that is ov Y t, Y t G t l, t + l, where the left hand side is a G t, t + l -measurable l random variable. Furthermore, Y j t G t l, t + ] l = Y j t l + Y j t + l 7 so that ] ] ov Y t G, Y t G = t, Y t + t, Y t + + t +, Y t + t +, Y t +, 9 where t + and t denote the parents of t, t l and t + = k, respectively. Thus, by the law of total l l covariance ov Y t, Y t = ov Y t, Y t G t l, t + ] l + t, Y t + t, Y t + + t +, Y t + = k l t +, Y t +. Note that the first term is negative, and the argument repeats for each of the other terms, rewriting them as a linear 35

6 ndpoint oncatenate inomial Midpoint III. ANTIORRLATD ONTINUOUS-TIM MAROV JUMP PROSSS We now produce variance reduced sample paths of continuous- Markov jump processes by driving these processes with anticorrelated unit-rate Poisson processes. onsider the random change representation of a Markov jump process, Xt = x + Y i ρ i s, Xs ds ζ i, Fig. 7. omparison of variance of a -sample unit-rate Poisson process mean estimator Ψ t constructed using -step binomial midpoint antithetic sample paths to the variances of a -step concatenated antithetic Ψ t, endpoint antithetic Ψ t and naive Ψ t equivalents ndpoint oncatenate inomial Midpoint Fig.. omparison of variance of a -sample unit-rate Poisson process mean estimator Ψ t constructed using -step binomial midpoint antithetic sample paths to the -step concatenated antithetic Ψ t, endpoint Ψ t antithetic and naive Ψ t equivalents. combination of some negative value plus a covariance of Y and Y evaluated at their parent s. These recursions will terminate with, Y and, Y, each of which is non-positive, so we have that ov Y t, Y t for each interior interval boundary point t. The claim holds at the remaining points t, T ] by 5, since t, Y t = + k t L L k L + t k L k L, Y k L k t L L k L, Y k, L for t k L, k L ]. where Y is a unit-rate Poisson process and ρ i and ζ i are the propensity function and jump vector of a reaction channel i. There are several ways to simulate such systems, for example Gillespie s stochastic simulation algorithm SSA 3] for exact simulation or tau-leaping ] for discrete- approximate simulation. Another way to exactly simulate the system is to simulate the unit-rate Poisson processes that drive the Markov process and compute the realized jump s and s using the propensity functions and jump vectors. Taking this approach, if we replace the use of i.i.d. unit-rate Poisson processes with anticorrelated identically distributed unit-rate Poisson processes, as constructed in Section II, we produce exact realizations of the Markov jump process that are negatively correlated and hence produce variancereduced mean estimates. Theorem 5 proves this result in the case of non-negative -dependent propensity functions, ρ i s, Xs = ρ i s. Theorem 5: For each i {,..., I}, let Y i, t and Y i, t be i.i.d. realizations of a unit-rate Poisson process Y. Let Y i, t and Y i, t be antithetically simulated realizations of Y using any of the above techniques such t for every t, T ] and Y i,j t and Y i,j t are independent for each i i. Let X t and X t be i.i.d. realizations of a Markov jump process Xt such that that i, X j t = x + t, Y i, Y i,j ρ i s ds ζ i, for every t, T ] where T is defined such that T = T max i ρi s ds. Let X t and X t be defined such that X j t = x + Y i,j ρ i s ds ζ i. 3 Finally, let Ψt := X t + X t and Ψ t := X t + X t be the unbiased -sample mean estimators of Xt. Then tr VarΨ t tr VarΨt for every t, T ]. In particular, continuous anticorrelated Markov jump process sampling reduces MS of mean estimators. Proof: It suffices to show that tr ovx t, X t for every t, T ]. 3

7 Then ovx t, X t = ov = i = ov Y i, i = Y i, ρ i s ds ζ i, Y i, ρ i s ds ρ i s ds, ζ i Y i, ρ i s ds ζ i ζ i. 5 The scalar covariance in 5 is negative by the Poisson process Lemmas above. Thus tr ovx t, X t = ov Y i, τ i t, Y i, τ i t ζ i, 7 for every t, T ], where τ i t := t ρi s ds. IV. RADIOATIV DAY While the requirement of Theorem 5 that the rate functions depend only on can somes be restrictive, this condition is sufficient but hardly necessary. In practice, these variance reduction techniques can be beneficial even in the -dependent or nonlinear cases. As an example, consider a simple model of radioactive decay with -dependent rates, defined by Xt = x Y t λxs ds, for some scalar rate parameter λ >. A pair of anticorrelated sample paths of this process are shown in Fig. 9. The antithetic -sample mean estimator using each of the above techniques were constructed and their estimated autocovariances are shown in Fig.. Note the almost 75% variance reduction in some regimes. RFRNS ] D. F. Anderson. An efficient finite difference method for parameter sensitivities of continuous Markov chains. SIAM Journal on Numerical Analysis, 5:37 5,. ]. riat and M. hammash. omputer control of gene expression: robust setpoint tracking of protein mean and variance using integral feedback. I onference on Decision and ontrol,. 3] D. T. Gillespie. A general method for numerically simulating the stochastic evolution of coupled chemical reactions. J. omput. Phys., :3 3, 97. ] D. T. Gillespie. Approximate accelerated stochastic simulation of chemically reacting systems. Journal of hemical Physics, 5:7 733,. 5] T. G. urtz. Approximation of population processes. MS-NSF Regional onf. Ser. in Appl. Math, 9. ] P. A. Maginnis. Variance reduction for Poisson and Markov jump processes. Master s thesis, University of Illinois at Urbana-hampaign,. X t X t Fig. 9. A pair of sample paths of the radioactive decay process, driven by binomial midpoint antithetic Poisson processes ndpoint oncatenate inomial MidPoint Fig.. Variance comparison -sample variance reduced mean estimators of a radioactive decay process using each type of antithetic sample path, with = /L =. 7] P. A. Maginnis, M. West, and G.. Dullerud. Application of variance reduction techniques for tau-leaping systems to particle filters. I onference on Decision and ontrol,. ] P. A. Maginnis, M. West, and G.. Dullerud. Anticorrelated discrete stochastic simulation. I onference on Decision and ontrol, 3. 9] M. Rathinam, P. Sheppard, and M. hammash. fficient computation of parameter sensitivities of discrete stochastic chemical reaction networks. Journal of hemical Physics, 33:33,. ] N. Riemer, M. West, R. A. Zaveri, and R.. aster. Simulating the evolution of soot mixing with a particle-resolved aerosol model. Journal of Geophysical Research,, 9. ]. P. Robert and G. asella. Monte arlo statistical methods. Springer, nd edition,. 37