Integrals for Continuous-time Markov chains

Transcription

1 Integrals for Continuous-time Markov chains P.K. Pollett Abstract This paper presents a method of evaluating the expected value of a path integral for a general Markov chain on a countable state space. We illustrate the method with reference to several models, including birth-death processes and the birth, death and catastrophe process. Introduction Let Xt, t 0 be a continuous-time Markov chain taking values in the nonnegative integers S = {0,,... } and consider the path integral Γ 0 f = τ0 0 fxt dt, where τ 0 = inf{t > 0 : Xt = 0} is the first hitting time of state 0, and f is a given function that maps S to [0,. Stefanov and Wang [0] derived an explicit expression for E i Γ 0 f := EΓ 0 f X0 = i, i, in the case of a birth-death process, under the condition that f is eventually non-decreasing. This built on earlier work of Hernández-Suárez and Castillo-Chavez [4], who considered the case i = and fx = x. However, results from potential theory subsume these results and allow one to extend substantially the work of these authors, thus allowing the expectation of Γ 0 f to be evaluated explicitly for a much wider variety of models. We shall provide several illustrations, giving special attention to birth-death processes and the birth, death and catastrophe process. We shall see that, in the case of a birthdeath process, the condition on f, imposed by Stefanov and Wang, can be relaxed completely. We will consider the more general problem of evaluating E i Γf for the path integral Γf = 0 fxt dt. 2 The solution to the original problem can then be obtained by making 0 an absorbing state and setting f0 = 0. As pointed out in [4], it is useful, particularly in biological contexts, to think of fx as being the cost or reward per unit time of staying in state x. The problem, then, is to evaluate the expected total cost over the life of the process. For convenience, we shall write f i = fi whenever this simplifies notation. Department of Mathematics, University of Queensland, Queensland 4072, Australia.

2 2 Expectation of the path integral Let Q = q ij, i, j S be the q-matrix of transition rates of the chain assumed to be stable and conservative, so that q ij represents the rate of transition from state i to state j, for j i, and q ii = q i, where q i := j i q ij < represents the total rate out of state i. For example, in the case of a birth-death process we would have q i,i+ = λ i and q i,i = µ i with µ 0 = 0, where λ i and µ i are the birth and death rates of the process. It will not be necessary to assume that Q is regular, so that there may actually be many processes with the given set of rates. However, we will take Xt, t 0 to be the minimal process associated with Q. Its transition function P t = p ij t, i, j S is the minimal solution to the Kolmogorov backward equations, and has the following interpretation: p ij t = PrXt = j, Nt < X0 = i, where Nt is the number of jumps of the process up to time t. For further technical details, see []. Whilst it is commonly assumed that Q is regular, which is equivalent to Nt being almost surely finite for all t, it is often useful in biological applications, and particularly when dealing with birth-death processes, to allow for the possibility that the process might explode by performing infinitely-many transitions in a finite time. The following result can be found in any text on Markov chains that deals with potential theory for example, Theorem of [8]. Proposition If y i = E i Γf, where Γf is given by 2, then y = y i, i S is the minimal non-negative solution to the system of equations q ij z j + f i = 0, i S, 3 j S in the sense that y satisfies these equations, and, if z = z i, i S is any nonnegative solution, then y i z i for all i S. Returning to the original problem formulated in the introduction, where we sought to evaluate expectations of the path integral Γ 0 f = τ 0 fxt dt with τ 0 0 being the first hitting time of state 0, we may simply apply Proposition to the chain modified so that 0 is an absorbing state, that is q 0 = 0 and q 0j = 0 for j, and set f0 = 0 when τ 0 <, there is no contribution to the path integral 2 for t > τ 0. Of course, this is the context in which we would most likely apply the result: where the expected value of the path integral is evaluated up to absorption. Corollary If e i = E i Γ 0 f, where Γ 0 f is given by, then e = e i, i is the minimal non-negative solution to the system of equations q ij z j + f i = 0, i. 4 j Remarks. i If we set fx =, then Γ 0 f records the time of first leaving the set E = {, 2,... }, that is Γ 0 f = min{τ 0, τ }, where τ = sup{t > 0 : Nt < }. If Q is assumed to be regular, so that τ is almost surely infinite, then Corollary 2

3 reduces to a well known and widely used result on expected hitting times that can be found in any text on Markov chains for example, Theorem of [8]. Despite its obvious use in evaluating expected extinction times, it is apparently not widely known to biologists; in their paper Four facts every conservation biologist should know about persistence, Mangel and Tier [6] implore their readers to use it: Fact 2 There is a simple and direct method for the computation of persistence times that virtually all biologists can use. ii If, more generally, τ 0 is replaced by the hitting time of a set A, that is τ A = inf{t > 0 : Xt A}, then the expected value of the resulting path integral Γ A f can be evaluated. If e i = E i Γ A f for i A c, then e = e i, i A c will be the minimal non-negative solution to j A q c ij z j + f i = 0, i A c. iii On dividing equation 4 by f i assuming here that f j > 0 for all j, we see that E i Γ 0 f is the same as the expected hitting time of state 0, starting in state i, for the Markov chain with transition rates Q = qij, i, j S given by qij = q ij /f i, for i, and q0j = q 0j. This was observed for birth-death processes by McNeil [7]. Indeed, he observed that, conditional on X0 = j, the distribution of Γ 0 f is the same as that for τ for the Markov chain with transition rates Q. A similar observation can be made in respect of equation 3: E i Γf is the same as the expected time to explosion, stating in state i, for the Markov chain with transition rates Q given by qij = q ij /f i, for i S. 3 Birth-death processes In the case of birth-death processes, explicit formulae can be obtained. This was done for E i Γ 0 f by Stefanov and Wang [0], assuming that there exists a positive number x 0 such that f is non-decreasing for x x 0, but, as we shall see, this condition can be relaxed. Let q i,i+ = λ i for i 0 and q i,i = µ i for i, where λ i, i 0 and µ i, i are sets of strictly positive birth and death rates, with all other transition rates being 0. Under these conditions, S is irreducible. For simplicity, set µ 0 = 0, so that q i = λ i + µ i, i S. First consider the system of equations 3. For the birth-death process, they can be written λ i i µ i i + f i = 0, i, and λ f 0 = 0, where i = z i+ z i, i 0, and can be solved iteratively to obtain i = f j π j, i 0, 5 where the potential coefficients π j, j S are given by π 0 = and π i = i j= j=0 λ j µ j, i. Summing 5 from i = 0 to j gives z j = z 0 C j f, where C j f = j i=0 3 f k π k.

4 Notice that C j f Cf as j, where Cf = i=0 f k π k. Thus, for a finite non-negative solution, we require Cf < and, then, z 0 Cf. The minimal non-negative solution is obtained on setting z 0 = Cf. We have proved the following result. Proposition 2 For the birth-death process, E j Γf = i=j f k π k, j S. Remark. On setting fx = we obtain the well known result that the expected time to explosion starting in state j is given by E j τ = i=j π k, j S. Indeed, for the birth-death process, τ is almost surely infinite if and only if E j τ = for some and then all j; see []. Next let us consider equations 4. We seek the minimal non-negative solution z i, i to the system λ i z i+ λ i + µ i z i + µ i z i + f i = 0, i 2, λ z 2 λ + µ z + f = 0. 6 If we set z 0 = 0 and i = z i+ z i, i 0, these can be written λ i i µ i i +f i = 0, i, and can be solved iteratively to obtain µ i = z f j π j = z µ f j π j, i, 7 j= where the potential coefficients π j, j are now given by π = and π i = i k=2 λ k µ k, i 2. Summing 7 from i = to j gives, for j 2, j j z j = z + µ f k π k = z + Using the fact that = µ i+ π i+, we arrive at j= j z µ f k π k. z j = j z µ B i f, where B i f = 4 i f k π k, 8

5 interpreting an empty sum as being 0. This expression is valid also for j =. Now let j A j = and observe that, as j, A j A and B j f Bf, where A = and Bf = f k π k. Both of these sums may converge or diverge. We shall consider the cases A = and A < separately. The case A =. Under this condition, the process is non-explosive and τ 0 is almost surely finite see []. Since A = the sequence {z j } will eventually become negative unless Bf z µ. So, if Bf =, the solution to 6 will be infinite. Otherwise, the minimal solution is obtained on setting z µ = Bf. Thus, we have proved the following result, which generalizes Proposition of Stefanov and Wang [0]. Proposition 3 For the birth-death process with A =, E j Γ 0 f = j f k π k, j, k=i and this is finite if and only if f kπ k <. Remark. On setting fx = we obtain the well known result that if the process hits 0 with probability, the expected first passage time to 0 starting in state j is given by j E j τ 0 = π k, j, and this is finite if and only if π k <. The case A <. In view of 8, we may write z j = z µ A j C j f, where, now, C f = 0 and, for j 2, k=i C j f = j i f k π k = i=2 j As j, A j A < and C j f Cf, where f k π k. Cf = i f k π k = i=2 f k π k. If Cf =, then certainly sequence {z j } will eventually become negative. Otherwise, we require µ z Mf, where Mf = sup j C j f/a j, in order to avoid this happening. The minimal solution is then obtained on setting z µ = Mf. Thus, we have proved the following result. 5

6 Proposition 4 For the birth-death process with A <, j i E j Γ 0 f = Mf f k π k, j, and this is finite if and only if Cf <. Remark. This result is not covered by Proposition of Stefanov and Wang [0], for they have effectively assumed that A =. Their method of proof relies on a state-space truncation argument, whereby the process is approximated by a finitestate birth-death process on S n = {0,,..., n} with a reflecting barrier at n. This latter stipulation means that the resulting process may not be the minimal process, though it will when C := π k =. However, the limit process will always have A = ; see [5]. Example. In order to illustrate the results of this section, we will consider the birth-death process on S = {0,,... } with λ i = λ and µ i = µ both strictly positive. This is a simple model for a population that allows immigration at rate λ and removal rate µ. It is also known as the M/M/ queue, and referred to as such in both [4] and [0]. Let us first apply Proposition 2. It is easy to see that π i = ρ i, i 0, where ρ = λ/µ, and so E j Γf = f k r i k, j 0, λ i=j where r = /ρ. To illustrate this further, take f k = α k, where α > 0. If α = r, then the expectation is finite if and only if r < that is λ > µ, in which case E j Γf = + rjrj λ r 2, j 0. If α r, then the expectation is finite if and only if max{α, r} <, which requires λ > µ, and in which case E j Γf = αrj+ rα j+ λr α α r, j 0. Next, we shall evaluate E j Γ 0 f using Propositions 3 and 4. First observe that A j = rj µ r, and so A < if and only if r < that is λ > µ, in which case A = /µ r. Consider first the case λ < µ. We now have π i = ρ i, i. Using Proposition 3, we find that E j Γ 0 f = µ j i=0 f k+ ρ k i, j. k=i 6

7 To illustrate this further, take f k = α k, where α > 0. The expectation is finite if and only if α < r, in which case E j Γ 0 f = r α j µr α α, j. Notice that {f j } is a decreasing sequence when α <, and so Proposition of Stefanov and Wang [0] cannot be used to establish this formula. Finally, we shall consider the case λ > µ, that is, r <. A simple calculation gives and so j r C j f = f k r j k and Cf = µ r C j f A j j = r r j k f k. r j r µ r f k. The expectation in Proposition 4 will therefore be finite if and only if f k <, in which case where E j Γ 0 f = rj µ r M = sup j j M r j r r j k f k, j, r j r j k f k. r j As before, take f k = α k, but assume that α <, so as to ensure that the expectation is finite. We find that rj r C j f j jr j +, if α = r, r = j r A j rα j α + α j r + α r j, if α r. r j αr α In both cases this defines an increasing sequence with limit r/ α. Therefore, if α = r, E j Γ 0 f = jrj µ r, j, while if α r, then E j Γ 0 f = rr j α j µ αr α, j. Example 2. Next we shall consider an example of a birth-death process that exhibits explosive behaviour. We begin by examining the pure birth process. This has transition rates q i,i+ = q i = λ i, i 0, where the birth rates λ i, i 0 are all 7

8 strictly positive. Equations 3 are λ i z i+ z i + f i = 0, i 0. On summing from i = 0 to j we get z j = z 0 C j f, where C j f = j i=0 f i/λ i. Since C j f Cf := i=0 f i/λ i, a finite non-negative solution is obtained whenever Cf < and z 0 Cf, in which case setting z 0 = Cf gives the minimal solution. We deduce that E i Γ = j=i f j/λ j. Setting f i = shows that E i τ = j=i /λ j. So, the expected time to explosion is infinite if and only if C := j=0 /λ j =. Indeed, the process is regular, that is τ is almost surely infinite for all starting states, if and only if this latter condition holds see []. Explosive behaviour is easily exhibited. For example, suppose that Xt describes the number in a population of individuals, pairs of whom produce offspring at per-interaction rate γ > 0. Then, γ i 2 will be the birth rate when the population size is i. Since at least two individuals are needed to get things going, it is convenient to relabel the state space so that Xt = i indicates that i+2 individuals are present at time t. So, we have a pure birth process of the kind described with λ i = λi + i + 2, where λ = γ/2. The process is explosive because C = i=0 and the path integral Γ has expectation E i Γ = λ λi + i + 2 = λ <, j=i For example, if f i = α i, where 0 < α, then E 0 Γ = λα f j j + j α log α α The introduction of a linear death component presents no particular difficulties. Let λ i = λii+ and define µ i by µ i = µi, where µ > 0. In this way, E = {, 2,... } is an irreducible class and 0 is the sole absorbing state. Let ρ = λ/µ. Then, in the notation established in connection with Propositions 3 and 4, we have π = and, for j 2, π j = j k=2 λ k µ k = ρ j j k=2 kk k. = ρ j j! This formula is valid also for j =. The j-th term of A is a j := µ j π j = µρ j j! and so A < because a j+ /a j = /ρj + 0. Thus, exit from E occurs with probability less than no matter what the ρ. Also, since λ i /µ i = ρi +, there is very strong positive drift. So, we might expect the process to be explosive. Indeed, this can be established. The j-th term of C is c j := λ j π j j π k = j λρ j j +! 8 ρ k k! = d j λjj +,

9 where d j = ρ j j! j ρ k k! = + j ρ j k j k! ρ j j! Since, for each k, the summand converges to 0 monotonically as j, monotone convergence implies that d j. Hence, because k j= {jj + } =, we may deduce that C <. To illustrate the evaluation of the path integral, suppose that the states in E have Poisson weights: f k = e α α k /k!, k, where α > 0. Then, if αρ, A j = ρ j ρ i and C j f = e α A j j α i, µ i! αρ αµ i! while if αρ =, A j = αµ j α i i! e α αρ and It is easy to show that C j f/a j M, where M = and. C j f = e α µ + αµa j µa j. e α αρe /ρ, if αρ, M = e α + α, if αρ =. e α Using Proposition 4, we get j e α α i E j Γ 0 f = αµ αρ i! eα j ρ i, if αρ, e /ρ i! and E j Γ 0 f = e α µ e α e α j α i j i! i=0 α i, if αρ =. i! 4 The birth, death and catastrophe process We shall briefly examine a model [9] that is equivalent to one described by Brockwell [2], which has been used to describe the behaviour of populations that are subject to catastrophic mortality or emigration events. In addition to simple birth and death transitions, it incorporates jumps down of arbitrary size catastrophes. The process has state space S = {0,,... } and q-matrix given by iρ k i d k, j = 0, i, iρd i j, j =, 2,... i, i 2, q ij = iρ, j = i, i 0, iρa j = i +, i 0, 0, otherwise, 9

10 where ρ> 0 is the rate per capita at which the population size changes, a> 0 is the probability of a birth and d i, the probability of a catastrophe of size i, is positive for at least one i a + i d i =. Notice that when d i = 0 for all i 2, we recover the simple linear birth-death process, with λ i = iρa and µ i = iρd. Clearly E = {, 2,... } is an irreducible class and 0 is an absorbing state that is accessible from E. It is easily shown that Q is regular. Let d be the probability generating function given by ds = a + d i s i+, s <, and assume that d <. Brockwell showed that the process is absorbed with probability if and only if the drift D, given by D = d, is less than or equal to 0. We shall evaluate E i Γ 0 f for the path integral, where τ 0 is the time to absorption. Considering equations 4, we seek the minimal non-negative solution to the system This can be written i iρaz i+ iρz i + iρ d i j z j + f i = 0, i. j= i ρaz i+ ρz i + ρ d i j z j + g i = 0, i, 9 j= where g i = f i /i. On multiplying by s i and then summing over i, we find that the generating function Hs = z is i of any solution z i, i to 9 must satisfy bshs aρz + gs = 0, where bs = ρds s and gs = g is i. It is easy to see that b is convex on 0,, with b0 = aρ and b = 0. Furthermore, as Brockwell showed, σ, the smallest zero of b in 0, ], satisfies σ = if D 0 and σ < if D < 0. We know, from Lemma V.2. of Harris [3], that /bs has a power series expansion with positive coefficients in a neighbourhood of 0. So, let us write /bs = j=0 e js j, s < σ, where e j > 0, noting that aρ = b0 = /e 0. Letting κ = aρz, we obtain and hence, for i 2, Hs = z + κe i g j e i j s i, j= i i 2 z i = κe i g j e i j = κe i g i j e j. j= Now, {e i } is an increasing sequence. This is because e 0 = /aρ, and, since, z i = κe i when g 0, we have from 9 that j=0 i ρae i ρe i + ρ d i j e j = 0, i, j= 0

11 and hence that i ae i e i = d i j e i e j + e i j= j=i d j, i. Therefore, {z i } is the difference of two non-negative increasing sequences, and so, in order to ensure that {z i } itself is non-negative, we require κ sup i h i, where h i = g j e i j. e i The minimal solution is obtained with equality here. Since {e i } is increasing, we have that 0 < e i j /e i for all i j 0. Moreover, because σ is the radius of convergence of j=0 e js j, we have σ = lim i e i /e i, whenever this limit exists, implying that e i j /e i σ j for each j. Hence, formally, h i gσ. If this can be justified, then provided gσ <, we may set κ = gσ to obtain the minimal non-negative solution to 9. By Fatou s lemma, we always have lim inf i h i = lim inf i e i g j e i j j= j= g j σ j = gσ, but, in order that lim i h i exists and equals gσ, further mild conditions must be imposed. For example, if j= g jb j < and e i j /e i b j for all i j, or if {e i j /e i } is monotonic in i, then dominated and monotone convergence, respectively, guarantee that h i gσ. If, for example, e 2i = 9 i = 3 2i and e 2i+ = 2.9 i = 2.3 2i for i = 0,,..., then it can be show that lim i h i does not exist. Finally, if gσ < and h i gσ, we get j= i 2 E i Γ 0 f = gσe i g i j e j, i, 0 j=0 or, if preferred, E i Γ 0 fs i = gσ gs/bs, s < σ. To illustrate this, take f i = α i, where α > 0, so that gs = /α log αs, s < /α, and hence gσ < provided α < /σ. For example, if α =, so that f i = for all i, then gσ < if D < 0 and gσ = g = if D 0. We may therefore deduce that, when D < 0, the expected time to extinction is given by E i τ 0 = log σe i i 2 j=0 e j i j, i, or, if preferred, E i τ 0 s i = bs log s, s < σ. σ

12 m= Expected Reward E i Γ 0 f m=4 m=6 m= m= Initial Population Size i Figure : The effect of varying the m, the mean catastrophe size, on E i Γ 0 f with Poisson weights f k = e α α k /k! a = 0.55 and α = 20. This is equation 3. of [2]. Explicit results can be obtained in the case where the catastrophe size follows a geometric law. Suppose that d i = b qq i, i, where b> 0 satisfies a+b =, and 0 q <. The simple linear birth-death process is recovered on setting q = 0. It is easy to see that D = a b/ q, and so D < 0 or D 0 according as c > or c, where c = q + b/a. We also have bs = b + qas2 + qas + a qs = a s cs qs, and hence if D < 0, then σ = /c <. The coefficients of the power series /bs = j=0 e js j are easily evaluated using partial fractions. If D = 0, then e j = + qj/a, j 0. Otherwise, if D > 0 or D < 0, then e j = q c qcj a c, j 0. Note that, in all cases, e j /e j σ and hence e j k /e j σ k. Thus, E i Γ 0 f may be evaluated explicitly by substituting these expressions into 0, remembering that the expectation will be finite whenever gσ <. For example, when f i = α i, where α > 0, the expectation is finite if α < max{, q + b/a}. In contrast, gσ is always finite in the case of Poisson weights: f k = e α α k /k!, where α > 0. 2

13 For this latter case, Figure illustrates the effect of varying the mean catastrophe size on the expected value of the path integral. Acknowledgements: I would like to thank the referees and the Editor for very helpful suggestions. I would also like to thank Valery Stefanov for useful conversations on this work. The support of the Australian Research Council Grant No. A is gratefully acknowledged. References [] W.J. Anderson, Continuous-Time Markov Chains: An Applications-Oriented Approach, Springer-Verlag, New York, 99. [2] P.J. Brockwell, The extinction time of a birth, death and catastrophe process and of a related diffusion model, Adv. Appl. Probab [3] T.H. Harris, The Theory of Branching Processes, Springer-Verlag, Berlin, 963. [4] C. M. Hernández-Suárez and C. Castillo-Chavez, A basic result on the integral for birth-death Markov processes, Math. Biosci [5] W. Ledermann and G.E.H. Reuter, Spectral theory for the differential equations of simple birth and death processes, Phil. Trans. R. Soc. London [6] M. Mangel and C. Tier, Four facts every conservation biologist should know about persistence, Ecology [7] D.R. McNeil, Integral functionals of birth and death processes and related limiting distributions, Ann. Math. Statist [8] J.R. Norris, Markov Chains, Cambridge University Press, Cambridge, 997. [9] P.K. Pollett, Quasi-stationarity in populations that are subject to large-scale mortality or emigration, Environment International [0] V.T. Stefanov and S. Wang, A note on integrals for birth-death processes, Math. Biosci