Markov Chains in Continuous Time

Transcription

1 Chapter 23 Markov Chains in Continuous Time Previously we looke at Markov chains, where the transitions betweenstatesoccurreatspecifietime- steps. That it, we mae time (a continuous variable) avance in fixe, iscrete steps. The transitions between each state, at each step, were efine by the transition matrix. In this chapter, we push the bounaries of theory a little further an allow time to be a continuous variable. So let us combine these ieas of a ranom (iscrete) variable X that may ump to other states at ranom times. Suppose X(t) isaiscreteranomvariable.sowecanlabelthestatesasfollows: X {0, 1, 2,...,n}. An then efine a past time (u), a present time (s) anafuture time (t + s). Let us efine a continuous time Markov process as follows: For all s, t 0ans + t>s>u 0 P (X(t + s) = X(u) =k an X(s) =i) =P (X(t + s) = X(s) =i). (23.1) 0 u s s+ t past present future time That is, the process of getting from X(u) tox(s) (thepasttothepresent)hasnoeffect on what happens from X(s) tox(s + t) (thepresenttothefuture). If, in aition, we make the following rule: P (X(t + s) = X(s) =i) =P (X(t) = X(0) = i) 161

2 162 CHAPTER 23. MARKOV CHAINS IN CONTINUOUS TIME then the process is sai to be stationary. That is, the transition probability oes not epen on the timing of the process (s), ust the time interval. We will assume (in this course) that theprocessesare stationary or homogeneous Markov processes. It is still possible to speak of a transition matrix, namely: P (X(t) = X(0) = i) =p i (t). (23.2) But now the matrix epens on time. Although the specific state of the system, at any time, is ranom, the system must be in one of the states. That is the total probability must a to 1. As in the iscrete case - this matrix is stochastic, that is for every state i: p i (t) =1. (23.3) An this must be true at all times, t. In other wors, the system has to go somewhere. Summing over all covers any possible future outcome Chapman-Kolmogorov relation Another important property that continuous-time chains share with iscrete time chains is the Chapman- Kolmogorov equation. If we consier the meaning of this relation for iscrete time, any transition from the present to the future, has to pass through some state at some intermeiate time (the near future ). So consier the present t =0somefuturetimes + t an some near future time, s. Conitioning on the (unkown an ranom) intermeiate state in the near future we have: p i (s+t) =P (X(s+t) = X(0) = i) = k P (X(s+t) = X(s) = k, X(0) = i)p (X(s) = k X(0) = i). But since the process is Markovian the conitional probability can be written as: (23.4) P (X(s + t) = X(s) =k, X(0) = i) =P (X(s + t) = X(s) =k) (23.5) an since the process is stationary, we can reset the time to start from zero: P (X(s + t) = X(s) =k) =P (X(t) = X(0) = k). (23.6) So then it follows that: P (X(s + t) = X(0) = i) = k P (X(t) = X(0) = k)p (X(s) = k X(0) = i). (23.7) or in other wors: p i (s + t) = k p ik (s)p k (t). (23.8) We recognize a matrix prouct on the right-han sie. So using matrix notation gives a more elegant

3 23.1. KOLMOGOROV EQUATIONS 163 statement of the Chapman-Kolmogorov relation: P(s + t) =P(s)P(t). (23.9) Acollateralconsequenceofthisrelationisthecommutivityofthetransitionmatrices: P(s)P(t) =P(t)P(s). (23.10) 23.1 Kolmogorov equations We woul like to have a simple formula for the probability of being in a certain state at a certain time. That is, something akin to the formula for the Poisson process forp (N(t) =n). This can be one, an we show how in this section. Let us consier the Chapman-Kolmogorov relation, equation (23.9), for a short time in the future: P(t + h) = P(h)P(t) = P(t)P(h). (23.11) Note that we can get to t + h from t =0,byeitherasmallsteph then a large step t or vice versa. this explains why there are two possible expressions. Now, subtracting P(t) from both sies we get: P(t + h) P(t) =(P(h) I)P(t) =P(t)(P(h) I). (23.12) The symbol s I enotes the unit (ientity) matrix. Then iviing by the time increment, h, antakingthelimith 0, gives: ( ) ( ) P(t + h) P(t) P(h) I P(h) I lim =lim P(t) =P(t) lim h 0 h h 0 h h 0 h. (23.13) Let us efine the ump rate matrix (also calle the generator matrix ): Q lim h 0 P(h) I h. (23.14) This gives us (a) the Kolmogorov forwar equation: an (b) the equivalent Kolmogorov backwar equation: P(t) =P(t)Q (23.15) t P(t) =QP(t). (23.16) t Obviously, accoring to (23.15) an (23.16), the matrices Q an P(t) commute,thatis QP(t) =P(t)Q. (23.17)

4 164 CHAPTER 23. MARKOV CHAINS IN CONTINUOUS TIME Since the sum along any row of the transition matrix is one, we note that the sum of every row in a ump-rate matrix is ZERO, since: 1 Q i =lim h 0 h (P i (h) δ i ). (23.18) That is: Q i =0, for all i. (23.19) Theorem: Thesolutionof(either/both)theKolmogorovequationsis: P(t) =exp(qt). (23.20) Proof: Firstly, we efine the exponential function for a matrix by the power series: exp(a) I + A 1! + A2 2! + A3 3! + Therefore: exp(0) = I. Then accoring to the propose solution (23.20): P(0) = exp (Q 0) = exp (0) =I. (23.21) This is correct, since it is known that: P (X(0) = X(0) = i) =1 i =, P(X(0) = X(0) = i) =0 i. That is, given we start in a state (i) att =0wearecertaintobeinthatstateatt =0annoother. So our propose solution satisfies the intial conitions. equations (23.16 an 23.15). Does it satisfy the Kolmogorov ifferential If we substitute (plug in) the solution (23.20) for P/t we get: t P = ( I + Qt t 1! + Q2 t 2 + Q3 t 3 ) + 2! 3!. (23.22) Then ifferentiation of each term in turn gives: that is: 0 + Q 1! + 2Q2 t 2! + 3Q3 t 2 3! + (23.23) ( t P = Q I + Qt 1! + Q2 t 2 ) + = QP(t) (23.24) 2! which is the Kolmogorov backwar equation (23.15) as require. This all seems very straightforwar. The only problem arises incalculatingtheexponentialofamatrix. This is quite easy on a computer, but not so easy with pencil an paper. Onlyforverysmallmatrices, or very simple matrices, can this be achieve. We will consier a few examples of this kin in the next chapter. Let us conclue the iscussion here with a wor on stationary/equilibrium states.

5 23.2. EQUILIBRIUM STATES Equilibrium states Suppose we have a continuous-time Markov chain, then we know that its ynamics are governe by the Kolmogorov equations: P(t) =QP(t) =P(t)Q. (23.25) t For an equilibrium in the system (or stationary state) the probabilities will not change in time. That is, for an equilibrium state, no matter what the initial conitions at t = 0: That is, at equilibirum, in the long run: lim P i(t) =π, for all i. (23.26) t lim t Then the equilibrium istribution must be such that: t P = 0. (23.27) PQ =0. (23.28) Recalling that, at extremely long times, the transition matrix becomes a matrix with each row equal to the equilibrium istribution. This then leas to: πq =0. (23.29) That is, the equilbrium state is in the (left) null space of the umpratematrix. More explicitly, for a system with states X {0, 1, 2, 3,...}, theequilibriumistributionissuchthat: π Q k =0, k =0, 1, 2,...,n. (23.30) An equilbrium may, or may not, exist in the system. If an equilibirum oes not exist, then the only solution of πq =0 (23.31) will be the trivial solution, π = Jump rates for the Poisson process Consier the Poisson process as an example of a continuous-time Markov chain. We can enote the state of the system, X most conveniently by the number of arrivals: N(t) {0, 1, 2, 3,..., }. so we take X = N(t). So for this example, what woul be the ump-rate matrix? Well, one can analyse the transition matrix over a short time h. Thiscanberepresentebythetransition graph (23.2) For the Poisson process P (N(h) = N(0) = i) is well known. This is ust the extra events that occur

6 166 CHAPTER 23. MARKOV CHAINS IN CONTINUOUS TIME Figure 23.1: Counting process for Poisson arrivals. Figure 23.2: Transition graph for a (homogeneous) Poisson process over a short time h. Note that, when the time is very short we never observe more than one arrival. So the umps are only between nearestneighbour states n n + 1, with probability proportional to the rate λ an the observation time h. By very short we mean λh 1, so the transition probabilities are very small to the neighbouring states. in the short time h, byveryshortwemeanλh 1. We know that, for <i,thereiszeroprobability. That is the number of events cannot ecrease (there are no negative events to reuce the number), N(t) always stays the same or increases. We know that there is a negligible chance of more than one eventhappeninginsuchashorttime. Infact, as iscusse, we always arrange the time h such that this is the case! Really, we only nee be concerne with one event of no event in this time. In mathematical terms we have: λh + o(h) = i +1 one event 1 λh + o(h) = i no event P (N(h) = N(0) = i) = o(h) >i+1 multiple events 0 <i negative events By efinition: Q i =lim h 0 P (X(h) = X(0) = i) δ i h where we use the Kronecker elta (δ i )short-hanfortheunitmatrix: I i = δ i = { 1 i = 0 i

7 23.3. JUMP RATES FOR THE POISSON PROCESS 167 Therefore the ump rate matrix is: Q i =lim h 0 P (N(h) = N(0) = i) δ i h = λ = i +1 one event λ = i no event 0 >i+1 multiple events 0 <i negative events That is, in explicit form we have a matrix with the following structure: λ λ λ λ 0 0 Q = 0 0 λ λ λ λ (23.32) The sum along every row of the ump-rate matrix is zero (as it shoul always be). There is no equilibrium state to this system. That is, as time goes on, arrivals will continue to a up inefinitely so that N(t), ast. Thesystemneversettlesintoastationary state (a time-average equilibrium). This is confirme by the solution to the equilibrium equation: πq =0 (23.33) that is: λ λ λ λ 0 0 ( ) ( ) π 0 π 1 π λ λ 0 = λ λ If we multiply out the vector (π) byfirstcolumnofq, weget: λπ 0 =0 which gives us: π 0 =0. Thenπ times the secon column of Q gives: λπ 0 λπ 1 =0 giving: π 1 =0. Continuinginthismannerweseethattheonlysolutionisthe trivial solution: π 0 = 0, π 1 =0, π 2 =0...Thatis,thereisnoequilibriumistributionforsuchasystem.

8 168 CHAPTER 23. MARKOV CHAINS IN CONTINUOUS TIME