Contnuous Tme Markov Chans Brth and Death Processes,Transton Probablty Functon, Kolmogorov Equatons, Lmtng Probabltes, Unformzaton Chapter 6 1
Markovan Processes State Space Parameter Space (Tme) Dscrete Contnuous Dscrete Markov chans (Chapter 4) Contnuous tme Markov chans (Chapters 5, 6) Contnuous Brownan moton process (Chapter 10) Chapter 6 2
Contnuous Tme Markov Chan A stochastc process {X(t), t 0} s a contnuous tme Markov chan (CTMC) f for all s, t 0 and nonnegatve ntegers, j, x(u), 0 u < s, ( ) ( ) ( ) ( ) { + = =, =,0 < } P X s t j X s X u x u u s ( ) ( ) { } = P X s+ t = j X s = and f ths probablty s ndependent of s, then the CTMC has statonary transton probabltes: P t = P X s+ t = j X s = s j () ( ) ( ) { } for all Chapter 6 3
Alternate Defnton Each tme the process enters state, The amount of tme t spends n state before makng a transton to a dfferent state s exponentally dstrbuted wth parameter v, and When t leaves state, t next enters state j wth probablty P j, where P = 0 and P = j j 1 Let q = vp, then v = q, j j j j ( h) Pj ( h) 1 P lm = v and lm = q h 0 h h 0 h j Chapter 6 4
Brth and Death Processes If a CTMC has states {0, 1, } and transtons from state n may go only to ether state n - 1 or state n + 1, t s called a brth and death process. The brth (death) rate n state n s λ n (µ n ), so v0 = λ0 v = λ + µ, > 0 P01 = 1 λ µ P, + 1 =, P, 1 =, > 0 λ + µ λ + µ λ o λ n-1 λ n λ 1 0 1 2 n-1 n n+1 µ 1 µ 2 µ n µ n+1 Chapter 6 5
Chapman-Kolmogorov Equatons In order to get from state at tme 0 to state j at tme t + s, the process must be n some state k at tme t From these can be derved two sets of dfferental equatons: Backward P j ( t) = qk Pkj ( t) vp j ( t) Forward ( ) ( ) ( ) P t s P t P s j k kj k = 0 + = k ( ) ( ) ( ) P t = q P t v P t j kj k j j k j Chapter 6 6
Lmtng Probabltes If All states of the CTMC communcate: For each par, j, startng n state there s a postve probablty of ever beng n state j, and The chan s postve recurrent: startng n any state, the expected tme to return to that state s fnte, then lmtng probabltes exst: P = lm P t (and when the lmtng probabltes exst, the chan s called ergodc) Can we fnd them by solvng somethng lke π = π P for dscrete tme Markov chans? j t j ( ) Chapter 6 7
Infntesmal Generator (Rate) Matrx Let R be a matrx wth elements (the rows of R sum to 0) Let t qj,f j = v,f = j n the forward equatons. In steady state: lm P t = lm q P t v P t ( ) ( ) ( ) j kj k j j t t k j 0 = q P v P k j These can be wrtten n matrx form as PR = 0 along wth P = j j 1 and solved for the lmtng probabltes. What do you get f you do the same wth the backward equatons? r kj k j j j Chapter 6 8
Balance Equatons The PR = 0 equatons can also be nterpreted as balancng: vp = q P j j kj k k j rate at whch process leaves j = rate at whch process enters j For a brth-death process, they are equvalent to levelcrossng equatons λ npn = µ n + 1Pn + 1 rate of crossng from n to n+ 1 = rate of crossng from n+ 1 to n so P n = λ λ λ µµ µ P and a steady state exsts f λλ λn µµ µ 0 1 n 1 0 1 2 n 0 1 1 n= 1 1 2 n < Chapter 6 9
Tme Reversblty A CTMC s tme-reversble f and only f Pq j = Pq j j when j There are two mportant results: 1. An ergodc brth and death process s tme reversble 2. If for some set of numbers {P }, P = 1 and Pq = Pq when j j j j then the CTMC s tme-reversble and P s the lmtng probablty of beng n state. Ths can be a way of fndng the lmtng probabltes. Chapter 6 10
Unformzaton Before, we assumed that P = 0,.e., when the process leaves state t always goes to a dfferent state. Now, let v be any number such that v v for all. Assume that all transtons occur at rate v, but that n state, only the fracton v /v of them are real ones that lead to a dfferent state. The rest are fcttous transtons where the process stays n state. Usng ths fcttous rate, the tme the process spends n state s exponental wth rate v. When a transton occurs, t goes to state j wth probablty v 1, j = * v Pj = v P j, j v Chapter 6 11
Unformzaton (2) In the unformzed process, the number of transtons up to tme t s a Posson process N(t) wth rate v. Then we can compute the transton probabltes by condtonng on N(t): j { } () () ( 0) P t = P X t = j X = n= 0 n= 0 n= 0 { () ( 0 ), () } () ( 0) { () ( 0 ), () } * n j vt ( vt) n! n { } = P X t = j X = N t = n P N t = n X = = P X t = j X = N t = n = P e e vt ( vt) n! n Chapter 6 12
More on the Rate Matrx ( ) ( t) Can wrte the backward dfferental equatons as P t = RP and ther soluton s ( ) ( 0 ) t t t = e R R P P = e snce P( 0) = I where n Rt n t e R n= 0 n! but ths computaton s not very effcent. We can also approxmate: e Rt = lm I+ R n t n n 1 t t R or e I R for large n n n Chapter 6 13