Markov chans M. Veeraraghavan; March 17, 2004 [Tp: Study the MC, QT, and Lttle s law lectures together: CTMC (MC lecture), M/M/1 queue (QT lecture), Lttle s law lecture (when dervng the mean response tme from mean number of customers), DTMC (MC lecture), M/M/1 queue dervaton usng DTMC analyss, derve dstrbuton of response tme n M/M/1 queue (QT lecture), relaton between Markov property and memoryless property (MC lecture), M/M/m queue (QT lecture).] Markov process [1, pg. 337] s a stochastc process whose dynamc behavor s such that probablty dstrbuton for ts future development depends only on ts present state and not how the process arrved n that state. If state space s dscrete, then t s a Markov chan,.e., the random X n varables or Xt () are dscrete. 1. CTMC Defnton of a CTMC: [2, page 381] A CTMC { Xt ()t 0} s a contnuous tme, dscrete value random process such that for an nfntesmal tme step of sze, PXt [ ( + ) Xt () ] q PXt [ ( + ) Xt () ] 1 q The above defnton s somewhat smplstc snce t treats transton rates q (1) to be ndependent of tme. A more general defnton s gven below. Reference [3] s defnton of a CTMC s { Xt ()t 0} s a contnuous tme, dscrete value random process f for t o < t 1 < < t n < t, wth t and t r 0 for r 12,,, n, PXt [ () Xt ( n ) Xt, ( n 1 ) n 1,, Xt ( 0 ) o ] PXt [ () Xt ( n ) ] (2) If t s a Markov process (meanng we allow for contnuous state, then the probablty should be PXt ( () x) nstead of PXt ( () ). 1
1.1 Transton probabltes, state probabltes and transton rates The behavor of the process s characterzed by the dstrbuton of the ntal state of the system gven by the pmf of Xt ( 0 ), PXt ( ( 0 ) k), k 012,,, and the transton probabltes p t) PXt [ () ( X( ν) ) ] for 0 ν t and, 012,,, where (3) p ( tt, ) 1 f 0 otherwse (4) The pmf of Xt ()(or the state probabltes at tme t ; remember n Markov chans X(t) s dscrete, hence we talk of a pmf) s denoted by P PXt [ () ] 012,,, t 0 (5) Usng the theorem of total probablty: Clearly, P 1 for each tme t 0 (6) P PXt [ () ] PXt [ () X( ν) ]PXν [ ( ) ] p t)p ( ν) (7) If ν 0, then P p ( 0, t)p ( 0) ; ths means that f we know the ntal state probabltes P ( 0) and the transton probabltes p ( 0, t), the Markov chan s completely determned. But for what values of t do we need to defne p ( 0, t)? All values of t? That s an number! Contrast wth DTMC. Meanwhle for CTMC, we defne two other functons, transton rates, whch are nonnegatve contnuous functons Defnton of transton rates: For, q p p t) ( t, t+ h) p ( tt, ) p lm ------------------------------------------------- ( t, t + h) lm -------------------------- (8) t ν t h 0 h h 0 h because as per (4), p ( tt, ) 0 f. q p ( p t t) ) ( t, t) p ( t, t+ h) lm ------------------------------------------------- ν t h 0 h 1 p ( tt, + h) lm -------------------------------- h h 0 (9) 2
agan by (4). Note that these are defntons and hence q can be defned to be the negatve of the dervatve of. t ( p t) ) ν t Relaton between transton probabltes and transton rates: p ( tt, + h) q h+ o( h) f (10) p ( tt, + h) 1 q h+ o( h) (11) where lm oh ---------- ( ) h 0 h 0 (12) 1.2 Chapman-Kolmogorov equaton Property of a Markov chan: The transton probabltes of a Markov chan { Xt ()t 0} satsfy the Chapman-Kolmogorov equaton: For all, I, where I s the state space of the process, p t) p k u)p k ( ut, ) 0 ν < u < t k I Proof: Use the theorem of total probablty: PXt ( () X( ν) ) PXt [ () Xu ( ) k, X( ν) ]PXu [ ( ) k X( ν) ] k Apply Markov property, whch s that the state of the system depends only on the last state. PXt ( () X( ν) ) PXt [ () Xu ( ) k]pxu [ ( ) k X( ν) ] k (13) (14) (15) 1.3 Kolmogorov s equatons: Brngng n the transton rate equatons (10) and (11) nto the Chapman-Kolmogorov equaton, we derve Kolmogorov s equatons: p t + h) p k u)p k ( ut, + h) 0 ν < u< t k I p t + h) p t) p k u) ( p k ( ut, + h) p k ( ut, )) k I (16) (17) Dvdng both sdes by h and takng the lmt as h 0 and u t, we get 3
p t) p t k u) ( pk ( ut, )) t The above s called Kolmogorov s forward equaton. Smlarly, Kolmogorov s backward equaton s: k I u t p k t)q k p t)q (18) p t) p ν k t)q k ( ν) p t)q ( ν) (19) Combnng (18) wth (7), we get a dfferental equaton for the state probabltes: dp -------------- dt p t)p t ( ν) P ( ν) p k t)q k p t)q (20) Flppng the order of the summatons: dp -------------- q dt k p k t)p ( ν) q P ( ν)p t) (21) dp Usng (7), -------------- q (22) dt k ()P t k q ()P t A Markov process s hence qute general - the q could be functons of tme; t could be nondecreasng (countng) or a process that ncreases and decreases. And hence M n M/M/1 should stand for Memoryless not Markov. The arrval process s Posson, whch s a specal case of Markov. 1.4 Tme-homogeneous Markov chans (3, page 361, 363): A Markov chan { Xt ()t 0} s sad to be tme-homogeneous f p t) depends only on the tme dfference t ν, where p t) PXt [ () Xν ( ) ]. In other words, p PXt [ ( + u) Xu ( ) ] for any u 0. (23) Or, equvalently n a tme-homogeneous Markov chan PXt [ () Xt ( n ) ] PXt [ ( t n ) X( 0) ] for any t n. (24) See defnton of strct-sense statonary processes where the ont CDF of random varables Xt ( 1 ), Xt ( 2 ), Xt ( m ) stays the same as the tme range t 1, t 2, t m shfts by a scalar τ. Pg. 275 of 4
[3] states that statonarty of the condtonal dstrbuton does not mply statonarty of the ont dstrbuton functon. Therefore, a tme-homogeneous Markov process need not be a statonary SP. In tme-homogeneous Markov chans, the transton rates q and q are ndependent of tme. We can see ths from (8) and (9). If p ( tt, + h) s ndependent of t n tme-homogeneous Markov chans, q s ndependent of t. In tme-homogeneous MCs, we can rewrte the Kolmogorov s forward equaton (18) and state probablty dfferental equaton (22) as follows: dp --------------- p dt k ()q t k p ()q t and (25) dp -------------- q dt k P k q P (26) 1.5 Why a Posson process s a specal case of a CTMC For example, n Posson process defnton, we see a smlar dependence on ust the nterval, and λ beng ndependent of tme. A Posson process s a pure-brth tme-homogeneous MC n whch p and the ntal state of the system s PXt [ ( 0 ) k] λt + 1 1 λt 0 otherwse ( λt 0 ) k e λt 0 ------------------------- k 012,,, k! (27) (28) 5
See Fg. 1 that shows a Posson process as a pure-brth CTMC. Transton probablty p 01 t) λ λ λ λ λ 0 1 2... n n+1... Fgure 1:Posson process as a pure-brth CTMC depends only on t ν (number of arrvals n a Posson process wthn an nterval only depends on the nterval). Therefore a Posson process s a tme-homogeneous process. Snce the number of arrvals wthn an nterval t ν s a Posson rv, p 01 ( t ν) PN [ 1], where N s the number of arrvals. PN [ 1] λ( t ν)e λ( t ν). Rewrtng p 01 ( t ν) as p 01, we can equate t to λt usng the Taylor s seres approxmaton for e λt f t s very small. p 00 can be smlarly approxmated to 1 λt. In CTMCs, the arcs are marked wth the transton rates. Usng (8) and (9), we can fnd the transton rates correpondng to the transton probabltes from (27) for the Posson process: q ( + 1) λ λ 1.6 Irreducble MCs and Global Balance equatons q q 0 for other cases Even n the tme-homogeneous case, t s hard to fnd tme-dependent probabltes, but for a specal knd of tme-homogeneous Markov chan, we can fnd the steady-state probabltes. Defntons: A state s reachable from state f for some tme t > 0, p > 0. A CTMC s sad to be rreducble f every state s reachable from every other state. Theorem 8.1 of [3], page 363: For an rreducble CTMC, the lmts p lm p lm P for, I (29) t t 6
always exst and are ndependent of the ntal state. If these lmts exst, then dp lm -------------- 0 dt t (30) Substtutng nto (26), we get dp lm -------------- dt t lm t q k P k + q P (31) p k q k q p 0 (32) Ths s the global balance equatons. Combned wth the equaton p 1, we can solve for p, steady-state probabltes of beng n the states. Does an rreducble MC have to be tme-homogeneous? Irreducble Markov chans that yeld postve lmtng probabltes { p } are called recurrent non-null or postve recurrent. The number of transtons n any fnte length of tme s fnte wth probablty 1 - chans wth ths property are referred to as regular. Most queueng systems have underlyng regular Markov chans. 2. DTMC: [3] Defnton of a DTMC: A DTMC { X n n 012,,, } s a dscrete tme dscrete value random sequence such that gven X 0, X 1,, X n 1, the next random varable X n depends only on X n 1 through the transton probablty PX [ n X n 1, X n 2 n 2,, X 0 o ] PX [ n X n 1 ] (33) The pmf of the random varable X n s p ( n) PX ( n ) (34) The transton probablty of the DTMC s defned as p k ( mn, ) PX ( n k ( X m ) ) 0 m n (35) 7
2.1 Transton probabltes and transton probablty matrx n DTMCs For (tme)-homogeneous DTMCs, p k ( mn, ) depends only upon the dfference n m. For such chans, The probablty p k ( n) PX ( m + n k ( X m ) ) for any m 0. (36) s called an n-step transton probablty,.e., t s the probablty that the MC wll move from state to state k n exactly n steps. If n 1, the 1-step transton probablty: p k ( n) p k ( 1) PX ( m + 1 k ( X m ) ) for any m 0. (37) We can also defne a 0-step transton probablty: p k ( 0) 1 f k 0 otherwse (38) Usng the generalzed multplcaton rule to obtan the ont probablty: PX ( 0 o, X 1 1,, X n n ) PX ( n n X n 1 n 1,, X 0 0 )PX ( n 1 n 1,, X 0 0 ) PX ( n n X n 1 n 1 )PX ( n 1 n 1,, X 0 0 ) PX ( n n X n 1 n 1 )( PX ( n 1 n 1 X n 2 n 2,, X 0 0 )) PX ( n n X n 1 n 1 )( PX ( n 1 n 1 X n 2 n 2 ))PX ( n 2 n 2,, X 0 0 ) p n n 1 p n 1 n 2 p 1 0 P 0 ( 0) In other words, as wth the CTMC, f we have the ntal pmf P 0 PX ( 0 0 ) and the one-step transton probabltes p, we can determne all the ont probabltes. The pmf of X 0 s called the ntal dstrbuton, specfed by a probablty vector p( 0) [ p 0 ( 0), p 1 ( 0), ] (39) The one-step transton probabltes s specfed n the form of a transton probablty matrx as: P p 00 p 01 p 02 p 0 p 1 p 2 (40) 8
2.2 Chapman-Kolmogorov equaton for DTMC Property of a DTMC: The Chapman-Kolmogorov equaton provdes a way to compute the n- step transton probabltes. p ( m + n) p k ( m)p k ( n) mn, 0, 0 k 0 Proof: Use the theorem of total probablty: PX [ m + n X 0 ] PX [ m + n X m k, X 0 ]PX [ m k X 0 ] k 0 Apply Markov property, whch s that the state of the system depends only on the last state. PX [ m + n X 0 ] PX [ m + n X m k]px [ m k X 0 ] k 0 (41) (42) (43) p ( m + n) p k ( m)p k ( n) mn, 0, 0 k 0 In the explanaton for the Chapman-Kolmogorov equaton for CTMCs we ddn t need them to be homogeneous. Same goes for DTMCs. The above proof s for the tme-homogeneous case, but t can be proven ust as easly for non-homogeneous DTMCs. We see that whle Chapman-Kolmogorov equaton holds for both CTMC and DTMC, the Kolmogorov forward and backward equatons are for transton rates and hence applcable only for CTMCs. (44) The matrx of n-step transton probabltes s obtaned from (44) by replacng m by 1 and n by n 1 and notng that p ( n) s an entry n the matrx P( n) as: P( n) P P( n 1) P n To obtan the pmf of random varable X n, usng (34), we get p ( n) PX ( n ) PX ( n X 0 )PX ( 0 ) p ( n)p ( 0) (45) If the pmf of X n s expressed as a vector: p( n) [ p 0 ( n), p 1 ( n),, p ( n), ], 9
where p ( n) s the probablty that the system s n state after n transtons, then p( n) p( 0)P( n) p( 0)P n (46) P( n) s a stochastc matrx because ts row sums are equal to 1,.e. 2.3 Irreducble DTMCs Defntons from [4]: p ( n ) 1 for all. Two states and communcate f for some n and n', we have p ( n) > 0 and p ( n' ) > 0. Irreducble Markov chan: If all states communcate, we say that the MC s rreducble. Perhaps we can see here that rreducble MC do not have to be tme-homogeneous. Couldn t we say that f for some m, n, m', n', p ( mn, ) > 0 and p ( m', n' ) > 0 then the non-homogeneous MC s rreducble? A state of a MC s perodc f there exsts some nteger m 1 such that p ( m) > 0 and some nteger d > 1 such that p ( n) > 0 only f n s a multple of d. A MC s aperodc f none of ts states s perodc. If p ( n) becomes ndependent of n and, then from (45), we see that p ( n) approaches a constant as n. Not all MC exhbt ths behavor. To determne whch ones do we need to classfy states of a MC [3]. We note that the n -step transton probabltes p ( n) of fnte, rreducble, aperodc MCs become ndependent of and n as n. Let q lm p ( n) n ν lm p ( n) lm p ( 0) p ( n) p ( 0) ( lm p ( n) ) p ( 0)q q p ( 0) n n n q In other words, ν lm p ( n) lm p ( n) q (47) n n Ths mples that P n converges to a matrx V, wth all rows beng equal to ( ν 0, ν 1, ) as n. From [4], page 259-260: 10
Ether ν 0 for all (whch happens n a queueng system when arrval rate exceeds the servce rate, n whch case the number of customers n the system approaches ; therefore the steady-state probablty of any state wth a fnte number of customers s 0 - example from [4], page 260), OR ν > 0 for all, n whch case { ν 0} s the unque statonary dstrbuton of the MC,.e. t s the only dstrbuton that satsfes the equaton ν ν p 01,, and ν 1 (see below for dervaton) (48) 0 The vector V s called the steady-state probablty vector of the MC. s nterpreted as the long-run proporton of the tme the MC spends n state. Dervaton of (48) s from [3], page 320: From the theorem of total probablty, ν From (47), we have p ( n) p ( n 1)p ( 1) (49) ν lm p ( n) lm p ( n 1). (50) n n Therefore, takng the lmt as n tends to nfnty of both sdes of (49), ν ν p, where we have replaced the 1-step transton probablty p ( 1) by the smpler notaton p. Note that whle ν s the steady-state probablty of beng n state, the probablty p s the one-step transton probablty. Theorem on pg. 260 of [4] states: In an rreducble aperodc MC, there are two possbltes for the scalars lm ( n), 1. q 0 for all 0, n whch case the MC has no statonary dstrbuton 2. q > 0 for all 0, n whch case { q 0} s the unque statonary dstrbuton of the chan (.e., t s the only probablty dstrbuton satsfyng (48). q p n 11
NOTE: The above theorem does not need the MC to be fnte. 2.4 Global balance equatons [4]: Multply both sdes of the equaton p 1 by ν and replace ν on RHS usng (48). We get the global balance equatons. 0 ν 0 p ν p 01,, 0 (51) These state that frequency of transtons out of state s equal to the frequency of transtons nto state. The long-term frequency of transtons from state to state s ν p. The global balance equatons can be generalzed to apply to a set of states. Consder a subset of states (51) over all S, we get: ν p S S 2.5 Detaled balance equatons for brth-death DTMCs ν p S S S By addng (52) A brth-death MC s one where two successve states can only dffer by unty,.e., p 0 f ( > 1) and P ( + 1) > 0 and p ( + 1) > 0 for all. Let the set of states S { 012,,,, n} as shown n Fg. 2. Applyng (52) yelds ν n p nn+ 1 n 01,, ( ) ν n + 1 p ( n + 1)n (53) Fgure 2:MC for a brth-death process... n n+1... Set S Equaton (53) s a specal caseof ν p ν p, 0 (54) 12
Queston: Do the detaled balance equatons only apply for brth-death MCs or other MCs too? [4] (page 262) says assume t holds and solve the MC. Ether an nconsstency wll be seen where ν 1 0 or you get a steady-state soluton. Compare global balance equatons of CTMC wth that of DTMC. Does the detaled balance equaton hold for a brth-death CTMC? 3. Relaton between Markov property and the memoryless property The relaton between these two propertes appears n a tme-homogeneous CTMC, where the holdng tme n a state has a dstrbuton that s memoryless (exponental). For a tme-homogeneous CTMC, f the state of the system Xt (), then the condtonal probablty of the tme Y spent n state can be characterzed as follows: PY ( r+ ty t) PY ( r) (55) To understand the above, consder t and t+ s as two tme nstants wthn the holdng tme of state, such that 0 s r. In ths case, because the CTMC s tme-homogeneous, PXt [ ( + s) Xt () ] PXs [ ( ) X( 0) ]. (56) From (56), we see that (55) follows. For a homogeneous MC, the past hstory of the process s completely summarzed n the current state ([1], page 296). For a non-homogeneous MC, t addtonally depends on tme spent n the state? If (55) holds, then PY ( r) Pt ( Y t+ r) F ------------------------------------ Y ( t + r) F Y or F (57) PY ( t) Y ( r) ---------------------------------------- 1 F Y Dvde by r and take the lmt as r 0, we get F' Y ( 0) F Y '() t ---------------------. (58) 1 F Y The above dfferental equaton has the unque soluton: F Y 1 e F' Y'0 ( )t, whch s the exponental dstrbuton. (59) 13
In other words, the tme spent n a gven state s exponentally dstrbuted for a tme-homogeneous MC (also referred to as a homogeneous MC). If the MC s a DTMC, the holdng tme dstrbuton s geometrc. Problem 5 n [1], page 421: Let the random varable denote the holdng tme n state k of the M/M/1 queue. Show that the dstrbuton of s exponental (for each k ). T k T k Answer: Consder T 0 frst. The only event leadng out of state 0 s a ob arrval. Snce ob nterarrval tmes are exponentally dstrbuted, tme to the next arrval s also EXP( λ), and hence T 0 EXP( λ). Next, consder the holdng tme n any state k, k 1. The tme X to the next arrval s EXP( λ), and tme Y to the servce completon of the ob n servce s EXP ( µ ). Clearly, T k mn{ X, Y}. If X and Y are exponentally dstrbuted random varables, then so s T k. Ths s because of the order statstcs result [1], page 157. Compute varables Y 1, Y 2,, Y n by reorderng the random varables X 1, X 2,, X n such that Y 1 mn{ X 1, X 2,, X n }, and (60) Y n max{ X 1, X 2,, X n }. (61) Y k s called the k-th order statstc. To derve the dstrbuton of Y k, exactly k le n the range (, y ] and n k le n the range [ y, ). F Yk ( y) PY ( k y) P( at least k of the X s le mn the nterval(, y ]) (62) F Yk ( y) n n F ( y) ( 1 Fy ( )) n < y < k (63) Therefore ( y) 1 [ 1 Fy ( )] n (64) F Y1 If all X s are greater than y (prob: [ 1 Fy ( )] n ), then Y 1 should be greater than y too. Applyng ths result to our problem of mn{ X, Y}, where X EXP( λ) and Y EXP ( µ ), we get F Tk T k ( y) 1 e λt e µt ( 1 e λ + µ )t. In other words, T k EXP( λ + µ ). But the above result s for an M/M/1 queue. It does not prove that ths holds for any tme-homogeneous CTMC. 14
For a tme-homogeneous DTMC, we can derve the dstrbuton of tmes between state changes as follows [1], page 356: Let the random varable represent the tme the system spends n state k durng a sngle vst to that state. T k n 1 P( T k n) ( 1 p kk )p kk, for all states k (65) Ths s a geometrc dstrbuton wth parameter p kk (one-step probablty). Apply same reasonng to a tme-homogeneous CTMC: q s the transton rate assocated wth stayng n state. Probablty of stayng n state from tme t to tme t + y s P( x < T < x + dx) lm ( 1 q ) 0 -- x ( q dx) (66) from (9) but for a tme-homogeneous system whch means q s ndependent of tme y PT ( y) lm ( 1 q ) q d x 1 e q y 0 0 -- x lm h 0 ( 1 + h) 1 -- h e ; set h q PT ( y) 1 e q y Interestng that ths holdng tme dstrbuton holds at any tme t - meanng t apples even n the transent state. H k ( t 1 ) s a random varable: start a clock as soon as the system enters state k and stop t when t leaves. Ths varable has ths exponental dstrbuton (memoryless property). It s not necessary that we only consder the holdng tmes n a state n the steady state for ths memoryless property to hold. The above leads to another defnton of CTMC: Reference [4] gves a defnton for CTMC that only holds for tme-homogeneous MCs. It states (on page 262): 15
A CTMC s a process { Xt ()t 0} takng values from the set of states 01,,, that has the property that each tme t enters state : 1. The tme t spends n state s exponentally dstrbuted wth parameter ν. We may vew ν as the rate (n transtons/sec) at whch the process makes a transton when at state. 2. When the process leaves state, t wll enter state wth probablty P, where P 1. It goes on to defne the transton rate q from state to state as q ν P. References [1] K. S. Trved, Probablty, Statstcs wth Relablty, Queueng and Computer Scence Applcatons, Second Edton, Wley, 2002, ISBN 0-471-33341-7. [2] R. Yates and D. Goodman, Probablty and Stochastc Processes, Wley, ISBN 0-471-17837-3. [3] K. S. Trved, Probablty, Statstcs wth Relablty, Queueng and Computer Scence Applcatons, Frst Edton, Prentce Hall, ISBN 0-13-711564-4r. [4] D. Bertsekas and R. Gallager, Data Networks, Prentce Hall, Second Edton, 1992. 16