Contnuous Tme Markov Chan Hu Jn Department of Electroncs and Communcaton Engneerng Hanyang Unversty ERICA Campus
Contents Contnuous tme Markov Chan (CTMC) Propertes of sojourn tme Relatons Transton probablty functon Transton rate matrx Kolmogorov's forward and backward dfferental equatons Propertes of CTMC Irreducblty Recurrence Tme average Statonary dstrbuton Lmtng dstrbuton 2
Defnton Introducton A contnuous-tme stochastc process {X t, t } wth a countable state space S s sad to be a contnuous-tme Markov chan (CTMC) f t satsfes the Markov Property:,, P X j X X x u s P X j X ts s u u ts s for any, j, x S, u s, and all s, t. u The condtonal dstrbuton of any future state X t+s gven the past states and the present state X s s ndependent of the past states and depends only on the present state. Tme homogeneous (nvarant) CTMC: P{X t+s = j X s = } s ndependent of s. 3
Introducton Y T Y 2 T 2 Y 1 T 1 1 2 3 tme 4
Transton Probablty Functon Let P(t) = (p j (t)) where p j (t) = P{X t+s = j X s = }, whch s called the transton probablty functon from state to state j. The ntal dstrbuton and the matrx P(t) unquely determne the future behavor of the CTMC. For t t... t n, 1 1,, n t 1 1,, 1 1,, n n tn n tn n t 1 1 1 1,, n n tn n tn n n n1 t n1,, n1 n n1 p t t p t t p t PX n1 n 1 2 P X X X t n t n n n1 n P X X X P X X P X X P X X p t t P X X n1 n2 n1 n2 1 1 5
Notatons for a CTMC The tmes of jumps (state transtons): τ = < τ 1 < τ 2, The sojourn tmes: T n = τ n+1 - τ n, n The sequence of states vsted: Y = X, Y 1 = X τ1, Y 2 = X τ2, Y T Y 2 Y 1 T 1 T 2 1 2 3 tme We assume that the sample path s rght contnuous. 6
Propertes of the Sojourn Tme Based on the homogeneousness and the Markov property, we have, P T t s T s X P T t X Memoryless property of the sojourn tme n CTMC. Let g (t) = P{T > t X = }. Then we have PT t s T s, X PT s X condtonal probablty PT t X PT s X Markov property g t g s g t s P T t s X The only functon that satsfes the above equaton s g t e qt for some constant q. Note that g 1 The sojourn tme s exponentally dstrbuted. 7
Propertes of the State Transton For j, we have 1, 1, PY j X PT t X Markov property P Y j T t X P Y j T t X P T t X re where r j = P{Y 1 = j Y = }. j 1 qt Frst jump probablty matrx: R = (r j ) It s also the one step transton probablty matrx of the embedded DTMC {Y n } where the embedded ponts are the jump tmes. Note that the dagonal elements of R are all. 8
Instantaneous state A state for whch q =. Absorbng state A state for whch q =. A Regular CTMC A CTMC s sad to be regular (non-explosve) f, wth probablty 1, the number of transtons n any fnte length of tme s fnte. Example of a non-regular CTMC r q, 1 1, 2 Suffcent condton for a regular CTMC sup q S 9
Transton Rate Matrx Transton rate from to j : q r q j j j Defne q j rjq f j q f j Transton rate matrx: Q=(q j ) q q1 q2 q r1q r2q q1 q11 q12 r1q1 q1 r12q1 Q q2 q21 q 22 r2q2 r21q2 q 2 Note that we have Qe= where e s a column vector whose elements are all 1. 1
Propertes of q j ( j) The probablty of two or more transtons n tme t s o t. j h PY j, T h, T T h X oh p h P X j X h 1 1 qt,, P Y j T T h T t X q e dt o h 1 1, j Y P T h t T t X P Y 1 1 h q ht qt j j j h j e r q e dt o h qh j j e r q e dt o h j j h q q t r q r q e j qh q e q q q jh oh q r q h o h qh j q j o h o h qt q e dt o h 11 o h lttle o functon: lm h h
h PT h X oh qh e oh 1 q h oh Propertes of q The probablty of two or more transtons n tme t s o t. p h P X X 12
We have Propertes of q j 1 1 p h r q h o h q h o h j j j j p h q h o h q h o h q : transton rate from state to state j q j : transton rate of leavng state Therefore, we have lm h lm h p j h h p 1 h q j h j q 13
Basc Equatons for a CTMC Chapman - Kolmogorov's equaton p t s p t p s j k kj ks Kolmogorov's backward dfferental equaton d d d P t pj t qk pkj t,.e., t t. dt dt P QP ks dt Kolmogorov's forward dfferental equaton d d d P t pj t pk tqkj,.e., t t. dt dt P P Q ks dt 14
Kolmogorov's Forward Dfferental Equaton From Chapman-Kolmogorov's equaton, we have p t s p t p s j k kj ks Consequently, j j k kj j k kj jj j ks ks, k j. p t s p t p t p s p t p t p s 1 p s p t. Hence, s p 1 j t s pj t pkj s p jj lm lm pk t p s s s ks, k j s s d dt t t. P P Q ks, k j ks, k j ks p k s p pk tlm p t s kj j s s p t q p t q k kj j jj t q kj. j 1 p lm s jj t s 15
Example: The Two-State Chan A CTMC spends an exponental tme wth rate λ n sate before gong to state 1 It spends an exponental tme wth rate μ n state 1 before gong to state 1. Q Compute p (t) and p 11 (t) 1 16
Example: The Two-State Chan From Kolmogorov's forward equaton, we have p t p t p t p ' t p t p t 1 Note that 1 1 e p ' t p t e t t d t e p t e t dt t t e p t e c 17
Example: The Two-State Chan t p t ce p 1 p t e In addton, we can observe lm p t. t By symmetry, we also have p11 t e t t 18
Tme Dependent Probablty From Kolmogorov's forward dfferental equaton, the vector P(t) s gven by P(t) = P() e Qt = e Qt (Note that P() = I), where the matrx exponent functon e Q s defned by e Q = I + Q + Q 2 /(2!) + = n= Q n /(n!) 19
Irreducblty and Recurrence Irreducblty A CTMC {X t } s rreducble f any one n the followng holds: The embedded DTMC {Y n } s rreducble For any,j S, we have p j (t) > for some t >. For any,j S, we have p j (t) > for all t >. Recurrence Let nf t X, lm t X s st A state s recurrent (transent) f 1 1 P X 2
Statonary Measure = (, 1, )( ) s a statonary measure f and P(t) = for all t. Suppose that a CTMC {X t } s rreducble and recurrent. Then there exsts the unque statonary measure up to a constant multplcaton. 1., for all S 2. For any state, f we defne E I X s j ds X then ther exsts aconstant c such that π= cπ. Note that j E X j 3. j where q j j π j s the statonary measure for the embedded DTMC Yn js 21
Statonary Dstrbuton The statonary measure s, n fact, a soluton of Q =. Proof: From the Kolmogorov's forward dfferental equaton, we have πp d t π t t dt πp πp Q πq The thrd property n the prevous slde From πq, for any S we have q q q q q r j j j j j j j js js, j js q s a statonary measure of the embedded DTMC Y S n whose transton probablty matrx R= r wth r j. q q j j js, j rate out = rate n Global balance equaton 22
Postve Recurrence A recurrent state s called postve recurrent (null recurrent) f E[τ() X = ] < (= ). Therefore, an rreducble and postve recurrent CTMC has the unque statonary dstrbuton: 1 E X 23
Ergodcty For a CTMC, we don't need to consder the perodcty because all the sojourn tmes are exponentally dstrbuted. An rreducble postve recurrent CTMC s called ergodc. If an rreducble regular CTMC {X t } has a nonnegatve numbers satsfyng Q = and e = 1, then {X t } s ergodc and hence t has the unque statonary dstrbuton. If a CTMC {X t } s ergodc and s the statonary dstrbuton, then for all, j S we have lm p j t j t 24