Couous me Markov chas have seay sae probably soluos f a oly f hey are ergoc, us lke scree me Markov chas. Fg he seay sae probably vecor for a couous me Markov cha s o more ffcul ha s he scree me case, bu he marx equao ha we use s, a leas o he surface, sgfcaly ffere from ha use for scree me chas. Ulmaely, however, he mehos come ow o he same basc ea for a Markov cha, scree or couous me, o have a seay sae soluo, he rae a whch he cha makes rasos o ay gve sae mus equal he rae a whch he cha makes rasos ou of ha same sae. Ths oo, as we wll see, s explc he marx equao for couous me chas. Thk of erms of pourg waer o a bucke wh a hole he boom. If we pour waer o he bucke more slowly ha ca ra hrough he hole, he bucke ever has ay waer. If we pour faser ha he waer ca ra, he level of he waer he bucke keeps geg hgher a hgher (ul fally sars flowg over he ses, bu we'll assume we have a fely eep bucke). For a Markov cha, he "level of waer he bucke" s aalogous o he probably of beg a sae. A sable waer level s aalogous o he probably for ha sae havg a seay sae value. Oe mpora sco bewee scree me a couous me Markov chas s ha he laer are, by efo, aperoc. Ths s because he me bewee sae chages s expoeally srbue, makg mpossble o resrc sae chages o occur oly a regularly space ervals. However, he more mpora sco has o o wh how we eal wh he sae raso probables. The ea of a "sgle sep raso probably" o loger makes sese, sce we o' have he oo of a sep. Defe h (, ) Pr[ S( ) = ks( ) = ],. H, h, [ ] ( ) ( ) For a homogeeous Markov cha, he raso probables are fucos oly of he fferece : ( ) = ( ) s H, H 0,,... 0 All of he Markov chas we coser wll be homogeeous, uless sae oherwse. Assume ha = s small. Page of 7
Elec 48 Recall ha a couous me Markov cha, he me bewee sae rasos s memoryless a hece s expoeally srbue. Ths mples ha he raso mes are geerae by a Posso process. A Posso process ca be efe several ways. We wll use he followg se of axoms:. Pr[ eve a erval of legh 0]= λ.. Pr[ 0 eves a erval of legh 0]= λ o( ( ) ). 3. Eves are epee. The frs axom says ha he probably of oe eve occurrg a very shor erval s proporoal o he legh of he erval. The seco axom saes ha he probably of o Posso eves occurrg urg a very shor erval s oe mus he probably of oe eve mus a erm whch s o. The "lle o" (( ) ) = (( ) ) oao meas ha lm o 0. Noe ha λ has us of probably ve by me, or rae of chage of probably. Defe P Pr eves a erval of legh. Here may be arbrarly large. We ca eerme hese probables for all values of by sarg a = 0 a workg up. () [ ] ( ) = () ( ) P + P P 0 0 0 = P0 () [ λ o( ( ) )] The las subsuo reles o beg very small. Mapulag hs equao a vg boh ses by, ( ) P P o 0( + ) 0() ( ) = λp0 () P Takg he lm of boh ses of he equao as 0, we ge { P0() } = λp0() The soluo o hs lear, frs-orer, me-vara ffereal equao s λ P0( )= ke 0 () Page of 7
Elec 48 for some cosa k. To eerme k, oe ha he probably of 0 eves 0 me s : λ 0 P 0 ke k 0 ( ) = = = Now coser =. ( ) = () ( )+ () ( ) P + P P0 P0 P = P () [ λ]+ e λ (( ) ) Noe ha we have roppe he o from he expresso for P0 ( ). We are gog o be playg he same game as before (vg by a akg he lm as 0), a he o wll sappear ayway. (( ) ) ( ) () = () P + P λe λp P + P lm λe λp P () λe λp ( ) () = () whch has he soluo λ P( )= λe I geeral, for, { } = () ( ) = () ( )+ () ( ) P + P P0 P P = P () [ λ]+ P () λ { P () } = λp () λp () P λ e! ()= ( ) λ You ca verfy hs soluo by subsug he ffereal equao. Ths shows ha he probably of eves a erval of legh has a Posso srbuo, wh λ he rae or parameer of he srbuo. Accorgly, he probably of eves s proporoal o ( ) for small (us use a power seres expaso for e λ ). Page 3 of 7
Elec 48 To show he coeco bewee he Posso process a he expoeal srbuo, we ee o prove ha he me bewee Posso eves s expoeally srbue. Le X be he raom varable for he me bewee wo cosecuve eves of a Posso process. Pr[ X ]= Pr[ X > ] = Pr[ 0 eves ]= P0 ()= e F = () X FX () s he cumulave srbuo fuco for a expoeally srbue raom varable X. Reurg o he scusso of homogeeous Markov chas, le p () be he sae probably vecor a me. Because he probably of eves s proporoal o ( ) for small, he probably of as oppose o more ha eve omaes as ges close o 0 (o surprsg - s a Posso process, afer all). Hece, as 0, lm H (, + ) = lm H( 0, ) ca be hough of as he sgle-sep raso marx for he couous me Markov cha. The lm p () H( 0, ) = lm p ( + ) () ( ) () ( + ) () = p H0, p p p lm lm H( 0, ) I p () lm = { p () } 0 p () Q= { p ( )} H( 0, ) I where Q = lm s he raso rae marx or rae geeraor marx or smply he geeraor marx of he homogeeous couous me Markov cha. The off-agoal elemes of Q are q h = lm ( 0, ), k As oe above, hese are o probables; hey are saaeous raes of chage probably. Because he cha s homogeeous a mus be memoryless, lm h 0, λ ( ) = Page 4 of 7
Elec 48 a hece q = λ λ s he rae of he Posso process goverg rasos from sae o sae k. Wha are he q s, he agoal elemes of he geeraor marx? q h ( 0, ) = lm ( ) s he probably ha he Markov cha s sae a me, gve h 0, ha was sae a me 0 (or, sce hs s a homogeeous cha, s he probably ha he Markov cha s sae a me +, gve ha was sae a me ). Sce he cha mus be some sae a me, gve ha was sae a me 0, a h 0, h 0, ( ) = ( ) = k lm h 0, lm h 0, 0 = k where λ ( ) = ( ) = = k λ = λ = λ = k s he sum of he raes of he Posso processes goverg he rasos ou of sae. Subsug hs o he expresso for q a aga usg he homogeey of he Markov cha, we ge q λ = lm = λ { } be a se of epee, Take a mome o coser λ. Le X, K, X expoeally srbue raom varables wh raes λ, K, λ, a le X m { X, K, X }. Page 5 of 7
Elec 48 [ ] [ ] Pr[ X > ]= Pr X > & X > & K& X > = Pr[ X > ] Pr[ X > ] K Pr X > λ = e e K e = e where λ = λ+ λ + K + λ. Hece X s also expoeally srbue, wh parameer (rae) λ. Ths says ha λ s he rae of a expoeally srbue raom varable ha s he mmum of he raom varables represeg he mes ul rasos from sae o all oher saes. Tha s, λ s he rae of a expoeally srbue raom varable ha represes he me spe sae. Oe way o look a s ha λ s he rae a whch probably mass "leaves" sae. We're almos here. So far, we have { } p() Q = p () We are erese he seay-sae probably vecor π = lm () { } = () { } = { } = π Q = lm p () lm p π 0 p. sce he ervave of he seay-sae probably vecor s by efo he 0 vecor. The marx equao π Q = 0 for couous me Markov chas s he aalog of π P = π for scree me Markov chas. The wo marces are que ffere. The elemes of P are probables; he elemes of Q are raes of chage probably. However, boh marces are sgular, sce each row of P sums o a each row of Q sums o 0. I he scree case, π P = π ca be erpree as meag ha he rae a whch he cha makes rasos "o" each sae s equal o he rae a whch he cha makes rasos "ou of" sae. The quoes are because we are clug rasos from sae o sae boh cases. Tha s, aoher way of lookg a he Chapma-Kolmogorov equao for sae s o rewre as: ( ) p π + p π + p π + K= p + p + p + Kπ 3 3 3 where he expresso pareheses o he rgh ha se clues p a he sum o he lef ha se clues he erm p π. The expresso pareheses o he rgh ha se sums o, gvg us he saar form of he Page 6 of 7
Elec 48 Chapma-Kolmogorov equao for hs sae. Sce he scree case, he cha makes a raso o every sep, eve f s a raso back o he same sae, π P = π ca also be erpree as a rae balace equao. I equaes he rae a whch rasos are mae o sae wh he rae a whch rasos are mae ou of sae. Page 7 of 7