MARKOV CHAINS. 5. Recurrence and transience. and the length of the rth excursion to i by. Part IB Michaelmas 2009 YMS.

Transcription

1 Part IB Mchaelmas 009 YMS MARKOV CHAINS E-mal: 5. Recurrence transence Recurrence transence; equvalence of transence summablty of n- step transton probabltes; equvalence of recurrence certanty of return. Recurrence as a class property, relaton wth closed classes. Smple rom wals n dmensons one, two three. the length of the rth excurson to by S r) = T r) When r =, we smply wrte T ) + when T r) T r ), f T r ) <, 0, f T r ) =. = +. See the dagram below. X n = T ; n addton, T r+) 5.3) s set to equal Gven a state I, we call t recurrent R) f P Xn = for nfntely many n ) =, the seres p n) dverges 5..) transent T) f P Xn = for nfntely many n ) = 0, the seres p n) converges. 5..) The last condton means that P Xn = for fntely many n ) =. ) S ) T ) T 4) T T T T S ) S 3) 3) S 4) S 5) 5) 6) S 6) tme The symbol means equvalence.) Note that all probabltes nvolved tae values 0 or but not from 0, ). The equvalence of propertes lsted n 5..) that of propertes lsted n 5..) s the subject of Theorem 5. below. Gven r = 0,,..., defne the rth passage tme T r) to state by T 0) = 0 T ) = nf [ n : X n = ], T r+) = nf [ n T r) + : X n = ], r, 5.) Next, defne the return probablty to state : f := P T < + ). 5.4) Ths probablty may be equal to or less than than one; n Theorem 5. t wll be proved that state s R ff f = 5.5.) state s T ff f < 5.5.)

2 Therefore, every state s ether R or T. In Theorem 5. we wll show that recurrence or transence are class propertes: two states from the same class are ether both R or both T. Before we proceed further, let us analyse the structure of the seres n 5..) 5..): where p n) p n) = P X n = ) = E X n = ) n n 5.6) = E X n = ) = E V n V = # of vsts to = tme spent n = X n = ). 5.7) Then equatons 5..) 5..) can be wrtten as ) nfntely = +, P X n = for many tmes = E V fntely < +. We see that, for a Marov chan, ) = + P V < + E V = + < + the case where P V < + ) = but E V = + does not occur. Theorem 5. dchotomy: p n) 5.8) 5.9) In a λ, P)-Marov chan, state I we have the = + P V = + ) = f = P T < + ) = : R = E V < + P V < + ) = f = P T < + ) < : T 5.0) Therefore, every state s ether recurrent or transent. 3 Proof property, The rom varable T s a stoppng tme. By the strong Marov P V ) = P X n = for at least two values of n ) = f, 5.0) more generally, PV ) = P X n = for at least values of n ) = f. 5.) Let us hghlght the event under consderaton: B ) := V } = Then, obvously, events B ) X n = for at least values of n }. are decreasng wth : B ) B )..., the event V = + } } = X n = for nfntely many values of n s the ntersecton B ). Hence, Next, QED P V = + ) = P Xn = for nfntely many n ) ) = P B ) = lm PB ) ) = lm f = p n) = E V = r 0 P V r) = r 0 f r, f = 0, f <. = +, f =,.e., P V = + ) =, < +, f <,.e., P V = + ) = 0. 5.) 5.3) Theorem 5. wll be repeatedly used n the analyss of recurrence transence of states of varous chans. 4

3 An alternatve proof of Theorem 5. explots the probablty generatng functons of rom varable T. Set f n) = P T = n) = P X n = but X l for l =,..., n ), n, then f = lm z Fz). On the other h, p n) ths mples that f then 5.4) Fz)= F z)) = Ez T = n z n f n), z <, 5.5) = P X n = ) = f n) + f n )p f )p n ) ; 5.6) Uz)= U z)) = n Uz) = Fz) + Fz)Uz),.e. Uz) = p n) z n, z <, 5.7) Fz) Fz). Hence, the lmtng value lm Uz) s fnte f only f lm Fz) <. That z z s, 5.3) holds true f only f f <. Remar 5. We establshed that state s recurrent f only f P T < ) =,.e. the return tme to s fnte wth probablty. However, the mean E T can be fnte ot nfnte. Ths dvdes recurrent states nto two dstnct categores: postve recurrent null recurrent see later). For convenence we repeat the defnton of a communcatng class: Defnton 5. States, j I belong to the same communcatng class f p n) j > 0 p n ) j > 0 for some n, n 0. The communcatng classes form 5 a partton of the state space I, when I s countable, some of them may be nfnte, the number of communcatng classes can also be nfnte. Next, a communcatng class C s called closed f C, f j then j C. Fnally, we say that the chan s rreducble f t has a unque communcatng class automatcally closed). In other words, n an rreducble chan, the whole of the state space I s a sngle closed) communcatng class. Remar 5. Observe that f the state space I s fnte, the defnton of a transent state concdes wth that of a non-essental state.e., a state from a non-closed communcatng class). In other words, n the fnte case every state from a non-closed class s transent, every state from a closed class s recurrent. However, as we noted n Remar., n the case of a countable DTMC a closed class can consst entrely of transent states, whch are, from a physcal pont of vew, non-essental. It shows that n the countable case the concept of transence s more relevant than that of a closed communcatng class. Our am now s to prove that recurrence transence are class propertes. Ths means that f states, j le n the same communcatng class then they are ether both recurrent or both transent. We therefore could use Defnton 5.3 A communcatng class s called recurrent transent) f all ts states are recurrent respectvely, transent). Theorem 5. Wthn the same communcatng class, all states are of the same type. Every fnte closed communcatng class s recurrent. Proof Let C be a communcatng class. Then, dstnct, j C, > 0 p n) j > 0 for some m, n. Then r 0: p m) j p n+m+r) p m) j p r) jj pn) j p n+m+r) jj p n) j pr) pm) j, 5.8) as the RHS n each nequalty taes nto account only a part of the possbltes of return. 6

4 Hence, for r n + m, p r) p r) jj pn+m+r) p m) j p n) j p r) jj p n) j pr n m) p m) 5.9.) j. 5.9.) Then the seres r r Now let C be a fnte closed communcatng class, j C. Then, wth X 0 = j C, X n C n. Hence, state C vsted nfntely often, wth P j V = ) > 0. In other words, t cannot be that P j V = ) = 0 for all C: we run the chan for nfntely many tmes among fntely many states, mplyng that P j V = ) =.) By the Strong Marov property, C p r) jj converge or dverge together. P j V = ) = P j T < ) P V = ). Then the probablty P V = ) cannot be 0 hence should be ). Thus, state s recurrent. Then every state from C s recurrent. QED Defnton 5.. A transton matrx P a λ, P) Marov chan) s called recurrent transent) f every state s recurrent respectvely, transent). We conclude ths secton wth one more statement nvolvng passage, or return, tmes. Theorem 5.3 Non-examnable) If P s rreducble recurrent then each rom varable T j the passage tme to state j) s fnte wth probablty. That s, PT j < ) = j ntal dstrbuton λ. Proof By the Marov property PT j < ) = 7 λ P T j < ). Gven, tae m wth p m) j > 0. Wrte = P j Vj = ) P j Xn = j for some n m ) obvously, there s equalty here, but the nequalty wll also do). Further, P j Xn = j for some n m ) = p m) j P j Xn = j for some n m Xm = ) = p m) j P Tj < ) p m) j =. We see that each summ p m) j P Tj < ) must be equal to p m) j ; otherwse.e. when p m) j P Tj < ) < p m) j for some ) we would have that <. Therefore, P Tj < ) p m) j = p m) j,.e. P Tj < ) =. Ths s true, hence ntal dstrbuton λ. Also, t s true j. QED Example 5. Math Trpos, Marov Chans, Part IB, 004, H) Let P = p j ) be a transton matrx. What does t mean to say that P s a) rreducble, b) recurrent? Suppose that P s rreducble recurrent that the state space contans at least two states. Defne a new transton matrx P = p j ) by 0 f = j, p j = p ) p j f j. Prove that P s also rreducble recurrent. Soluton A Marov chan s called rreducble f t has a sngle communcatng class. Equvalently, every par of states, j communcate; that s, the n-step transton probablty p n) j > 0 for some n= n, j)). Another equvalent condton s that a fnte sequence of non-repeated states = 0,,..., m = j wth p l l+ > 0. A chan s called recurrent f state : P Xn = for nfntely many n ) =, 8

5 or equvalently, P Xn = for some n ) =. When the chan s rreducble, t s enough to chec that a state wth the above property. Also, f the chan s rreducble, t s recurrent f only f state, the mnmal non-negatve soluton to the equaton hp = h satsfyng h = has h j = j. If P = p j ) s rreducble then p < state unless the total number of states s one). Matrx P descrbes the Marov chan obtaned from the orgnal DTMC by recordng the jumps to the new state only; clearly t s rreducble. Formally, tae the sequence 0,..., m as above, then p l l+ > 0. Now chec the recurrence of P: f n the orgnal chan p = 0 then the return to state occurs n both chans on the same event, hence the return probablty to state wll be the same. If p > 0 then n the new chan, the return probablty s equal to p P return to after tme n the orgnal chan) = p p ) whch s. Alternatvely, h P = h f only f hp = h,.e. the solutons to both equatons are the same. Hence, the mnmal soluton to h P = h wth h = s the same as that to hp = h. Therefore, t s, the new chan s recurrent f only f the orgnal one s. Example 5. Consder a homogeneous brth death process, where p + = p, p = p,, p 0 = r, p 00 = r. It s often called a homogeneous rom wal on the set of non-negatve ntegers Z + = 0,,,...} wth a barrer at 0 absorbng when r = 0, reflectng when r =, or a rom combnaton of the two when r 0, )). Case : 0 < r 0 < p /. Here we have a sngle communcaton class whch s closed. That s, the chan s rreducble. For state = 0: the return probablty s f 0 = P 0 T 0 < + ) = r + r P H 0) < + ) condtonal on the st step = r + r h = r + r = : state 0 s R ) Hence, all states are R, the whole process s R. Case : 0 < r / < p <. Agan, the chan s rreducble. The return probablty to state = 0: f 0 = r + r h = r + r p p < r + r = : state 0 s T. 5.) The remanng cases are non-generc: ether p or r equal or 0. For brevty, consder just one of them: Case 3: 0 r p = 0: from state the process always jumps to whle from 0 t can jump to when 0 < r bac to 0 when 0 r < when 0 < r <, both possbltes occur). The chan s reducble: every state > forms an open CC so s T. For 0 < r : the par 0, } forms a closed CC, both = 0 = are R-states. For r = 0 state = s T state = 0 R. Example 5.3 Rom wals on lattces) Rom wals on cubc lattces are popular nterestng models of countable Marov chans. Here we have a partcle that jumps at tmes n =,,... from ts current poston Z d to another ste j Z d wth probablty p j, regardless of the past trajectory. We wll mostly focus on homogeneous nearest-neghbour rom wals where probabltes p j are > 0 only when j are neghbourng stes depend only on the drecton from to j.e. are determned by p 0,j where j s a neghbour of the orgn 0 = 0,..., 0)). For d = lattce Z d s smply the set of ntegers; a rom wal here s specfed by probabltes p q = p of jumps to the rght the left. q= _ p _ p 0 Ths s an ntutvely appealng extended verson of the drunard model 0 p

6 or brth--death process); see Example.. Here, the state space s I = Z= Z ), the transton probablty matrx s nfnte has a dstnct dagonal structure q 0 p P = q 0 p 0.., 5.) q 0 p wth entres p above q below the man dagonal, the rest flled wth zeros. The probablty p n) 00 s obvously zero when n s odd you can t return after an odd number of jumps). If n = s even then the return s equvalent to have exactly jumps to the rght jumps to the left. Hence, p ) 00 = )!!! p q. recurrent when p = /. As the model s homogeneous, the same holds for each state Z; one smply says that for p = / the one-dmensonal nearest-neghbour rom wal s recurrent for p / transent. The case p = / s called symmetrc. For d =, Z s a plane square lattce; here we wll consder the symmetrc nearest-neghbour rom wal where probabltes to jump n every drecton are the same equal /4. /4 0 =0,0) For large, by Strlng formula n! πnn n e n, )!!! = Hence, the seres p n) = n π ) e π e = π 4. p ) 00 converges or dverges together wth Every closed path on Z must have equally many jumps to the left the rght equally many jumps up down. ) 4p p). Now, the parabola p p p) reaches ts maxmum on [0, ] at p = /; the maxmal value equals /4. Hence, 4p p), wth equalty ff p =. Therefore the above seres dverges when p = / converges when p /. We conclude that for d =, state 0 s transent when p /

7 In the new coordnates, up to a factor, the jumps are along dagonals of 0 = 0,0) the unt square. /4 Hence agan p n) 00 = 0 when n s odd. A useful dea s to project the rom wal onto orthogonal axes rotated by π/4. It means that the chan X n) n the new coordnates s formed by a par of ndependent symmetrc nearest-neghbour rom wals on Z n the horzontal vertcal drectons). Return to 0 = 0, 0) means return to 0 n each of them. Therefore, for n = 0 =0,0) Hence, ) p ) )! 00 =!! π. 5.6) p ) 00 =, the rom wal s recurrent. For d = 3, Z 3 s the three-dmensonal cubc lattce; we may thn that t s an nfntely extended crystal. Then our walng partcle may model a soltary quantum electron movng between heavy ons or atoms fxed at the stes of the lattce. The probablty of movng to one of the sx neghbours equals /6. The moves are n one-to-one correspondence old coordnates: chan X n ) move by vector ±; 0) move by vector ±0; ) new coordnates: chan X n) move by vector ± / ); ) move by vector ± / ) ; ). 3 4

8 /6 0 = 0,0,0) In fact, suppose that < m < l. Then when you pass to!j!l! from m!) 3, you ether a) replace the tals + )...m j + )...l of m! by the product m + )...m + m j),.e. m + )...l when j < m or b) replace the tal +)...m by the product m+)...jm+)...3m j), that s m + )...jm + )...l when j > m. Ether way you ncrease the denomnator, hence decrease the rato. Then, for n = = 6m: p 6m) 00 6m)! 3m)! ) ) 6m 3m)! m!) 3 ) 3m 5.9) 3 Stll, p n) 00 = 0 when n s odd. If n s even, a path returns to 0 = 0, 0, 0) f only f t maes equal numbers of jumps n each of three pars of opposte drectons up/down, east/west, north/south). So, p ) 00 = Now, the sum, j, l 0 : + j + l = = )!!) )!!) )!!) j!) l!), j, l 0 : + j + l = max!!j!l!!!j!l! 6 ) ) ) 6 ) 3, j, l 0 : + j + l =!!j!l! 3.!!j!l! = 3, 5.7), j, l 0 : + j + l = the number of ways to place balls nto 3 boxes. Also, for = 3m 3m)! m!m!m! 3m)!!j!l! whenever + j + l = 3m. 5.8) 5 whch, by Strlng, s ) 3 π m3/. 5.30) Hence, p 6m) 00 <. m But for m : p 6m) 00 /6) p 6m ) 00 p 6m) 00 /6) 4 p 6m 4) 00,.e. Thus, p 6m ) 00 6 p 6m) 00 p 6m 4) p 6m) 00. the wal s transent. p ) 00 m p 6m) ) <, 5.3) A smlar approach can be used n hgher dmensons. But there s another way to establsh transence n all dmensons d > 3. Namely, project the rom wal Xn) d on Z d to three dmensons by dscardng all coordnates but the frst three. The projected chan ) Xn proj on Z 3 stays where t s wth probablty d 3)/d when the orgnal wal jumps n one of the dscarded drectons), but when t jumps, t behaves as the nearest-neghbour symmetrc wal n dmenson 3 wth P X proj n+ = ± e α ) X proj n = = /d) d 3)/d =, α =,, 3, 5.3) 6 e = ; 0; 0), e = 0; ; 0), e 3 = 0; 0; ). 6

9 d 3) d d Soluton ) If = E T h = P X T Y T = 0) then 0,0) = +,0),,0) = + 0,0) /4 +, ) /,, ) = +,0) /, h 0,0) = h,0), h,0) = /4 + h 0,0) /4 + h, ) /, h, ) = h,0) /, by condtonng on the frst step, the Marov property symmetry. Clearly, f the orgnal d-dmensonal wal returns to 0 = 0,...0) then the projected wal returns to 0, 0, 0). Hence, f the orgnal d-dmensonal wal Xn d) s recurrent then the projected chan ) Xn proj s recurrent. But then consder the rom wal on Z 3 obtaned from ) Xn proj by dscardng the stays recordng the jumps only. The latter s the nearest-neghbour symmetrc ) rom wal on Z 3 whch s transent. By Theorem 6.3 below, X proj n s also transent. Then so s X d n ). Nearest-neghbour symmetrc rom wals are often called smple wals. Re-phrasng a famous sayng, we could state that n two dmensons every road of a smple rom wal wll lead you to the orgn or any other gven ste) whle n three dmensons hgher t s no longer so. The dfference between two three dmensons emerges n a countless varety of stuatons n vrtually all domans of mathematcs. Example 5.4 Math Trpos, Marov Chans, Part IIA, 003, A0J Part IIB, 003, B0J) ) Let X n, Y n ) be a smple symmetrc rom wal n Z, startng from 0, 0), set T = nf n 0 : max X n, Y n } = }. Determne the quanttes ET PX T = Y T = 0). ) Let X n ) be a Marov chan wth state space I transton matrx P. What does t mean to say that a state I s recurrent? Prove that s recurrent f only f p n) =, where p n) denotes the, ) entry n P n. n=0 Show that the smple symmetrc rom wal n Z s recurrent. Hence, By symmetry, O ET = 0,0) = 9, h 0,0) =. PX T = Y T = 0) = 4 h 0,0) = 8. ) State s recurrent f f = P T < ) = where T = nf n : X n = }. If V s the total tme spent n then Then On the other h, P V + ) = P V ) P V + V ) = P V )f =... = f +. E V ) = E V = E PV ) = 0 f. X n = ) = p n). 7 8

10 Hence, f = f only f p n) =. Now let X n ) be a smple symmetrc rom wal n Z. It s rreducble, hence t suffces to chec that p n) = for a sngle Z, say the orgn 0, 0). Wrte X n ± ) for the projecton of X n ) on the dagonal x = ±y} n Z. Then X n ± ) are ndependent smple symmetrc rom wals on Z, return to 0, 0) n X n ) means return to 0 n each of X n ± ). Next, ) P 0 X ± = 0) =, p ) 00 = P 0 X + = 0)P 0X = 0). Then Strlng s formula asserts that p ) Hence, 00 s ) ) =, as. π π p n) 00 = p ) 00 =. 0 Example 5.5 Math Trpos, Marov Chans, Part IIA, 994, A0K) ) Let X n ) be a smple symmetrc rom wal on the ntegers. Show that X n ) s recurrent. ) Let X n ), Y n ) Z n ) be smple symmetrc rom wals on the ntegers. Then V n = X n, Y n, Z n ) s a Marov chan. What are the transton probabltes for ths chan? Show that, wth probablty one, V n ) vsts 0, 0, 0) only fntely many tmes. [Strlng s formula states n!e/n) n / πn as n. 9 Other stard results may be used wthout proofs f clearly stated.] Soluton. ) We use the followng result: state : f P T < ) =, then p n) =, s R, f P T < ) <, then n=0 n=0 p n) <, s T. Also, recurrence transence are class propertes, the smple symmetrc rom wal defnes an rreducble chan. So, t suffces to chec recurrence or transence for a sngle state, say 0. We have: n p n) 00 = n p n) 00 = n n)! n!n! use Strlng s formula to obtan the summs 4πn n/e) n πn n/e) n =. n πn ) n As the seres / πn) dverges, the rom wal s R. n ) The transton probabltes for V n ) are p,j,)l,m,n) = we am to show that /8, f l = j m = l n =, 0, otherwse. n=0 p n) 0,0,0)0,0,0) <. As before, p n) 0,0,0)0,0,0) = 0 when n s odd. Furthermore, p n) 0,0,0)0,0,0) = ) 3 ) 3 p n) n) 00 where p 00 s as n part ). Hence, p n) 00, the πn) 3/ seres converges. So, V n ) s T; hence the answer. Example 5.6 Math Trpos, Marov Chans, Part IIA, 996, A30E) ) A rom sequence of non-negatve ntegers F n ) s obtaned by settng 0,

11 F 0 = 0 F =, once F 0,...,F n are nown, tang F n+ to be ether the sum or the dfference of F n F n, each wth probablty /. Is F n ) a Marov chan? By consderng the Marov chan X n = F n, F n ), fnd the probablty that F n ) reaches 3 before frst returnng to 0. ) Draw enough of the flow dagram for X n ) to establsh a general pattern. Hence, usng the strong Marov property, show that the httng probablty for, ), startng from, ), s 3 5)/. Soluton. See the dagram...,0) 0,) 0,),),).,),)...,),) 3,). 3,)..,3),3),3) 3,5) ,).,3).,) One can see a trangular pattern emergng from the dagram, wth treele symmetres. In partcular, P,) ht, ) ) = P,3) ht, ) ) = P,3) ht, ) ) := p P,) ht, ) ) = P3,) ht, ) ) := p. Condtonng on the frst jump, by the strong Marov property, we can wrte Ths yelds p = p = p + P ),3) ht, ) = p + p, p = + P ),3) ht, ) = + pp whence p = p. p) + p,.e., p 3 4p + 4p = p )p 3p + ) = 0. The roots are p = 3 ± 5)/. We are nterested n the mnmal non-negatve root,.e., p = 3 5)/. Consequently, p = / + 5). Obvously, 0 < p, p <. F n ) s not a Marov chan, as F n+ depends on F n F n, but the par F n, F n ) s. The ntal part of the dagram shows that the level F n = 3 can be reached from F 0, F ) = 0, ) ether at, 3) or, 3). To ht ths level before vstng level F n = 0.e.,, 0)), we have two straght paths, supplemented wth a number of adjacent trangular cycles. The frst possblty gves the probablty the second whch adds to 3/ ) = 7, ) +... = 7,