Pla Group Work 0. The title says it all Next Tie: MCMC ad Geeral-state Markov Chais Midter Exa: Tuesday 8 March i class Hoework 4 due Thursday Uless otherwise oted, let X be a irreducible, aperiodic Markov chai o coutable state-space S with iitial distributio µ ad trasitio probabilities P. You are free to use the Referece ites at the ed. Note i particular Referece which stregthes the coditio we saw last tie. Exaple. Defie a Markov chai o {,..., 6} with trasitio probability atrix 0.5 0 0.5 0 0 0 0.5 0.5 0.5 0 0 0 0.5 0 0.5 0 0 0 0 0 0 0.5 0.75 0 0 0 0 0.5 0.5 0 0 0 0.5 0.5 0 0.5 () Tell e everythig you ca about this chai. Exaple. Defie a Markov chai o {,..., 6} with trasitio probability atrix 0 0 0.4 0.6 0 0 0 0 0.4 0.6 0 0 0 0 0 0 0.5 0.75 0 0 0 0 0.5 0.75 0.8 0. 0 0 0 0 0.8 0. 0 0 0 0 () Tell e everythig you ca about this chai. Exaple 3. A Queueig Exaple Let (u k ) k 0 be a o-egative sequece with k u k =. Let Z, for 0, deote the legth of a queue at tie, where at each tie, oe custoer arrives ad k custoers are served with probability u k, if there are at least k i the queue. The Z is a Markov chai o Z. (a) Fid its trasitio probabilities. (b) Decopose the state space (c) Fid the period of each couicatig class. (d) Fid coditios for recurrece or trasiece of the chai. 7 Mar 006
Exaple 4. Cosider a Markov chai with two states ad trasitio probabilities [ ] p p p p Show by iductio that [ P = + (p ) ] (p ) (p ) + (p ) What s the liitig distributio of this chai? (3) (4) Exaple 5. Cosider the followig odel for the diffusio of gas. Suppose that M olecules are distributed betwee two chabers that are separated by a pereable boudary. At each tie, oe of the olecules with all equally likely to be chose crosses the boudary fro oe chaber to the other. Tell e everythig you ca about this process. Exaple 6. Cosider o-egative sequeces p i, q i, ad r i for i 0 satisfyig q 0 = 0 (5) r 0 + p 0 = (6) q i + r i + p i = for i > 0 (7) Defie a rado walk o the itegers with trasitio atrix P (i, i ) = q i P (i, i) = r i P (i, i + ) = p i, (8) ad all other etries zero. Let α 0 = ad α = q q p p. Show that the chai is trasiet if α < ad recurret if α =. As a optioal additio, show that the chai is positive recurret if p 0/(α p ) <. Exaple 7. A spider is hutig a fly, ad the fly is tryig to survive. The spider starts i locatio ad oves betwee locatios ad accordig to the Markov trasitios [ ] 0.7 0.3. (9) 0.3 0.7 The fly starts i locatio ad oves betwee the locatios with trasitios [ ] 0.4 0.6. (0) 0.4 0.6 The hut eds if the two ever lad o the sae locatio, i which case the fly is eate. Show that this progress of the hut ca be described (except for kowig at which locatio the hut eds) by a three-state Markov chai. Fid the trasitio probabilities for this chai. What is the expected duratio of the hut? 7 Mar 006
Exaple 8. Cosider a syetric rado walk i d diesios. The drift operator = P V V for this chai is a oralized (by /d) versio of a operator called the discrete Laplacia because o fier ad fier grids, it approxiates the stadard Laplacia differetial operator = = / x + / y + / z. Thus, the behavior of rado walks o Euclidea space coects quite deeply to the solutios of differetial equatios. For exaple, we ve see that the existece of bouded fuctios V with V satisfyig certai iequalities correspods to recurrece or trasiece of the chai. As aother exaple, cosider a regio S 0 S with boudary poits S 0. If f is a fuctio o S 0 such that f = 0 o S 0 S 0 ad f = g o S 0, the f(s) = E s g(x T S0 ). So for exaple, i the oe-diesioal syetric rado walk. Let S 0 = {0,..., } ad let g(0) = ad g() = 0. The, the solutio f gives the Gabler s rui probabilities for every iitial wealth. But solutios are just of the for f(s) = s () by direct calculatio. (a) What is for the diesioal rado walk? Express it i ters that are reiiscet of the cotiuous Laplacia describd above. See Referece 3 to check your aswer, but do t look before tryig it. Note that we ca thik of as a operator or as a atrix. What does the atrix look like i the oe-diesioal case? (b) I the Gabler s rui exaple above, we coputed the solutio by ispectio because a liear fuctio has zero secod differeces. But we ca use the theore i Referece 4 as well. Show that πjk φ k (j) = si () is a eigefuctio (of the operator, or equivaletly eigevector of the atrix for) with eigevalue λ k = cos πk/. The, usig the equatio i the theore, we get we get f(j) = k= si πk πkj si cos πk = j, (3) as we foud above. (c) Now, let d = ad let S 0 S be a fiite set with boudary poits S 0. We wat to show here that if f is a fuctio o S 0 such that f = 0 o Iterior(S 0 ) S 0 S 0 ad f = g o S 0, the f(s) = E s g(x T S0 ). We ll do this i several steps. i. Draw a picture of a regio S 0 satisfyig the coditios above. ii. If s S 0, the there are two possibilities, either the chai stays i Iterior(S 0 ) forever or it hits the boudary S 0. Show that the latter has probability. iii. Let U = g(x T S0 ). For ay s S, let s N, s E, s W, s S deote the eighbors of s (to the orth, east, west, ad south, respectively). By coditioig o the first step, show that E s U = 4 E s N U + 4 E s E U + 4 E s S U + 4 E s W U. (4) iv. Show f, defied above, satisfies f = 0 o Iterior(S 0 ) ad f = g o S 0. v. What ca you say about the uiqueess of this solutio? 3 7 Mar 006
Exaple 9. Storage Model Cosider a storage syste (da, warehouse, isurace policy) that receives iputs at rado ties but otherwise drais at a regular rate. Let T 0 = 0 ad let the rest of the T i s be iid Z -valued with cdf G. These are iter-arrival ties for the iputs to our storage syste. Let the S s be iid Z with cdf H. These are the aouts iput at the tie Z = T 0 + + T. Assue that the S s ad T s are idepedet of each other as well. Suppose also that the storage syste drais or outputs at rate r betwee iputs. Defie a process (V ) 0 by V + = (V + S rt + ) +. (5) Here, V represets the cotets of the storage syste just before the th iput (that is, at tie Z ). Is V a Markov chai? What is the structure of the trasitio probabilities? What ca we say about the log-ru behavior of the chai? What is special about the state {0}? How ight we geeralize this odel to ake it ore realistic? 4 7 Mar 006
Referece 0. Let D deote the set of states i o-absorbig couicatig classes ad C,..., C be the absorbig couicatig classes i the usual state decopositio. Let h r (s) = H(s, C r ) for i S. By coditioig o the first step, we get for i D that h r (s) = P (s, s ) + P (s, u)h r (u). (6) s C r u D Hece, h r is a o-egative solutio to this equatio. It ca be show (by iductio o tie ) that h r is the sallest o-egative solutio of this equatio. Let s D ad s C r for soe r. The, If C r is trasiet or ull recurret, li P (s, s ) = 0. If C r is positive recurret ad aperiodic, li P (s, s ) = π r (s )h r (s), where π r is the statioary distributio for the chai restricted to C r. If C r is positive recurret ad periodic with period d r li as above, but P (s, s ) itself does ot coverge. P (s, s ) = π r (s )h r (s) (7) =0 Referece. Aother Drift Criterio for Trasiece Let Z be a irreducible, coutable-state Markov chai o S with drift operator. The, Z is trasiet if ad oly if there exists a s 0 S ad a bouded, ocostat fuctio V such that V (s) = 0 for s s 0. (8) Referece. Rado Walks We ve already see that the oe-diesioal rado walk is recurret. To see that it s ull recurret, ote that the trasitio probabilities are doubly stochastic. (Why?) So, ρ = ρp has a solutio ρ(i). This is ot oralizable, so the chai is ull recurret. Let s cosider the case of a syetric, two diesioal rado walk which oves up, dow, left, or right with probability /4 each. The chai is irreducible because we ca fid a o-zero probability path fro ay oe state to aother. It is also has period for the sae reaso as i the oe-diesioal case. Thus, to assess the recurrece of the chai, we ca look at oly oe poit, let s say 0. We kow, the, that P + (0, 0) = 0 for all 0 ad, by cosiderig all paths two ad fro zero, we get that P (0, 0) = ()!. (9) i!j!i!j! 4 i+j= Why? Our goal is to copute P (0, 0) to test recurrece for the chai. By ultiplyig P (0, 0) by (!) /(!), we get P (0, 0) = P (0, 0) (0) 5 7 Mar 006
= i+j= ()! () i!j!i!j! 4 = ()! () i!( i)!i!( i)! 4 i=0 = 4 (3) i i i=0 = 4 (4) i i i=0 = 4 (why?) (5) π (6) =. (7) The peultiate equatio follows by Stirlig s approxiatio! ( e π, fro which ) / π. Hece, the chai is recurret, ad because agai the atrix is doubly stochastic, the ivariat easure has ifiite ass. The chai is thus positive recurret. Next, cosider three diesioal case. Usig the sae logic, we have that P (0, 0) = = = = P (0, 0) (8) 0 i+j ( ( ( ()! (9) i!j!( i j)!i!j!( i j)! 6 ) ) ) 0 i+j c 3 3 (30) i j i j 0 i+j 3 (3) i j c 3, (3) where c = ax 0 i+j. We ca show that c ( i j /3 /3). We thus get that P (0, 0)! (/3)!(/3)!(/3)! 3 C 3/, (33) by Stirlig s approxiatio. Thus, the chai is trasiet. 6 7 Mar 006
Referece 3. Let be the drift operator V = P V V for the syetric rado walk i d diesios. Whe d =, V (i) = (V (i + ) + V (i )) V (i) (34) I geeral, we have that = = V (i + ) V (i) + V (i ) (35) (V (i + ) V (i)) (V (i) V (i )). (36) V (i,..., i d ) = (37) d d k= V (i,..., i k, i k +, i k+,..., i d ) V (i,..., i d ) + V (i,..., i k, i k, i k+,..., i d ), (38) which looks uch worse tha it is. Notice that we have secod-order divided differeces i each variable, reiiscet of secod derivatives. Referece 4. Let X be a syetric rado walk o S = Z d. Let S 0 S have boudary S 0. The, restricted to S 0 correspods to a syetric atrix o S 0 ad thus has eigevalues (λ k ) ad eigefuctios (φ k ) ad the solutio to the above proble takes the for f(s) = d k u S 0,v S 0 u,vadjacet φ k (u)g(v) φ k(s). (39) λ k Agai, this looks bad, but do t worry about the details. The key poit is that give the coputable properties of the operator ad give boudary coditios, we ca fid what we eed quite siply. 7 7 Mar 006