Solutions of soe selected probles of Hoework 4 Sangchul Lee May 7, 2018 Proble 1 Let there be light A professor has two light bulbs in his garage. When both are burned out, they are replaced, and the next day starts with two working light bulbs. Suppose that when both light bulbs are working, one of the two will burn out with probability 0.02 that is, each light bulb has probability 0.01 of burning out each day and we ignore the possibility of losing two on the sae day. However, when only one light bulb is still working, it will burn out with probability 0.05. What is the expected tie between light bulb replaceents? Solution. The diagra for this chain is quite siple: 0.05 0.02 0 1 2 The chain only oves in the order 0 2 1 0 and leaves the state 0 iediately. So 1 E 0 [T 0 ] 1 + E 2 [T 1 ] + E 1 [T 0 ]. Also, when the chain starts at 2, T 1 is the first tie you jup fro 2 to 1. So T 1 has the geoetric distribution with paraeter 0.02 under P 2 and we obtain E 2 [T 1 ] 1 0.02 50. Siilar arguent yields E 1[T 0 ] 1 0.05 20 and therefore E 0 [T 0 ] 71. Proble 2 Stopping tie identity Let X n n 0 be a Markov chain with a finite state space S and transition atrix P. Set T y in{n 1 : X n y}. Show that, for all x,y S, P x [T y < ] n1 Pn x,y 1 + n1 Pn y,y. Reark. The ration in the right-hand side does not ake sense when the nuerator is infinite, since this forces that the denoinator is also infinite. It is ore natural to prove P n x,y P x [T y < ] 1 + P n y,y. 1 n1 n1 1
Solution. Recall that E x [Ny] n1 Pn x,y where Ny n1 1 {X n y} is the nuber of visits to y and that it can be coputed by E x [Ny] ρ xy /1 ρ yy. Then 1 follows by [RHS of 1] ρ xy 1 + ρ yy ρ xy [LHS of 1]. 1 ρ yy 1 ρ yy Alternatively, one ay establish 1 anually, and this will be a good review on strong Markov property. The idea is to utilize the strong Markov property to split your chain at the first tie it vistis y. Using the fact that {X n y} iplies {T y < }, we decopose the expectation into [LHS of 1] n1 n1 P x [X n y] k1 n1 P x [X n y,t y k] P x [X n y,t y < ] k1 nk P x [X n y T y k]p x [T y k]. In the last equality, we interchanged the order of suation and used the fact that P x [X n y,t y k] 0 if n < k. Then by strong Markov property, as desired. nk k1 k1 P y [X n k y]p x [T y k] P x [T y k] P x [T y < ] Proble 3 Lazy Markov chain 1 n 0 n 1 P y [X n y] P y [X n y] [RHS of 1]. In the following, you are not allowed to use the results about existence of stationary distributions that we have shown in class. Let X n n 0 be a Markov chain with a finite state space S and transition atrix P. Consider the lazy Markov chain with the sae state space S and transition atrix Q : 1 2 I + P, where I is the identity atrix. i Bonus Show that there is always a row vector vx x S such that v vp. ii Show that v vp if and only if v vq. In particular, a row vector πx x S is a stationary distribution of P if and only if π is a stationary distribution of Q. Fro now on suppose that S is irreducible. iii Show that there exists k 1 such that Q k x,y > 0 for every x,y S. iv Bonus Suppose that v vp for a row vector vx x S. Show that the coefficients vx have the sae sign for each x S. v Bonus Suppose that v vp for a vector v 0 with vx 0 for each x S. Prove that vx > 0 for each x S. vi Suarize all the above to conclude that each irreducible Markov chain has a stationary distribution π with πx > 0, for each x S. 2
Solution. i Write 1 1 x S for the colun vector whose coefficients are all equal to 1. Being a stochatic atrix, P satisfies P1 1 and hence it has the right-eigenvalue 1. Since detp λi detp T λi, the right-eigenvalues of P exactly atch the left-eigenvalues of P. Therefore there exists an eigenvector v of P for the eigenvalue 1. ii If v vp, then vq 1 2 v + vp v. Conversely, if v vq then vp 2Qv v v. iii Since P is irreducible, for each x,y S there exists k x,y 1 such that P k x,y x,y > 0. Now set k ax{k x,y : x,y S}. Then by the binoial theore, Q k x,y 1 2 k k j0 iv Here we present a nice solution by a student in the class. k P j x,y 1 k j 2 k P k x,y x,y > 0. k x,y Assue that the row vector v solves v vp. Let A {y S : vy > 0}. If A S, then we are done. So it suffices to consider the case where A is a proper subset of S and we asssue so. We clai that this iplies vx 0 for all x S. Indeed, let k be chosen as in iii. Then it follows that vy y A y A x S vxq k x,y y A x A vxq k x,y x A 1 Q k x,y vx. y S\A The second equality follows by excluding all the non-positive ters. But since Q k has only positive coefficients, it follows that 1 y S\A Q k x,y < 1 for each x A. So if A is non-epty, this leads to the contradiction y A vy < x A vx. Therefore A is epty and all the coefficients of v ust be non-positive as claied. v Choose k 1 so as to satisfy iii. Then by ii, we have v Qv Q k v. Now pick y 0 S such that vy 0 > 0. Such y 0 exists because v 0. Then vx vyq k y,x vy 0 Q k y 0,x > 0. y S vi By i v, for each irreducible stochastic atrix P on S, there exists a row vector v vx x S such that vx > 0 for all x S and v vp. Then π πx x S defined by πx vx/ y S vy is a distribution vector with πx > 0 for all x S and π πp, hence is a stationary distribution. 3
Proble 4 Bernoulli-Laplace odel of diffusion Consider two urns each of which contains balls; b of these 2 balls are black, and the reaining 2 b are white. We say that the syste is in state i if the first urn contains i black balls and i white balls while the second contains b i black balls and b + i white balls. Each trial consists of choosing a ball at rando fro each urn and exchanging the two. Let X n be the state of the syste after n exchanges have been ade. i Show that X n is a Markov chain and find the transition atrix. ii Verify that the stationary distribution is given by b 2 b πi i i 2 Solution. i Once we know the nuber of black balls at the first urn, then we have the coplete inforation on how black and white balls are distributed over two urns. Also, each exchange is perfored independently of all the other exchanges and depends only on the current distribution of balls. Therefore X n is a Markov chain. To describe its transition atrix, we first set up the state space. Since each urn contains at least ax{0,b } black balls and cannot hold ore than in{b,} black balls, it is natural to set S {ax{0,b },in{b,}}. Now for each i, j S, we have Pi, j i i b i i b i b i, if j i + 1,, if j i 1, + i b i, if j i, 0, otherwise. ii It suffices to show that π satisfies the detailed balanced condition. And since the transition occures only to nearest neighbors, it suffices to check πipi,i + 1 πi + 1Pi + 1,i for each i S such that i + 1 S as well. Pulling out soe coon factors, we find that 2 2 πipi,i + 1 b 2 b i i b! i!b i 1! ib i 2 b! b + i! i 1! and 2 b 2 b 2 πi + 1Pi + 1,i i + 1 b + i + 1 i + 1 i 1 b! i!b i 1! 2 b! b + i! i 1! are equal. Therefore π satisfies the detailed balanced condition and hence is a stationary distribution. 4
Proble 5 Overhand shuffling A deck of 52 cards is shuffled using overhand shuffle, which aounts to splitting the deck in halves at a rando place and switching the top half with the botto half. Copute the expected inial nuber of shuffles needed to see again the initial state of the deck. Solution. Label cards in a deck fro to to botto by 0,1,,51. Now assue that the deck is split at between the card k 1 and k. Of course, 1 k 51. Then the result of shuffle is 0 1. k 1 ḳ. 51 ḳ. 51 0 1. k 1 Notice that the relative position between any two cards is preserved by this shuffling, if we declare that the top-ost card lies right below the botto-ost card. Matheatically speaking, the resulting of this shuffling is siply the addition by k odulo 52. In particular, knowing the label of the top-ost card is enough to deterine the entire card configuration of the deck. This suggests us a very siple choice of the state space, naely S {0,,51}. Let X n be the label of the top-ost card in the deck after the n-th shuffle. Then X n n 0 is a Markov chain with the transition atrix { 1 Pi, j 51, if i j 0, otherwise. Since this atrix is doubly stochastic, P is irreducible and π 1 52 i S is the unique stationary distribution. Therefore E 0 [T 0 ] 1 π0 52. 5
Appendix: Two additional solutions of Proble 3.iv 1 st Solution. We adopt the following powerful theore: Theore Brouwer fixed point theore. If f : [0,1] n [0,1] n is continuous, then there exists x [0,1] n such that f x x. Now let {vx x S : vx 0 and x S vx 1}. This set is hoeoorphic to [0,1] S 1 and the apping v vp is a continuous function on. So there exists v 0 such that v 0 v 0 P. By the part v, which can be proved independently of this part, we know that v 0 x > 0 for all x S. Now let v 0 be any row vector solving v vp. Pick α 0 such that w v 0 αv satisfies wx 0 for all x S and wx 0 for soe x S. Since w solves w wp, again by v we find that w 0. Therefore v α 1 v 0 and the desired clai follows. 2 nd Solution. We first establish the following lea. Lea. Let A be a d d stochastic atrix such that A i j > 0 for all i, j. Then for any colun vector u u 1,,u d, there exists c such that A n u i j A n i j u j c as n for each i. Proof of Lea. We write ε in i, j A i j and notice that ε > 0. Also, for each colun vector u, we define u axu inu. Now pick i, j such that Au i axau and u j inu and. Then axau Au i A i j u j + 1 A i j axu axu A i j u axu ε u and likewise we have inau inu + ε u. Fro this, we know that axa n u is onotone decreasing in n. ina n u is onotone increasing in n. Au 1 2ε u. In particular, A n u converges to 0 as n. So it follows that both axa n u and ina n u converge to the sae liit c. Therefore A n u i converges to c by the squeezing theore. Let us return to the original proof. Notice that A Q k with k chosen as in iii satisfies the assuption of the lea. Now for each y S, we apply the lea to the colun vector Qx,y x S to find a constant py such that x S : Q nk+1 x,y Q k n x,zqz,y py. n z S Moreover, py 0 since it is the liit of non-negative quantities. So if a row vector v solves v vp, then y S : vy vxq nk+1 x,y n vx py. x S x S So v cpy y S, where c x S vx. Since py s have the sae sign, the sae is true for v. 6