11.4. RECURRENCE AND TRANSIENCE 93

Size: px

Start display at page:

Download "11.4. RECURRENCE AND TRANSIENCE 93"

Bethanie Newton
5 years ago
Views:

1 11.4. RECURRENCE AND TRANSIENCE 93 Similar arguments show that simple smmetric random walk is also recurrent in 2 dimensions but transient in 3 or more dimensions. Proposition If x is recurrent and r(x, ) > 0, then is recurrent and r(, x) =1. Proof. First we show r(, x) = 1. Suppose not. Since r(x, ) > 0, there is a smallest n and 1,..., n 1 such that p(x, 1 )p( 1, 2 ) p( n 1,) > 0. Since this is the smallest n, noneofthe i can equal x. Then P x (T x = 1) p(x, 1 ) p( n 1,)(1 r(, x)) > 0, acontradictiontox being recurrent. Next we show that is recurrent. Since r(, x) > 0, there exists L such that p L (, x) > 0. Then p L+n+K (, ) Summing over n, p L+n+K (, ) n p L (, x)p n (x, x)p K (x, ). p L (, x)p K (x, ) n p n (x, x) =1. We sa a subset C of S is closed if x 2 C and r(x, ) > 0implies 2 C. AsubsetD is irreducible if x, 2 D implies r(x, ) > 0. Proposition Let C be finite and closed. Then C contains a recurrent state. From the preceding proposition, if C is irreducible, then all states will be recurrent. Proof. If not, for all we have r(, ) < 1and 1 1 E x N() = P x (N() k) = P x (T k < 1) = k=1 k=1 1 r(x, )r(, ) k 1 = k=1 r(x, ) 1 r(, ) < 1.

2 94 CHAPTER 11. MARKOV CHAINS Since C is finite, then P E x N() < 1. But that is a contradiction since E x N() = p n (x, ) = p n (x, ) n n = P x ( n 2 C) = 1=1. n n Theorem Let R = {x : r(x, x) = 1}, the set of recurrent states. Then R = [ 1 i=1r i, where each R i is closed and irreducible. Proof. Sa x if r(x, ) > 0. Since ever state is recurrent, x x and if x, then x. If x and z, thenp n (x, ) > 0andp m (, z) > 0 for some n and m. Then p n+m (x, z) > 0orx z. Therefore we have an equivalence relation and we let the R i be the equivalence classes. Looking at our examples, it is eas to see that in the Ehrenfest urn model all states are recurrent. For the branching process model, suppose p(x, 0) > 0 for all x. Then 0 is recurrent and all the other states are transient. In the renewal chain, there are two cases. If {k : a k > 0} is unbounded, all states are recurrent. If K =max{k : a k > 0}, then{0, 1,...,K 1} are recurrent states and the rest are transient. For the queueing model, let µ = P ka k,theexpectednumberofpeople arriving during one customer s service time. We ma view this as a branching process b letting all the customers arriving during one person s service time be considered the progen of that customer. It turns out that if µ apple 1, 0 is recurrent and all other states are also. If µ>1allstatesaretransient Stationar measures Aprobabilitµ is a stationar distribution if µ(x)p(x, ) =µ(). (11.8) x

3 11.5. STATIONARY MEASURES 95 In matrix notation this is µp = µ,orµ is the left eigenvector corresponding to the eigenvalue 1. In the case of a stationar distribution, P µ ( 1 = ) =µ(), which implies that 1, 2,... all have the same distribution. We can use (11.8) when µ is a measure rather than a probabilit, in which case it is called a stationar measure. Note µp n =(µp )P n 1 = µp n 1 = = µ. If we have a random walk on the integers, µ(x) =1forallx serves as a stationar measure. In the case of an asmmetric random walk: p(i, i +1)= p, p(i, i 1) = q =1 p and p 6= q, settingµ(x) =(p/q) x also works. To check this, note µp (x) = µ(i)p(i, x) =µ(x 1)p(x 1,x)+µ(x +1)p(x +1,x) p x 1p p x+1q = + q q = px q q + px x q q = px x q. x r In the Ehrenfest urn model, µ(x) =2 r works. One wa to see this x is that µ is the distribution one gets if one flips r coins and puts a coin in the first urn when the coin is heads. A transition corresponds to picking a coin at random and turning it over. Proposition Let a be recurrent and let T = T a. Set Then µ is a stationar measure. T 1 µ() =E a 1 (n=). The idea of the proof is that µ() istheexpectednumberofvisitsto b the sequence 0,..., T 1 while µp is the expected number of visits to b 1,..., T.Theseshouldbethesamebecause T = 0 = a. Proof. Let p n (a, ) =P a ( n =, T > n). So 1 µ() = P a ( n =, T > n) = 1 p n (a, )

4 96 CHAPTER 11. MARKOV CHAINS and µ()p(, z) = 1 p n (a, )p(, z). First we consider the case z 6= a. Then p n (a, )p(, z) = P a (hit in n steps without first hitting a and then go to z in one step) = p n+1 (a, z). So µ()p(, z) = p n (a, )p(, z) n 1 1 = p n+1 (a, z) = p n (a, z) = µ(z) since p 0 (a, z) =0. Second we consider the case a = z. Then p n (a, )p(, z) = P a (hit in n steps without first hitting a and then go to z in one step) = P a (T = n +1). Recall P a (T =0)=0,andsincea is recurrent, T<1. So µ()p(, z) = p n (a, )p(, z) n 1 1 = P a (T = n +1)= P a (T = n) =1.

5 11.5. STATIONARY MEASURES 97 On the other hand, T 1 1 (n=a) =1 (0 =a) =1, hence µ(a) =1. Therefore,whetherz 6= a or z = a, wehaveµp (z) =µ(z). Finall, we show µ() < 1. If r(a, ) =0,thenµ() =0. Ifr(a, ) > 0, choose n so that p n (a, ) > 0, and then 1=µ(a) = µ()p n (a, ), which implies µ() < 1. We next turn to uniqueness of the stationar distribution. We give the stationar measure constructed in Proposition the name µ a.weshowed µ a (a) =1. Proposition If the Markov chain is irreducible and all states are recurrent, then the stationar measure is unique up to a constant multiple. Proof. Fix a 2S.Letµ a be the stationar measure constructed above and let be an other stationar measure. Since = P, then (z) = (a)p(a, z)+ 6=a ()p(, z) Continuing, = (a)p(a, z)+ 6=a (a)p(a, )p(, z)+ x6=a = (a)p a ( 1 = z)+ (a)p a ( 1 6= a, 2 = z) + P ( 0 6= a, 1 6= a, 2 = z). (z) = (a) (x)p(x, )p(, z) 6=a n P a ( 1 6= a, 2 6= a,..., m 1 6= a, m = z) m=1 + P ( 0 6= a, 1 6= a,..., n 1 6= a, n = z) n (a) P a ( 1 6= a, 2 6= a,..., m 1 6= a, m = z). m=1

6 98 CHAPTER 11. MARKOV CHAINS Letting n!1, we obtain (z) (a)µ a (z). We have (a) = x (x)p n (x, a) (a) x µ a (x)p n (x, a) = (a)µ a (a) = (a), since µ a (a) =1(seeproofofProposition11.13). Thismeansthatwehave equalit and so for each n and x either p n (x, a) =0or (x) = (a)µ a (x). Since r(x, a) > 0, thhen p n (x, a) > 0forsomen. Consequentl (x) (a) = µ a(x). Proposition If a stationar distribution exists, then µ() > 0 implies is recurrent. Proof. If µ() > 0, then = µ() = µ(x)p n (x, ) = 1 µ(x) p n (x, ) n=1 n=1 x x n=1 = 1 µ(x) P x ( n = ) = µ(x)e x N() x n=1 x = x µ(x)r(x, )[1 + r(, )+r(, ) 2 + ]. Since r(x, ) apple 1andµ is a probabilit measure, this is less than µ(x)(1 + r(, )+ ) apple 1+r(, )+. Hence r(, ) must equal 1. x Recall that T x is the first time to hit x.

7 11.5. STATIONARY MEASURES 99 Proposition If the Markov chain is irreducible and has stationar distribution µ, then µ(x) = 1 E x T x. Proof. µ(x) > 0 for some x. If 2S,thenr(x, ) > 0andsop n (x, ) > 0 for some n. Hence µ() = µ(x)p n (x, ) > 0. x Hence b Proposition 11.15, all states are recurrent. B the uniqueness of the stationar distribution, µ x is a constant multiple of µ, i.e.,µ x = cµ. Recall and so µ x () = = n 1 µ x () = 1 P x ( n =, T x >n), P x ( n =, T x >n)= n P x (T x >n)=e x T x. P x ( n =, T x >n) Thus c = E x T x.recallingthatµ x (x) =1, µ(x) = µ x(x) c = 1 E x T x. We make the following distinction for recurrent states. If E x T x < 1, then x is said to be positive recurrent. If x is recurrent but E x T x = 1, x is null recurrent. An example of null recurrent states is the simple random walk on the integers. If we let g() =E T x,themarkovproperttellsusthat Some algebra translates this to g() =1+ 1g( 1) + 1 g( +1). 2 2 g() g( 1) = 2 + g( +1) g().

8 100 CHAPTER 11. MARKOV CHAINS If d() =g() g( 1), we have d( +1)=d() 2. If d( 0 )isfinitefor an 0,theng() willbelessthan 1forall larger than some 1, which implies that g() will be negative for su cientl large, acontradiction. We conclude g is identicall infinite. Proposition Suppose a chain is irreducible. (a) If there exists a positive recurrent state, then there is a stationar distribution. (b) If there is a stationar distribution, all states are positive recurrent. (c) If there exists a transient state, all states are transient. (d) If there exists a null recurrent state, all states are null recurrent. Proof. To show (a), suppose x is positive recurrent. We have seen that is a stationar measure. Then µ x (S) = T x 1 µ x () =E x T x 1 µ x () =E x 1 (n=) 1=E x T x < 1. Therefore µ() =µ()/e x T x will be a stationar distribution. definition of µ x we have µ x (x) =1,henceµ(x) > 0. From the For (b), suppose µ(x) > 0forsomex. If is another state, choose n so that p n (x, ) > 0, and then from µ() = x µ(x)p n (x, ) we conclude that µ() > 0. Then 0 <µ() =1/E T,whichimpliesE T < 1. We showed that if x is recurrent and r(x, ) > 0, then is recurrent. So (c) follows. Suppose there exists a null recurrent state. If there exists a positive recurrent or transient state as well, then b (a) and (b) or b (c) all states are positive recurrent or transient, a contradiction, and (d) follows.

9 11.6. CONVERGENCE Convergence Our goal is to show that under certain conditions p n (x, )! (), where is the stationar distribution. (In the null recurrent case p n (x, )! 0.) Consider a random walk on the set {0, 1}, wherewithprobabilitoneon each step the chain moves to the other state. Then p n (x, ) =0ifx 6= and n is even. A less trivial case is the simple random walk on the integers. We need to eliminate this periodicit. Suppose x is recurrent, let I x = {n 1:p n (x, x) > 0}, andletd x be the g.c.d. (greatest common divisor) of I x. d x is called the period of x. Proposition If r(x, ) > 0, then d = d x. Proof. Since x is recurrent, r(, x) > 0. Choose K and L such that p K (x, ),p L (, x) > 0. p K+L+n (, ) p L (, x)p n (x, x)p K (x, ), so taking n =0,wehavep K+L (, ) > 0, or d divides K + L. Sod divides n if p n (x, x) > 0, or d is a divisor of I x. Hence d divides d x.bsmmetr d x divides d. Proposition If d x =1, there exists m 0 such that p m (x, x) > 0 whenever m m 0. Proof. First of all, I x is closed under addition: if m, n 2 I x, p m+n (x, x) p m (x, x)p n (x, x) > 0. Secondl, if there exists N such that N,N +1 2 I x,letm 0 = N 2. If m m 0,thenm N 2 = kn + r for some r<n and m = r + N 2 + kn = r(n +1)+(N r + k)n 2 I x. Third, pick n 0 2 I x and k>0suchthatn 0 + k 2 I x. If k =1,weare done. Since d x =1,thereexistsn 1 2 I x such that k does not divide n 1.

10 102 CHAPTER 11. MARKOV CHAINS We have n 1 = mk + r for some 0 <r<k. Note (m +1)(n 0 + k) 2 I x and (m +1)n 0 + n 1 2 I x. The di erence between these two numbers is (m +1)k n 1 = k r<k.sonowwehavetwonumbersini k di ering b less than or equal to k 1. Repeating at most k times, we get two numbers in I x di ering b at most 1, and we are done. We write d for d x.achainisaperiodicifd =1. If d>1, we sa x if p kd (x, ) > 0forsomek>0WedivideS into equivalence classes S 1,...S d.everd steps the chain started in S i is back in S i.sowelookatp 0 = p d on S i. Theorem Suppose the chain is irreducible, aperiodic, and has a stationar distribution. Then p n (x, )! () as n!1. Proof. The idea is to take two copies of the chain with di erent starting distributions, let them run independentl until the couple, i.e., hit each other, and then have them move together. So define 8 >< p(x 1,x 2 )p( 1, 2 ) if x 1 6= 1, q((x 1, 1 ), (x 2, 2 )) = p(x 1,x 2 ) if x 1 = 1,x 2 = 2, >: 0 otherwise. Let Z n =( n,y n )andt =min{i : i = Y i }.Wehave while Subtracting, P( n = ) =P( n =, T apple n)+p( n =, T > n) = P(Y n =, T apple n)+p( n =, T > n), P(Y n = ) =P(Y n =, T apple n)+p(y n =, T > n). P( n = ) P(Y n = ) apple P( n =, T > n) P(Y n =, T > n) Using smmetr, apple P( n =, T > n) apple P(T >n). P( n = ) P(Y n = ) applep(t >n).

11 11.6. CONVERGENCE 103 Suppose we let Y 0 have distribution and 0 = x. Then p n (x, ) () applep(t >n). It remains to show P(T > n)! 0. To do this, consider another chain W n =( n,y n ), where now we take n,y n independent. Define r((x 1, 1 ), (x 2, 2 )) = p(x 1,x 2 )p( 1, 2 ). The chain under the transition probabilities r is irreducible. To see this, there exist K and L such that p K (x 1,x 2 ) > 0andp L ( 1, 2 ) > 0. If M is large, p L+M (x 2,x 2 ) > 0andp K+M ( 2, 2 ) > 0. So p K+L+M (x 1,x 2 ) > 0and p K+L+M ( 1, 2 ) > 0, and hence we have r K+L+M ((x 1,x 2 ), ( 1, 2 )) > 0. It is eas to check that 0 (a, b) = (a) (b) isastationardistributionfor W. Hence W n is recurrent, and hence it will hit (x, x), hence the time to hit the diagonal {(, ) : 2S}is finite. However the distribution of the time to hit the diagonal is the same as T.

Lecture 7. µ(x)f(x). When µ is a probability measure, we say µ is a stationary distribution.

Lecture 7. µ(x)f(x). When µ is a probability measure, we say µ is a stationary distribution. Lecture 7 1 Stationary measures of a Markov chain We now study the long time behavior of a Markov Chain: in particular, the existence and uniqueness of stationary measures, and the convergence of the distribution