Transience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient:

Size: px

Start display at page:

Download "Transience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient:"

Clemence Turner
5 years ago
Views:

1 Stochastic2010 Page 1 Long-Time Properties of Countable-State Markov Chains Tuesday, March 23, :14 PM Homework 2: if you turn it in by 5 PM on 03/25, I'll grade it by 03/26, but you can turn it in by class on 03/26 with no late penalty. Can turn in to my office Amos Eaton 310 or mailbox in Amos Eaton 301. We need to revisit the classification of classes in a Markov chain when the number of states is not necessarily finite. We have the following qualitative differences: Transience: Whereas a finite closed communication class must be recurrent, an infinite closed communication class can be transient: Classical example is a biased random walk: Recurrence: One needs to distinguish between two kinds of behavior of a recurrent state (or a recurrent class) when it's infinite in size: A state j is said to be positive recurrent provided that: A state j is said to be null recurrent provided that: To see why null recurrence can happen note:

2 Stochastic2010 Page 2 One can show through a tedious combinatorial calculation (we won't do; but see the texts like Karlin & Taylor or Resnick) that for an unbiased random walk in d dimensions. That means that random walks in dimensions less than or equal to two are null recurrent. This is sort of counterintuitive to the general public -- see Feller, Introduction to Probability Theory, Vol 1. Ch. 3. On the other hand (and this requires a closer inspection of the calculation), random walks in more than two dimensions are transient. For finite communication classes, recurrence is equivalent to positive recurrence, so you can only have null recurrence in an infinite communication class. Positive recurrence and null recurrence (as well as transience) are class properties,meaning that they are the same for all states in a communication class. (So just need to compute for one representative state in each class). This leads us to two fundamental questions: a. How does a Markov chain behave in the long run when it starts in a communication class of one of these three types? b. How does one determine the type (transience, null recurrence, positive recurrence) of a communication class in an infinite Markov chain? We can reduce the discussion to the case of irreducible Markov chains because: a closed communication class can be treated as its own irreducible Markov chain a communication class which is not closed is automatically transient Let's turn to the first question and assume for the moment we know the type of our irreducible Markov chain. Positive recurrent Markov chain: This is the generalization of recurrent finite-state Markov chains (since any recurrent finitestate irreducible Markov chain is positive recurrent), and so we are guaranteed a unique stationary distribution:

3 Stochastic2010 Page 3 Law of large numbers for Markov chains applies. If the Markov chain is aperiodic, then the stationary distribution is also a limit distribution: See Resnick Secs and 2.13 for proofs. Null recurrent Markov chain: These are a little bit unusual but as we said do apply to some canonical models like unbiased random walks. Null recurrent Markov chains do not have a stationary distribution but instead have an invariant measure: but need not be normalized! An irreducible null recurrent Markov chain has an invariant measure which is unique up to constant positive multiple. (Resnick Sec. 2.12) In fact, it is true that: Long-time behavior (Resnick Sec. 2.12, 2.13):

4 Stochastic2010 Page 4 So what this means is that in an irreducible null recurrent Markov chain, if I start at a given state i, then I will repeatedly visit any other state j infinitely often but the probability to be seen at state j at a particular epoch is vanishingly small for large time. This just means that a null recurrent Markov chain has the proclivity to wander off far away for long periods of time before inevitably returning once in a while to where it was before. Transient Markov chain: Moreover, the number of times the Markov chain returns to state j is given by a geometric distribution. Transient Markov chains can be analyzed in the countable state case with the same formulas that were developed for the finite state case; one just is working with infinite sums and infinite matrices, but all the operations are guaranteed to converge meaningfully. For example, absorption probability calculations and expected accumulated reward/loss calculations work the same way as before. Classification of Countable-State Markov Chains First consider, through purely topological considerations, how the Markov chain decomposes into communication classes.

5 Stochastic2010 Page 5 Next one looks at classes that are not closed -- these must be transient (C 1 and C 3 ). Finite closed communication classes are automatically positive recurrent. Infinite closed communication classes (C 2 ) require closer analysis - -can be any of the three types. Again we can think of closed communication classes as irreducible Markov chains so we just concentrate on irreducible Markov chains from hereon. 1. Existence of a stationary distribution for an irreducible Markov chain is equivalent to the Markov chain being positive recurrent. Why is this true? We've already shown that positive recurrent irreducible Markov chains have a (unique) stationary distribution. Why does the existence of a stationary distribution imply positive recurrence? Because for null recurrent and transient Markov chains, we have: This can be justified by dominated convergence theorem since

6 Stochastic2010 Page 6 since This in particular means that stationary distributions are incompatible with null recurrent and transient Markov chains because for a stationary distribution: 2. Every recurrent Markov chain has an invariant measure, so in other words, an irreducible Markov chain that doesn't have an invariant measure must be transient. These two facts together may still not distinguish between null recurrent and transient case if we happen to find an invariant measure that we can't normalize into a stationary distribution. 3. Decisive test for transience (Karlin and Taylor Sec. 3.4) In an irreducible Markov chain, choose any reference state and let Q be the matrix obtained by deleting the i * th row and column from the probability transition matrix P. If the only bounded, nonnegative (column vector) solution x to the equation x = Q x is x=0 (the column vector with all zero entries), then the Markov chain is recurrent. Otherwise it is transient. The derivation of this transience criterion will be presented in next lecture. Actually a related criterion is also useful: With the same setup, if the equation Qx=x has any solution with then the Markov chain is recurrent. (But not an if and only if). Summary of classification procedure for countable-state Markov chains 1. Decompose the Markov chain into communication classes based on topology i. Any communication class that is not closed must be transient 2. Any communication class that is closed can be considered as its own irreducible Markov chain with state space identified with that communication class. i. Look for an invariant measure

7 Stochastic2010 Page 7 ii. 1) If you find an invariant measure that can be normalized into a stationary distribution, then that class is positive recurrent. 2) If you prove no invariant measure exists, then the class must be transient. 3) If you find an invariant measure that can't be normalized, then must try something else; could be null recurrent or transient. Now choose any convenient reference state i *, delete the corresponding row and column from the probability transition matrix P and call this matrix Q. Look for solutions to x=qx. 1) If you find a nonzero, nonnegative, bounded solution, then the class must be transient. 2) If you find an unbounded solution, then the Markov chain must be recurrent. 3) If you prove that the only solution that is nonnegative and bounded is the trivial solution x=0, then the class is recurrent. Assuming you can analyze both of the equations, then you can definitely classify the Markov chain, and you can do parts i) and ii) in either order.

Classification of Countable State Markov Chains

Classification of Countable State Markov Chains Friday, March 21, 2014 2:01 PM How can we determine whether a communication class in a countable state Markov chain is: transient null recurrent positive