Lecture 9 Classification of States

Transcription

1 Lecture 9: Classification of States of 27 Course: M32K Intro to Stochastic Processes Term: Fall 204 Instructor: Gordan Zitkovic Lecture 9 Classification of States There will be a lot of definitions and some theory before we get to examples. You might want to peek ahead notions are being introduced; it will help your understanding. THE COMMUNICATION RELATION Let {X n } n N0 be a Markov chain on the state space S. For a given set B of states, define the hitting time τ(b) of B as τ B = min{n N 0 : X n B}. (9.) We know that τ B is, in fact, a stopping time with respect to {X n } n N0. When B consists of only one element B = {i}, we simply write τ i for τ {i} ; τ i is the first time the Markov chain {X n } n N0 hits the state i. As always, we allow τ B to take the value +; it means that no state in B is ever hit. The hitting times are important both for immediate applications of {X n } n N0, as well as for better understanding of the structure of Markov chains. Example 9.. Let {X n } n N0 be the chain which models a game of tennis (see Lecture 8). The probability of winning for (say) Amélie can be phrased in terms of hitting times: P[Amélie wins] = P[τ ia < τ ib ], where i A = Amélie wins and i B = Björn wins (the two absorbing states of the chain). We will learn how to compute such probabilities in the subsequent lectures. Having introduced the hitting times τ B, let us give a few more definitions. Recall that the notation P i [A] is used to mean P[A X 0 = i] (for any event A). In practice, we use P i to signify that we are starting the chain from the state i, i.e., P i corresponds to a Markov chain whose transition matrix is the same as the one of {X n } n N0, but the initial distribution is given by P i [X 0 = j] = 0 if j = i and P i [X 0 = i] =.

2 Lecture 9: Classification of States 2 of 27 Definition 9.2. A state i S is said to communicate with the state j S, denoted by i j if P i [τ j < ] > 0. Intuitively, i communicates with j is there is a non-zero chance that the Markov chain X will eventually visit j if it starts from i. Sometimes we also say that j is a consequent of i, that j is accessible from i, or that j follows i. Example 9.3. In the tennis example, every state is accessible from (0, 0) (the fact that p (0, ) is important here), but (0, 0) is not accessible from any other state. The consequents of (40, 40) are (40, 40) itself, (40, Adv), (Adv, 40), Amélie wins and Björn wins. Before we examine some properties of the relation, here is a simple but useful characterization and it is based on the following fact about probability: let {A n } n N0 be an increasing sequence of events, i.e., A n A n+, for all n N 0. Then P[ n N0 A n ] = lim n P[A n ], and the sequence inside the limit is non-decreasing. Proposition 9.4. i j if and only if > 0 for some n N 0. Proof. The event A = {τ j < } can be written as an increasing union Therefore, A = n N A n, where A n = {τ j n}. P i [τ j < ] = P i [A] = lim n P i [A n ] = lim n P i [τ j n], and the sequence P i [τ j n], n N is non-decreasing. In particular, P i [τ j < ] P i [A n ], for all n N. (9.2) Suppose, first, that visited, we have > 0 for some n. Since τ j is the first time j is P i [A n ] = P i [τ j n] P i [X j = n] = > 0. By (9.2), we have P i [τ j < ] > 0, and so, i j. Conversely, suppose that i j, i.e., P i [A] > 0. Since P i [A] = lim n P i [A n ], we must have P i [A n0 ] > 0 for some n 0 (and then all larger

3 Lecture 9: Classification of States 3 of 27 n), i.e., P i [τ j n 0 ] > 0. When τ j n 0, we must have X 0 = j or X = j or... or X n = j, i.e., and so {τ j n 0 } n 0 k=0 {X k = j}, 0 < P i [τ j n 0 ] P i [ n n 0 0 k=0 {X k = j}] P i [X k = j]. k=0 Therefore, P i [X n = j] for at least one n {0,,..., n 0 }. In other words, > 0, for at least one n {0,,..., n 0 }. Proposition 9.5. For all i, j, k S, we have. i i, 2. i j, j k implies i k. Proof.. If we start from state i S we are already there (note that 0 is allowed as a value for τ B in (9.)), i.e., τ i = 0 when X 0 = i. 2. Using Proposition 9.4, it will be enough to show that ik > 0 for some n N. By the same Proposition, we know that p (n ) > 0 and p (n 2) jk > 0 for some n, n 2 N 0. By the Chapman-Kolmogorov relations, with n = n + n 2, we have ik = p (n ) il p (n 2) lk p (n ) p (n 2) jk > 0. l S Remark 9.. The inequality ik p (n ) il p (n 2) lk is valid for all i, l, k S, as long as n + n 2 = n. It will come in handy later. CLASSES Definition 9.7. We say that the states i and j in S intercommunicate, denoted by i j if i j and j i. A set B S of states is called irreducible if i j for all i, j S. Unlike the relation of communication, the relation of intercommunication is symmetric. Moreover, we have the following three properties (the result follows directly from Proposition 9.4, so we omit it): Proposition 9.8. The relation is an equivalence relation of S, i.e., for all i, j, k S, we have

4 Lecture 9: Classification of States 4 of 27. i i (reflexivity), 2. i j implies j i (symmetry), and 3. i j, j k implies i k (transitivity). The fact that is an equivalence relation allows us to split the state-space S into equivalence classes with respect to. In other words, we can write S = S S 2 S 3..., where S, S 2,... are mutually exclusive (disjoint) and all states in a particular S n intercommunicate, while no two states from different equivalence classes S n and S m do. The sets S, S 2,... are called classes of the chain {X n } n N0. Equivalently, one can say that classes are maximal irreducible sets, in the sense that they are irreducible and no class is a subset of a (strictly larger) irreducible set. A cookbook algorithm for class identification would involve the following steps:. Start from an arbitrary state (call it ). 2. Identify all states j that communicate with it - don t forget that always i i, for all i. 3. That is your first class, call it C. If there are no elements left, then there is only one class C = S. If there is an element in S \ C, repeat the procedure above starting from that element. The notion of a class is especially useful in relation to another natural concept: Definition 9.9. A set B S of states is said to be closed if i j for all i B and all j S \ B. A state i S such that the set {i} is closed is called absorbing. Here is a nice characterization of closed sets: Proposition 9.0. A set B of states is closed if and only if p = 0 for all i B and all j B c = S \ B. Proof. Suppose, first, that B is closed. Then for i B and j B c, we have i j, i.e., = 0 for all n N. In particular, p = 0. Conversely, suppose that p = 0 for all i B, j B c. We need to show that i j (i.e. = 0 for all n N) for all i B, j B c. Suppose, to the contrary, that there exist i B and j B c such that > 0 for some n N. Since p = 0, we must have n >. Out

5 Lecture 9: Classification of States 5 of 27 of all n > such that ik > 0 for some k B c, we pick the smallest one (let us call it n 0 ), so that that p (n 0 ) ik = 0 for all k B c. By the Chapman-Kolmogorov relation, we have = k S p (n ) ik p kj = k B p (n ) ik p kj + p (n ) ik p kj. (9.3) k B c The terms in the first sum in the second line of (9.3) are all zero because p kj = 0 (k B and j B c ), and the terms of the second one are also all zero because p (n ) ik contradiction. = 0 for all k B c. Therefore, = 0 - a Intuitively, a set of states is closed if it has the property that the chain {X n } n N0 stays in it forever, once it enters it. In general, if B is closed, it does not have to follow that S \ B is closed. Also, a class does not have to be closed, and a closed set does not have to be a class. Here are some examples: Example 9.. Consider the tennis chain of the previous lecture and consider the following three sets of states:. B = { Amélie wins }: closed and a class, but S \ B is not closed 2. B = S \ {(0, 0)}: closed, but not a class, and 3. B = {(0, 0)}: class, but not closed. There is a relationship between classes and the notion of closedness: Proposition 9.2. Every closed set B is a union of classes. Proof. Let ˆB be the union of all classes C such that C B =. In other words, take all the elements of B and throw in all the states which intercommunicate with them. I claim that ˆB = B. Clearly, B ˆB, so we need to show that ˆB B. Suppose, to the contrary, that there exists j ˆB \ B. By construction, j intercommunicates with some i B. In particular i j. By closedness of B, we must have j B. This is a contradiction with the assumptions that j ˆB \ B. Example 9.3. A converse of Proposition 9.2 is not true. Just take the set B = {(0, 0), (0, 5)} in the tennis example. It is a union of classes, but it is not closed.

6 Lecture 9: Classification of States of 27 TRANSIENCE AND RECURRENCE It is often important to know whether a Markov chain will ever return to its initial state, and if so, how often. The notions of transience and recurrence address this questions. Definition 9.4. The (first) visit time to state j, denoted by τ j () is defined as τ j () = min{n N : X n = j}. As usual τ j () = + if X n = j for all n N. Note that the definition of the random variable τ j () differs from the definition of τ j in that the minimum here is take over the set N of natural numbers, while the set of non-negative integers N 0 is used for τ j. When X 0 = j, the hitting time τ j and the visit time τ j () coincide. The important difference occurs when X 0 = j. In that case τ j = 0 (we are already there), but it is always true that τ j (). It can even happen that P i [τ i () = ] =. Definition 9.5. A state i S is said to be. recurrent if P i [τ i () < ] =, 2. positive recurrent if E i [τ i ()] < (E i means expectation when the probability is P i ), 3. null recurrent if it is recurrent, but not positive recurrent, 4. transient if it is not recurrent. A state is recurrent if we are sure we will come back to it eventually (with probability ). It is positive recurrent if the time between two consecutive visits has finite expectation. Null recurrence means the we will return, but the waiting time may be very long. A state is transient is there is a positive chance (however small) that the chain will never return to it. Remember that the greatest common denominator (GCD) of a set A of natural numbers if the largest number d such that d divides each k A, i.e., such that each k A is of the form k = ld for some l N. Definition 9.. A period d(i) of a state i S is the greatest common denominator of the return-time set R(i) = {n N : > 0} of the state i. When R(i) =, we set d(i) =. A state i S is called aperiodic if d(i) =.

7 Lecture 9: Classification of States 7 of 27 Example 9.7. Consider the Markov chain with three states and the transition matrix 0 0 P = The return set for each state i {, 2, 3} is given by R(i) = {3,, 9, 2,... }, so d(i) = 3 for all i {, 2, 3}. However, if we change the probabilities a bit: 0 0 ˆP = 0 0, the situation changes drastically: so that d(i) = for i {, 2, 3} R() = {3, 4, 5,,... }, R(2) = {2, 3, 4, 5,,... }, R(3) = {, 2, 3, 4, 5,,... }, A CRITERION FOR RECURRENCE The definition of recurrence from above is conceptually simple, but it gives us no clue about how to actually go about deciding whether a particular state in a specific Markov chain is recurrent. A criterion stated entirely in terms of the transition matrix P would be nice. Before we give it, we need to introduce some notation. For two (not necessarily different) states, i, j S, let f (n) be the probability that it will take exactly n steps for the first visit to j (starting from i) to occur: f (n) = P i [τ j () = n] = P[X n = j, X n = j, X n 2 = j,..., X 2 = j, X = j X 0 = i]. Let f (no superscript) denote the probability that j will be reached from i eventually, i.e., f = P i [τ j () < ] = Clearly, we have the following equivalence: The reason the quantities f (n) relationship: f (n). n= i is recurrent if and only if f =. are useful lies in the following recursive

8 Lecture 9: Classification of States 8 of 27 Proposition 9.8. For n N and i, j S, we have = n m= f (m) p (n m). Proof. In preparation for the rest of the proof, let us reiterate that the event {τ j () = m} can be written as {τ j () = m} = {X m = j, X m = j, X m 2 = j,..., X = j}. Using the law of total probability, we can split the event {X n = j} according to the value of the random variable τ j () (the values of τ j () larger than n do not appear in the sum since X n cannot be equal to j in those cases): n = P i [X n = j] = P i [X n = j τ j () = m] P i [τ j () = m] m= = = n m= P i [X n = j X m = j, X m = j, X m 2 = j,..., X = j] f (m) n P i [X n = j X m = j] f (m) = m= Where did we use the Markov Property? n m= p (n m) f (m) An important corollary to Proposition 9.8 is the following characterization of recurrence: Proposition 9.9. A state i S is recurrent if and only if = +. n= Proof. For i = j, Proposition 9.8 states that = n m= p (n m) f (m). (9.4) For N N, we define N s N =, n= so that, if we sum (9.4) over all n from to N N, we get s N = N n= n m= p (n m) f (m) = N n= N m= {m n} p (n m) f (m) (9.5) Then, we interchange the order of summation in the last sum in (9.5) and remember that p (0) = to get s N = N m= N n=m p (n m) f (m) = N f (m) m= N m = n=0 N f (m) ( + s N m ) m=

9 Lecture 9: Classification of States 9 of 27 The sequence {s N } N N is non-decreasing, so s N N f (m) ( + s N ) ( + s N ) f. m= Therefore, f lim N s N + s N. If n= p(n) = +, then lim N s N =, and, so, lim N s N +s N =, so f. On the other hand, f = P i [τ i () < ], so f. Therefore, f = and the state i is recurrent. Conversely, suppose that i is recurrent, i.e., f =. We can repeat the procedure from above (but with + instead of N and using Tonelli s theorem to interchange the order of the sums) to get that = f (m) n= m= ( + This can happen only if m= p(n) n= = +. ) = + n=. Here is an application of our recurrence criterion - a beautiful and unexpected result of George Pólya from 92. Example 9.20 (Optional). In addition to the simple symmetric random walk on the line (d = ) we studied before, one can consider random walks whose values are in the plane (d = 2), the space (d = 3), etc. These are usually defined as follows: the random walk in d dimensions is the Markov chain with the state space S = Z d ; starting from the state (x,..., x d ), it picks one of its 2d neighbors (x +,..., x d ), (x,..., x d ), (x, x 2 +,..., x d ), (x, x 2,..., x d ),..., (x,..., x d + ), (x,..., x d ) randomly and uniformly and moves there. In this problem, we are interested in recurrence properties of the d-dimensional random walk and and their dependence on d. We already know that the simple symmetric random walk on Z is recurrent (i.e., every i Z is a recurrent state). The easiest way to proceed when d 2 is to use Proposition 9.8, i.e., to estimate the values, for n N. By symmetry, we can focus on the origin, i.e., it is enough to compute, for each n N, = 00 = P 0[X n = (0, 0,..., 0)]. For that, we should count all trajectories from (0,..., 0) that return to (0,..., 0) in n steps. First of all, it is clear that n needs to be even, i.e., n = 2m, for some m N. It helps if we think of any trajectory as a sequence of increments ξ,..., ξ n, where each ξ i takes its value in the set {,, 2, 2,..., d, d}. In words, ξ i = +k if the k-th coordinate

10 Lecture 9: Classification of States 0 of 27 increases by on the i-th step, and ξ i = k, if the k-th coordinate decreases. This way, the problem becomes combinatorial: In how many ways can we put one element of the set {,, 2, 2,..., d, d} into each of n = 2m boxes so that the number of boxes with k in them equals to the number of boxes with k in them? To get the answer, we start by fixing a possible count (i,..., i d ), satisfying i + + i d = m of the number of times each of the values in {, 2,..., d} occurs. These values have to be placed in m of the 2m slots and their negatives (possibly in a different order) in the remaining m slots. So, first, we choose the positive slots (in ( 2m m ) ways), and then distribute i ones, i 2 twos, etc., in those slots; this can be done in 2 ( ) m i i 2... i d ways. This is also the number of ways we can distribute the negative ones, twos, etc., in the remaining slots. All in all, for fixed i, i 2,..., i d, all of this can be done in For d = 2 we could have used the values up, down, left and right, for,, 2 or 2, respectively. In dimension 3, we could have added forward and backward, but we run out of words for directions for larger d. 2 ( m i...i ) is called the multinomial coefficient. It counts the number of ways we d can color m objects into one of d colors such that there are i objects of color, i 2 of color 2, etc. It is a generalization of the binomial coefficient and is given by ( m i i 2... i d ) = m! i!i 2!... i d!. ( )( 2m m ) 2 m i i 2... i d ways. Remembering that each path has the probability (2d) 2m, and summing over all i,..., i d with i + + i d = m, we get p (2m) = (2d) 2m ( ) 2m m i + +i d =m For d =, the situation is somewhat simpler: p (2m) = ( ) 2m 4 m m ( m i i 2... i d It is still too complicated sum over all m N, but we can simplify it further by using Stirling s formula n! 2πn ( n e ) n, where a n b n means lim n a n /b n =. Indeed, from there, ( ) 2m 4m, (9.) m πm ) 2. and so p (2m) mπ. It is easy to apply our criterion: p (2m) =, m=

11 Lecture 9: Classification of States of 27 and we recover our previous conclusion that the simple symmetric random walk is recurrent. Moving on to the case d = 2, we notice that the sum of the multinomial coefficients no longer equals ; in fact it is given by 3 m i=0 ( ) m 2 = i ( ) 2m, (9.7) m 3 Why is this identity true? Can you give a counting argument? and, so, and p (2m) = m ( 4 m πm ) 2 πm, p (2m) = +, m= which implies that the two-dimensional random walk is also recurrent. How about d 3? Things are even more complicated now. The multinomial sum does not admit a nice closed-form expression as in (9.7), so we need to do some estimates; these are a bit tedious so we skip them, but report the punchline, which is that p (2m) C( 3m ) 3/2, for some constant C. This is where it gets interesting: this series converges: p (2m) <, m= and, so, the random walk is transient for d = 3. This is enough to conclude that the random walk is transient for all d 3, too (why?). In a nutshell, the random walk is recurrent for d =, 2, but transient for d 3. CLASS PROPERTIES Certain properties of states are shared between all elements in a class. Knowing which properties have this feature is useful for a simple reason - if you can check them for a single class member, you know automatically that all the other elements of the class share it. Definition 9.2. A property is called a class property it holds for all states in its class, whenever it holds for any one particular state in the that class. Put differently, a property is a class property if and only if either all states in a class have it or none does.

12 Lecture 9: Classification of States 2 of 27 Proposition Transience and recurrence are class properties. Proof. Suppose that the state i is recurrent, and that j is in its class, i.e., that i j. Then, there exist natural numbers m and k such that p (m) > 0 and p (k) ji > 0. By the Chapman-Kolmogorov relations, for each n N, we have p (n+m+k) = l S l 2 S p (k) jl l l 2 p (m) l 2 m p(k) ji p (m). In other words, there exists a positive constant c (take c = p (k) ji p (m) ), independent of n, such that p (n+m+k) c. Therefore, by recurrence of i we have n= p(n) n= = n=m+k+ p (n+m+k) n= c = +, and n= = +, and so, j is recurrent. Therefore, recurrence is a class property. Since transience is just the opposite of recurrence, it is clear that transience is also a class property. Proposition Period is a class property, i.e., all elements of a class have the same period. Proof. Let d = d(i) be the period of the state i, and let j i. Then, there exist natural numbers m and k such that p (m) > 0 and p (k) ji > 0. By Chapman-Kolmogorov, p (m+k) p (m) p (k) ji > 0, and so m + k R(i). Similarly, for any n R(j), p (m+k+n) p (m) p (k) ji > 0, so m + k + n R(i). By the definition of th period, we see now that d(i) divides both m + k and m + k + n, and, so, it divides n. This works for each n R(j), so d(i) is a common divisor of all elements of R(j); this, in turn, implies that d(i) d(j). The same argument with roles of i and j switched shows that d(j) d(i). Therefore, d(i) = d(j). THE CANONICAL DECOMPOSITION Now that we know that transience and recurrence are class properties, we can introduce the notion of canonical decomposition of a Markov

13 Lecture 9: Classification of States 3 of 27 chain. Let S, S 2,... be the collection of all classes; some of them contain recurrent states and some transient ones. Proposition 9.22 tells us that if there is one recurrent state in a class, than all states in the class must be recurrent. This, it makes sense to call the whole class recurrent. Similarly, the classes which are not recurrent consists entirely of transient states, so we call them transient. There are at most countably many states, so the number of all classes is also at most countable. In particular, there are only countably (or finitely) many recurrent classes, and we usually denote them by C, C 2,.... Transient classes are denoted by T, T 2,.... There is no particular rule in the choice of indices, 2, 3,... for particular classes. The only point is that they can be enumerated because there are at most countably many of them. The distinction between different transient classes is usually not very important, so we pack all transient states together in a set T = T T Definition Let S be the state space of a Markov chain {X n } n N0. Let C, C 2,... be its recurrent classes, and T, T 2,... the transient ones, and let T = T T 2... be their union. The decomposition S = T C C 2 C 3..., is called the canonical decomposition of the (state space of the) Markov chain {X n } n N0. The reason that recurrent classes are important is simple - they can be interpreted as Markov chains themselves. In order for such an interpretation to be possible, we need to make sure that the Markov chain stays in a recurrent class if it starts there. In other words, we have the following important proposition: Proposition Recurrent classes are closed. Proof. Suppose, contrary to the statement of the Proposition, that there exist two states i = j such that. i is recurrent, 2. i j, and 3. j i. The idea of the proof is the following: whenever the transition i j occurs (perhaps in several steps), then the chain will never return to i, since j i. By recurrence of i, that is not possible. Therefore i j - a contradiction.

14 Lecture 9: Classification of States 4 of 27 More formally, the recurrence of i means that (starting from i, i.e., under P i ) the event A = {X n = i, for infinitely many n N} has probability, i.e., P i [A] =. The law of total probability implies that = P i [A] = P i [A τ j () = n]p i [τ j () = n]+ n N + P i [A τ j () = +] P i [τ j () = +]. (9.8) On the event τ j () = n, the chain can visit the state i at most n times, because it cannot hit i after it visits j (remember j i). Therefore, P[A τ j () = n] = 0, for all n N. It follows that = P i [A τ j () = +] P i [τ j () = +]. Both of the terms on the right-hand side above are probabilities whose product equals, which forces both of them to be equal to. In particular, P i [τ j () = +], or, phrased differently, i j - a contradiction. Together with the canonical decomposition, we introduce the canonical form of the transition matrix P. The idea is to order the states in S with the canonical decomposition in mind. We start from all the states in C, followed by all the states in C 2, etc. Finally, we include all the states in T. The resulting matrix looks like this P P P = 0 0 P , Q Q 2 Q where the entries should be interpreted as matrices: P is the transition matrix within the first class, i.e., P = (p, i C, j C ), etc. Q k contains the transition probabilities from the transient states to the states in the (recurrent) class C k. Note that Proposition 9.25 implies that each P k is a stochastic matrix, or, equivalently, that all the entries in the row of P k outside of P k are zeros. We finish the discussion of canonical decomposition with an important result and one of its consequences. Proposition 9.2. Suppose that the state space S is finite. Then there exists at least one recurrent state.

15 Lecture 9: Classification of States 5 of 27 Proof. The transition matrix P is stochastic, and so are all its powers P n. In particular, if we sum the entries in each row, we get =, for all i S. j S Summing the above equality over all n N and switching the order of integration gives + = = n N n N j S = j S n N. We conclude that n N = + for at least one state j (this is where the finiteness of S is crucial). We claim that j is a recurrent state. Suppose, to the contrary, that it is transient. If we sum the equality from Proposition 9.8 over n N, we get = n N n N = m N n m= f (m) p (n m) f (m) = n=0 m N n=m = f. n=0 p (n m) f (m) (9.9) Transience of j implies that n= p(n) < and, by definition, f, for any i, j. Therefore, n N < - a contradiction. Remark If S is not finite, it is not true that recurrent states must exist. Just remember the Deterministically-Monotone Markov Chain example, or the random walk with p = 2. All states are transitive there. In a finite state-space case, we have the following dichotomy: Corollary A class of a Markov chain on a finite state space is recurrent if and only if it is closed. Proof. We know that recurrent classes are closed. In order to show the converse, we need to prove that transient classes are not closed. Suppose, to the contrary, the there exists a finite state-space Markov chain with a closed transient class T. Since T is closed, we can see it as a state space of the restricted Markov chain. This, new, Markov chain has a finite number of states so there exists a recurrent state. This is a contradiction with the assumption that T consists only of transient states. Remark Again, finiteness is necessary. For a random walk on Z, all states intercommunicate. In particular, there is only one class Z itself and it it trivially closed. If p = 2, however, all states are transient, and, so, Z is a closed and transient class.

16 Lecture 9: Classification of States of 27 EXAMPLES Random walks: p (0, ). Communication and classes. Clearly, it is possible to go from any state i to either i + or i in one step, so i i + and i i for all i S. By transitivity of communication, we have i i + i + 2 i + k. Similarly, i i k for any k N. Therefore, i j for all i, j S, and so, i j for all i, j S, and the whole S is one big class. Closed sets. The only closed set is S itself. Transience and recurrence We studied transience and recurrence in the lectures about random walks (we just did not call them that). The situation highly depends on the probability p of making an up-step. If p > 2, there is a positive probability that the first step will be up, so that X =. Then, we know that there is a positive probability that the walk will never hit 0 again. Therefore, there is a positive probability of never returning to 0, which means that the state 0 is transient. A similar argument can be made for any state i and any probability p = 2. What happens when p = 2? In order to come back to 0, the walk needs to return there from its position at time n =. If it went up, the we have to wait for the walk to hit 0 starting from. We have shown that this will happen sooner or later, but that the expected time it takes is infinite. The same argument works if X =. All in all, 0 (and all other states) are null-recurrent (recurrent, but not positive recurrent). Periodicity. Starting from any state i S, we can return to it after 2, 4,,... steps. Therefore, the return set R(i) is always given by R(i) = {2, 4,,... } and so d(i) = 2 for all i S. Gambler s ruin: p (0, ). Communication and classes. The winning state a and the losing state 0 are clearly absorbing, and form one-element classes. The other a states intercommunicate among each other, so they form a class of their own. This class is not closed (you can - and will - exit it and get absorbed sooner or later). Transience and recurrence. The absorbing states 0 and a are (trivially) positive recurrent. All the other states are transient: starting from any state i {, 2,..., a }, there is a positive probability (equal to p a i ) of winning every one of the next a i games and, thus, getting absorbed in a before returning to i.

17 Lecture 9: Classification of States 7 of 27 Periodicity. The absorbing states have period since R(0) = R(a) = N. The other states have period 2 (just like in the case of a random walk). Deterministically monotone Markov chain Communication and classes. A state i communicates with the state j if and only if j i. Therefore i j if and only if i = j, and so, each i S is in a class by itself. Closed sets. The closed sets are precisely the sets of the form B = i, i +, i + 2,..., for i N. Transience and recurrence All states are transient. Periodicity. The return set R(i) is empty for each i S, so d(i) =, for all i S. A game of tennis Communication and classes. All the states except for those in E = {(40, Adv), (40, 40), (Adv, 40), Amélie wins, Björn wins} intercommunicate only with themselves, so each i S \ E is in a class by iteself. The winning states Amélie wins and Björn wins are absorbing, and, so, also form classes with one element. Finally, the three states in {(40, Adv), (40, 40), (Adv, 40)} intercommunicate with each other, so they form the last class. Periodicity. The states i in S \ E have have the property that = 0 for all n N, so d(i) =. The winning states are absorbing so d(i) = for i {Amélie wins, Björn wins}. Finally, the return set for the remaining three states is {2, 4,,... } so their period is 2. PROBLEMS Problem 9.. Suppose that all classes of a Markov chain are recurrent, and let i, j be two states such that i j. Only one of the statements below is necessarily true; which one? (a) for each state k, either i k or j k (b) j i (c) p ji > 0 or p > 0 (d) n= p(n) <

18 Lecture 9: Classification of States 8 of 27 (e) None of the above Solution: The correct answer is (b). (a) False. Take a chain with two states, 2 where p = p 22 =, and set i = j =, k = (b) True. Recurrent classes are closed, so i and j belong to the same class. Therefore j i. (c) False. Take a chain with 4 states, 2, 3, 4 where p 2 = p 23 = p 34 = p 4 =, and set i =, j = (d) False. That would mean that j is transient. (e) False Problem 9.2. Consider a Markov Chain whose transition graph is given on the right Identify the classes Find transient and recurrent states. 3. Find periods of all states Compute f (n) 3, for all n N Using Mathematica, we can get that, approximately, P 20 = , where P is the transition matrix of the chain. Compute the probability P[X 20 = 3], if the initial distribution (the distribution of X 0 ) is given by P[X 0 = ] = /2 and P[X 0 = 3] = /2. Solution:. The classes are T = {}, T 2 = {2}, C = {3, 4, 5, } and C 2 = {7, 8} 2. The states in T and T 2 are transient, and the others are recurrent. 3. The periods are all.

19 Lecture 9: Classification of States 9 of For n =, f (n) 3 = 0, since you need at least two steps to go from to 3. For n = 2, the chain needs to follow 2 3, for f (n) 3 = (/2) 2. For n larger than 2, the only possibility is for the chain to stay in the state for n 2 periods, jump to 2 and finish at 3, so f (n) 3 = (/2) n in that case. All together, we have f (n) 3 = 0, n = (/2), n We know from the notes that the distribution of X 20, when represented as a vector a (20) = (a (20), a (20) 2,..., a (20) 8 ) satisfies a (20) = a (0) P 20. By the assumption a (0) = ( 2, 0, 2, 0, 0, 0, 0, 0), so P[X 20 = 3] = a (20) 3 = = Problem 9.3. Identify the recurrent states in the setting of. Problem 8.2, Lecture 8, 2. Problem 8.3, Lecture 8. Solution:. The state (0, 00) is clearly recurrent (it is absorbing). Let us show that all other (0200 of them) are transient. It will be enough to show that i (0, 00), for each i S, because (0, 00) is in a recurrent class by itself. To do that, we just need to find a path from any state (r, b) to (0, 00) through the transition graph: (r, b) (r, b) (0, b) (0, b + ) (0, 00). 2. Once the chain moves to the next state, it can never go back. Therefore, it cannot happen that i j unless i = j, and so, every state is a class of its own. The class {m} is clearly absorbing and, therefore, recurrent. All the other classes are transient since it is possible (and, in fact, certain) that the chain will move on to the next state and never come back. As for periods, all of them are, since is in the return set of each state. Problem 9.4. A fair -sided die is rolled repeatedly, and for n N, the outcome of the n-th roll is denoted by Y n (it is assumed that {Y n } n N are independent of each other). For n N 0, let X n be the remainder (taken in the set {0,, 2, 3, 4}) left after the sum n k= Y k is divided by 5, i.e. X 0 = 0, and X n = n Y k ( mod 5 ), for n N, k=

20 Lecture 9: Classification of States 20 of 27 making {X n } n N0 a Markov chain on the state space {0,, 2, 3, 4} (no need to prove this fact). Write down the transition matrix of the chain, classify the states, separate recurrent from transient ones, and compute the period of each state. Solution: The outcomes, 2, 3, 4, 5, leave remainders, 2, 3, 4, 0,, when divided by 5, so the transition matrix P of the chain is given by P = Since p > 0 for all i, j S, all the states belong to the same class, and, because there is at least one recurrent state in a finite-state-space Markov chain and because recurrence is a class property, all states are recurrent. Finally, is in the return set of every state, so the period of each state is. Problem 9.5. Let {Z n } n N0 be a Branching process (with state space S = {0,, 2, 3, 4,... } = N 0 ) with the offspring probability given by p 0 = /2, p 2 = /2. Classify the states (find classes), and describe all closed sets. Solution: If i = 0, then i is an absorbing state (and, therefore, a class in itself). For i N, p > 0 if j is of the form 2k for some k i (out of i individuals, i k die and the remaining k have two children). In particular, no one ever communicates with odd i N. Therefore, each odd i is a class in itself. For even i, it communicates with all even numbers from 0 to 2i; in particular, i i + 2 and i i 2. Therefore, all positive even numbers intercommunicate and form a class. Clearly, {0} is a closed set. Every other closed set will contain a positive even number; indeed, after one step we are in an even state, no matter where be start from, and this even number could be positive, unless the initial state was 0). Therefore, any closed set different from {0} will have to contain the set of all even numbers 2N 0 = {0, 2, 4,... }, and, 2N 0 is a closed set itself. On the other hand, if we add any collection of odd numbers to 2N 0, the resulting set will still be closed (exiting it would mean hitting an odd number outside of it). Therefore, closed sets are exactly the sets of the form C = {0} or C = 2N 0 D, where D is any set of odd numbers.

21 Lecture 9: Classification of States 2 of 27 Problem 9.. Which of the following statements is true? Give a short explanation (or a counterexample where appropriate) for your choice. {X n } n N0 is a Markov chain with state space S.. f (n) for all i, j S and all n N. 2. If states i and j intercommunicate, then there exists n N such that > 0 and ji > If all rows of the transition matrix are equal, then all states belong to the same class. 4. If f < and f ji <, then i and j do not intercommunicate. 5. If P n I, then all states are recurrent. (Note: We say that a sequence {A n } n N of matrices converges to the matrix A, and we denote it by A n A, if (A n ) A, as n, for all i, j.) Solution:. TRUE. is the probability that X n = j ( given that X 0 = i). On the other hand f (n) is the probability that X n = j and that X k = j for all k =,..., n (given X 0 = i). Clearly, the second event is a subset of the first, so f (n). 2. FALSE. Consider the Markov chain with the transition graph depicted on the right. All states intercommunicate, but 2 > 0 if and only if n is of the form 3k +, for k N 0. On the other hand 2 > 0 if and only if n = 3k + 2, for some k N 0. Thus, 2 and 2 are never simultaneously positive FALSE. Consider a Markov chain with the following transition matrix: [ ] 0 P =. 0 Then is an absorbing state and it is in a class of its own, so it is not true that all states belong to the same class FALSE. Consider the Markov chain with the transition graph on the right: Then the states 2 and 3 clearly intercommunicate. In the event that (starting from 2), the first transition happens to be 2, we are never visiting 3. Otherwise, we visit 3 at time. Therefore, f 23 = /2 <. Similarly, f 32 = /2 <

22 Lecture 9: Classification of States 22 of TRUE. Suppose that there exists a transient state i S. Then n <, and, in particular, 0, as n. This is a contradiction with the assumption that, for all i S. Problem 9.7. Let C be a class in a Markov chain. Only one of the statements below is necessariliy true; which one? (a) C is closed, (b) C c is closed, (c) at least one state in C is recurrent, (d) for all states i, j C, p > 0, (e) none of the above. Solution: The answer is (e). (a) False. Take C = {(0, 0)} in the Tennis example. (b) False. Take C = {Amélie wins} in the Tennis example. (c) False. This is only true in finite chains. Take the Deterministically Monotone Markov Chain as an example. All of its states are transient. (d) False. This would be true if it read for each pair of states i, j C, there exists n N such that > 0. Otherwise, we can use the Tennis chain and the states i = (40, Adv) and j = (Adv, 40). They belong to the same class, but p = 0 (you need to pass through (40, 40) to go from one to another). (e) True. Problem 9.8. In a Markov chain whose state space has n elements (n N). Only one of the statements below is necessariliy true; which one? (a) all classes are closed (b) at least one state is transient, (c) not more than half of all states are transient, (d) there are at most n classes, (e) none of the above. Solution: The answer is (d).

23 Lecture 9: Classification of States 23 of 27 (a) False. In the Tennis example, there are classes that are not closed. (b) False. Just take the Regime Switching with 0 < p 0, p 0 <. Both of the states are recurrent there. Or, simply take a Markov chain with only one state (n = ). (c) False. In the Tennis example, 8 states are transient, but n = 20. (d) True. Classes form a partition of the state space, and each class has at least one element. Therefore, there are at most n classes. (e) False. Problem 9.9. Let C and C 2 be two (different) classes. Only one of the statements below is necessariliy true; which one? (a) i j or j i, for all i C, and j C 2, (b) C C 2 is a class, (c) if i j for some i C and j C 2, then k l for all k C 2 and l C, (d) if i j for some i C and j C 2, then k l for some k C 2 and l C, (e) none of the above. Solution: The answer is (c). (a) False. Take C = {Amélie wins} and C 2 = {Björn wins}. (b) False. Take the same example as above. (c) True. Suppose, to the contrary, that k l for some k C 2 and l C. Then, since j and k are in the same class, we must have j k. Similarly, l i (l and i are in the same class). Using the transitivity property of the communication relation, we get j k l i, and so j i. By the assumption i j, and so, i j. This is a contradiction, however, with the assumption that i and j are in different classes. (d) False. In the Tennis example, take i = (0, 0), j = (30, 0), C = {i} and C 2 = {j}. (e) False.

24 Lecture 9: Classification of States 24 of 27 Problem 9.0. Let i be a recurrent state with period 5, and let j be another state. Only one of the statements below is necessariliy true; which one? (a) if j i, then j is recurrent, (b) if j i, then j has period 5, (c) if i j, then j has period 5, (d) if j i then j is transient, (e) none of the above. Solution: The answer is (c) (a) False. Take j = 0 and i = in the chain in the picture. (b) False. Take the same counterexample as above. (c) True. We know that i is recurrent, and since all recurrent classes are closed, and i j, h must belong to the same class as i. Period is a class property, so the period of j is also 5. (d) False. Take j =, i = 0 in the chain in the picture. (e) False. Problem 9.. Let i and j be two states such that i is transient and i j. Only one of the statements below is necessariliy true; which one? (a) n= p(n) = n= p(n), (b) if i k, then k is transient, (c) if k i, then k is transient, (d) period of i must be, (e) none of the above. Solution: The answer is (c)

25 Lecture 9: Classification of States 25 of 27 (a) False. In the chain above, take i = and j = 2. Clearly, i j, and both of them are transient. A theorem from the notes states that n= p(n) < and n= p(n) <, but these two sums do not need to be equal. Indeed, the only way for i = to come back to itself in n steps is to move to 2, come back to 2 in n 2 steps and then jump to. Therefore, n 3, p (2) = 0.25 and p () = p = 0. Therefore, = p () + p (2) n= + n=3 = p (n 2) 0.25 for = n=. This equality implies that the two sums are equal if and only if 0.75 n= = 0.25, i.e., = = 3. n= n= We know however, that one of the possible ways to go from 2 to 2 in n steps is to just stay in 2 all the time. The probability of this trajectory is (/2) n, and so, (/2) n for all n. Hence, n= n= (/2) n =. Therefore, the two sums cannot be equal. (b) False. Take i = 2 and j = 3 in the chain above. (c) True. Suppose that k i, but k is recurrent. Since recurrent classes are closed, i must be in the same class as k. That would mean, however, that i is also recurrent. This is a contradiction with the assumption that i is transient. (d) False. Take i = 2 in the modification of the example above in which p 22 = 0 and p 2 = p 23 = /2. The state 2 is still transient, but its period is 2. (e) none of the above. Problem 9.2. Suppose there exists n N such that P n = I, where I is the identity matrix and P is the transition matrix of a finite-state-space Markov chain. Only one of the statements below is necessarily true. Which one? (a) P = I, (b) all states belong to the same class, (c) all states are recurrent (d) the period of each state is n,

26 Lecture 9: Classification of States 2 of 27 (e) none of the above. Solution: The answer is (c). (a) False. Take the Regime-switching chain with [ ] 0 P = 0 Then P 2 = I, but P = I. (b) False. If P = I, all states are absorbing, and, therefore, each is in a class of its own. (c) True. By the assumption P kn = (P n ) k = I k = I, for all k N. Therefore, p (kn) = for all k N, and so lim m p (m) = 0 (maybe it even doesn t exist). In any case, the series m= p(m) cannot be convergent, and so, i is recurrent, for all i S. Alternatively, the condition P n = I means that the chain will be coming back to where it started - with certainty - every n steps, and so, all states must be recurrent. (d) False. Any chain satisfying P n = I, but with the property that the n above is not unique is a counterexample. For example, if P = I, then P n = I for any n N. (e) False. Problem 9.3. Let i be a recurrent state with period 3, and let j be another state. Only one of the statements below is necessarily true. Which one? (a) if j i, then j is recurrent (b) if j i, then j has period 3 (c) if i j, then j has period 3 (d) if j i then j is transient (e) none of the above. Solution: The true statement is (c). Since i is recurrent and i j, we have j i, and so, i and j belong to the same class. Periodicity is a class property so the period of i and j are the same, namely 3. Problem 9.4. Let C and C 2 be two (different) communication classes of a Markov chain. Only one of the statements below is necessarily true. Which one?

27 Lecture 9: Classification of States 27 of 27 (a) i j or j i, for all i C, and j C 2 (b) C C 2 is a class (c) if i j for some i C and j C 2, then k l for all k C 2 and l C (d) if i j for some i C and j C 2, then k l for some k C 2 and l C, (e) none of the above. Solution: The correct answer is (c). Suppose that i j and k l for some k C 2 and some l C. Since both j and k are in the same class (and so are i and l), we have j k (and l i), and then, by transitivity i j k l i which implies that j k. Therefore, j and k are in the same class, which is in contradiction with the assumption that C and C 2 are different classes.