December 15, 2011
In last discussion we studied the transience and recurrence of Markov chains There are 2 other closely related issues about Markov chains that we address Is there an invariant distribution? Does the chain converge to this distribution? (Asymptotic behavior) For irreducible finite MC, if R is an aperiodic recurrent class, then there is an invariant dist. π such that π(x) > 0 for all x [p. 16, (1.9)] pn (x, y) π(y) as n. π is unique (by irreduciblity).
But for the countable MC, the case is different. For example for the Simple Random Walk (d=1) we have p2n = (2n)!, n!n!2 2n p 2n+1 = 0. Therefore: n=1 p n (0, 0) =, then it is recurrent (1 recurrent class) and lim pn (0, 0) = 0 (p. 47), then the invariant dist. does not exist, why? bc, then π = 0, which is not a probability dist. Therefore we must have lim n p n (y, x) > 0. This condition is not possible for the transient chains because it contradicts p n < But it is possible for recurrent chains to have lim p n (x, y) > 0. We say a MC is positive recurrent if lim n p n (x, y) > 0 Remember that lim n p n (x, y) > 0 implies p n (x, y) =. A MC which is not positive recurrent is called null recurrent.
Positive recurrent MC behave similar to the finite MC s. For example: Assume a countable MC {X n } is aperiodic, and irreducible. If X n is positive recurrent with π(x) = lim n p n (y, x), then π(x) is a probability dist. i.e x π(x) = 1. π(x) is an invariant dist. i.e if P(X 0 = x) = π(x), x, then P(X 1 = x) = π(x) x. OR equivalently π(x) = π(y)p(y, x) y (Next page we repeat this fact in a stronger form) Remember that in the finite case, we discussed the return time T defined by T x = inf{n 1 X n = x} If X n is transient then P x (T x = ) > 0 (bc by definition ρ xx < 1) Therefore if X n is transient then E x T x =.
If X n is recurrent, then { Ex T x = 1 π(x) < positive recurrent null recurren Consider the irreducible and aperiodic MC. The following 3 are equivalent State x is positive recurrent ( y limn p n (y, x) = π(x) > 0) There is an invariant dist. π(x) > 0: π(x) = y π(y)p(y, x) All states are positive recurrent This theorem shows that being positive recurrent is a class property
Branching Process Assume a population starts with n individuals at time 0: X 0 = n Between time n and n + 1, each individual gives birth to a random number of children Y 1, Y 1,, Y n, and die. Furthermore, {Y k } n k=1 are iid. Y n are the number of children, then they are integer-valued let p 0 = P(Y i = 0), p 1 = P(Y i = 1), p 2 = P(Y i = 2). Let X n denote the population in time n. Then X n = Y 1 + + Y Xn 1 If we know the population at n, then the MC holds: P(X n+1 = j X n = j, X n 1 = i 1,, X 0 = 1) = P(X n+1 = j X n = j) indeed we can compute the r-h-s p(k, j) = P(X n+1 = j X n = j) = P(Y 1 + + Y k = j)
Therefore, p(k, j) is the probability of having k individuals in state n + 1, given we have k individuals in state n. We can calculate a few things about the expectations: Let µ = EYi = j=1 jp j. The average number of individuals at time n The conditional average is easy to compute E(X n X n 1 = k) = E(Y 1 + + Y k ) = key 1 = kµ EX n = E(X n X n 1 = k)p(x n 1 = k) = µ kp(x n 1 = k) k=0 k=0 = µex n 1 By iteration we get EX n = µ n EX 0 Remember that in the Branching process {0} is recurrent.
Consider a few paths of the Branching process. S n S 4 3 2 1 1 2 3 4 5 6 7 8 n
Two important sets Define: A n = {ω X n (ω) = 0}. When X n = 0, then X n+k = 0 for all k 1. A n are increasing: A n A n+1 Define: A = {ω ω dies out} = {X n = 0 for some n} Then we have A = A n (1) n=1 Next, we find a way to compute P(A ) in terms of P(A n ).
Continuity from below of P Here we state an important fact about the continuity of P Theorem Assume A n are increasing events: A n A n+1. Then Therefore, P( A n ) = lim P(A n ) n=1 n A n P(A ) = P(population dies out) = P( A n ) (2) n=1 = lim P(A n ) n
Theorem (µ < 1) If µ < 1, then the populations dies out with probability one: P(X n = 0 for some n) = 1 Proof. P(X n 1) = k=1 P(X n = k) k=1 kp(x n = k) = EX n = µ n EX 0 Note that {X n 1} = A c n. Therefore Therefore P(A n ) = 1 P(A c n) 1 µ n EX 0 1 P(A n ) 1 µ n EX 0 Let n. Using the squeeze theorem lim P(A n ) = 1. Then (2) finishes the proof.
When µ > 1, then EX n = µ n. Is it possible that X n 0: lim P(X n = 0) > 0? n Example Let X n {0, 2 n }. P(X n = 0) = 1 1 n, and P(X n = 2 n ) = 1 n. Then P(X n = 0) 1 EX n = 2 n 1 n. Therefore, we have to be careful with the situation: The case that p 0 = 0 is trivially survives with probability 1 If p 0 > 0, but p 0 + p 1 = 1, then µ = EX n < 1 Therefore, the only nontrivial case is p 0 > 0; p 0 + p 1 < 1
Assume p 0 > 0, and p 0 + p 1 < 1. Furthermore, let Then a n (k) = P(A n X 0 = k) and a(k) = P(A X 0 = k) a(k) = P k (A ) = lim n P k (A n ) = lim n a n (k) (3) a n (k) is the probability of extinction, given we started with k individuals but these k individuals form k independent Branching processes, each starting from one individuals Therefore, Probability of extinction of a B-P starting with k individuals is equal to the extinction of k independent B-P each starting with one individual, ie, Let a = a(1). a(k) = [a(1)] k = [ lim n a n (1)] k
Fixed Point Property of a Define the moment generating function of X by ϕ X (s) = Es X. Starting from 1, after taking one step we will have k children with probability p k : = p 0 + k=1 a = k=0 P 1 (A X 1 = k)p k = p 0 + P 1 (A X 0 = k)p k = p 0 + k=1 k=1 a(k)p k = P 1 (A X 1 = k)p k p k a k = Ea X 1 k=0 The last equality holds because when X 0 = 1, then X 1 = Y 1 a = Ea X 1 or a = ϕ X1 (a) (4) Therefore, a is a fixed point of the mgf ϕ
But equation (4) is not enough for finding a = P(population dies out) This is because ϕ X1 (α) = α might have more than one solution. Let us define ϕ n (s) = ϕ Xn (s) n 1 i.e φ n denotes the generating function φ Xn where {X n } is the B-P : ϕ n (s) = Es Xn of X n Write the definition for Es Xn to see that a n = P(A n ) = P(X n = 0) = ϕ n (0). (5) If we work a bit harder we find that ϕ Xn (s) is ϕ X1 at ϕ n 1 (s) evaluated ϕ n (s) = ϕ Xn (s) = ϕ (ϕ X n 1 (s)) = ϕ(ϕ n 1 (s))
We claim a = P 1 (A ) is the smallest root of ϕ(s) = s: Theorem a = P 1 (A ) is the smallest root of ϕ(s) = s. We know that a = lim n a n. We also know that a n = ϕ n (0) = ϕ(ϕ n 1 (0)) = ϕ(a n 1 ). We prove this theorem by induction. Let â be the smallest root of ϕ(s) = s. We want to show by induction that a n â for all n. Because then we can take a limit and show that a â, which forces a = â. a 0 = 0 â. Next assume a n 1 â. Then a n = ϕ(a n 1 ) ϕ(â) = â The inequality follows from the fact that φ is an increasing function.
here we state some of the properties of of ϕ. Most of them are easy to check Lemma For the positive random variable X, let ϕ(s) = ϕ X (s) ϕ(s) is increasing on [0, ) ϕ(0) = p0 = P(X = 0) ϕ (1) = EX Since p 0 + p 1 < 1 by assumption (µ 1), then ϕ (s) 0 If X 1,, X n are iid, then ϕ X1 + +X n (s) = ϕ X1 (s) ϕ Xn (s)
Lemma Let ϕ n (s) = ϕ Xn (s). Then ϕ n (s) = ϕ(ϕ n 1 (s)) Proof. ϕ n (s) = E 1 s Xn = j=0 p k k=0 j=0 P 1 (X n = j)s j = j=0 P k (X n 1 = j)s j = s j k=0 P k (X n 1 = j)p 1 (X 1 = k) = p k [ϕ n 1 (s)] k = ϕ(ϕ n 1 (s)) k=0 P k (X n = j) is the probability of starting from k individual, and having j individual at time n.