MS&E 321 Spring Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10

Transcription

1 MS&E 321 Spring Stochastic Systems June 1, 2013 Prof. Peter W. Glynn Page 1 of 10 Section 3: Regenerative Processes Contents 3.1 Regeneration: The Basic Idea A Necessary and Sufficient Condition for Recurrence Branching Chains Regenerative Processes Positive Recurrent Discrete Time Markov Chains Regeneration: The Basic Idea We describe an approach to studying the long-run behavior of Markov chains that will work for: all discrete state space Markov chains (both finite and infinite state space) the (significant) majority of continuous state space Markov chains The idea is to recognize that a discrete state space Markov chain regenerates every time it returns to a fixed state z S. These z-cycles are iid (a careful proof of this depends, not surprisingly, on the strong Markov property), so the path evolution of the chain is thereby split up into iid segments. As a consequence, we can exploit the extensive theory for iid sequences to study Markov chains/processes. By studying the chain on the regenerative time scale, the Markov dependence structure is simplified to the point where iid theory applies. In order to exploit such an idea, we need to know that infinitely many z-cycles occur. In other words, we will need to determine whether for x S or not. P x (X n z infinitely often) 1 Definition The state z is said to be recurrent if P z (X n z infinitely often) 1. Otherwise, z is said to be transient. Let N(z) I(X k z) be the total number of visits to z. Observe that the strong Markov property implies that k0 P z (N(z) k) P z (τ(z) < ) k 1, so z is recurrent if and only if P z (τ(z) < ) 1, where τ(z) inf{n 1 : X n z}. (Of course, z is transient if and only if P z (τ(z) < ) < 1.) It turns out that recurrence/transience is a class property. We need a definition. 1

2 Definition A Markov chain X (X n : n 0) is said to be irreducible if, for each x, y S, there exists n n(x, y) such that P n (x, y) > 0. Proposition Suppose X is irreducible. transient. Either all states are recurrent or all states are Hence, it makes sense to talk about an irreducible recurrent Markov chain or an irreducible transient Markov chain (as opposed to having to characterize recurrence state-by-state). 3.2 A Necessary and Sufficient Condition for Recurrence Theorem Suppose X is irreducible. Then, X is recurrent if and only if there exists x, y S such that P n (x, y). n0 Proof: Note that for x y E x N(y) kp x (N(y) k) k1 kp x (τ(y) < )P y (τ(y) < ) k 1 P y (τ(y).) k1 Because X is irreducible, P x (τ(y) < ) > 0. It follows that E x N(y) < if and only if P y (τ(y) < ) < 1 (i.e. y is transient). But E x N(y) E x I(X k y) k1 P k (x, y). k1 A beautiful example with which to illustrate this result is that of mean-zero random walk. Definition A sequence (X n : N 0) is said to be a random walk if X n X 0 + Z Z n, where the Z i s are iid and independent of X 0. It is a mean-zero random walk if EZ i 0. Definition A nearest neighbor random walk on Z is a random walk for which X 0 Z and Z i { 1, 1}. A nearest neighbor random walk on Z d is a sequence (X n : n 0), where X n (X n (1),..., X n (d)) and X(i) (X n (i) : n 0) is a collection of independent one-dimensional nearest neighbor random walks. The random walk is said to be symmetric if P (Z i 1) P (Z i 1) 1/2. Use of Stirling s formula establishes that: 2

3 Nearest neighbor symmetric random walk is recurrent in d 1, 2. Nearest neighbor symmetric random walk is transient in d 3. This result generalizes to much more general mean-zero increment distributions; see, for example, K. L. Chung s A Course in Probability Theory. This is one of the major results of 20th century probability. Exercise Suppose that X is a random walk on Z d for which EZ 0. Prove that (X n : n 0) is transient. 3.3 Branching Chains The branching chain model arises in many applications (biological population modeling, nuclear reactions, etc.). The simplest version is a Markov chain X (X n : n 0) on Z + with dynamics described by: X n X n+1 i1 Z n+1,i where (Z ni : n 1, i 1) is a collection of iid Z + -valued rv s independent of X 0. A key quantity of interest is the probability of extinction u (x) P x (T < ), where T inf{n 0 : X n 0}. Clearly, if P (Z 0) > 0, P x (T < ) > 0. Because each individual s progeny evolve independently, u (1) P (Z x)u (1) x (3.3.1) x0 ϕ(u (1)), where ϕ(y) Ey Z for y 0. Assume that EZ <, and note that (3.3.1) asserts that u (1) is a root of ϕ(x) x. Of course, 1 is a root. But this non-linear equation may have multiple roots. Observe that ϕ( ) is non-decreasing, convex, with ϕ(0) P (Z 0) > 0 and ϕ (1) EZ. If: ϕ (1) 1: unique root of ϕ(x) x on [0, 1] occurs at x 1. So, P x (T < ) 1. ϕ (1) > 1: unique root z of ϕ(x) x on (0, 1), with second root 1. Note that both ( z x : x 0) and (1 x : x 0) are solutions of u(x) P (Z 0) x + y1 P (X n y X 0 0)u(y); (3.3.2) (u (x) : x 0) is the minimal non-negative solution of (3.3.2). So, u (x) z x (since z < 1) when EZ > 1. Exercise a) Prove that P y (X n y infinitely often) 0 for y 1. b) Prove that X n a.s. on {T }. 3

4 3.4 Regenerative Processes We now formalize the notion of a regenerative process. Given an S-valued stochastic process X (X(t) : t 0) with an associated increasing sequence of finite-valued random times (T (n) : n 0), define the cycles (W n : n 0) via { X(T (n 1) + t) : 0 t < τ n W n (t) ; t τ n, where T ( 1) 0, τ n T (n) T (n 1), and / S. Definition We say that X is regenerative (with respect to (T (n) : n 0)) if: W 0, W 1,... are independent W 1, W 2,... are identically distributed. We say that X is non-delayed if T (0) 0; otherwise, X is delayed. We say that X is positive recurrent if Eτ 1 <. Let N(t) max{n 1 : T (n) t} be the index of the last cycle completed prior to time t. Remark When T (0) 0, N(t) is a renewal counting process. Proposition N(t)/t 1/Eτ 1 a.s. as t. This is the strong law of large numbers for renewal counting processes. Theorem (Strong Law for Regenerative Processes) Let X be regenerative with respect to (T (n) : n 0), and let f : S R +. If: T (0) f(x(s)) ds < a.s. E 0 T (1) T (0) f(x(s))ds and Eτ 1 are not simultaneously infinite then as t, where 1 t t 0 f(x(s))ds EY 1(f) Eτ 1 Y i (f) T (i) T (i 1) f(x(s))ds. a.s. 4

5 Outline of Proof: N(t) a.s. (since T (n) < a.s. for n 0) Use path-by-path argument based on squeeze N(t) i0 Y i(f) N(t)+1 i0 τ i 1 t t 0 f(x(s))ds N(t)+1 i0 Y i (f) N(t) i0 τ. i Proposition EN(t)/t 1/Eτ 1 as t. This is the so-called elementary renewal theorem. The key to the proof is the Wald identity. Wald Identity: Let (Z n : n 1) be a sequence of independent non-negative rv s for which EZ i µ i for i 1. If T is a stopping time adapted to (Z n : n 1), then T T E Z i E µ i. i1 i1 Proof of Wald s Identity: T E Z i E Z i I(T i) i1 i1 EZ i I(T i) i1 (by Fubini) EZ i (1 I(T i 1)) i1 EZ i (1 q i 1 (Z 1,..., Z i 1 )) i1 EZ i E(1 q i 1 (Z 1..., Z i 1 )) i1 µ i EI(T i) i1 E µ i I(T i) (by Fubini) i1 T E µ i. i1 (since T is a stopping time) (independence) Corollary: If the Z i s are also identically distributed, E T i1 Z i ET EZ 1. 5

6 Outline of proof of Proposition N(t) + 1 is a stopping time adapted to (τ i : i 0). (Note that N(t) is not a stopping time) Wald s identity implies that N(t)+1 E i0 τ i Eτ 0 + E(N(t) + 1) Eτ 1 Since t N(t)+1 i0 τ i, lim inf t t 1 EN(t) 1/Eτ 1 when Eτ 0 <. For lim sup, truncate the τ i s to the (τ i n) s and let N n (t)+1 min{k 0 : k in τ i n t}. Note that N( ) N n ( ) and N n(t)+1 i0 τ i n t + n Apply Wald s identity and divide by t: lim sup t So, lim sup t t 1 EN(t) 1/Eτ 1 n. 1 t EN n(t) 1 Eτ 1 n Send n and apply the Monotone Convergence Theorem (MCT). Deal separately with Eτ 0. Theorem Let X be regenerative with respect to (T (n) : n 0) and let f : S R +. If E(Y 0 (f) + Y 1 (f)) <, then 1 t Ef(X(s))ds EY 1(f) t Eτ 1 as t. 0 Outline of proof of Theorem t t 0 f(x(s))ds 1 N(t)+1 t i0 Y i (f) Same type of argument as in Theorem proves that N(t)+1 t 1 i0 Y i (f) EY 1(f) Eτ 1 a.s. as t Wald s identity proves that N(t)+1 t 1 E Y i (f) t 1 EY 0 (f) + t 1 E(N(t) + 1)EY 1 (f) i0 6

7 Elementary renewal theorem shows that t 1 E N(t)+1 i0 Y i (f) EY 1 (f)/eτ 1 as t, so (t 1 N(t)+1 i0 Y i (f) : t 0) is uniformly integrable So, (t 1 t 0 f(x(s))ds : t 0) is uniformly integrable So, t 1 E t 0 f(x(s))ds EY 1(f)/Eτ Positive Recurrent Discrete Time Markov Chains Assume X (X n : n 0) is irreducible and recurrent. For y S, let { 1 if x y f(x) 0 else Note that τ(z) 1 P x (Y 0 (f) k) P x ( I(X i y) k) i0 P x (τ(y) < τ(z))p y (τ(y) < τ(z)) k 1 P y (τ(z) < τ(y)) so E x Y 0 (f) <, E x Y 1 (f) <. Hence, Theorem implies that I(X i y) E z n 1 i0 τ(z) 1 i0 I(X i y) E z τ(z) a.s. as n, and Theorem implies that P i (x, y) E z n 1 i0 τ(z) 1 i0 I(X i y) E z τ(z) (3.5.1) as n. Note that: right-hand side of (3.5.1) is independent of z (since left-hand side is) limit is independent of x if E z τ(z), limit is zero (i.e., null ) if E z τ(z) <, limit is positive Definition A recurrent state z S is said to be positive recurrent if E z τ(z) < and null recurrent otherwise. Since the right-hand side of (3.5.1) is independent of z, positive recurrence/null recurrence is a class property (i.e. either holds for all states in S or no states in S). When X is positive recurrent, set π(y) E τ(z) 1 z i0 I(X i y) (3.5.2) E z τ(z) 7

8 π (π(y) : y S) is a probability mass function on S For f : S R +, EY 1 (f) Eτ 1 y π(y)f(y). τ(z) 1 τ(z) 1 If E z j0 f(x j ) <, then E x j0 f(x j ) < for all x S (so E x Y 1 ( f ) < and E x Y 0 ( F ) < for each x S); Theorem then implies that as n 1 E x f(x j ) n j0 y π(y)f(y) Exercise Suppose X is positive recurrent and y π(y) f(y) <. Show, by example, that 1 E µ f(x j ) n j0 y π(y)f(y) as n, if µ (µ(x) : x S) does not have finite support. For π as defined via (3.5.2), π satisfies the linear system π πp. The proof is similar to that used in the compactness argument for Chapter 1; (3.5.1) implies that π(y) lim n lim n z n P i (x, y) i1 P i (x, z)p (z, y). n 1 To bring the limit inside the sum over z: For ɛ > 0, select a finite set K S such that y K c π(y) < ɛ. Then, select n 0 so that for n n 0, i0 for z K. It follows that for n n 0. On the other hand, n 1 P i (x, z) π(z) < ɛ/ K i0 n 1 z K c i0 P i (x, z)p (z, y) 2ɛ z K n 1 i0 P i (x, z)p (z, y) z K Since ɛ was arbitrary, π(y) z π(z)p (z, y). 8 π(z)p (z, y).

9 Exercise a) Prove that X is positive recurrent if and only if π πp has a solution π within the class of probability mass functions. b) Prove that if X is positive recurrent, the solution π to π πp is unique within the class of probability mass functions. c) Give an example that shows that π πp may have multiple linearly independent solutions when π is not restricted to be a probability mass function. (Note that your example must be irreducible and positive recurrent.) We are now ready to state a theorem that summarizes the above discussion. Theorem Let X (X n : n 0) be an irreducible Markov chain taking values in S. a) If X is null recurrent and f : S R + is such that EY 1 (f) <, then 1 f(x j ) 0 n j0 a.s. as n. If, in addition, E x Y 0 (f) <, then as n. 1 E x f(x j ) 0 n j0 b) X is positive recurrent if and only if there exists a probability mass function solution to π πp ; the solution π is unique within the class of probability mass functions. c) If X is positive recurrent, then for each f : S R +, 1 f(x j ) n j0 y π(y)f(y) P x a.s. (3.5.3) as n and 1 E x f(x j ) n j0 y π(y)f(y) (3.5.4) as n. If f : S R and y π(y) f(y) <, then (3.5.3) and (3.5.4) continue to hold. Remark One implication of (3.5.2) is that for a positive recurrent chain, π(z) 1/E z τ(z). Remark Suppose that X is null recurrent. Consider τ(z) 1 η(y) E z j0 I(X j y). It turns out that η (η(y) : y S) is an invariant measure, in the sense that η satisfies η ηp ; see Resnick, p (Of course, η can not be summable, since otherwise X would be positive recurrent.) Does this invariant measure have a probability interpretation? 9

10 The answer is: Yes! Suppose that at time n 0, we independently assign to each y S a Poisson number of particles to that state having mean η(y). Let each of the particles (across all states) independently evolve according to the transition dynamics of the Markov chain. Then, the distribution of the particles at time 1 (and all future times) will be identical to the distribution at time n 0. Remark We have seen that all recurrent Markov chains have invariant measures. A transient irreducible Markov chain may either have an invariant measure η (i.e. a positive solution η to η ηp or not. In particular, existence of an invariant measure of X does not imply that X is recurrent. 10