Discrete Time Stochastic Processes

Transcription

1 Discrete Time Stochastic Processes Joseph C. Watkins May 5, 2007 Contents Basic Concepts for Stochastic Processes 3. Conditional Expectation Stochastic Processes, Filtrations and Stopping Times Coupling Random Walk 6 2. Recurrence and Transience The Role of Dimension Simple Random Walks Renewal Sequences Waiting Time Distributions and Potential Sequences Renewal Theorem Applications to Random Walks Martingales Examples Doob Decomposition Optional Sampling Theorem Inequalities and Convergence Backward Martingales Markov Chains Definition and Basic Properties Examples Extensions of the Markov Property Classification of States Recurrent Chains Stationary Measures Asymptotic Behavior Rates of Convergence to the Stationary Distribution Transient Chains Asymptotic Behavior

2 CONTENTS ρ C Invariant Measures Stationary Processes Definitions and Examples Birkhoff s Ergodic Theorem Ergodicity and Mixing Entropy

3 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 3 Basic Concepts for Stochastic Processes In this section, we will introduce three of the most versatile tools in the study of random processes - conditional expectation with respect to a σ-algebra, stopping times with respect to a filtration of σ-algebras, and the coupling of two stochastic processes.. Conditional Expectation Information will come to us in the form of σ-algebras. The notion of conditional expectation E[Y G] is to make the best estimate of the value of Y given a σ-algebra G. For example, let {C i ; i } be a countable partitiion of Ω, i. e., C i C j =, whenever i j and i C i = Ω. Then, the σ-algebra, G, generated by this partition consists of unions from all possible subsets of the partition. For a random variable Z to be G-measurable, then the sets {Z B} must be such a union. Consequently, Z is G-measurable if and only if it is constant on each of the C i s. The natural manner to set E[Y G](ω) is to check which set in the partition contains ω and average Y over that set. Thus, if ω C i and P (C i ) > 0, then E[Y G](ω) = E[Y C i ] = E[Y ; C i] P (C i ). If P (C i ) = 0, then any value E[Y G](ω) (say 0) will suffice, In other words, E[Y G] = i E[Y C i ]I Ci. (.) We now give the definition. Check that the definition of E[Y G] in (.) has the two given conditions. Definition.. Let Y be an integrable random variable on (Ω, F, P ) and let G be a sub-σ-algebra of F. The conditional expectation of Y given G, denoted E[Y G] is the a.s. unique random variable satisfying the following two conditions.. E[Y G] is G-measurable. 2. E[E[Y G]; A] = E[Y ; A] for any A G. To see that that E[Y G] as defined in (.) follows from these two conditions, note that by condition, E[Y G] must be constant on each of the C i. By condition 2, for ω C i, E[Y G](ω)P (C i ) = E[E[Y G]; C i ] = E[Y ; C i ] or E[Y G](ω) = E[Y ; C i] P (C i ) = E[Y C i ]. For the general case, the definition states that E[Y G] is essentially the only random variable that uses the information provided by G and gives the same averages as Y on events that are in G. The existence and uniqueness is provided by the Radon-Nikodym theorem. For Y positive, define the measure ν(a) = E[Y ; A] for A G. Then ν is a measure defined on G that is absolutely continuous with respect to the underlying probability P restricted to G. Set E[Y G] equal to the Radon-Nikodym derivative dν/dp G. The general case follows by taking positive and negative parts of Y. For B F, the conditional probability of B given G is defined to be P (B G) = E[I B G].

4 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 4 Exercise.2.. Let C F, then P (B σ(c)) = P (B C)I C + P (B C c )I C c. 2. If G = σ{c,, C n }, the σ-algebra generated by a finite partition, then P (B G) = n P (B C i )I Ci. i= In other words, if ω C i then P (A G)(ω) = P (A C i ). Theorem.3 (Bayes Formula). Let B G, then P (B A) = E[P (A G)I B]. E[P (A G)] Proof. E[P (A G)I B ] = E[I A I B ] = P (A B) and E[P (A G)] = P (A). Exercise.4. Show the Bayes formula for a finite partition {C,, C n } is P (C j A) = P (A C j )P (C j ) n i= P (A C i)p (C i ). If G = σ(x), then we usually write E[Y G] = E[Y X]. For these circumstances, we have the following theorem which can be proved using the standard machine. Theorem.5. Let X be a random variable. Then Z is a measurable function on (Ω, σ(x)) if and only if there exists a measurable function h on the range of X so that Z = h(x). This shows that the deifinition of conditional expectation with respect to a σ-algebra extends the definition of conditional expectation with respect to a random variable. We now summarize the properties of conditional expectation. Theorem.6. Let Y, Y, Y 2, have finite absolute mean on (Ω, F, P ) and let a, a 2 R. In addition, let G and H be σ-algebras contained in F. Then. If Z is any version of E[Y G], then EZ = EY. (E[E[Y G]] = EY ). 2. If Y is G measurable, then E[Y G] = Y, a.s. 3. (linearity) E[a Y + a 2 Y 2 G] = a E[Y G] + a 2 E[Y 2 G], a.s. 4. (positivity) If Y 0, then E[Y G] 0, a.s. 5. (conditional monotone convergence theorem) If Y n Y, then E[Y n G] E[Y G], a.s. 6. (conditional Fatous s lemma) If Y n 0, then E[lim inf n Y n G] lim inf n E[Y n G]. 7. (conditional dominated convergence theorem) If lim n Y n (ω) = Y (ω), a.s., if Y n (ω) V (ω) for all n, and if EV <, then lim n E[Y n G] = E[Y G], almost surely.

5 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 5 8. (conditional Jensen s inequality) If φ : R R is convex, and E φ(y ) <, then E[φ(Y ) G] φ(e[y G]), almost surely. In particular, 9. (contraction) E[Y G] p Y p for p. 0. (tower property) If H G, then E[E[Y G] H] = E[Y H], almost surely.. (conditional constants) If Z is G-measurable, and E ZY <, then E[ZY G] = ZE[Y G]. 2. (role of independence) If H is independent of σ(σ(y ), G), then E[Y σ(g, H)] = E[Y G], almost surely. In particular, 3. if Y is independent of H, then E[Y H] = EY. Proof.. Take A = Ω in the definition. 2. Check that Y satisfies both and 2 in the definition. 3. Clearly a E[Y G] + a 2 E[Y 2 G] is G-measurable. Let A G. Then yielding property 2. E[a E[Y G] + a 2 E[Y 2 G]; A] = a E[E[Y G]; A] + a 2 E[E[Y 2 G]; A] = a E[Y ; A] + a 2 E[Y 2 ; A] = E[a Y + a 2 Y 2 ; A], 4. Because Y 0 and {E[Y G] n }] is G-measurable, we have that 0 E[Y ; {E[Y G] n }] = E[E[Y G]; {E[Y G] n }] n P {E[Y G] n } and Consequently, P {E[Y G] n } = 0. ( ) P {E[Y G] < 0} = P {E[Y G] n } = 0. n=

6 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 6 5. Write Z n = E[Y n G]. Then, by positivity of conditional expectation, Z = lim n Z n exists almost surely. Note that Z is G-measurable. Choose A G. By the monotone convergence theorem, E[Z; G] = lim E[Z n; G] = lim E[Y n; G] = E[Y ; G]. n n Thus, Z = E[Y G]. Because Y Y Y, we have E[ Y G] E[Y G] E[ Y G] and E[Y G] E[ Y G]. Consequently, E[ G] : L (Ω, F, P ) L (Ω, G, P ) is a continuous linear mapping. 6. Repeat the proof of Fatou s lemma replacing expectation with E[ G] and use both positivity and the conditional monotone convergence theorem. 7. Repeat the proof of the dominated convergence theorem from Fatou s lemma again replacing expectation with conditional expectation. 8. Follow the proof of Jensen s inequality. 9. Use the conditional Jensen s inequality on the convex function φ(y) = y p, y 0, E[Y G] p p = E[ E[Y G] p ] E[E[ Y G] p ] E[E[ Y p G]] = E[ Y p ] = Y p p. 0. By definition E[E[Y G] H] is H-measurable. Let A H, then A G and E[E[E[Y G] H]; A] = E[E[Y G]; A] = E[Y ; A].. Use the standard machine. The case Z an indicator function follows from the definition of conditional expectation. 2. We need only consider the case Y 0, EY > 0. Let Z = E[Y G] and consider the two probability measures E[Y ; A] E[Z; A] µ(a) =, ν(a) =. EY EY Note that EY = EZ and define C = {A : µ(a) = ν(a)}. If A = B C, B G, C H, then Y I B and C are independent and E[Y ; B C] = E[Y I B ; C] = E[Y I B ]P (C) = E[Y ; B]P (C). Z is G-measurable and thus ZI B and C are independent and E[Z; B C] = E[ZI B ; C] = E[ZI B ]P (C) = E[Z; B]P (C) = E[Y ; B]P (C).

7 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 7 Consequently, D = {B C; B G, C H} C. Now, D is closed under pairwise intersection. Thus, by the the Sierpinski Class Theorem, µ and ν agree on σ(d) = σ(g, H) and E[Z; A] = E[Y ; A], for all A σ(g, H). Because Z is σ(g, H)-measurable, this completes the proof. 3. Take G to be the trivial σ-algebra {, Ω}. However, this property can be verified directly. Exercise.7. Let X be integrable, then the collection is uniformly integrable. {E[X G]; G a sub-σalgebra of F}. Remark.8. If the sub-σ-algebra G F, then by Jensen s inequality, E[ G] : L 2 (Ω, F, P ) L 2 (Ω, G, P ). If E[Y 2 ] <, we can realize the conditional expectation E[Y G] as a Hilbert space projection. Note that L 2 (Ω, G, P ) L 2 (Ω, F, P ). Let Π G be the orthogonal projection operator onto the closed subspace L 2 (Ω, G, P ). Thus Π G Y is G-measurable and Y Π G Y is orthogonal to any element in X L 2 (Ω, G, P ) Consequently, E[(Y Π G Y )X] = 0, or E[Π G Y X] = E[Y X]. take X = I A, A G to see that Π G Y = E[Y G]. As before, this can be viewed as a minimization problem min{e[(y Z) 2 ]; Z L 2 (Ω, G, P )}. The minimum is acheived uniquely by the choice Z = E[Y G]. Exercise.9. then. Prove the conditional Chebyshev inequality: Let g : R [0, ) be increasing on [a, ), 2. Let G be a sub-σ-algebra and let EY 2 <, then where Var(Y G)] = E[(Y E[Y G]) 2 G]. P {g(x) > a G} E[g(X) G]. a Var(Y ) = E[Var(Y G)] + Var(E[Y G]).

8 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 8 Example.0.. Let S j be a binomial random variable, the number of heads in j flips of a biased coin with the probability of heads equal to p. Thus, there exist independent 0- valued random variables {X j : j } so that S n = X + + X n. Let k < n, then S k = X + + X k and S n S k = X k+ + + X n are independent. Therefore, E[S n S k ] = E[(S n S k ) + S k S k ] = E[(S n S k ) S k ] + E[S k S k ] = E[S n S k ] + S k = (n k)p + S k. Now to find E[S k S n ], note that x = E[S n S n = x] = E[X S n = x] + + E[X n S n = x]. Each of these summands has the same value, namely x/n. Therefore E[S k S n = x] = E[X S n = x] + + E[X k S n = x] = k n x and E[S k S n ] = k n S n. 2. Let {X n : n } be independent random variables having the same distribution. Let µ be their common mean and σ 2 their common variance. Let N be a non-negative valued random variable and define S = N n= X n, then ES = E[E[S N]] = E[Nµ] = µen. Var(S) = E[Var(S N)] + Var(E[S N]) = E[Nσ 2 ] + Var(Nµ) = σ 2 EN + µ 2 Var(N). Alternatively, for an N-valued random variable X, define the generating function G X (z) = Ez X = z x P {X = x}. Then G X () =, G X() = EX, G X() = E[X(X )], Var(X) = G X() + G X() G X() 2. Now, G S (z) = Ez S = E[E[z S N]] = E[z S N = n]p {N = n} = E[z (X+ +X N ) N = n]p {N = n} Thus, and = = n=0 n=0 E[z (X+ +Xn) ( ]P {N = n} = E[z X ] E[z Xn ] ) P {N = n} n=0 G X (z) n P {N = n} = G N (G X (z)) n=0 G S(z) = G N(G X (z))g X(z), x=0 n=0 ES = G S() = G N()G X() = µen G S(z) = G N (G X (z))g X(z) 2 + G N (G X (z))g X(z) G S() = G N ()G X() 2 + G N()G X() = G N ()µ 2 + EN(σ 2 µ + µ 2 ). Now, substitute to obtain the formula above.

9 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 9 3. The density with respect to Lebesgue measure for a bivariate standard normal is f (X,Y ) (x, y) = ( 2π ρ exp x2 2ρxy + y 2 ) 2 2( ρ 2. ) To check that E[Y X] = ρx, we must check that for any Borel set B Complete the square to see that E[(Y ρx)i B (X)] = E[Y I B (X)] = E[ρXI B (X)] or E[(Y ρx)i B (X)] = 0. = 2π ρ 2 2π ρ 2 B B ( ( (y ρx) exp ( ( z exp because the integrand for the z integration is an odd function. Now ) ) (y ρx)2 2( ρ 2 dy e x2 /2 dx ) z 2 ) ) 2( ρ 2 dz e x2 /2 dx = 0, z = y ρx ) Cov(X, Y ) = E[XY ] = E[E[XY X]] = E[XE[Y X]] = E[ρX 2 ] = ρ. Definition.. On a probability space (Ω, F, P ), let G, G 2, H be sub-σ-algebras of F. Then the σ-algebras G and G 2 are conditionally independent given H if for A i G i, i =, 2. P (A A 2 H) = P (A H)P (A 2 H), Exercise.2 (conditional Borel-Cantelli lemma). For G a sub-σ-algebra of F, let {A n : n } be a sequence of conditionally independent events given G. Show that for almost all ω Ω, { 0 P {A n i.o. G}(ω) = according as Consequently, n= P {A n i.o.} = P {ω : { = P (A n G)(ω) <, P (A n G)(ω) = }. n= Proposition.3. Let H, G and D be σ-algebras. Then. If H and G are conditionally independent given D, then H and σ(g, D) are conditionally independent given D. 2. Let H and G be sub-σ-algebras of H and G, respectively. Suppose that H and G are independent. Then H and G are conditionally independent given σ(h, G ). 3. Let H G. Then G and D are conditionally independent given H if and only if, for every D D, P (D H) = P (D G). Exercise.4. Prove the previous proposition. The Sierpinski class theorem will be helpful in the proof.

10 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 0.2 Stochastic Processes, Filtrations and Stopping Times A stochastic process X (or a random process, or simply a process) with index set Λ and a measurable state space (S, B) defined on a probability space (Ω, F, P ) is a function such that for each λ Λ, X : Λ Ω S X(λ, ) : Ω S is an S-valued random variable. Note that Λ is not given the structure of a measure space. In particular, it is not necessarily the case that X is measurable. However, if Λ is countable and has the power set as its σ-algebra, then X is automatically measurable. X(λ, ) is variously written X(λ) or X λ. Throughout, we shall assume that S is a metric space with metric d. A realization of X or a sample path for X is the function X(, ω 0 ) : Λ S for some ω 0 Ω. Typically, for the processes we study Λ will be the natural numbers, and [0, ). Occasionally, Λ will be the integers or the real numbers. In the case that Λ is a subset of a multi-dimensional vector space, we often call X a random field. The distribution of a process X is generally stated via its finite dimensional distributions µ (λ,...,λ n)(a) = P {(X λ,..., X λn ) A}. If this collection probability measures satisfy the consistency consition of the Daniell-Kolmogorov extension theorem, then they determine the distribution of the process X on the space S Λ. We will consider the case Λ = N. Definition.5.. A collection of σ-algebras {F n : n 0} each contained in F is a filtration if F n F n+, n = 0,,.... F n is meant to capture the information available to an observer at time n. 2. The natural filtration for the process X is F X n = σ{x k ; k n}. 3. This filtration is complete if F is complete and {A : P (A) = 0} F A process X is adapted to a filtration {F n : n 0} (X is F n -adapted.) if X n is F n measurable. In other words, X is F n -adapted if and only if for every n N, Fn X F n 5. If two processes X and X have the same finite dimensional distributions, then X is a version of X. 6. If, for each n, P {X n = X n } =, then we say that X is a modification of X. In this case P {X = X} =. This conclusion does not necessarily hold if the index set is infinite but not countably infinite. 7. A non-negative integer-valued random variable τ is an F n -stopping time if {τ n} F n for every n N. Exercise.6. τ is an F n -stopping time if and only if {τ = n} F n for every n N.

11 BASIC CONCEPTS FOR STOCHASTIC PROCESSES Exercise.7. Let X be F n -adapted and let τ be an F n -stopping time. For B B, define σ(b, τ) = inf{n τ : X n B} and σ + (B, τ) = inf{n > τ : X n B}. Then σ(b, τ) and σ + (B, τ) are F n -stopping times. Proposition.8. Let τ, τ 2,..., be F n -stopping times and let c 0. Then. τ + c and min{τ, c} are F n -stopping times. 2. sup k τ k is an F n -stopping time. 3. inf k τ k is an F n -stopping time. 4. lim inf k τ k and lim sup k τ k are F n -stopping times. Proof.. {τ + c n} = {τ n c} F max{0,n c} F n. If c n, {min{τ, c} n} = Ω F n, If c > n, {min{τ, c} n} = {τ n} F n, 2. {sup k τ k n} = k {τ k n} F n. 3. {inf k τ k n} = k {τ k n} F n. (This statement does not hold if τ k is a continuous random variable.) 4. This follows from parts 2 and 3. Definition.9.. Let τ be an F n -stopping time. Then F τ = {A F : A {τ n} F n for all n}. 2. Let X be a random process then X τ, the process X stopped at time τ, is defined by X τ n = X min{τ,n} The following exercise explains the intuitive idea that F τ gives the information known to observer up to time τ. Exercise.20.. F τ is a σ-algebra. 2. Fτ X = σ{xn τ : n 0}. Proposition.2. Let σ and τ be F n -stopping times and let X be an F n -adapted process.. τ and min{τ, σ} are F τ -measurable. 2. If τ σ, then F τ F σ. 3. X τ is F τ -measurable. Proof.. {min{τ, σ} c} {τ n} = {min{τ, σ} min{c, n}} {τ n} = ({τ min{c, n}} {σ min{c, n}}) {τ n} F n.

12 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 2 2. Let A F τ, then Hence, A F σ. 3. Let B B, then A {σ n} = (A {τ n}) {σ n} F n. {X τ B} {τ n} = n k=({x k B} {τ = k}) F n. Exercise.22. Let X be an F n adapted process. For n = 0,,, define F τ n = F min{τ,n}. Then. {F τ n : n 0} is a filtration. 2. X τ is both F τ n-adapted and F n -adapted..3 Coupling Definition.23. Let ν and ν be probability distributions (of a random variable or of a stochastic process). Then a coupling of ν and ν is a pair of random variable X and X defined on a probability space (Ω, F, P ) such that the marginal distribution of X is ν and the marginal distribution of X is ν As we shall soon see, coupling are useful in comparing two distribution by constructing a coupling and comparing random variables. Definition.24. The total variation distance between two probability measures ν and ν on a measurable space (S, B) is ν ν T V = sup{ ν(b) ν(b) ; B B}. (.2) Example.25. Let ν be a Ber(p) and let ν be a Ber( p) distribution, p p. If X and X are a coupling of independent random variables, then P {X X} = P {X = 0, X = } + P {X =, X = 0} = ( p) p + p( p) = p + p 2p p. Now couple ν and ν as follows. Let U be a U(0, ) random variable. Set X = if and only if U p and X = if and only if U p. Then Note that ν ν T V = p p. P {X X} = P {p < U p} = p p. We now turn to the relationship between coupling and the total variation distance. Proposition.26. If S is a countable set and B is the power set then ν ν T V = ν{x} ν{x}. (.3) 2 x S

13 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 3 ν ν I II III A A c Figure : Distribution of ν and ν. The event A = {x; ν{x} > ν{x}}. Regions I and II have the same area, namely ν ν T V = ν(a) ν(a) = ν(a c ) ν(a c ). Thus, the total area of regions I and III and of regions II and III is. Proof. (This is one half the area of region I plus region II in Figure.) Let A = {x; ν{x} > ν{x}} and let B B. Then ν(b) ν(b) = (ν{x} ν{x}) (ν{x} ν{x}) = ν(a B) ν(a B) x B x A B because the terms eliminated to create the second sum are negative. Similarly, ν(a B) ν(a B) = x A B(ν{x} ν{x}) (ν{x} ν{x}) = ν(a) ν(a) x A because the additional terms in the second sum are positive. Repeat this reasoning to obtain However, because ν and ν are probability measures, ν(b) ν(b) ν(a c ) ν(a c ). ν(a c ) ν(a c ) = ( ν(a)) ( ν(a)) = ν(a) ν(a). Consequently, the supremum in (.2) is obtained in choosing either the event A or the event A c and thus is equal to half the sum of the two as given in (.3). Exercise.27. Let f = sup x S f(x). Show that ν ν T V = 2 sup{ x S f(x)(ν{x} ν{x}); f = }. Theorem.28. If S is a countable set and A is the power set, then ν ν T V = inf{p {X X}; (P, X, X) C(ν, ν)} (.4) where the infimum is taken over all possible couplings C(ν, ν) of ν and ν.

14 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 4 Proof. For any coupling (P, X, X), Thus, P {X B} = P {X B, X B} + P {X B, X B c } P { X B} + P {X B, X B c } P {X X} P {X B, X B c } P {X B} P { X B} = ν(b) ν(b). Thus, P {X X} is at least big as the total variation distance. Next, we construct a coupling that achieve the infimum. First, set p = x S min{ν{x}, ν{x}} = x S 2 (ν{x} + ν{x} ν{x} ν{x} ) = ν ν T V, the area of region III in Figure. If p = 0, then ν and ν takes values on disjoint set and the total variation distance between the two is. For p > 0, define a random variable Y, P {Y = x} = min{ν{x}, ν{x}}/p. In addition, define the probability distributions, µ{x} = ν{x} ν{x} I {ν{x}> ν{x}}, and µ{ x} = p ν{ x} ν{ x} I { ν{ x}>ν{ x}}. p Now, flip a coin that lands heads with probability p. If the coin lands heads, set X = X = Y. If the coin lands tails, then let X and X be independent, X has distribution µ and X has distribution µ. Checking, we see that P {X = x} = P {X = x coin lands heads}p {coin lands heads} + P {X = x coin lands tails}p {coin lands tails} = min{ν{x}, ν{x}} ν{x} ν{x} p + I {ν{x}> ν{x}} ( p) = ν{x} p p Similarly, P { X = x} = ν{ x}. Finally, P {X = X} = P {X = X coin lands heads}p {coin lands heads} + P {X = X coin lands tails}p {coin lands tails} = p + x S P {X = x, X = x coin lands tails}( p) = p + x S P {X = x coin lands tails}p { X = x coin lands tails}( p) = p + x S µ{x} µ{x}( p) = p. Note that each term in sum is zero. Either the first term, the second term or both terms are zero based on the sign of ν{x} ν{x}. Thus, P {X X} = p = ν ν T V. We will typically work to find successful couplings of stochastic processes. They are defined as follows:

15 BASIC CONCEPTS FOR STOCHASTIC PROCESSES 5 Definition.29. A coupling (P, X, X) is called a successful coupling of two adapted stochastic processes, if the stopping time τ = inf{t 0; X t = X t } is almost surely finite and if X t = X t for all t τ. Then, we say that τ is the coupling time and that X and X are coupled at time τ. If ν t and ν t are the distributions of the processes at time t, then (P, X t, X t ) is a coupling of ν t and ν t and, for a successful coupling, ν X t ν Y t T V P {X t Y t } = P {τ > t}. Thus, the earlier the coupling time, the better the estimate of the total variation distance of the time t distributions.

16 2 RANDOM WALK 6 2 Random Walk Definition 2.. Let X be a sequence of independent and identically distributed R d -valued random variables and define n S 0 = 0, S n = X k. Then the sequence S is called a random walk with steps X, X 2,.... If the steps take on only the values ±e j, the standard unit basis vectors in R d, then the random sequence S is called a simple random walk. If each of these 2d steps are equally probable, the S is called a symmetric simple random walk. Theorem 2.2. Let X, X 2,... be independent with distribution ν and let τ be a stoppping time. Then, conditioned on {τ < }, {X τ+n, n } is independent of Fτ X and has the same distribution as the original sequence. Proof. Choose A F τ and B j B(R d ), then because the finite dimensional distributions determine the process, it suffices to show that P (A {τ <, X τ+j B j, j k}) = P (A {τ < }) ν(b k ). k= j k To do this note that A {τ = n} F X n and thus is independent of X n+, X n+2,.... Therefore, P (A {τ = n, X τ+j B j, j k}) = P (A {τ = n, X n+j B j, j k}) = P (A {τ = n})p {X n+j B j, j k}) = P (A {τ = n}) ν(b j ). j k Now, sum over n and note that each of the events are disjoint for differing values of n, Exercise Assume that the steps of a real valued random walk S have finite mean and let τ be a stopping with finite mean. Show that ES τ = µeτ. 2. In the exercise, assume that µ = 0. For c > 0 and τ = min{n 0; S n c}, show that Eτ is infinite. 2. Recurrence and Transience The strong law of large numbers states that if the step distribution has finite mean µ, then P { lim n n S n = µ} =. So, if the mean exists and is not zero, the walk will drift in the direction of the mean. If the mean does not exist or if it is zero, we have the possibility that the walk visits sites repeatedly. Noting that the walk could only return to possible sites motivates the following definitions.

17 2 RANDOM WALK 7 Definition 2.4. Let S be a random walk. A point x R d is called a recurrent state if for every neighborhood U of x, P {S n U i. o.} =. Definition 2.5. If, for some random real valued random variable X and some l > 0, n= P {X = nl} =, then X is said to be distributed on the lattice L l = {nl : n Z} provided that the equation hold for no larger l. If this holds for no l > 0, then X is called non-lattice and is said to be distributed on L 0 = R. Exercise 2.6. Let the steps of a random walk be distributed on a lattice L l, l > 0. Let τ n be the n-th return of the walk to zero. Then P {τ n < } = P {τ < } n. Theorem 2.7. If the step distribution is on L l, then either every state in L l is recurrent or no states are recurrent. Proof. Let G be the set of recurrent points. Claim I. The set G is closed. Choose a sequence {x k : k 0} G with limit x and choose U a neighborhood of x. Then, there exists an integer K so that x k U whenever k K and hence U is a neighborhood of x k. Because x k G, P {S n U i. o.} = and x G. Call y be a possible state (y C) if for every neighborhood U of y, there exists k so that P {S k U} > 0. Claim II. If x G and y C, then x y G. Let U = (y ɛ, y + ɛ) and pick k so that P { S k y < ɛ} > 0. Use f. o., finitely often, to indicate the complement of i. o., infinitely often. Then k, and P { S n x < ɛ f. o.} P { S k+n x < ɛ f. o., S k y < ɛ}. If S k is within ɛ of y and S k+n is within ɛ of x, then S k+n S k is within 2ɛ of x y. Therefore, for each { S k+n x < ɛ, S k y < ɛ} { (S k+n S k ) (x y) < 2ɛ, S k y < ɛ} { S k+n x < ɛ f. o., S k y < ɛ} { (S k+n S k ) (x y) < 2ɛ f. o., S k y < ɛ}. Use the fact that S k and S k+n S k are independent and that the distribution of S n and S k+n S k are equal. P { S n x < ɛ f. o.} P { (S n+k S k ) (x y) < 2ɛ f. o., S k y < ɛ} = P { (S n+k S k ) (x y) < 2ɛ f. o.}p { S k y < ɛ} = P { S n (x y) < 2ɛ f. o.}p { S k y < ɛ}. Because x G, 0 = P { S n x < ɛ f. o.}. Recalling that P { S k y < ɛ} > 0, this forces P { S n (x y) < 2ɛ f. o.} = 0,

18 2 RANDOM WALK 8 P { S n (x y) < 2ɛ i. o.} =. Finally, choose U, a neighborhood of x y and ɛ so that (x y 2ɛ, x y + 2ɛ) U, then and x y G. Claim III. G is a group. If x G and y G, then y C and x y G. Claim IV. If G, and if l > 0, then G = L l. P {S n U i. o.} P { S n (x y) < 2ɛ i. o.} =, Clearly L l G. Because G, 0 G. Set S X = {x : P {X = x} > 0}. If x S X, then x C and x = 0 x G and x G. By the definition of l, all integer multiples of l can be written as positive linear sums from x, x, x S X and G L l Claim V. If l = 0, and G, then G = L 0. 0 = inf{y > 0 : y G}. Otherwise, we are in the situation of a lattice valued random walk. Let x R and let U be a neighborhood of x. Now, choose ɛ > 0 so that (x ɛ, x + ɛ) U. By the definition of infimum, there exists l < ɛ so that l G. Thus, L l G and L l U. Consequently, and x G. P {S n U i. o.} =, Corollary 2.8. Let S be a random walk in R d. Then, either the set of recurrent states forms a closed subgroup of R d or no states are recurrent. This subgroup is the closed subgroup generated by the step distribution. Proof. For claims I, II, and III, let stand for some norm in R d and replace open intervals by open balls in that norm. Theorem 2.9. Let S be a random walk in R d.. If there exists an open set U containing 0 so that then no states are recurrent. 2. If there is a bounded open set U such that then 0 is recurrent. P {S n U} <, n=0 P {S n U} =, n=0

19 2 RANDOM WALK 9 Proof. If n=0 P {S n U} <, then by the Borel-Cantelli, P {S n U i. o.} = 0. Thus 0 is not recurrent. Because the set of recurrent states is a group, then no states are recurrent. Then, For the second half, let ɛ > 0 and choose a finite set F so that U B(x, ɛ). Consequently, for some x, {S n U} x F x F{S n B(x, ɛ)}, P {S n U} x F P {S n B(x, ɛ)} =. n=0 P {S n B(x, ɛ)}. Define the pairwise disjoint sets, A k, the last visit to B(x, ɛ) occurred for S k. { {Sn / B(x, ɛ), n =, 2,...} for k = 0 A k = {S k B(x, ɛ), S n+k / B(x, ɛ), n =, 2,...} for k > 0 Then {S n B(x, ɛ) f. o.} = A k, and P {S n B(x, ɛ) f. o.} = P (A k ). k=0 For k > 0, if S k B(x, ɛ) and S n+k S k > 2ɛ, then S n+k / B(x, ɛ). Consequently, P (A k ) P {S k B(x, ɛ), S n+k S k > 2ɛ, n =, 2,...} k=0 = P {S k B(x, ɛ)}p { S n+k S k > 2ɛ, n =, 2,...} = P {S k B(x, ɛ)}p { S n > 2ɛ, n =, 2,...} Summing over k, we find that ( ) P {S n B(x, ɛ) f. o.} P {S k B(x, ɛ)} P { S n > 2ɛ, n =, 2,...}. k=0 The number on the left is at most one, the sum is infinite and consequently for any ɛ > 0, P { S n > 2ɛ, n =, 2,...} = 0. Now, define the sets A k as before with x = 0. By the argument above P (A 0 ) = 0. For k, define a strictly increasing sequence {ɛ i : i } with limit ɛ. By the continuity property of a probability, we have, Pick a term in this sequence, then P (A k ) = lim i P {S k B(0, ɛ i ), S n+k / B(0, ɛ), n =, 2,...}. P {S k B(0, ɛ i ), S n+k / B(0, ɛ), n =, 2,...} P {S k B(0, ɛ i ), S n+k S k ɛ ɛ i, n =, 2,...} = P {S k B(0, ɛ i )}P { S n+k S k ɛ ɛ i, n =, 2,...} = P {S k B(0, ɛ i )}P { S n ɛ ɛ i, n =, 2,...} = 0.

20 2 RANDOM WALK 20 Therefore, P (A k ) = 0 for all k 0. P {S n B(0, ɛ) f. o.} = P (A k ) = 0. Because every neighborhood of 0 contains a set of the form B(0, ɛ) for some ɛ > 0, we have that 0 is recurrent. 2.2 The Role of Dimension Lemma 2.0. Let ɛ > 0, m 2, and let be the norm x = max i d x i. Then Proof. Note that and consider the stopping times Consequently, P { S n < mɛ} (2m) d n=0 P { S n < mɛ} K=0 n=0 P { S n < ɛ}. α { m,...,m } d P {S n αɛ + [0, ɛ) d } τ α = min{n 0 : S n αɛ + [0, ɛ) d }. P {S n αɛ + [0, ɛ) d } = n=0 = P {S n αɛ + [0, ɛ) d, τ α = k} k=0 n=0 k=0 n=0 k=0 n=k P { S n S k < ɛ, τ α = k} P { S n S k < ɛ}p {τ α = k} ( ) P { S n < ɛ} P {τ α = k} P { S n < ɛ} k=0 n=0 n=0 because the sum on k is at most. terms. The proof ends upon noting that the sum on α consists of (2m) d Theorem 2. (Chung-Fuchs). Let d =. If the weak law of large numbers holds, then S n is recurrent. n S n P 0

21 2 RANDOM WALK 2 Proof. Let A > 0. Using the lemma above with d = and ɛ =, we have that P { S n < } P { S n < m} Am { P S n < n }. 2m 2m A n=0 n=0 n=0 Note that, for any A, we have by the weak law of large numbers, lim { S P n < n } =. n A If a sequence has a limit, then its Cesaro sum has the same limit and therefore, lim m Am { P S n < n } = A 2m A 2. n=0 Because A is arbitrary, the first sum above is infinite and the walk is recurrent. Theorem 2.2. Let S be an random walk in R 2 and assume that the central limit theorem holds in that { } lim P n S n B = n(x) dx n where n is a normal density whose support is R 2, then S is recurrent. Proof. By the lemma, with ɛ = and d = 2. P { S n < } 4m 2 P { S n < m}. Let c > 0, then n=0 n=0 n and m n c implies P { S n < m} = P Call this function ρ(c), then for θ > 0, and we can write m 2 lim P { S [θm m 2 ] < m} = ρ(θ /2 ). P { S n < m} = n=0 Let m and use Fatou s lemma. lim inf m 4m 2 P { S n < m} 4 n=0 0 B { n S n < m } n(x) dx. n [ c.c] 2 P { S [θm2 ] < m} dθ. 0 ρ(θ /2 ) dθ. The proof is complete if we show the following. Claim. ρ(θ /2 ) dθ diverges. 0 Choose ɛ so that n(x) n(0)/2 > 0 if x < ɛ, then for θ > /ɛ 2 ρ(θ /2 ) = n(x) dx 2 n(0)4 θ, showing that the integral diverges. [ θ /2,θ /2 ]

22 2 RANDOM WALK Simple Random Walks To begin on R, let the steps X k take on integer values. Define the stopping time τ 0 = min{n > 0; S n = 0} and set the probability of return to zero p n = P {S n = 0} and the first passage probabilities Now define their generating functions n=0 f n = P {τ 0 = n}. G p (z) = p n z n and G f (z) = E[z τ0 ] = f n z n. Note that τ 0 may be infinite. In this circumstance, P {τ 0 < } = G f () <. Proposition 2.3. Using the notation above, we have Proof. Note that p 0 = and f 0 = 0. For n, p n = P {S n = 0} = = G p (z) = + G p (z)g f (z). n P {S n = 0, τ 0 = k} = k=0 n=0 n P {S n = 0 τ 0 = k}p {τ 0 = k} k=0 n P {S n k = 0}P {τ 0 = k} = k=0 n p n k f k. Now multiply both sides by z n, sum on n, and use the property of multiplication of power series. Proposition 2.4. For a simple random walk, with the steps X k taking the value + with probability p and with value q = p.. G p (z) = ( 4pqz 2 ) /2. 2. G f (z) = ( 4pqz 2 ) /2. Proof.. Note that p n = 0 if n is odd. For n even, we must have n/2 steps to the left and n/2 steps to the right. Thus, ( ) n p n = 2 n (pq) n/2. Now, set x = pqz and use the binomial power series for ( x) /2. 2. Apply the formula in with the identity in the previous proposition. k=0 Corollary 2.5. P {τ 0 < } = p q Proof. G f () = 4pq. Now a little algebra gives the result.

23 2 RANDOM WALK 23 Exercise 2.6. Show that Eτ 0 = G f () =. Exercise 2.7. Let S be a simple symmetric random walk on Z. Show that lim πnp {S2n = 0} =. n and find c n so that Hint: Use the Stirling formula. lim c np {τ 0 = 2n} =. n Exercise 2.8. Let S be a random walk in R d having the property that its components form d independent simple symmetric random walks on Z. Show that S is recurrent if d = or 2, but is not recurrent if d 3. Exercise 2.9. Let S be the simple symmetric random walk in R 3. Show that P {S 2n = 0} = ( ) 2n n n j ( ) 2 n 6 2n n = 0,,.... n j, k, (n j k) j=0 k= Exercise The simple symmetric random walk in R 3 is not recurrent. Hint: For a nonnegative sequence a,..., a m, n l= a2 l (max l m a l ) n l= a l. For S, a simple symmetric random walk in R d, d > 3. coordinates. Define the stopping times τ 0 = 0, τ k+ = inf{n > τ k : S n S τk }. Let S be the projection onto the first three Then { S τk : k 0} is a simple symmetric random walk in R 3 and consequently return infinitely often to zero with probability 0. Consequently, S is not recurrent.

24 3 RENEWAL SEQUENCES 24 3 Renewal Sequences Definition 3... A random {0, }-valued sequence X is a renewal sequence if X 0 =, P {X n = ɛ n, 0 < n r + s} = P {X n = ɛ n, 0 < n r}p {X n r = ɛ n, r n r + s} for all positive integers r and s and sequences (ɛ,..., ɛ r+s ) {0, } (r+s) such that ɛ r =. 2. The random set Σ X = {n; X n = } is called the regenerative set for the renewal sequence X. 3. The stopping times are called the renewal times, their differences are called the waiting or sojourn times and T 0 = 0, T m = inf{n > T m ; X n = } W j = T j T j, j =, 2,.... T m = n W j. A renewal sequence is a random sequence whose distribution beginning at a renewal time is the same as the distribution of the original renewal sequence. We can characterize the renewal sequence in any one of four equivalent ways, through. the renewal sequence X, 2. the regenerative set Σ X, 3. the sequence of renewal times T, or 4. the sequence of sojourn times W. Check that you can move easily from one description to another. j= Definition For a sequence, {a n : n }, define the generating function G a (z) = a k z k. k=0 2. For pair of sequences, {a n : n } and {b n : n }, define the convolution sequence n (a b) n = a k b n k. 3. For a pair of measures A and B on N, define the convolution measure (A B)(K) = A{j}B{k}. k=0 j+k K

25 3 RENEWAL SEQUENCES 25 By the uniqueness theorem for power series, a sequence is uniquely determined by its generating function. Convolution is associative and commutative. We shall write a 2 = a a and a m for the m-fold convolution. Exercise G a b (z) = G a (z)g b (z). 2. If X and Y are independent N-valued random variables, then the mass function for X + Y is the convolution of the mass functions for X and the mass function for Y. Exercise 3.4. The sequence {W j : j } are independent and identically distributed. Thus, T is a random walk on Z + having on strictly positive integral steps. If we let R be the waiting time distribution for T, then R m (K) = P {T m K}. We can allow for P {X n = 0 for all n } > 0. In this case the step size will be infinite with positive probability, R{ }. This can be determined from the generating function because R{ } = G R (). There are many regenerative sets in a random walk. Exercise 3.5. Let S be a random walk on R, then the following are regenerative sets.. {n; S n = 0} 2. {n; S n > S k for 0 k < n} (strict ascending ladder times) 3. {n; S n < S k for 0 k < n} (strict descending ladder times) 4. {n; S n S k for 0 k < n} (weak ascending ladder times) 5. {n; S n S k for 0 k < n} (weak decending ladder times) If S is a random walk in Z, then the following are regenerative sets. 6. {S n ; S n > S k for 0 k < n} 7. { S n ; S n > S k for 0 k < n} Theorem 3.6. Let X and Y be independent renewal sequencces, then so is {X 0 Y 0, X Y,...}. In other words, the coincident times of independent renewal sequences is itself a renewal sequence. Proof. Choose natural numbers r and s and fix a sequence (ɛ 0,..., ɛ r+s ) {0, } (r+s+) with ɛ r =. Let A be the sets of pairs of sequences (ɛ X 0,..., ɛ X r+s) and (ɛ Y 0,..., ɛ Y r+s) in {0, } (r+s+) whose term by term

26 3 RENEWAL SEQUENCES 26 product is (ɛ 0,..., ɛ r+s ). Note that ɛ X r = ɛ Y r =. P {X n Y n = ɛ n, n =,..., r + s} = P {X n = ɛ X n, Y n = ɛ Y n, n =,..., r + s} A = A = A = A P {X n = ɛ X n, n =,..., r + s}p {Y n = ɛ Y n, n =,..., r + s} P {X n = ɛ X n, n =,..., r}p {X n r = ɛ X n, n = r +,..., r + s} P {Y n = ɛ Y n, n =,..., r}p {Y n r = ɛ Y n, n = r +,..., r + s} P {X n = ɛ X n, Y n = ɛ Y n, n =,..., r}p {X n r = ɛ X n, Y n = ɛ Y n, n = r +,..., r + s} = P {X n Y n = ɛ n, n =,..., r}p {X n r Y n r = ɛ n, n = r +,..., r + s} 3. Waiting Time Distributions and Potential Sequences Definition For a renewal sequence X define the renewal measure N(B) = n B X n = I B (T n ). The σ-finite random measure N gives the number of renewels during a time set B. 2. U(B) = EN(B) = n=0 P {T n B} = n=0 R n (B) is the potential measure. Use the monotone convergence theorem to show that it is a measure. 3. The sequence {u k : k 0} defined by is called the potential sequence for X. n=0 u k = P {X k = } = U{k} Note that Exercise u k = P {T n = k}. n=0 G U = /( G R ) (3.) 2. U = (U R) + δ 0 where δ 0 denotes the delta distribution. Consequently, the waiting time distribution, the potential measure and the generating function for the waiting time distribution each uniquely determine the distribution of the renewal sequence.

27 3 RENEWAL SEQUENCES 27 Theorem 3.9. For a renewal process, let R denote the waiting time distribution, {u n ; n 0} the potential sequence, and N the renewal measure. For non-negative integers k and l, P {N[k +, k + l] > 0} = k R[k + n, k + l n]u n n=0 = k R[k + l + n, ]u n = n=0 k+l n=k+ R[k + l + n, ]u n. Proof. If we decompose the event according to the last renewal in [k +, k + l], we obtain P {N[k +, k + l] > 0} = = = = = P {T j k, k + T j+ k + l} j=0 k P {T j = n, k + T j+ k + l} j=0 n=0 k P {T j = n, k + n W j+ k + l n} j=0 n=0 k P {T j = n}p {k + n W j+ k + l n} j=0 n=0 k R[k + n, k + l n]u n. n=0 On the other hand, P {N[k +, k + l] = 0} = = = k P {T j k, T j+ > k + l} = P {T j = n, T j+ > k + l} j=0 j=0 n=0 k k P {T j = n, W j+ > k + l n} = P {T j = n}p {W j+ > k + l n} j=0 n=0 j=0 n=0 k R[k + l + n, ]u n. n=0 Now, take complements.

28 3 RENEWAL SEQUENCES 28 Finally, if we decompose the event according to the first renewal in [k +, k + l], we obtain P {N[k +, k + l] > 0} = = = P {k + T j k + l, T j+ > k + l} j=0 k+l j=0 n=k+ k+l j=0 n=k+ P {T j = n, T j+ > k + l} = k+l j=0 n=k+ P {T j = n}p {W j+ > k + l n} = P {T j = n, W j+ > k + l n} k+l n=k+ R[k + l + n, ]u n. Taking l = 0 for the second statement in the theorem above gives: Corollary 3.0. For each k 0, = k R[k + n, ]u n = n=0 k R[n +, ]u k n. (3.2) Exercise 3.. Let X be a renewal sequence with potential measure U and waiting time distribution R. If U(Z + ) <, then N(Z + ) is a geometric random variable with mean U(Z + ). If U(Z + ) =, then N(Z + ) = a.s. In either case, U(Z + ) = R{ }. Definition 3.2. A renewal sequence and the corresponding regenerative set are transient if the corresponding renewal measure is finite almost surely and they are recurrent otherwise. Theorem 3.3 (Strong Law for Renewal Sequences). Given a renewal sequence X, let N denote the renewal measure, U, the potential measure and µ [, ] denote the mean waiting time. Then, n=0 lim n n U[0, n] = µ, lim n n N[0, n] = µ a.s. (3.3) Proof. Because N[0, n] n +, the first equality follows from the second equality and the bounded convergence theorem. To establish this equality, note that by the strong law of large numbers, T m m = m m W j = µ j= a.s.. If T m n < T m, then N[0, n] = m, and m T m n N[0, n] = m n and the equality follows by taking limits as n. m T m = m m m T m

29 3 RENEWAL SEQUENCES 29 Definition 3.4. Call a recurrent renewal sequence positive recurrent if the mean waiting time is finite and null recurrent if the mean waiting time is infinite. Definition 3.5. Let X be a random sequence X n {0, } for all n and let τ = min{n 0; X n = }. Then X is a delayed renewal seqence if either P {τ = } = or, conditioned on the event {τ < }, is a renewal sequence. τ is called the delay time. {X τ+n ; n 0} Exercise 3.6. Let S be a random walk on Z and let X n = I {x} (S n ). Then X is a delayed renewal sequence. Exercise 3.7. Let X be a renewal sequence. Then a {0, }-valued sequence, X, is a delayed renewal sequence with the same waiting time distribution as X if and only if P { X n = ɛ n for 0 n r + s} = P { X n = ɛ n for 0 n r}p {X n r = ɛ n for r < n r + s} (3.4) for all r, s N and (ɛ 0,..., ɛ r+s ) {0, } (r+s+) satisfying ɛ r =. Theorem 3.8. Let X and Ỹ be independent delayed renewal sequences, both with waiting time distribution R. Define the F (X,Y ) -stopping time σ = min{n 0 : X n = Ỹn = }. Define { Xn if n σ, X n = Ỹ n if n > σ Then X and X have the same distribution. In other words, X and Ỹ is a coupling of the renewal processes having marginal distributions X and Y with coupling time σ. Proof. By the Daniell-Kolmogorov extension theorem, it is enough to show that they have the same finite dimensional distributions. With this in mind, fix n and (ɛ 0,..., ɛ n ) {0, } (n+). Define σ = min{n, min{k, Ỹk = ɛ k = }}. Note that on the set { X 0 = ɛ 0,..., X n = ɛ n }, σ = min{n, σ}. Noting that X and (Ỹ, σ) are independent, we have P { X 0 = ɛ 0..., X n = ɛ n } = = = n P { σ = k, X 0 = ɛ 0,..., X n = ɛ n } k=0 n P { σ = k, X 0 = ɛ 0,..., X k = ɛ k, Y k+ = ɛ k+,..., Y n = ɛ n } k=0 n P {X 0 = ɛ 0,..., X k = ɛ k }P { σ = k, Ỹk+ = ɛ k+,..., Ỹn = ɛ n } k=0 Claim. Let Y be a non-delayed renewal sequence with waiting distribution R. Then for k n P { σ = k, Ỹk+ = ɛ k+,..., Ỹn = ɛ n } = P { σ = k}p {Y k+ = ɛ k+,..., Y n = ɛ n }.

30 3 RENEWAL SEQUENCES 30 If k = n, then because k + > n, the given event is impossible and both sides of the equation equal zero. For k < n and ɛ k = 0, then { σ = k} = and again the equation is trivial. For k < n and ɛ k =, then { σ = k} is the finite union of events of the form {Ỹ0 = ɛ 0,..., Ỹk = } and the claim follows from the identity (3.4) above on delayed renewal sequences. Use this identity once more to obtain P { X 0 = ɛ 0..., X n = ɛ n } = = n (P { σ = k}p {X 0 = ɛ 0,..., X k = ɛ k }P {Y k+ = ɛ k+,..., Y n = ɛ n }) k=0 n P { σ = k}p {X 0 = ɛ 0,..., X n = ɛ n } = P {X 0 = ɛ 0,..., X n = ɛ n }. k=0 Exercise 3.9. Let W be an N-valued random variable with mean µ, then P {W n} = µ. n= More generally, set then n f(n) = a n, k= Ef(W ) = a n P {W n}. n= Theorem Let R be a waiting distribution with finite mean µ and let Ỹ be a delayed renewal sequence having waiting time distribution R and delay distribution D{n} = µ R[n +, ), n Z+. Then P {Ỹk = } = /µ. Proof. By the exercise above, D is a distribution. Let T 0, the waiting time, be a random variable having this distribution. Let {u n ; n } be the potential sequence corresponding to R. Then P {X n = T 0 = k} = u n k provided k n and 0 otherwise. Hence, by the identity (3.2). P {Ỹk = } = k P {Ỹk = T 0 = n}p {T 0 = n} = n=0 k n=0 u k n µ R[n +, ] = µ

31 3 RENEWAL SEQUENCES Renewal Theorem Definition 3.2. The period of a renewal sequence is A renewal process is aperiodic if its period is one. gcd{n > 0; u n > 0}, gcd( ) =. Note that if B is a set of positive integers, gcd(b) =, then there exists K such that every integer k > K can be written as a positive linear combination of elements in B. Theorem 3.22 (Renewal). Let µ [, ] and let X be an aperiodic renewal sequence, then lim P {X k = } = k µ. Proof. (transient case) In this case R{ } > 0 and thus µ =. Recall that u k = P {X k = }. Because lim k u k = 0 = /µ. u k <, k=0 (positive recurrent) Let Ỹ be a delayed renewal sequence, independent of X, having the same waiting time distribution as X and satisfying Define the stopping time and the process P {Ỹk = } = µ. σ = min{k 0; X k = Ỹk = }. X k = { Xk if k σ, Ỹ k if k > σ. Then X and X have the same distribution. If P {σ < } =, then P {X k = } µ = P { X k = } P {Ỹk = } = (P { X k =, σ k} P {Ỹk =, σ k}) + (P { X k =, σ > k} P {Ỹk =, σ > n}) = P { X k =, σ > k} P {Ỹk =, σ > k} max{p { X k =, σ > k}, P {Ỹk =, σ > k}}p {σ > k}. Thus, the renewal theorem holds in the positive recurrent case if σ is finite almost surely. With this in mind, define the F Y t -stopping time τ = min{k; Ỹk = and u k > 0}. Because u k > 0 for sufficiently large k, and Ỹ has a finite waiting, then Ỹ has a finite delay, τ is finite with probability. Note that X, τ and {Ỹτ+k; k = 0,,...} are independent. Consequently {X k Ỹ τ+k ; k = 0,,...} is a renewal sequence independent of τ with potential sequence P {X k Ỹ τ+k = } = P {X k = }P {Ỹτ+k = } = u 2 k.

32 3 RENEWAL SEQUENCES 32 Note that k= u2 k =. Otherwise, lim k u k = 0 contradicting from (3.3), the strong law for renewal n sequences, that lim n n k= u k = /µ. Thus, the renewal process is recurrent. i.e., P {X k = Ỹτ+k = i.o} =. Let {σ m ; m } denote the times that this holds. Because this sequence is strictly increasing, σ 2mj + m < σ 2k(m+). Consequently, the random variables {X σ2km +m; j } are independent and take the value one with probability u m. If u m > 0, then P {X σ2km +m = } = u m =. k= Thus, by the second Borel-Cantelli lemma, k= P {X σ2mj+m = i.o} =. Recall that τ is independent of X and {Ỹτ+k; k } and so P {X σ2τj+τ = i.o} = E[P {X σ2τj+τ = i.o. τ}] = Each one of these occurences is an example of σ <. P {X σ2mj+m = i.o}p {τ = m} =. m= (null recurrent) By way of contradiction, choose ɛ > 0, so that Now pick an integer q so that lim inf k u k > ɛ. q R[n, ) > 2 ɛ. n= Introduce the independent delayed renewal processes Ỹ 0,..., Ỹ q with waiting distribution R and fixed delay r for Ỹ r. Thus X and Ỹ 0 have the same distribution. Guided by the proof of the positive recurrent case, define the stopping time σ = min{k : Ỹ r k =, for all r = 0,..., q}, and set X r k = { Ỹ r k if k σ, X k if k > σ. We have shown that X r and Ỹ r have the same distribution. Because u k > ɛ infinitely often, k= uq+ k =. Modify the argument about to see that σ < with probability one. Consequenty, there exists an integer k > q so that P {σ < k} /2 and u k = P {X k = } > ɛ.

33 3 RENEWAL SEQUENCES 33 Thus, u k r = P { X r k = } P { X r k =, σ < k} = P { X r k = σ < k}p {σ k} ɛ 2. Consequently q+ u k+ n R[n, ) >. n= However identity (3.2) states that this sum is at most one. This contradiction proves the null recurrent case. Putting this altogether, we see that a renewal sequence is recurrent if and only if n=0 u n =, null recurrent if and only if it is recurrent and lim n u n = 0. Exercise Fill in the details for the proof of the renewal theorem in the null recurrent case. Corollary Let µ [, ] and let X be an renewal sequence with period γ, then lim k P {X γk = } = γ µ. Proof. The process Y defined by Y k = X γk is an aperiodic renewal process with renewal measure R Y (B) = R X (γb) and mean µ/γ. 3.3 Applications to Random Walks We now apply this to random walks. Lemma Let n 0. For a real-valued random walk, S, the probability that n is a strict ascending ladder time equals the probability that there is not positive ladder time less than or equal to n Proof. Let {X k : k } be the steps in S. Then { n } {n is a strict ascending ladder time for S} = X k > 0; for m =, 2,..., n. and { m } {, 2,..., n is not a weak descending ladder time for S} = X k > 0; for m =, 2,..., n. By replacing X k with X n+ k, we see that these two events have the same probability. Theorem Let G ++ and G be the generating functions for the waiting time distributions for strict ascending ladder times and weak descending latter times for a real-valued random walk. Then k=m k= ( G ++ (z)) ( G (z)) = ( z).

34 3 RENEWAL SEQUENCES 34 Proof. Let u ++ n be the probability that n is a strict ascending ladder time and r k be the probability that k is the smallest positive weak ascending ladder time. Then the lemma above states that u ++ n = n r k. By the generation function identity (3.), ( G ++ (z)) is the generating function of the sequence {u ++ n : n }, we have for z [0, ), k=0 G ++ (z) = = ( n=0 n k=0 z k=0 r k )zn = z r k z k k=0 n=k z = G (z). z r k zn The case z = now follows from the monotone convergence theorem. Exercise Either the set of strict ascending ladder times and the set of weak descending ladder times are both null recurrent or one of these sets is transient and the other is positive recurrent. Remark If the step distribution in a random walk assigns zero probability to each one-point set, the weak and strict ladder times are almost surely equal. In this case, we have, with the obvious notational addition, ( G ++ (z)) ( G (z)) = ( z). If the step distribution is symmetric about zero, then G ++ (z) = G (z) = z and a Taylor s series expansion gives the waiting time distribution of ladder times.