Stochastic Processes in Discrete Time

Transcription

1 Stochastic Processes in Discrete Time May 5, 2005 Contents Basic Concepts for Stochastic Processes 3. Conditional Expectation Stochastic Processes, Filtrations and Stopping Times Random Walk 3 2. Recurrence and Transience The Role of Dimension Simple Random Walks Renewal Sequences 2 3. Waiting Time Distributions and Potential Sequences Renewal Theorem Applications to Random Walks Martingales Examples Doob Decomposition Optional Sampling Theorem Inequalities and Convergence Backward Martingales Markov Chains Definition and Basic Properties Examples Extensions of the Markov Property Classification of States Recurrent Chains Stationary Measures Asymptotic Behavior Transient Chains Asymptotic Behavior ρ C Invariant Measures

2 6 Stationary Processes Definitions and Examples Birkhoff s Ergodic Theorem Ergodicity and Mixing Entropy

3 Basic Concepts for Stochastic Processes In this section, we will introduce two of the most versatile tools in the study of random processes - conditional expectation with respect to a σ-algebra and stopping times with respect to a filtration of σ-algebras.. Conditional Expectation We will begin with a general definition of conditional expectation and show that this agrees with more elementary notions. We have previously seen this definition in the definition of conditional expectation with respect to a random variable. Definition. Let Y be an integrable random variable on (Ω, F, P ) and let G be a sub-σ-algebra of F. The conditional expectation of Y given G, denoted E[Y G] is the a.s. unique random variable satisfying the following two conditions.. E[Y G] is G-measurable. 2. E[E[Y G]; A] = E[Y ; A] for any A G. Thus, E[Y G] is essentially the only random variable that uses the information provided by G and gives the same averages as Y on events that are in G. The existence and uniqueness is provided by the Radon-Nikodym theorem. For Y positive, define the measure ν(a) = E[Y ; A] for A G. Then ν is a measure defined on G that is absolutely continuous with respect to the underlying probability restricted to G. Set E[Y G] equal to the Radon-Nikodym derivative dν/dp G. The general case follows by taking positive and negative parts of Y. For B F, the conditional probability of B given G is defined to be P (B G) = E[I B G]. Exercises.. Let C F, then P (B σ(c)) = P (B C)I C + P (B C c )I C c. 2. If G = σ{c,, C n }, the σ-algebra generated by a finite partition, then P (B G) = Theorem. (Bayes Formula) Let B G, then n P (B C i )I Ci. i= P (B A) = E[P (A G)I B]. E[P (A G)] Proof. E[P (A G)I B ] = E[I A I B ] = P (A B) and E[P (A G)] = P (A). Exercise. Show the Bayes formula for a finite partition {C,, C n } is P (C j A) = P (A C j )P (C j ) n i= P (A C i)p (C i ). 3

4 If G = σ(x), then we usually write E[Y G] = E[Y X]. For these circumstances, we have the following theorem which can be proved using the standard machine. Theorem. Let X be a random variable. Then Z is a measurable function on (Ω, σ(x)) if and only if there exists a measurable function h on the range of X so that Z = h(x). This shows that the deifinition of conditional expectation with respect to a σ-algebra extends the definition of conditional expectation with respect to a random variable. We now summarize the properties of conditional expectation. Theorem. Let Y, Y, Y 2, have finite absolute mean on (Ω, F, P ) and let a, a 2 R. In addition, let G and H be σ-algebras contained in F. Then. If Z is any version of E[Y G], then EZ = EY. (E[E[Y G]] = EY ). 2. If Y is G measurable, then E[Y G] = Y, a.s. 3. (Linearity) E[a Y + a 2 Y 2 G] = a E[Y G] + a 2 E[Y 2 G], a.s. 4. (Positivity) If Y 0, then E[Y G] 0, a.s. 5. (conditional monotone convergence theorem) If Y n Y, then E[Y n G] E[Y G], a.s. 6. (conditional Fatous s lemma) If Y n 0, then E[lim inf n Y n G] lim inf n E[Y n G]. 7. (conditional dominated convergence theorem) If lim n Y n (ω) = Y (ω), a.s., if Y n (ω) V (ω) for all n, and if EV <, then lim n E[Y n G] = E[Y G], almost surely. 8. (conditional Jensen s inequality) If φ : R R is convex, and E φ(y ) <, then almost surely. In particular, 9. (Contraction) E[Y G] p Y p for p. 0. (Tower Property) If H G, then almost surely. E[φ(Y ) G] φ(e[y G]), E[E[Y G] H] = E[Y H],. (conditional constants) If Z is G-measurable, and E ZY <, then E[ZY G] = ZE[Y G]. 4

5 2. (Role of Independence) If H is independent of σ(σ(y ), G), then almost surely. In particular, 3. if Y is independent of H, then E[Y H] = EY. Proof.. Take A = Ω in the definition. E[Y σ(g, H)] = E[Y G], 2. Check that Y satisfies both and 2 in the definition. 3. Clearly a E[Y G] + a 2 E[Y 2 G] is G-measurable. Let A G. Then yielding property 2. E[a E[Y G] + a 2 E[Y 2 G]; A] = a E[E[Y G]; A] + a 2 E[E[Y 2 G]; A] = a E[Y ; A] + a 2 E[Y 2 ; A] = E[a Y + a 2 Y 2 ; A], 4. Because Y 0 and {E[Y G] n }] is G-measurable, we have that 0 E[Y ; {E[Y G] n }] = E[E[Y G]; {E[Y G] n }] n P {E[Y G] n } and Consequently, P {E[Y G] n } = 0. ( ) P {E[Y G] < 0} = P {E[Y G] n } = 0. n= 5. Write Z n = E[Y G]. Then, by positivity of conditional expectation, Z = lim n Z n exists almost surely. Note that Z is G-measurable. Choose A G. By the monotone convergence theorem, E[Z; G] = lim E[Z n; G] = lim E[Y n; G] = E[Y ; G]. n n Thus, Z = E[Y G]. Because Y Y Y, we have E[ Y G] E[Y G] E[ Y G] and E[Y G] E[ Y G]. Consequently, E[ G] : L (Ω, F, P ) L (Ω, G, P ) is a continuous linear mapping. 5

6 6. Repeat the proof of Fatou s lemma replacing expectation with E[ G] and use both positivity and the conditional monotone convergence theorem. 7. Repeat the proof of the dominated convergence theorem from Fatou s lemma again replacing expectation with conditional expectation. 8. Follow the proof of Jensen s inequality. 9. Use the conditional Jensen s inequality on the convex function φ(y) = y p, y 0, E[Y G] p p = E[ E[Y G] p ] E[E[ Y G] p ] E[E[ Y p G]] = E[ Y p ] = Y p p. 0. By definition E[E[Y G] H] is H-measurable. Let A H, then A G and E[E[E[Y G] H]; A] = E[E[Y G]; A] = E[Y ; A].. Use the standard machine. The case Z an indicator function follows from the definition of conditional expectation. 2. We need only consider the case Y 0, EY > 0. Let Z = E[Y G] and consider the two probability measures E[Y ; A] E[Z; A] µ(a) =, ν(a) =. EY EY Note that EY = EZ and define C = {A : µ(a) = ν(a)}. If A = B C, B G, C H, then Y I B and C are independent and E[Y ; B C] = E[Y I B ; C] = E[Y I B ]P (C) = E[Y ; B]P (C). Z is G-measurable and thus ZI B and C are independent and Consequently, E[Z; B C] = E[ZI B ; C] = E[ZI B ]P (C) = E[Z; B]P (C) = E[Y ; B]P (C). D = {B C; B G, C H} C. Now, D is closed under pairwise intersection. Thus, by the the Sierpinski Class Theorem, µ and ν agree on σ(d) = σ(g, H) and E[Z; A] = E[Y ; A], for all A σ(g, H). Because Z is σ(g, H)-measurable, this completes the proof. 3. Take G to be the trivial σ-algebra {, Ω}. However, this property can be verified directly. Exercise. Let X be integrable, then the collection {E[X G] : G a sub-σalgebra of F}. 6

7 is uniformly integrable. As before, if E[Y 2 ] <, we can realize the conditional expectation E[Y G] as a Hilbert space projection. For the sub-σ-algebra G F, L 2 (Ω, G, P ) L 2 (Ω, F, P ). Let Π G be the orthogonal projection operator onto the closed subspace L 2 (Ω, G, P ). Then, mimicking the analysis of conditional expectation with respect to a random variable, we have that Π G Y = E[Y G]. As before, this can be viewed as a minimization problem min{e[(y Z) 2 ]; Z L 2 (Ω, G, P )}. The minimum is acheived uniquely by the choice Z = E[Y G]. Exercises.. Prove the conditional Chebyshev inequality: Let g : R [0, ) be increasing on [a, ), then 2. Let G be a sub-σ-algebra and let EY 2 <, then where Var(Y G)] = E[(Y E[Y G]) 2 G]. Examples. P {g(x) > a G} E[g(X) G]. a Var(Y ) = E[Var(Y G)] + Var(E[Y G]).. Let S j be a binomial random variable, the number of heads in j flips of a biased coin with the probability of heads equal to p. Thus, there exist independent 0- valued random variables {X j : j } so that S n = X + + X n. Let k < n, then S k = X + + X k and S n S k = X k+ + + X n are independent. Therefore, E[S n S k ] = E[(S n S k ) + S k S k ] = E[(S n S k ) S k ] + E[S k S k ] = E[S n S k ] + S k = (n k)p + S k. Now to find E[S k S n ], note that x = E[S n S n = x] = E[X S n = x] + + E[X n S n = x]. Each of these summands have the same value, namely x/n. Therefore E[S k S n = x] = E[X S n = x] + + E[X k S n = x] = k n x and E[S k S n ] = k n S n. 2. Let {X n : n } be independent random variables having the same distribution. Let µ be their common mean and σ 2 their common variance. Let N be a non-negative valued random variable and define S = N n= X n, then ES = E[E[S N]] = E[Nµ] = µen. 7

8 Var(S) = E[Var(S N)] + Var(E[S N]) = E[Nσ 2 ] + Var(Nµ) = σ 2 EN + µ 2 Var(N). Alternatively, for an N-valued random variable X, define the generating function Then Now, G X (z) = Ez X = z x P {X = x}. G X () =, G X() = EX, G X() = E[X(X )], Var(X) = G X() + G X() G X() 2. G S (z) = Ez S = E[E[z S N]] = E[z S N = n]p {N = n} = E[z (X+ +X N ) N = n]p {N = n} Thus, and = = E[z (X+ +Xn) ( ]P {N = n} = E[z X ] E[z Xn ] ) P {N = n} G X (z) n P {N = n} = G N (G X (z)) G S(z) = G N (G X (z))g X(z), x=0 ES = G S() = G N()G X() = µen G S(z) = G N (G X (z))g X(z) 2 + G N(G X (z))g X(z) G S() = G N()G X() 2 + G N ()G X() = G N ()µ 2 + EN(σ 2 µ + µ 2 ). Now, substitute to obtain the formula above. 3. The density with respect to Lebesgue measure for a bivariate standard normal is f (X,Y ) (x, y) = ( 2π ρ exp x2 2ρxy + y 2 ) 2 2( ρ 2. ) To check that E[Y X] = ρx, we must check that for any Borel set B E[Y I B (X)] = E[ρXI B (X)] or E[(Y ρx)i B (X)] = 0. Complete the square to see that E[(Y ρx)i B (X)] = 2π ρ 2 = 2π ρ 2 B B ( ( (y ρx) exp ( ( z exp ) ) (y ρx)2 2( ρ 2 dy e x2 /2 dx ) z 2 ) ) 2( ρ 2 dz e x2 /2 dx = 0, z = y ρx ) because the integrand for the z integration is an odd function. Now Cov(X, Y ) = E[XY ] = E[E[XY X]] = E[XE[Y X]] = E[ρX 2 ] = ρ. 8

9 Definition. On a probability space (Ω, F, P ), let G, G 2, H be sub-σ-algebras of F. Then the σ-algebras G and G 2 are conditionally independent given H if for A i G i, i =, 2. P (A A 2 H) = P (A H)P (A 2 H), Exercise. (conditional Borel-Cantelli lemma) For G a sub-σ-algebra of F, let {A n : n } be a sequence of conditionally independent events given G. Show that for almost all ω Ω, { 0 P {A n i.o. G}(ω) = according as Consequently, n= P {A n i.o.} = P {ω : { = P (A n G)(ω) <, P (A n G)(ω) = }. n= Proposition. Let H, G and D be σ-algebras. Then. If H and G are conditionally independent given D, then H and σ(g, D) are conditionally independent given D. 2. Let H and G be sub-σ-algebras of H and G, respectively. Suppose that H and G are independent. Then H and G are conditionally independent given σ(h, G ). 3. Let H G. Then G and D are conditionally independent given H if and only if, for every D D, P (D H) = P (D G). Exercise. Prove the previous proposition. The Sierpinski class theorem will be helpful in the proof..2 Stochastic Processes, Filtrations and Stopping Times A stochastic process X (or a random process, or simply a process) with index set Λ and a measurable state space (S, B) defined on a probability space (Ω, F, P ) is a function such that for each λ Λ, X : Λ Ω S X(λ, ) : Ω S is an S-valued random variable. Note that Λ is not given the structure of a measure space. In particular, it is not necessarily the case that X is measurable. However, if Λ is countable and has the power set as its σ-algebra, then X is automatically measurable. 9

10 X(λ, ) is variously written X(λ) or X λ. Throughout, we shall assume that S is a metric space with metric d. A realization of X or a sample path for X is the function X(, ω 0 ) : Λ S for some ω 0 Ω. Typically, for the processes we study Λ will be the natural numbers, and [0, ). Occasionally, Λ will be the integers or the real numbers. In the case that Λ is a subset of a multi-dimensional vector space, we often call X a random field. The distribution of a process X is generally stated via its finite dimensional distributions µ (λ,...,λ n)(a) = P {(X λ,..., X λn ) A}. If this collection probability measures satisfy the consistency consition of the Daniell-Kolmogorov extension theorem, then they determine the distribution of the process X on the space S Λ. We will consider the case Λ = N. Definition.. A collection of σ-algebras {F n : n 0} of set in F is a filtration if F n F n+, n = 0,,.... F n is meant to capture the information available to an observer at time n. 2. The natural filtration for the process X is F X n = σ{x k ; k n}. 3. This filtration is complete if F is complete and {A : P (A) = 0} F A process X is adapted to a filtration {F n : n 0} (X is F n -adapted.) if X n is F n measurable. In other words, X is F n -adapted if and only if for every n N, Fn X F n 5. If two processes X and X have the same finite dimensional distributions, then X is a version of X. 6. If, for each n, P {X n = X n } =, then we say that X is a modification of X. In this case P {X = X} =. This conclusion does not necessarily hold if the index set is infinite but not countably infinite. 7. A non-negative random variable τ is an F n -stopping time if {τ n} F n for every n N. Exercise. τ is an F n -stopping time if and only if {τ = n} F n for every n N. Exercise. Let X be F n -adapted and let τ be an F n -stopping time. For B B, define Then σ(b, τ) is an F n -stopping time. σ(b, τ) = inf{n τ : X n B}. Proposition. Let τ, τ,..., be F n -stopping times and let c 0. Then. τ + c and min{τ, c} are F n -stopping times. 2. sup k τ k is an F n -stopping time. 3. inf k τ k is an F n -stopping time. 0

11 4. lim inf k τ k and lim sup k τ k are F n -stopping times. Proof.. {τ + c n} = {τ n c} F max{0,n c} F n. If c n, {min{τ, c} n} = Ω F n, If c > n, {min{τ, c} n} = {τ n} F n, 2. {sup k τ k n} = k {τ k n} F n. 3. {inf k τ k n} = k {τ k n} F n. (This statement does not hold if τ k is a continuous random variable.) 4. This follows from parts 2 and 3. Definition.. Let τ be an F n -stopping time. Then F τ = {A F : A {τ n} F n for all n}. 2. Let X be a random process then X τ, the process X stopped at time τ, is defined by Xn τ = X min{τ,n} The following exercise explains the intuitive idea that F τ gives the information known to observer up to time τ. Exercise.. F τ is a σ-algebra. 2. F X τ = σ{x τ n : n 0}. Proposition. Let σ and τ be F n -stopping times and let X be an F n -adapted process.. τ and min{τ, σ} are F τ -measurable. 2. If τ σ, then F τ F σ. 3. X τ is F τ -measurable. Proof.. {min{τ, σ} c} {τ n} = {min{τ, σ} min{c, n}} {τ n} = ({τ min{c, n}} {σ min{c, n}}) {τ n} F n. 2. Let A F τ, then Hence, A F σ. A {σ n} = (A {τ n}) {σ n} F n.

12 3. Let B B, then {X τ B} {τ n} = n k=({x k B} {τ = k}) F n. Exercise. Let X be an F n adapted process. For n = 0,,, define F τ n = F min{τ,n}. Then. {F τ n : n 0} is a filtration. 2. X τ is both F τ n-adapted and F n -adapted. 2

13 2 Random Walk Definition. Let X be a sequence of independent and identically distributed R d -valued random variables and define n S 0 = 0, S n = X k. Then the sequence S is a random walk with steps X, X 2,.... If the steps take on only the values ±e j, the standard unit basis vectors in R d, then S is called a simple random walk. If each of these 2d steps are equally probable, the S is called a symmetric simple random walk. k= Theorem. Let X, X 2,... be independent with distribution ν and let τ be a stoppping time. Then, conditioned on {τ < }, {X τ+n, n } is independent of Fτ X and has the same distribution as the original sequence. Proof. Choose A F τ and B j B(R d ), then because the finite dimensional distributions determine the process, it suffices to show that P (A {τ <, X τ+j B j, j k}) = P (A {τ < }) ν(b k ). j k To do this note that A {τ = n} F X n and thus is independent of X n+, X n+2,.... Therefore, P (A {τ = n, X τ+j B j, j k}) = P (A {τ = n, X n+j B j, j k}) = P (A {τ = n})p {X n+j B j, j k}) = P (A {τ = n}) ν(b j ). j k Now, sum over n and note that each of the events are disjoint for differing values of n, Exercise.. Assume that the steps of a real valued random walk S have finite mean and let τ be a stopping with finite mean. Show that ES τ = µeτ. 2. In the exercise, assume that µ = 0, c 0 and τ = min{n 0; S n c}. Show that Eτ is infinite. 2. Recurrence and Transience The strong law of large numbers states that if the step distribution has finite mean µ, then P { lim n n S n = µ} =. So, if the mean exists and is not zero, the walk will drift in the direction of the mean. If the mean does not exist or if it is zero, we have the possibility that the walk visits sites repeatedly. Noting that the walk could only return to possible sites motivates the following definitions. 3

14 Definition. Let S be a random walk. A point x R d is called a recurrent state if for every neighborhood U of x, P {S n U i. o.} =. Definition. If, for some random real valued random variable X and some l > 0, n= P {X = nl} =, then X is said to be distributed on the lattice L l = {nl : n Z} provided that the equation hold for no larger l. If this holds for no l > 0, then X is called non-lattice and is said to be distributed on L 0 = R. Exercise. Let the steps of a random walk be distributed on a lattice L l, l > 0. Let τ n be the n-th return of the walk to zero. Then P {τ n < } = P {τ < } n. Theorem. If the step distribution is on L l, then either every state in L l is recurrent or no states are recurrent. Proof. Let G be the set of recurrent points. Claim I. The set G is closed. Choose a sequence {x k : k 0} G with limit x and choose U a neighborhood of x. Then, there exists an integer K so that x k U whenever k K and hence U is a neighborhood of x k. Because x k G, P {S n U i.o.} = and x G. Call y be a possible state (y C) if for every neighborhood U of y, there exists k so that P {S k U} > 0. Claim II. If x G and y C, then x y G. Let U = (y ɛ, y + ɛ) and pick k so that P { S k y < ɛ} > 0. Use f.o., finitely often, to indicate the complement of i.o., infinitely often. Then k, and P { S n x < ɛ f.o.} P { S k+n x < ɛ f.o., S k y < ɛ}. If S k is within ɛ of y and S k+n is within ɛ of x, then S k+n S k is within 2ɛ of x y. Therefore, for each { S k+n x < ɛ, S k y < ɛ} { (S k+n S k ) (x y) < 2ɛ, S k y < ɛ} { S k+n x < ɛ f.o., S k y < ɛ} { (S k+n S k ) (x y) < 2ɛ f.o., S k y < ɛ}. Use the fact that S k and S k+n S k are independent and that the distribution of S n and S k+n S k are equal. P { S n x < ɛ f.o.} P { (S n+k S k ) (x y) < 2ɛ f.o., S k y < ɛ} = P { (S n+k S k ) (x y) < 2ɛ f.o.}p { S k y < ɛ} = P { S n (x y) < 2ɛ f.o.}p { S k y < ɛ}. 4

15 Because x G, 0 = P { S n x < ɛ f.o.}. This forces P { S n (x y) < 2ɛ f.o.} = 0, P { S n (x y) < 2ɛ i.o.} =. Finally, choose U, a neighborhood of x y and ɛ so that (x y 2ɛ, x y + 2ɛ) U, then and x y G. Claim III. G is a group. If x G and y G, then y C and x y G. P {S n U i.o.} P { S n (x y) < 2ɛ i.o.} =, Claim IV. If G, and if l > 0, then G = L l. Clearly L l G. Because G, 0 G. Set S X = {x : P {X = x} > 0}. If x S X, then x C and x = 0 x G and x G. By the definition of l. All integer multiples of l can be written as positive linear sums from x, x, x S X and G L l Claim V. If l = 0, and G, then G = L 0. 0 = inf{y > 0 : y G}. Otherwise, we are in the situation of a lattice valued random walk. Let x R and let U be a neighborhood of x. Now, choose ɛ > 0 so that (x ɛ, x + ɛ) U. By the definition of infimum, there exists l < ɛ so that l G. Thus, L l G and L l U. Consequently, and x G. P {S n U i.o.} =, Corollary. Let S be a random walk in R d. Then, either the set of recurrent states forms a closed subgroup of R d or no states are recurrent. This subgroup is the closed subgroup generated by the step distribution. Proof. For claims I, II, and III, let stand for some norm in R d and replace open intervals by open balls in that norm. Theorem. Let S be a random walk in R d.. If there exists an open set U containing 0 so that then no states are recurrent. P {S n U} <, 2. If there is a bounded open set U such that then 0 is recurrent. P {S n U} =, 5

16 Proof. If P {S n U} <, then by the Borel-Cantelli, P {S n U i.o.} = 0. Thus 0 is not recurrent. Because the set of recurrent states is a group, then no states are recurrent. Then, For the second half, let ɛ > 0 and choose a finite set F so that U B(x, ɛ). Consequently, for some x, {S n U} x F x F{S n B(x, ɛ)}, P {S n U} x F P {S n B(x, ɛ)} =. P {S n B(x, ɛ)}. Define the pairwise disjoint sets, A k, the last visit to B(x, ɛ) occurred for S k. { {Sn / B(x, ɛ), n =, 2,...} for k = 0 A k = {S k B(x, ɛ), S n+k / B(x, ɛ), n =, 2,...} for k > 0 Then {S n B(x, ɛ) f.o.} = A k, and P {S n B(x, ɛ) f.o.} = P (A k ). k=0 For k > 0, if S k B(x, ɛ) and S n+k S k > 2ɛ, then S n+k / B(x, ɛ). Consequently, k=0 P (A k ) P {S k B(x, ɛ), S n+k S k > 2ɛ, n =, 2,...} = P {S k B(x, ɛ)}p { S n+k S k > 2ɛ, n =, 2,...} = P {S k B(x, ɛ)}p { S n > 2ɛ, n =, 2,...} Summing over k, we find that P {S n B(x, ɛ) f.o.} ( P {S k B(x, ɛ)})p { S n > 2ɛ, n =, 2,...}. k=0 The number on the left is at most one, the sum is infinite and consequently for any ɛ > 0, P { S n > 2ɛ, n =, 2,...} = 0. Now, define the sets A k as before with x = 0. By the argument above P (A 0 ) = 0. For k, define a strictly increasing sequence {ɛ i : i } with limit ɛ. By the continuity property of a probability, we have, Pick a term in this sequence, then P (A k ) = lim i P {S k B(0, ɛ i ), S n+k / B(0, ɛ), n =, 2,...}. P {S k B(0, ɛ i ), S n+k / B(0, ɛ), n =, 2,...} P {S k B(0, ɛ i ), S n+k S k ɛ ɛ i, n =, 2,...} = P {S k B(0, ɛ i )}P { S n+k S k ɛ ɛ i, n =, 2,...} = P {S k B(0, ɛ i )}P { S n ɛ ɛ i, n =, 2,...} = 0. 6

17 Therefore, P (A k ) = 0 for all k 0. P {S n B(0, ɛ) f.o.} = P (A k ) = 0. Because every neighborhood of 0 contains a set of the form B(0, ɛ) for some ɛ > 0, we have that 0 is recurrent. 2.2 The Role of Dimension K=0 Lemma. Let ɛ > 0, m 2, and let be the norm x = max i d x i. Then Proof. Note that and consider the stopping times P { S n < mɛ} (2m) d P { S n < mɛ} P { S n < ɛ}. α { m,...,m } d P {S n αɛ + [0, ɛ) d } τ α = min{n 0 : S n αɛ + [0, ɛ) d }. Consequently, P {S n αɛ + [0, ɛ) d } = = P {S n αɛ + [0, ɛ) d, τ α = k} k= k= k= P { S n S k < ɛ, τ α = k} P { S n S k < ɛ}p {τ α = k} ( P { S n < ɛ})p {τ α = k} P { S n < ɛ} k= because the sum on k is at most. Because the sum on α consists of (2m) d terms, the proof is complete. Theorem. (Chung-Fuchs) Let d =. If the weak law of large numbers holds, then S n is recurrent. n S n P 0 Proof. Let A > 0. Using the lemma above with d = and ɛ =, we have that P { S n < } 2m P { S n < m} Am P { S n < n 2m A }. 7

18 Note that, for any A, we have by the weak law of large numbers, lim n P { S n < n A } =. If a sequence has a limit, then its Cesaro sum has the same limit and therefore, lim m Am P { S n < m 2m A } = A 2. Because A is arbitrary, the first sum above is infinite and the walk is recurrent. Theorem. Let S be an random walk in R 2 and assume that the central limit theorem holds in that lim P { S n B} = n(x) dx n n where n is a normal density whose support is R 2, then S is recurrent. Proof. By the lemma, with ɛ = and d = 2. Let c > 0, then P { S n < } 4m 2 B P { S n < m}. n and m n c implies P { S n < m} = P { n S n < m n } [ c.c] 2 n(x) dx. Call this function ρ(c), then for θ > 0, and we can write m 2 lim P { S [θm m 2 ] < m} = ρ(θ /2 ). P { S n < m} = Let m and use Fatou s lemma. lim inf m 4m 2 P { S n < m} 4 The proof is complete if we show the following. Claim. 0 ρ(θ /2 ) dθ diverges. P { S [θm 2 ] < m} dθ. 0 ρ(θ /2 ) dθ. Choose ɛ so that n(x) n(0)/2 > 0 if x < ɛ, then for θ > /ɛ 2 ρ(θ /2 ) = n(x) dx 2 n(0)4 θ, [ θ /2,θ /2 ] showing that the integral diverges. 8

19 2.3 Simple Random Walks To begin on R, let the steps X k take on the value + with probability and with value q = p. Define the stopping time τ 0 = min{n > 0; S n = 0} and set Now define their generating functions p n = P {S n = 0}, and f n = P {τ 0 = n}. G p (z) = p n z n and G f (z) = E[z τ ] = f n z n. Note that τ 0 may be infinite. In this circumstance, P {τ 0 < } = G f () <. Proposition. For a simple random walk, using the notation above, we have. G p (z) = + G p (z)g f (z). 2. G p (z) = ( 4pqz 2 ) /2. 3. G f (z) = ( 4pqz 2 ) /2. Proof.. For n, p n = P {S n = 0} = = n n P {S n = 0, τ 0 = k} = P {S n = 0 τ 0 = k}p {τ 0 = k} k= n k= k= P {S n k = 0}P {τ 0 = k} = k= p n k f k. Now multiply both sides by z n and use the property of multiplication of power series. 2. Note that p n = 0 if n is odd. For n even, we must have n/2 steps to the left and n/2 steps to the right. Thus, ( ) n p n = 2 n (pq) n/2. Now, set x = pqz and use the binomial power series for ( x) /2. 3. Apply the formula in 2 with the identity in. Corollary. P {τ 0 < } = p q Proof. G f () = 4pq. Now a little algebra gives the result. Exercise. Show that Eτ 0 = G f () =. Exercise. 9

20 . Let S be a simple symmetric random walk on Z. Show that lim πnp {S2n = 0} =. n and find c n so that Hint: Use the Stirling formula. lim c np {τ 0 = 2n} =. n 2. Let S be a random walk in R d having the property that its components form d independent simple symmetric random walks on Z. Show that S is recurrent if d = or 2, but is not recurrent if d Let S be the simple symmetric random walk in R 3. Show that P {S 2n = 0} = ( ) 2n n n j ( ) 2 n 6 2n n = 0,,.... n j, k, (n j k) j=0 k= 4. The simple symmetric random walk in R 3 is not recurrent. Hint: For a nonnegative sequence a,..., a m, n l= a2 l (max l m a l ) n l= a l. For S, a simple symmetric random walk in R d, d > 3. coordinates. Define the stopping times Let S be the projection onto the first three τ 0 = 0, τ k+ = inf{n > τ k : S n S τk }. Then { S τk : k 0} is a simple symmetric random walk in R 3 and consequently return infinitely often to zero with probability 0. Consequently, S is not recurrent. 20

21 3 Renewal Sequences Definitions.. A random {0, }-valued sequence X is a renewal sequence if X 0 =, P {X n = ɛ n, 0 < n r + s} = P {X n = ɛ n, 0 < n r}p {X n r = ɛ n, r < n r + s} for all positive integers r and s and sequences (ɛ,..., ɛ r+s ) {0, } (r+s) such that ɛ r =. 2. The random set Σ X = {n : X n = } is called the regenerative set for the renewal sequence X. 3. The stopping times T 0 = 0, T m = inf{n > T m : X n = } are called the renewal times, their differences W j = T j T j, j =, 2,.... are called the waiting or sojourn times and m T m = W j. j= A renewal sequence is a random sequence whose distribution beginning at a renewal time is the same as the distribution of the original renewal sequence. We can characterize the renewal sequence in any one of four equivalent ways, through. the renewal sequence X, 2. the regenerative set Σ X, 3. the renewal times T, or 4. the sojourn times W. Check that you can move easily from one description to another. Definition. For a sequence, {a n : n }, define the generating function G a (z) = a k z k. 2. For pair of sequences, {a n : n } and {b n : n }, define the convolution sequence (a b) n = k=0 n a k b n k. k= 2

22 3. For a pair of measures A and B on N, define the convolution measure (A B)(K) = A({j})B({k}). j+k K By the uniqueness theorem for power series, a sequence is uniquely determined by its generating function. Convolution is associative and commutative, write a 2 = a a and a m for the m-fold convolution. Exercise.. G a b (z) = G a (z)g b (z). 2. If X and Y are independent N-valued random variables, then the mass function for X + Y is the convolution of the mass functions for X and the mass function for Y. Exercise. The sequence {W j : j } are independent and identically distributed. Example. Let X be an independent sequence of Ber(p) random variables. Then X is a renewal sequence. The waiting times have a Geo(p) distribution. Thus, T is a random walk on Z + having on strictly positive integral steps. If we let R be the waiting time distribution for T, then R m (K) = P {T m K}. We can allow for P {X n = 0 for all n } > 0. In this case the step size will be infinite with positive probability, i.e., R{ } > 0. This can be determined from the generating function because R{ } = G R (). There are many regenerative sets in a random walk. Exercise. Let S be a random walk on R, then the following are regenerative sets.. {n : S n = 0} 2. {n : S n > S k for 0 k < n} (strict ascending ladder times) 3. {n : S n < S k for 0 k < n} (strict descending ladder times) 4. {n : S n S k for 0 k < n} (weak ascending ladder times) 5. {n : S n S k for 0 k < n} (weak decending ladder times) If S is a random walk in Z, then the following are regenerative sets. 6. {S n : S n > S k for 0 k < n} 7. { S n : S n < S k for 0 k < n} 22

23 Theorem. Let X and Y be independent renewal sequencces, then is {X 0 Y 0, X Y,...} also a renewal sequence. Proof. Choose natural numbers r and s and fix a sequence (ɛ,..., ɛ r+s ) {0, } (r+s) with ɛ r =. Let A be the sets of pairs of sequences (ɛ X,..., ɛ X r+s) and (ɛ Y,..., ɛ Y r+s) in {0, } (r+s+) whose term by term product is (ɛ 0,..., ɛ r+s ). Note that ɛ X r = ɛ Y r =. P {X n Y n = ɛ n, n =,..., r + s} = A P {X n = ɛ X n, Y n = ɛ Y n, n =,..., r + s} = A P {X n = ɛ X n, n =,..., r + s}p {Y n = ɛ Y n, n =,..., r + s} = A P {X n = ɛ X n, n =,..., r}p {X n r = ɛ X n, n = r +,..., r + s} P {Y n = ɛ Y n, n =,..., r}p {Y n r = ɛ Y n, n = r +,..., r + s} = A P {X n = ɛ X n, Y n = ɛ Y n, n =,..., r} P {X n r = ɛ X n, Y n = ɛ Y n, n = r +,..., r + s} = P {X n Y n = ɛ n, n =,..., r}p {X n r Y n r = ɛ n, n = r +,..., r + s} 3. Waiting Time Distributions and Potential Sequences Definition.. For a renewal sequence X define the renewal measure N(B) = n B X n = I B (T n ). The σ-finite random measure N gives the number of renewels during a time set B. 2. U(B) = EN(B) = P {T n B} = R n (B) is the potential measure. Use the monotone convergence theorem to show that it is a measure. 3. The sequence {u k : k 0} defined by is called the potential sequence for X. u k = P {X k = } = U({k}) Note that Exercises. u k = P {T n = k}. 23

24 . G U = /( G R ). 2. U = (U R) + δ 0 where δ 0 denotes the delta distribution. Consequently, the waiting time distribution, the potential measure and the generating function for the waiting time distribution each uniquely determine the distribution of the renewal sequence. Theorem. For a renewal process, let R denote the waiting time distribution, {u n : n 0} the potential sequence, and N the renewal measure. For non-negative integers k and l, P {N[k +, k + l] > 0} = k R[k + n, k + l n]u n = k R[k + l + n, ]u n = k+l n=k+ R[k + l + n, ]u n. Proof. P {N[k +, k + l] > 0} = = = = P {T j k, k + T j+ k + l} j=0 k P {T j = n, k + T j+ k + l} j=0 k P {T j = n}p {k + n T k + l n} j=0 k R[k + n, k + l n]u n. P {N[k +, k + l] = 0} = = k P {T j k, T j+ > k + l} = P {T j = n, T j+ > k + l} j=0 j=0 k k P {T j = n}p {T > k + l n} = R[k + l + n, ]u n. j=0 Now, take complements. P {N[k +, k + l] > 0} = = P {k + T j k + l, T j+ > k + l} = j=0 j=0 k P {T j = n}p {T > k + l n} = k+l j=0 n=k+ P {T j = n, T j+ > k + l} k R[k + l + n, ]u n. 24

25 Taking l = 0 for the second statement in the theorem above gives: Corollary. For each k 0, k k = R[k + n, ]u n = R[n +, ]u k n. Exercise. Let X be a renewal sequence with potential measure U and waiting time distribution R. If U(Z + ) <, then N(Z + ) is a geometric random variable with mean U(Z + ). If U(Z + ) =, then N(Z + ) = a.s. In either case, U(Z + ) = R{ }. Definition. A renewal sequence and the corresponding regenerative set are transient if the corresponding renewal measure is finite almost surely and they are recurrent otherwise. Theorem. (Strong Law for Renewal Sequences) Given a renewal sequence X, let N denonte the renewal measure, U, the potential measure and µ [, ] denote the mean waiting time. Then, lim n U[0, n] = lim n n n N[0, n] = µ a.s. Proof. Because N[0, n] n, the equality of the first and third terms follows from the equality of the second and third terms and the bounded convergence theorem. To establish this equality, note that by the strong law of large numbers, T m m = m m S j = µ j= a.s.. If T m n < T m, then m T m n N[0, n] = m n and the equality follows by taking limits as n. m T m = m m m T m Definition. Call a recurrent renewal sequence positive recurrent is the mean waiting time is finite and null recurrent if the mean waiting time is infinite. Definition. Let X be a random sequence X n {0, } for all n and let τ = min{n 0 : X n = }. Then X is a delayed renewal seqence if either P {τ = } = or, conditioned on the event {τ < }, is a renewal sequence. τ is called the delay time. {X τ+n : n 0} Exercise. Let S be a random walk on Z and let X n = I {x} (S n ). Then X is a delayed renewal sequence. 25

26 Exercise. Let X be a renewal sequence. Then a {0, }-valued sequence, X, is a delayed renewal sequence with the same waiting time distribution as X if and only if P { X n = ɛ n for 0 n r + s} = P { X n = ɛ n for 0 n r}p {X n r = ɛ n for r < n r + s} for all r, s N and (ɛ 0,..., ɛ r+s ) {0, } (r+s+) satisfying ɛ r =. Theorem. Let X and Y be independent delayed renewal sequences, both with waiting time distribution R. Define the F (X,Y ) -stopping time σ = min{k 0 : X k = Y k = }. Define { Xk if k σ, Z k = if k > σ Then X and Z have the same distribution. Y k Proof. By the Daniell-Kolmogorov extension theorem, it is enough to show that they have the same finite dimensional distributions. With this in mind, fix n and (ɛ 0,..., ɛ n ) {0, } (n+). Define σ = min{n, min{k, Y k = ɛ k = }}. Note that on the set {Z 0 = ɛ 0,..., Z n = ɛ n }, σ = min{n, σ}. Noting that X and (Y, σ) are independent, we have n P {Z 0 = ɛ 0..., Z n = ɛ n } = P { σ = k, Z 0 = ɛ 0,..., Z n = ɛ n } = = k=0 n P { σ = k, X 0 = ɛ 0..., X k = ɛ k, Y k+ = ɛ k+,... Y n = ɛ n } k=0 n P {X 0 = ɛ 0,..., X k = ɛ k }P { σ = k, Y k+ = ɛ k+,... Y n = ɛ n } k=0 Claim. Let Ỹ be a non-delayed renewal sequence with waiting distribution R. Then for k n P { σ = k, Y k+ = ɛ k+,... Y n = ɛ n } = P { σ = k}p {Ỹk+ = ɛ k+,... Ỹn = ɛ n }. If k = n, the statement is trivial because the statements concerning Y and Ỹ are impossible. For k < n and ɛ k = 0, then { σ = k} = and again the equation is trivial. For k < n and ɛ k =, then { σ = k} is the finite union of events {Y 0 = ɛ 0,..., Y k = } and the identity follows from the exercise above. Use this exercise once more to obtain, P {Z 0 = ɛ 0..., Z n = ɛ n } = = n k=0 ( ) P { σ = k}p {X 0 = ɛ 0,..., X k = ɛ k }P {Ỹk+ = ɛ k+,... Ỹn = ɛ n } n P { σ = k}p {X 0 = ɛ 0,..., X n = ɛ n } k=0 = P {X 0 = ɛ 0,..., X n = ɛ n }. 26

27 Exercise. Let W be an N valued random variable with mean µ, then P {W > n} = µ. n= More generally, set then k f(k) = a n, n= Ef(W ) = a n P {W > n}. n= Theorem. Let R be a waiting distribution with finite mean µ and let X be a delayed renewal sequence having waiting time distribution R and delay distribution Then P {X n = } = /µ. D{k} = µ R[k +, ), n Z+. Proof. By the exercise above, D is a distribution. Let T 0 be a random variable having this distribution. Let {u n : n } be the renewal sequence corresponding to R. Then P {X n = T 0 = k} = u n k provided k n and 0 otherwise. Hence, by the corollary. P {X n = } = n P {X n = T 0 = k}p {T 0 = k} = k=0 3.2 Renewal Theorem Definition. The period of a renewal sequence is A renewal process is aperiodic if its period is one. n k=0 gcd{n > 0 : u n > 0}, gcd( ) =. u n k µ R[k +, ) = µ Note that if B is a set of positive integers, gcd(b) =, then there exists K such that every integer k > K can be written as a positive linear combination of elements in B. Theorem. (Renewal) Let µ [, ] and let X be an aperiodic renewal sequence, then lim P {X k = } = k µ. 27

28 Proof. (transient case) Recall that u k = P {X k = }. If u k <, then lim k u k = 0 and by the strong law for renewal sequences, µ =. k=0 (positive recurrent) Let Y be a delayed renewal sequence, independent of X, having the same waiting time distribution as X and satisfying Define the stopping time and the process P {Y n = } = µ. σ = min{n 0 : X n = Y n = }. { Xn if n σ, Z n = if n > σ Then X and Z have the same distribution. If P {σ < } =, then P {X n = } µ = P {Z n = } P {Y n = } Y n = (P {Z n =, σ n} P {Y n =, σ n}) + (P {Z n =, σ > n} P {Y n =, σ > n}) = P {Z n =, σ > n} P {Y n =, σ > n} 2P {σ > n} 0 as n and the theorem follows in the positive recurrent case. With this in mind, define the F Y t -stopping time τ = min{k : Y k = and u k > 0}. Because u k > 0 for sufficiently large k, and Y has a finite delay and finite waiting distribution, τ is finite with probability. Note that X, τ and {Y τ+n : n = 0,,...} are independent. Consequently {X n Y τ+n : n = 0,,...} is a renewal sequence independent of τ. Note that n= u2 n =. Otherwise, lim n u n = 0 contradicting the fact that lim n n n= u n = /µ. Thus, the renewal process is recurrent. i.e., P {X n = Y τ+n = i.o} =. Let {σ k : k } denote the times that this holds. Because this sequence is strictly increasing, σ 2kj + k < σ 2k(j+). Consequently, the random variables {X σ2kj +k : j } are independent and take the value one with probability u k. Recall that τ is independent of X and so {X σ2τj+τ : j } are conditionally independent given τ and take the value one with probability u τ > 0. 28

29 By the conditional Borel-Cantelli lemma, P {X σ2τj+τ = i.o} = E[P {X σ2τj+τ = i.o. τ}] = Each one of these occurences is an example of σ <. (null recurrent) By way of contradiction, choose ɛ > 0, so that Now pick an integer q so that lim inf n u n > ɛ. q R[n, ) > 2 ɛ. n= Introduce the independent delayed renewal processes Y 0,..., Y q with waiting distribution R and fixed delay r for Y r. Thus X and Y 0 have the same distribution. Guided by the proof of the positive recurrent case, define the stopping time set σ = min{n : Y r n =, for all r = 0,..., q}, Z r n = { Y r n if n σ X n if n > σ. and note that Z r and Y r have the same distribution. Because u n > ɛ infinitely often, n= uq+ n =. Modify the argument about to see that σ < with probability one. Consequenty, there exists an integer k > q so that Thus, P {σ < k} /2 and u k = P {X k = } > ɛ. u k r = P {Z r k = } P {Z r k =, σ k} = P {Z r k = σ k}p {σ k} = P {X k = }P {σ k} ɛ 2. Consequently q+ u k+ n R[n, ) >. n= One of the identities relating the potential sequence and the renewal measure gives this sum to at most one. This contradiction proves the null recurrent case. Putting this altogether, we see that a renewal sequence is recurrent if and only if u n = null recurrent if and only if it is recurrent and lim n u n = 0 Corollary. Let µ [, ] and let X be an renewal sequence with period γ, then lim P {X γk = } = γ k µ. Proof. The process Y defined by Y k = X γk is an aperiodic renewal process with renewal measure R Y (B) = R(γB) and mean µ/γ. 29

30 3.3 Applications to Random Walks We now apply this to random walks. For a real-valued random walk S, we set the notation: strictly ascending weakly descending renewal process ladder times ladder times potential sequence u ++ n waiting time distribution R ++ u n R waiting time generating function G ++ R G R and Lemma. Let n 0. u ++ n = R [n +, ] Proof. Let {X k : k } be the steps in S. Then {n is a strict ascending ladder time for S} = { n X k > 0 for m =, 2,..., n}. m {, 2,..., n is not a weak descending ladder time for S} = { X k > 0 for m =, 2,..., n}. By replacing X k with X n+ k, we see that these two events have the same probability. k=m k= Theorem. ( G ++ R (z)) ( G R (z)) = ( z). Proof. Then the lemma above states that u ++ n = n R {k}. k= Recall that ( G ++ R (z)) is the generating function of the sequence {u ++ n : n }, we have for z [0, ), G ++ R (z) = = ( n k= R {k})z n = z k= n=k z R {k} zn z = G R (z). z The case z = now follows from the monotone convergence theorem. k= R {k}z n Exercise. Either the set of strict ascending ladder times and the set of weak descending ladder times are both null recurrent or one of these sets is transient and the other is positive recurrent. If the step distribution in a random walk assigns probability to each one-point set, the weak and strict ladder times are almost surely equal. In this case, we have, with the obvious notational addition, ( G ++ R (z)) ( G R (z)) = ( z). 30

31 If the step distribution is symmetric about zero, then G ++ R (z) = G (z) = z and a simple Taylor s series expansion gives the waiting time distribution of ladder times. R 3

32 4 Martingales Definition. A real-valued sequence X with E X n < for all n 0 and adapted to a filtration {F n : n 0} is an F n -martingale if E[X n+ F n ] = X n for n = 0,,..., an F n -submartingale if E[X n+ F n ] X n for n = 0,,..., and an F n -supermartingale if the inequality above is reversed. Thus, X is a supermartingale if and only if X is a submartingale and X is a martingale if and only if X and X are submartingales. If X is an F X n -martingale, then we simply say that X is a martingale. Exercise.. Show that X is an F n -martingale if and only if for every j < k, The case A = Ω shows that EX k is constant. 2. Give a similar characterization for submartingales. E[X k ; A] = E[X j ; A], for all A F j. 3. Show that if X is an F n -martingale, then X is a martingale. Proposition.. Suppose X is an F n -martingale, φ is convex and E φ(x n ) < for all n. Then φ X is an F n -submartingale. 2. Suppose X is an F n -submartingale, φ is convex and non-decreasing, E φ(x n ) < for all n. Then φ X is an F n -submartingale. Proof. By Jensen s inequality, E[φ(X n+ ) F n ] φ(e[x n+ F n ]) φ(x n ). For part, the last inequality is an equality. for part 2, the last inequality follows from the assumption that φ is nondecreasing. 4. Examples Examples.. Let S be a random walk whose step sizes have mean µ. Then S is submartingale, martingale, or supermartingale according to the property µ > 0, µ = 0, and µ < 0. To see this, note that and E S n n E X k = ne X k= E[S n+ F X n ] = E[X n+ + S n F X n ] = E[X n+ F X n ] + E[S n F X n ] = µ + S n. 32

33 2. In any situation, {S n nµ : n 0} is a martingale. 3. Assume µ = 0 and that Var(X ) = σ 2 let Y n = S 2 n nσ 2. Then E[Y n+ F X n ] = E[(X n+ S n ) 2 F X n ] (n + )σ 2 = E[X 2 n+ 2X n+ S n + S 2 n F X n ] (n + )σ 2 = E[X 2 n+ F X n ] 2E[X n+ F X n ]S n + S 2 n (n + )σ 2 = E[X 2 n+] 2 0 S n + S 2 n (n + )σ 2 = σ 2 + S 2 n (n + )σ 2 = Y n 4. (Wald s martingale) Let φ(t) = E[exp(itX )] be the characteristic function for the step distribution. Fix t R and define Y n = exp(its n )/φ(t) n. Then E[Y n+ F X n ] = E[exp(it(S n + X n+ ) F X n ]/φ(s) n+ = exp(iss n )E[exp(itX n+ ) F X n ]/φ(t) n+ = exp(its n )/φ(t) n = Y n. A similar martingale can be constructed using the Laplace transform L(α) = E[exp( αx )]. 5. (Likelihood ratios) Let X 0, X,... be independent and let f 0 and f be probability density functions with respect to a σ-finite measure µ on some state space S. A stochastic process of fundamental importance in the theory of statistical inference is the sequence of likelihood ratios L n = f (X 0 )f (X ) f (X n ), n = 0,,.... f 0 (X 0 )f 0 (X ) f 0 (X n ) To assure that L n is defined assume that f 0 (x) > 0 for all x S. Then E[L n+ F X n ] = E[L n f (X n+ ) f 0 (X n+ ) F X n ] = L n E[ f (X n+ ) f 0 (X n+ ) ]. If f 0 is the probability density function for X k, then E[ f (X n+ ) f 0 (X n+ ) ] = f (x) f 0 (x) f 0(x) µ(dx) = and L is a martingale. S S f (x) µ(dx) =, 6. Let {X n ; n 0} be independent mean random variables. Then Z n = n k= X k is an F X n -martingale. 7. (Doob s martingale) Let X be a random variable, E X < and define X n = E[X F n ], then by the tower property, E[X n+ F n ] = E[E[X F n+ ] F n ] = E[X F n ] = X n. 33

34 8. (Polya urn) An urn has initial number of balls, colored green and blue with at least one of each color. Draw a ball at random from the urn and return it and c others of the same color. Repeat this procedure, letting X n be the number of blue and Y n be the number of green after n iterations. Let V n be the fraction of blue balls. X n V n =. X n + Y n Then E[V n+ F (X,Y ) n ] = = X n + c X n X n Y n + X n + Y n + c X n + Y n X n + Y n + c X n + Y n (X n + c)x n + X n Y n (X n + Y n + c)(x n + Y n ) = V n 9. (martingale transform) For a filtration {F n : n 0}, Call a random sequence H predictable if H n is F n -measurable. Define (H X) 0 = 0 and (H X) n = n H k (X k X k ). k= Assume that H n is non-negative and bounded. Then E[(H X) n+ F n ] = (H X) n + E[H n+ (X n+ X n ) F n ] = (H X) n + H n+ E[(X n+ X n ) F n ]. Thus, (H X) is a martingale, submartingale, or supermartingale according to what property X has. Exercises.. Let f : [0, ) R be measurable with 0 f(x) dx <. Let F n = σ{[(k )/2 n, k2 n ), k =, 2,..., 2 n }. Let Z n = E[f F n ]. Describe Z n. 2. Let τ be an F n -stopping time and define H n = I {n τ}. Show that H is predictable and describe (H X) n. 4.2 Doob Decomposition Theorem. (Doob Decomposition) Let X be an F n -submartingale. There exists unique random sequences M and V such that. X n = M n + V n for all n M is an F n -martingale = V 0 V V is F n -predictable. 34

35 Proof. Define U k = E[X k X k F k ] and V n = n U k. k= then 3 follows from the submartingale property of X and 4 follows from the basic properties of conditional expectation. Define M so that is satisfied, then E[M n M n F n ] = E[X n X n F n ] E[V n V n F n ] = E[X n X n F n ] E[U n F n ] = E[X n X n F n ] U n = 0. Thus, there exists processes M and V that satisfy -4. To show uniqueness, choose at second pair M and Ṽ. Then M n M n = Ṽn V n. is F n -measurable. Therefore, M n M n = E[M n M n F n ] = M n M n by the martingale property for M and M. Thus, the difference between M n and M n is constant, independent of n. This difference M 0 M 0 = Ṽ0 V 0 = 0 by property 3. Theorem. Each of the components in the Doob decomposition of a uniformly integrable submartingale is uniformly integrable. Proof. Use the Doob decomposition to write X = M + V, where M is an F n -martingale and V is an increasing F n -predictable process. Call Then, by the monotone convergence theorem, V = lim n V n. EV = lim EV n = lim E[X n M n ] = lim E[X n M 0 ] sup E X n + E M 0 < n n n by the uniform integrability of X. Because the sequence V n is dominated by an integrable random variable V, it is uniformly integrable. M = X V is uniformly integrable because the difference of uniformly integrable sequences is uniformly integrable. n 35

36 4.3 Optional Sampling Theorem Definition. Let X be a random sequence of real valued random variables with finite absolute mean. A N { }-valued random variable τ that is a.s. finite satisfies the sampling integrability conditions for X if the following conditions hold:. E X τ < ; 2. lim inf n E[ X n ; {τ > n}] = 0. Theorem. (Optional Sampling) Let τ 0 τ be an increasing sequence of F n -stopping times and let X be an F n -submartingale. Assume that for each n, τ n is an almost surely finite random variable that satisfies the sampling integrability conditions for X. Then {X τk : k 0} is an F τk -submartingale. Proof. {X τn : n 0} is F τn -adapted and each random variable in the sequence has finite absolute mean. The theorem thus requires that we show that for every A F τn With this in mind, set E[X τn+ ; A] E[X τn ; A]. B m = A {τ n = m}. These sets are pairwise disjoint and because τ n is almost surely finite, their union is almost surely A. Thus, the theorem will follow from the dominated convergence theorem if we show Choose p > m. Then, E[X τn+ ; B m ] = E[X m ; B m ] + Note that for each l, E[X τn+ ; B m ] E[X τn+ ; B m ] = E[X m ; B m ]. p l=m E[(X l+ X l ); B m {τ n+ > l}] + E[(X τn+ X p ); B m {τ n+ > p}]. {τ n+ > l} = {τ n+ l} c F l, and B m F m F l. Consequently, by the submartingale property for X, each of the terms in the sum is non-negative. To check that the last term can be made arbitrarily close to zero, Note that {τ n+ > p} a.s.as p and X τn+ I Bm {τ n+>p} X τn+, which is integrable by the first property of the sampling integability condition. convergence theorem, lim E[ X τ n+ ; B m {τ n+ > p}] = 0. p Thus by the dominated For the remaining term, by property 2 of the sampling integribility condition, we are assured of a subsequence {p(k) : k 0} so that lim k E[ X p(k) ; {τ n+ > p(k)}] = 0. 36

37 A reveiw of the proof shows that we can reverse the inequality in the case of supermartingales and replace it with an equality in the case of martingales. The most frequent use of this theorem is: Corollary. Let τ be a F n -stopping time and let X be an F n -(sub)martingale. Assume that τ is an almost surely finite random variable that satisfies the sampling integrability conditions for X. Then E[X τ ] = ( )E[X 0 ]. Example. Let X be a random sequence satisfying X 0 = and P {X n+ = x Fn X 2 if x = 0, } = 2 if x = 2X n, 0 otherwise. Thus, after each bet the fortune either doubles or vanishes. Let τ be the time that the fortune vanishes, i.e. τ = min{n > 0 : X n = 0}. Note that P {τ > k} = 2 k. However, EX τ = 0 < = EX 0 and τ cannot satisfy the sampling integrability conditions. Because the sampling integrability conditions are a technical relationship between X and τ, we look to find easy to verify sufficient conditions. Proposition. If τ is bounded, then τ satifies the sampling integrability conditions for any sequence X of integrable random variables. Proof. Let b be a bound for τ. Then. X τ = which is integrable and property follows. b X k I {τ=k } k=0 2. For n > b, {τ > n} = and property 2 follows. b X b, Exercise. Show that if 0 X 0 X, then implies 2 in the sampling integrability conditions. Proposition. Let X be an F n -adapted sequence of real-valued random variables and let τ be an almost surely finite F n -stopping time. Suppose for each n > 0, there exists m n so that k=0 E[ X k X k F k ] m k a.s. on the set {τ k}. Set f(k) = n k= m k. If Ef(τ) <, then τ satisfies the sampling integrability conditions for X. So, large values for the conditional mean of the absolute value of the step size forces the stopping time to have more stringent integrability conditions in order to satisfy the sufficient condition above. 37