17. Convergence of Random Variables

Transcription

1 7. Convergence of Random Variables In elementary mathematics courses (such as Calculus) one speaks of the convergence of functions: f n : R R, then lim f n = f if lim f n (x) = f(x) for all x in R. This is called pointwise convergence of functions. A random variable is of course a function (X: Ω R for an abstract space Ω), and thus we have the same notion: a sequence X n : Ω R converges pointwise to X if lim X n (ω) =X(ω), for all ω Ω. This natural definition is surprisingly useless in probability. The next example gives an indication why. Example : Let X n be an i.i.d. sequence of random variables with (X n = ) = p and (X n =0)= p. For example we can imagine tossing a slightly unbalanced coin (so that p> 2 ) repeatedly, and {X n =} corresponds to heads on the n th toss and {X n =0} corresponds to tails on the n th toss. In the long run, we would expect the proportion of heads to be p; this would justify our model that claims the probability of heads is p. Mathematically we would want X (ω)+...+ X n (ω) lim = p for all ω Ω. n This simply does not happen! For example let ω 0 = {T,T,T,...}, the sequence of all tails. For this ω 0, lim n More generally we have the event Then n X j (ω 0 )=0. j= A = {ω : only a finite number of heads occur}. lim n n X j (ω) = 0 for all ω A. j= We readily admit that the event A is very unlikely to occur. Indeed, we can show (Exercise 4) that (A) = 0. In fact, what we will eventually show (see the Strong Law of Large Numbers [Chapter 20]) is that

2 38 7. Convergence of Random Variables ω : lim n n X j (ω) =p =. j= This type of convergence of random variables, where we do not have convergence for all ω but do have convergence for almost all ω (i.e., the set of ω where we do have convergence has probability one), is what typically arises. Caveat: In this chapter we will assume that all random variables are defined on a given, fixed probability space (Ω,A,) and takes values in R or R n. We also denote by x the Euclidean norm of x R n. Definition 7.. We say that a sequence of random variables (X n ) n converges almost surely to a random variable X if { } N = ω : lim X n(ω) X(ω) has (N) =0. Recall that the set N is called a null set. Note that N c = Λ = { } ω : lim X n(ω) =X(ω) and then (Λ) =. We usually abbreviate almost sure convergence by writing lim X n = X a.s. We have given an example of almost sure convergence from coin tossing preceding this definition. Just as we defined almost sure convergence because it naturally occurs when pointwise convergence (for all points ) fails, we need to introduce two more types of convergence. These next two types of convergence also arise naturally when a.s. convergence fails, and they are also useful as tools to help to show that a.s. convergence holds. Definition 7.2. A sequence of random variables (X n ) n converges in L p to X( p< ) if X n, X are in L p and: lim E{ X n X p } =0. Alternatively one says X n converges to X in p th mean, and one writes X n L p X. The most important cases for convergence in p th mean are when p = and when p = 2. When p = and all r.v. s are one-dimensional, we have

3 7. Convergence of Random Variables 39 E{X n X} E{ X n X } and E{ X n } E{ X } E{ X n X } because x y x y. Hence X n L X implies E{X n } E{X} and E{ X n } E{ X }. (7.) Similarly, when X n L p X for p (, ), we have that E{ X n p } converges to E{ X p }: see Exercise 5 for the case p =2. Definition 7.3. A sequence of random variables (X n ) n probability to X if for any ε>0 we have converges in lim ({ω : X n(ω) X(ω) >ε}) =0. This is also written and denoted lim ( X n X >ε)=0, X n X. Using the epsilon-delta definition of a limit, one could alternatively say that X n tends to X in probability if for any ε>0, any δ>0, there exists N = N(δ) such that ( X n X >ε) <δ for all n N. Before we establish the relationships between the different types of convergence, we give a surprisingly useful small result which characterizes convergence in probability. Theorem 7.. X n X if and only if lim E { Xn X + X n X } =0. roof. There is no loss of generality by taking X =0.Thuswewanttoshow X n 0 if and only if lim E{ Xn + X } = 0. First suppose that X n n 0. Then for any ε>0, lim ( X n >ε) = 0. Note that X n + X n X n + X n { X n >ε} + ε { Xn ε} { Xn >ε} + ε. Therefore { } Xn E E { { Xn >ε}} + ε = ( Xn >ε)+ε. + X n Taking limits yields

4 40 7. Convergence of Random Variables { } lim E Xn ε; + X n since ε was arbitrary we have lim E{ Xn + X } =0. n Next suppose lim E{ Xn + X n } = 0. The function f(x) = x derivative (+x) > 0 and is increasing. Therefore 2 +x has ε +ε { X n >ε} X n + X n { X n >ε} X n + X n. Taking expectations and then limits yields ε +ε lim ( X n >ε) lim E { } Xn =0. + X n Since ε>0 is fixed, we conclude lim ( X n >ε)=0. Remark: What this theorem says is that X n X iff E{f( Xn X )} 0 for the function f(x) = x] + x. A careful examination of the proof shows that the same equivalence holds for any function f on R + which is bounded, strictly increasing on (0, ), continuous, and with f(0) = 0. For example we have X n X iff E{ Xn X } 0and also iff E{arctan( X n X )} 0. The next theorem shows that convergence in probability is the weakest of the three types of convergence (a.s., L p, and probability). Theorem 7.2. Let (X n ) n be a sequence of random variables. a) If X n L p X, then X n X. b) If X n a.s. X, then X n X. roof. (a) Recall that for an event A, (A) =E{ A }, where A is the indicator function of the event A. Therefore, { X n X >ε} = E { { Xn X >ε}}. Note that Xn X p ε p > on the event { X n X >ε}, hence { Xn X p } E ε p { Xn X >ε} = ε p E { X n X p { Xn X >ε}}, and since X n X p 0 always, we can simply drop the indicator function to get:

5 7. Convergence of Random Variables 4 ε p E{ X n X p }. The last expression tends to 0 as n tends to (for fixed ε>0), which gives the result. X (b) Since n X + X n X always, we have { } lim E Xn X + X n X { = E lim } X n X = E{0} =0 + X n X by Lebegue s Dominated Convergence Theorem (9.(f)). We then apply Theorem 7.. The converse to Theorem 7.2 is not true; nevertheless we have two partial converses. The most delicate one concerns the relation with a.s. convergence, and goes as follows: Theorem 7.3. Suppose X n X. Then there exists a subsequence nk such that lim k X nk = X almost surely. roof. Since X n we have that lim E{ Xn X + X n X } = 0 by Theorem 7.. Choose a subsequence n k such that E{ Xn k X + X nk X } <. Then 2 k k= E{ Xn k X } < and by Theorem 9.2 we have that k= + X nk X X nk X + X nk X < a.s.; since the general term of a convergent series must tend to zero, we conclude lim X k X = 0 a.s. n n Remark 7.. Theorem 7.3 can also be proved fairly simply using the Borel Cantelli Lemma (Theorem 0.5). Example 2: X n X does not necessarily imply that Xn converges to X almost surely. For example take Ω =[0, ], A the Borel sets on [0, ], and the uniform probability measure on [0, ]. (That is, is just Lebesgue measure restricted to the interval [0, ].) Let A n be any interval in [0, ] of length a n, and take X n = An. Then ( X n >ε)=a n, and as soon as a n 0 we deduce that X n 0 (that is, Xn tends to 0 in probability). More precisely, let X n,j be the indicator of the interval [ j n, j n ], j n, n. We can make one sequence of the X n,j by ordering them first by increasing n, and then for each fixed n by increasing j. Call the new sequence Y m.thus the sequence would be: X,,X 2,,X 2,2,X 3,,X 3,2,X 3,3,X 4,,... Y,Y 2,Y 3,Y 4,Y 5,Y 6,Y 7,...

6 42 7. Convergence of Random Variables Note that for each ω and every n, there exists a j such that X n,j (ω) =. Therefore lim sup m Y m = a.s., while lim inf m Y m = 0 a.s. Clearly then the sequence Y m does not converge a.s. However Y n is the indicator of an interval whose length a n goes to 0 as n, so the sequence Y n does converge to 0 in probability. The second partial converse of Theorem 7.2 is as follows: Theorem 7.4. Suppose X n X and also that Xn Y,alln, and Y L p. Then X is in L p L and X p n X. roof. Since E{ X n p } E{Y p } <, wehavex n L p.forε>0wehave { X n >Y + ε} { X > X n + ε} { X X n >ε} { X X n >ε}, hence ( X >Y + ε) ( X X n >ε), and since this is true for each n, wehave ( X >Y + ε) lim ( X X n >ε)=0, by hypothesis. This is true for each ε>0, hence ( X >Y) lim m ( X >Y + m )=0, from which we get X Y a.s. Therefore X L p too. Suppose now that X n does not converge to X in L p. There is a subsequence (n k ) such that E{ X n X p } ε for all k, and for some ε>0. The subsequence X nk trivially converges to X in probability, so by Theorem 7.3 it admits a further subsequence X nkj which converges a.s. to X. Now,the r.v. s X nkj X tend a.s. to 0 as j, while staying smaller than 2Y,soby Lebesgue s Dominated Convergence we get that E{ X nkj X p } 0, which contradicts the property that E{ X k X p } ε for all k: hence we are done. The next theorem is elementary but also quite useful to keep in mind. Theorem 7.5. Let f be a continuous function. a) If lim X n = X a.s., then lim f(x n )=f(x) a.s. b) If X n X, then f(xn ) f(x). roof. (a) Let N = {ω : lim X n (ω) X(ω)}. Then (N) = 0 by hypothesis. If ω N, then

7 7. Convergence of Random Variables 43 ( ) lim f(x n(ω)) = f lim X n(ω) = f(x(ω)), where the first equality is by the continuity of f. Since this is true for any ω N, and (N) = 0, we have the almost sure convergence. (b) For each k>0, let us set: { f(x n ) f(x) >ε} { f(x n ) f(x) >ε, X k} { X >k}. (7.2) Since f is continuous, it is uniformly continuous on any bounded interval. Therefore for our ε given, there exists a δ>0 such that f(x) f(y) ε if x y δ for x and y in [ k, k]. This means that { f(x n ) f(x) >ε, X k} { X n X >δ, X k} { X n X >δ}. Combining this with (7.2) gives { f(x n ) f(x) >ε} { X n X >δ} { X >k}. (7.3) Using simple subadditivity ( (A B) (A)+ (B)) we obtain from (7.3): { f(x n ) f(x) >ε} ( X n X >δ)+ ( X >k). However { X > k} tends to the empty set as k increases to so lim k ( X >k) = 0. Therefore for γ > 0 we choose k so large that ( X >k) <γ. Once k is fixed, we obtain the δ of (7.3), and therefore lim ( f(x n) f(x) >ε) lim ( X n X >δ)+γ = γ. Since γ>0 was arbitrary, we deduce the result.

8 44 7. Convergence of Random Variables Exercises for Chapter 7 7. Let X n,j be as given in Example. Let Z n,j = n p Xn,j. Let Y m be the sequence obtained by ordering the Z n,j as was done in Example 2. Show that Y m tends to 0 in probability but that (Y m ) m does not tend to 0 in L p, although each Y n belongs to L p. 7.2 Show that Theorem 7.5(b) is false in general if f is not assumed to be continuous. (Hint: Take f(x) = {0} (x) and the X n s tending to 0 in probability.) 7.3 Let X n be i.i.d. random variables with (X n =)= 2 and (X n = ) = 2. Show that n X j n j= converges to 0 in probability. (Hint: Let S n = n j= X j, and use Chebyshev s inequality on { S n >nε}.) 7.4 Let X n and S n be as in Exercise 7.3. Show that n S 2 n 2 converges to zero a.s. (Hint: Show that n= { n S 2 n 2 >ε} < and use the Borel- Cantelli Lemma.) *7.5 Suppose X n Y a.s., each n, n =, 2, 3...Ṡhow that sup n X n Y a.s. also. 7.6 Let X n X. Show that ϕxn converges pointwise to ϕ X (where ϕ X (u) = E{e iux }, the characteristic function of X). (Hint: Use Theorem 7.4.) 7.7 Let X,...,X n be i.i.d. Cauchy random variables with parameters α = 0 and β =. (That is, their density is f(x) = π(+x 2 ), <x<.) Show that n n j= X j also has a Cauchy distribution. (Hint: Use Characteristic functions: See Exercise 3..) 7.8 Let X,...,X n be i.i.d. Cauchy random variables with parameters α = 0 and β =. Show that there is no constant γ such that n n j= X j γ. (Hint: Use Exercise 7.7.) Deduce that there is no constant γ such that n lim n j= X j = γ a.s. as well. 7.9 Let (X n ) n have finite variances and zero means (i.e., Var(X n ) = σx 2 n < and E{X n } = 0, all n). Suppose lim σx 2 n = 0. Show X n converges to 0 in L 2 and in probability. 7.0 Let X j be i.i.d. with finite variances and zero means. Let S n = n j= X j. Show that n S n tends to 0 in both L 2 and in probability. *7. Suppose lim X n = X a.s. and X< a.s. Let Y = sup n X n. Show that Y< a.s.

9 Exercises 45 *7.2 Suppose lim X n = X a.s. Let Y = sup n X n X. Show Y< a.s. (see Exercise ), and define a new probability measure Q by Q(A) = { } { } c E A, where c = E. +Y +Y Show that X n tends to X in L under the probability measure Q. 7.3 Let A be the event described in Example. Show that (A) =0. (Hint: Let A n = { Heads on n th toss }. Show that n= (A n)= and use the Borel-Cantelli Theorem (Theorem 0.5.) ) 7.4 Let X n and X be real-valued r.v. s in L 2, and suppose that X n tends to X in L 2. Show that E{Xn} 2 tends to E{X 2 } (Hint: use that x 2 y 2 = (x y) 2 +2 y x y and the Cauchy-Schwarz inequality). *7.5 (Another Dominated Convergence Theorem.) Let (X n ) n be random variables with X n X (lim X n = X in probability). Suppose X n (ω) C for a constant C>0and all ω. Show that lim E{ X n X } =0.(Hint: First show that ( X C) =.)

10