7.1 Convergence of sequences of random variables

Transcription

1 Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite sequece of distributios it is possible to costruct a probability space with a correspodig sequece of radom variables is a otrivial fact, whose proof is due to Kolmogorov (see for example Billigsley). 7.1 Covergece of sequeces of radom variables For every poit!, (X (!)) is a umber sequece ad X(!) is a umber. It might be that! is such that the sequece (X (!)) coverges to X(!), but it might also be that this sequece does ot coverge at all, or that it does ot coverge to X(!). The set! lim X (!) = X(!) is a evet; as such, it has a probability, which, i priciple, could be either zero, oe or ay itermediate umber. The followig defiitio provides a termiology to oe of those cases: Defiitio 7.1 The sequece of radom variable (X ) is said to coverge to the radom variable X almost-surely ($*/; )3/,) (or, w.p. 1) if We write X a.s X. P! lim X (!) = X(!) = 1.

2 164 Chapter 7 We ca write this mode of covergece i more explicit form. (X (!)) exists ad equals X(!) if ad oly if The limit of > 0, N N, > N, X (!) X(!) <. Note that we ca replace this coditio by the equivalet coditio k N, N N, > N, X (!) X(!) 1 k. Equivaletly, X(!) is ot the limit of (X (!)) if ad oly if k N, N N, > N, X (!) X(!) > 1 k. It follows that the coditio X a.s X ca be reformulated as P! k N, N N, > N, X (!) X(!) > 1 = 0, k or equivaletly, N=1 =N+1 P This ca further be writte as {! X (!) X(!) > 1k} = 0. P lim sup {! X (!) X(!) > 1k} = 0. Note that the sequece of evets {! X (!) X(!) > 1k} is icreasig as a fuctio of k, hece also their lim-sups. Thus, this equality is equal to lim k P lim sup {! X (!) X(!) > 1k} = 0. But if the limit of a icreasig o-egative sequece teds to zero, it must be a.s that the sequece is idetically zero, amely X X if ad oly if P lim sup {! X (!) X(!) > 1k} = 0 k N. a.s I words, X X if for every k, the probability that X (!) deviates from X(!) by more tha 1k for ifiitely may s is zero.

3 Limit Theorems 165 Just like sequeces of fuctios ca coverge to a limitig fuctios i more tha oe way (e.g., poitwise versus uiformly), so a sequece of radom variables ca coverge to a limitig radom variable i may di eret ways. Sayig that X coverges to X is like sayig that the sequece of radom variables (X X) coverges to the (costat) radom variable zero. The zero radom variable has the property that its secod momet is zero. This leads us to the followig defiitio: Defiitio 7.2 The sequece (X ) is said to coverge to X i the mea-square (;-(&;") if lim E X X 2 = 0. We write X m.s X. I words: the sequece (X X) coverges to zero i the mea-square if its secod momets coverge to zero. A third mode of covergece higes of the fact that we would relate the fact that (X ) coverges to X with the fact that for every > 0, the probability that X X > should teds to zero as : Defiitio 7.3 The sequece of radom variables (X ) is said to coverge to the radom variable X i probability (;&9";2%") if for every > 0, We write X Pr X. lim P({! X (!) X(!) > }) = 0. You might thik that this coicides with the mode of covergece we have already defied almost-sure covergece. We will see that this is ot the case. Covergece i probability di ers substatially from almost-sure covergece. Also, here too, we might replace > 0 by 1k for k N. Fially, we might say that the sequece (X ) coverges to X is the distributio of X coverges to the distributio of X. I this case, we do t eve eed all the radom variables to be defied o the same probability space; each variable could, i priciple, belog to a separate world. Defiitio 7.4 The sequece of radom variables (X ) is said to coverge to the radom variable X i distributio (;&#-5;%") if for every cotiuity poit a R of

4 166 Chapter 7 F X, lim F X (a) = F X (a), i.e., if the sequece of distributio fuctios of the X coverges poit-wise to the D distributio fuctio of X at all poits where F X is cotiuous. We write X X. The first questio to be addressed is whether there exists a hierarchy of modes of covergece. We wat to kow which modes of covergece imply which. The aswer is that both almost-sure ad mea-square covergece imply covergece i probability, which i tur implies covergece i distributio. O the other had, almost-sure ad mea-square covergece do ot imply each other. Propositio 7.1 Almost-sure covergece implies covergece i probability. Proof : If X a.s X, the By the Fatou lemma, hece P lim sup {! X (!) X(!) > } = 0 > 0. lim sup P (A ) P lim sup A, lim sup P ({! X (!) X(!) > }) 0 > 0. Sice the sequece P ({! X (!) X(!) > }) is o-egative, it follows as oce that lim P ({! X (!) X(!) > }) = 0 > 0, i.e., X Pr X. Propositio 7.2 Mea-square covergece implies covergece i probability.

5 Limit Theorems 167 m.s Proof : This is a immediate cosequece of the Markov iequality. Let X X, the for every > 0, P({! X (!) X(!) > }) = P({! X (!) X(!) 2 > 2 }) EX X 2 2, i.e., which implies that X lim P({! X (!) X(!) > }) = 0, Pr X. Propositio 7.3 Mea-square covergece does ot imply almost-sure covergece. Proof : All we eed is a couter example. Cosider a family of idepedet Beroulli variables X with atomistic distributios, 1 x= 1 p X (x) = 1 1 x= 0. The larger, the more it is likely that X(!) = 0. Thus, it seems sesible to guess that the sequece of radom variables (X ) coverges to the (costat) radom variable X = 0. The questio is i what sese does this covergece occur. m.s First, we show that X X. Ideed, E[X X 2 ] = E[X 2 ] = 1 0. O the other had, (X ) does ot coverge to X almost-surely. Sice for = 12, =1 P({! X (!) X(w) > 12}) = it follows from the secod lemma of Borel-Catelli that 1 =1 =, P lim sup({! X (!) X(w) > 12} = 1.

6 168 Chapter 7 Propositio 7.4 Almost-sure covergece does ot imply mea-square covergece. Proof : Agai, we costruct a couter example, this time takig 1 p X (x) = 2 x = x = 0., Oce agai, the larger, the more it is likely that X(!) = 0. We immediately see that X does ot coverge to X i the mea-square, sice Yet, X a.s X. For every > 0, =1 EX X 2 = E[X 2 ] = 6 2. P({! X (!) X(!) > }) = hece by the first lemma of Borel-Catelli, =1 1 2 <, P lim sup({! X (!) X(!) > } = 0. Corollary 7.1 Covergece i probability does ot imply either almost-sure covergece ot covergece i the mea-square. Proof : Suppose, for example, that covergece i probability implies almostsure covergece. This would mea that covergece i the mea-square implies almost-sure covergece, which cotradicts the last propositio. Fially, we show that covergece i probability implies covergece i distributio, hece both almost-sure covergece ad covergece i the mea-square imply covergece i distributio.

7 Limit Theorems 169 Propositio 7.5 Covergece i probability implies covergece i distributio. Proof : Let a R be give, ad set > 0. O the oe had F X (a) = P (X a) + P (X a, X a + ) + P (X a, X > a + ) = P (X ax a + ) P (X a + ) + P (X a, X > a + ) P (X a + ) + P (X < X ) F X (a + ) + P (X X > ), where we have used the fact that if A B the P(A) P(B). By a similar argumet F X (a ) = P (X a,x a) + P (X a,x > a) Thus, we have obtaied that = P (X a X a) P (X a) + P (X a,x > a) P (X a) + P (X < X ) F X (a) + P (X X > ), F X (a ) P (X X > ) F X (a) F X (a + ) + P (X X > ). Takig ow we have F X (a ) lim if F X (a) lim sup F X (a) F X (a + ). Fially, sice this iequality holds for ay > 0 we coclude that provided that a is a cotiuity poit of F X, lim F X (a) = F X (a). To coclude, the various modes of covergece satisfy the followig scheme:

8 170 Chapter 7 almost surely i the mea square i probability i distributio. Exercise 7.1 Prove that if X coverges i distributio to a costat c, the X coverges i probability to c.. Exercise 7.2 Prove that if X coverges to X i probability the it has a subsequece that coverges to X almost-surely. 7.2 The weak law of large umbers Theorem 7.1 (Weak law of large umbers (.*-&$#%.*952/% -: :-(% 8&(%)) Let X be a sequece of idepedet idetically-distributed radom variables o a probability space (, F, P) ad let µ = E[X i ]. Defie the sequece of ruig averages, S = X 1 + +X. The, S coverges to µ i probability, i.e., for every > 0, lim P ({! S (!) µ > }) = 0. Commet: Take the particular case where X 1, X 2,... are i.i.d., A R ad The, S = 1 i=1 1 X i (!) A Y i = I Xi A = 0 X i (!) A. Y i = fractio of times X i (!) A for 1 i.

9 Limit Theorems 171 The weak law of large umbers states that the fractio of times the outcome is i a give set coverges i probability to E[Y 1 ], which is the probability of this set, P X1 (A), amely, for every > 0, lim P({! S (!) P X1 (A) > }) = 0. Proof : We will prove the weak law uder the additioal assumptio that the radom variables have fiite variace 2 = Var[X i ]. The, the weak law of large umbers is a immediate cosequece of the Chebyshev iequality: by the additivity of the expectatio ad the variace (for idepedet radom variables), The, E[S ] = µ ad Var[S ] = 2. P ({! S (!) µ > }) Var[S ] = 2 which teds to zero as. 2 2, Commet: the first proof is due to Jacob Beroulli (1713), who proved it for the particular case of biomial variables. 7.3 The strog law of large umbers Our ext limit theorem is the strog law of large umber, which states that the ruig average of a sequece of i.i.d. variables coverges to the mea almostsurely (thus stregtheig the weak law of large umbers, which oly provides covergece i probability). Theorem 7.2 (Strog law of large umbers (.*-&$#%.*952/% -: 8'(% 8&(%)) Let X be a sequece of idepedet, idetically distributed radom variables with fiite expectatio µ = E[X i ]. Defie the sequece of ruig averages, S = X 1 + +X. The, S coverges to µ almost-surely, i.e., for every > 0, P({! lim S (!) = µ}) = 1.

10 172 Chapter 7 Proof : We have to prove that for every > 0, P lim sup{! S (!) µ > } = 0, which by the lemma of first Borel-Catelli lemma holds if =1 P ({! S (!) µ > }) <. We will prove the theorem uder the additioal assumptio that the radom variables are bouded, i.e., there is M < such that X i M. Set Y k = X k µ 2M, The reverse relatio is X k = 2MY k + µ. Clearly E[Y k ] = 0, ad by the boudedess assumptio, Y k 1. Hece, we ca use Hoe dig s iequality ad get that for every ad a By our defiitio of Y k, P Y k a exp a2 2 {! S (!) µ > } =! 1 X k µ > =! 1 =! 2M! =! 2M (X k µ) > Y k > Y k! 2M Y k 2M! By Hoe dig s iequality (oce for (Y k ) ad oce for ( Y k )), Y k Y k 2M. which is ideed summable. P ({! S (!) µ > }) 2 exp 2 8M 2,

11 7.4 The cetral limit theorem Limit Theorems 173 Theorem 7.3 (Cetral limit theorem (*',9/% -&"#% )5:/)) Let (X ) be a sequece of i.i.d. radom variables with E[X i ] = 0 ad Var[X i ] = 1. The, the sequece of radom variables S = X 1 + +X coverges i distributio to a radom variable X N (0, 1). That is, for every a R, lim P (S 1 a a) = e y22 dy. 2 Commets: If E[X i ] = µ ad Var[X i ] = 2 the the same applies for S = X 1 + +X µ = 1 X i µ. i=1 The cetral limit theorem (CLT) is about a ruig average rescaled by a factor of. If we deote by Y the ruig average, the the CLT states that Y = X 1 + +X, P Y a (a), i.e., it provides a estimate of the distributio of Y at distaces O( 12 ) from its mea. It is a theorem about small deviatios from the mea. There exist more sophisticated theorems about the distributio of Y far from the mea, part of the so-called theory of large deviatios. There are may variats of this theorem.

12 174 Chapter 7 Proof : We will use the followig fact, which we wo t prove: if the sequece of momet geeratig fuctios M X (t) of a sequece of radom variables (X ) coverges for every t to the momet geeratig fuctio M X (t) of a radom variable X, the X coverges to X i distributio. I other words, M X (t) M X (t) for all t implies that X D X. Thus, we eed to show that the momet geeratig fuctios of the S s teds as to exp(t 2 2), which is the momet geeratig fuctio of a stadard ormal variable. Recall that the pdf of a sum of two radom variables is the covolutio of their pdf, but the momet geeratig fuctio of their sum is the product of the their momet geeratig fuctio. Iductively, M X1 +X ,+X (t) = i=1 M Xi (t) = [M X1 (t)], where we have used the fact that they are i.i.d., Now, if a radom variable Y has a momet geeratig fuctio M Y, the but sice f Ya (y) = af Y (ay) we get that M Ya (t) = R e ty f Ya (y) dy, M Ya (t) = a R e ty f Y (ay) dy = R e atya f Y (ay) d(ay) = M Y (ta), from which we deduce that M S (t) = M X1 t. Take the logarithm of both sides, ad write the left had side explicitly, log M S (t) = log R e tx f X1 (x) dx. Taylor expadig the expoetial about t = 0 we have, log M S (t) = log R 1 + tx + t2 x t3 x 3 6 = log t2 2 + O( 32 ) = t2 2 + O( 32 ) t2 2. e x f 32 X1 (x) dx

13 Limit Theorems 175 Example: Suppose that a experimetalist wats to measure some quatity. He kows that due to various sources of errors, the result of every sigle measuremet is a radom variable, whose mea µ is the correct aswer, ad the variace of his measuremet is 2. He therefore performs idepedet measuremets ad averages the results. How may such measuremets does he eed to perform to be sure, withi 95% certaity, that his estimate does ot deviate from the true result by 4? The questio we re askig is how large should be i order for the iequality P µ 4 1 X k µ to hold. This is equivalet to askig what should be for P 4 1 X k µ By the cetral limit theorem the right had side is, for large, approximately which turs out to be larger tha 0.95 for 62. e y2 2 dy, The problem with this argumet that it uses the assumptio that is large, but it is ot clear what large is. Is = 62 su cietly large for this argumet to hold? This problem could have bee solved without this di culty but resortig istead to the Chebyshev iequality: P 4 1 X k µ 4 = 1 P 1 X k µ , ad the right had side is larger tha 0.95 if = 320.

14 176 Chapter 7 Example: The umber of studets X who are goig to fail i the exam is a Poisso variable with mea 100, i.e, X Poi(100). I am goig to admit that the exam was too hard if more tha 120 studet fail. What is the probability for it to happe? We kow the exact aswer, P (X 120) = e k k!, which is a quite useless expressio. Let s base our estimate o the cetral limit theorem as follows: a Poisso variable with mea 100 ca be expressed as the sum of oe hudred idepedet variables X k Poi(1) (the sum of idepedet Poisso variables is agai a Poisso variable), that is X = 100 X k. Now, P (X 120) = P X k , which by the cetral limit theorem equals approximately, P (X 120) e y2 2 dy Example: Let us examie umerically a particular example. Let X i Exp (1) be idepedet expoetial variable ad set S = 1 (X i 1). i=1 A sum of idepedet expoetial variables has distributio Gamma (, 1), i.e., its pdf is x 1 e x (). The desity for this sum shifted by is with x > ad after dividig by, (x + ) 1 e (x+), () f S (x) = ( x + ) 1 e ( x+), ()

15 Limit Theorems =1 0.6 = = = Figure 7.1: The approach of a ormalized sum of 1, 2, 4 ad 16 expoetial radom variables to the ormal distributio.

16 178 Chapter 7 with x >. See Figure 7.1 for a visualizatio of the approach of the distributio of S toward the stadard ormal distributio.