556: MATHEMATICAL STATISTICS I

Transcription

1 556: MATHEMATICAL STATISTICS I CHAPTER 5: STOCHASTIC CONVERGENCE The following efinitions ae state in tems of scala anom vaiables, but exten natually to vecto anom vaiables efine on the same obability sace with measue P. Fo examle, some esults ae state in tems of the Eucliean istance in one imension X = ( X) 2, o fo sequences of k-imensional anom vaiables = (1,..., k ), 5.1 Convegence in Distibution k X = (j X j ) 2 j=1 Consie a sequence of anom vaiables X 1, X 2,... an a coesoning sequence of cfs, F X1, F X2,... so that fo n = 1, 2,.. F Xn (x) =P[ x]. Suose that thee exists a cf, F X, such that fo all x at which F X is continuous, lim F (x) = F X (x). Then X 1,..., conveges in istibution to anom vaiable X with cf F X, enote X an F X is the limiting istibution. Convegence of a sequence of mgfs o cfs also inicates convegence in istibution, that is, if fo all t at which M X (t) is efine, if as n, we have 1/2 M Xi (t) M X (t) Definition : DEGENERATE DISTRIBUTIONS The sequence of anom vaiables X 1,..., conveges in istibution to constant c if the limiting istibution of X 1,..., is egeneate at c, that is, X an P [X = c] = 1, so that { 0 x < c F X (x) = 1 x c Inteetation: A secial case of convegence in istibution occus when the limiting istibution is iscete, with the obability mass function only being non-zeo at a single value, that is, if the limiting anom vaiable is X, then P [X = c] = 1 an zeo othewise. We say that the sequence of anom vaiables X 1,..., conveges in istibution to c if an only if, fo all ϵ > 0, lim P [ c < ϵ] = 1 This efinition inicates that convegence in istibution to a constant c occus if an only if the obability becomes inceasingly concentate aoun c as n. Note: Points of Discontinuity To show that we shoul ignoe oints of iscontinuity of F X in the efinition of convegence in istibution, consie the following examle: let { 0 x < ϵ F ϵ (x) = 1 x ϵ 1.

2 be the cf of a egeneate istibution with obability mass 1 at x = ϵ. Now consie a sequence {ϵ n } of eal values conveging to ϵ fom below. Then, as ϵ n < ϵ, we have { 0 x < ϵn F ϵn (x) = 1 x ϵ n which conveges to F ϵ (x) at all eal values of x. Howeve, if instea {ϵ n } conveges to ϵ fom above, then F ϵn (ϵ) = 0 fo each finite n, as ϵ n > ϵ, so lim F ϵ n (ϵ) = 0. Hence, as n, F ϵn (ϵ) 0 1 = F ϵ (ϵ). Thus the limiting function in this case is F ϵ (x) = { 0 x ϵ 1 x > ϵ which is not a cf as it is not ight-continuous. Howeve, if { } an X ae anom vaiables with istibutions {F ϵn } an F ϵ, then P [ = ϵ n ] = 1 conveges to P [X = ϵ] = 1, howeve we take the limit, so F ϵ oes escibe the limiting istibution of the sequence {F ϵn }. Thus, because of ight-continuity, we ignoe oints of iscontinuity in the limiting function. 5.2 Convegence in Pobability Definition : CONVERGENCE IN PROBABILITY TO A CONSTANT The sequence of anom vaiables X 1,..., conveges in obability to constant c, enote c, if lim P [ c < ϵ] = 1 o lim P [ c ϵ] = 0 that is, if the limiting istibution of X 1,..., is egeneate at c. Inteetation : Convegence in obability to a constant is ecisely equivalent to convegence in istibution to a constant. THEOREM (WEAK LAW OF LARGE NUMBERS) Suose that X 1,..., is a sequence of i.i.. anom vaiables with exectation µ an finite vaiance σ 2. Let Y n be efine by Y n = 1 X i n then, fo all ϵ > 0, lim P [ Y n µ < ϵ] = 1, that is, Y n µ, an thus the mean of X 1,..., conveges in obability to µ. Poof. Using the oeties of exectation, it can be shown that Y n has exectation µ an vaiance σ 2 /n, an hence by the Chebychev Inequality, fo all ϵ > 0. Hence an Y n µ. P [ Y n µ ϵ] σ2 nϵ 2 0 P [ Y n µ < ϵ] 1 as n as n 2

3 Definition : CONVERGENCE IN PROBABILITY TO A RANDOM VARIABLE The sequence of anom vaiables X 1,..., conveges in obability to anom vaiable X, enote X, if, fo all ϵ > 0, lim P [ X < ϵ] = 1 o equivalently lim P [ X ϵ] = 0 To unestan this efinition, let ϵ > 0, an consie A n (ϵ) {ω : (ω) X(ω) ϵ} Then we have X if lim P (A n(ϵ)) = 0 that is, if thee exists an n such that fo all m n, P (A m (ϵ)) < ϵ. 5.3 Convegence Almost Suely The sequence of anom vaiables X 1,..., conveges almost suely to anom vaiable X, enote X if fo evey ϵ > 0 [ ] P lim X < ϵ = 1, that is, if A {ω : (ω) X(ω)}, then P (A) = 1. Equivalently, X if fo evey ϵ > 0 [ ] P lim X > ϵ = 0. This can also be witten lim (ω) = X(ω) fo evey ω Ω, excet ossibly those lying in a set of obability zeo une P. Altenative chaacteization: Let ϵ > 0, an the sets A n (ϵ) an B m (ϵ) be efine fo n, m 0 by A n (ϵ) {ω : (ω) X(ω) > ϵ} Then X if an only if P (B m (ϵ)) 0 as m. Inteetation: B m (ϵ) n=m A n (ϵ). The event A n (ϵ) coesons to the set of ω fo which (ω) is moe than ϵ away fom X. The event B m (ϵ) coesons to the set of ω fo which (ω) is moe than ϵ away fom X, fo at least one n m. The event B m (ϵ) occus if thee exists an n m such that X > ϵ. X if an only if an only if P (B m (ϵ)) 0. X if an only if P [ X > ϵ infinitely often ] = 0 that is, X if an only if thee ae only finitely many fo which if ω lies in a set of obability geate than zeo. (ω) X(ω) > ϵ 3

4 Note that X if an only if lim P (B m(ϵ)) = lim P m m ( n=m A n (ϵ) in contast with the efinition of convegence in obability, whee Clealy lim P (A m(ϵ)) = 0. m A m (ϵ) n=m an hence almost sue convegence is a stonge fom. A n (ϵ) ) = 0 X if Altenative teminology: a.e. X almost eveywhee, X w..1 X with obability 1, X Inteetation: A anom vaiable is a eal-value function fom (a sigma-algeba efine on) samle sace Ω to R. The sequence of anom vaiables X 1,..., coesons to a sequence of functions efine on elements of Ω. Almost sue convegence equies that the sequence of eal numbes (ω) conveges to X(ω) (as a eal sequence) fo all ω Ω, as n, excet ehas when ω is in a set having obability zeo une the obability istibution of X. THEOREM (STRONG LAW OF LARGE NUMBERS) Suose that X 1,..., is a sequence of i.i.. anom vaiables with exectation µ an (finite) vaiance σ 2. Let Y n be efine by Y n = 1 X i n then, fo all ϵ > 0, [ ] P lim Y n µ < ϵ = 1, that is, Y n µ, an thus the mean of X 1,..., conveges almost suely to µ. 5.4 Convegence In th Mean The sequence of anom vaiables X 1,..., conveges in th mean to anom vaiable X, enote X if lim E [ X ] = 0. Fo examle, if then we wite lim E [ ( X) 2] = 0 =2 In this case, we say that { } conveges to X in mean-squae o in quaatic mean. 4

5 THEOREM Fo 1 > 2 1, = X 1 = n X = 2 Xn X Poof. By Lyaunov s inequality so that as n, as 2 < 1. Thus E[ X 2 ] 1/ 2 E[ X 1 ] 1/ 1 E[ X 2 ] E[ X 1 ] 2/ 1 0 E[ X 2 ] 0 = an X 2 n The convese oes not hol in geneal. THEOREM (RELATING THE MODES OF CONVERGENCE) Fo sequence of anom vaiables X 1,...,, following elationshis hol X o = X = X X so almost sue convegence an convegence in th mean fo some both imly convegence in obability, which in tun imlies convegence in istibution to anom vaiable X. No othe elationshis hol in geneal. THEOREM (Patial Conveses: NOT EXAMINABLE) (i) If fo evey ϵ > 0, then (ii) If, fo some ositive intege, then P [ X > ϵ ] < n=1 E[ X ] < n=1 THEOREM (Slutsky s Theoem) Suose that Then (i) + Y n X + c X an Y n c (ii) Y n (iii) /Y n cx X/c ovie c 0. 5

6 5.5 The Cental Limit Theoem THEOREM (THE LINDEBERG-LÉVY CENTRAL LIMIT THEOREM) Suose X 1,..., ae i.i.. anom vaiables with mgf M X, with exectation µ an vaiance σ 2, both finite. Let the anom vaiable Z n be efine by Z n = X i nµ nσ 2 = n(xn µ) σ whee = 1 n an enote by M Zn the mgf of Z n. Then, as n, X i, M Zn (t) ex{t 2 /2} iesective of the fom of M X. Thus, as n, Z n Z N (0, 1). Poof. Fist, let Y i = (X i µ)/σ fo i = 1,..., n. Then Y 1,..., Y n ae i.i.. with mgf M Y say, an E fy [Y i ] = 0, Va Y [Y i ] = 1 fo each i. Using a Taylo seies exansion, we have that fo t in a neighbouhoo of zeo, M Y (t) = 1 + te Y [Y ] + t2 2! E Y [Y 2 ] + t3 3! E Y [Y 3 ] +... = 1 + t2 2 + O(t3 ) using the O(t 3 ) notation to catue all tems involving t 3 an highe owes. Re-witing Z n as Z n = 1 n as Y 1,..., Y n ae ineenent, we have by a stana mgf esult that M Zn (t) = n Y i { ( )} } n } n t M Y = {1 + t2 n 2n + O(n 3/2 ) = {1 + t2 2n + o(n 1 ). so that, by the efinition of the exonential function, as n M Zn (t) ex{t 2 /2} Z n Z N (0, 1) whee no futhe assumtions on M X ae equie. Altenative statement: The theoem can also be state in tems of so that Z n = X i nµ Z n an σ 2 is teme the asymtotic vaiance of Z n. n = n( µ) Z N (0, σ 2 ). 6

7 Notes : (i) The theoem equies the existence of the mgf M X. (ii) The theoem hols fo the i.i.. case, but thee ae simila theoems fo non ientically istibute, an eenent anom vaiables. (iii) The theoem allows the constuction of asymtotic nomal aoximations. Fo examle, fo lage but finite n, by using the oeties of the Nomal istibution, S n = AN (µ, σ 2 /n) X i AN (nµ, nσ 2 ). whee AN (µ, σ 2 ) enotes an asymtotic nomal istibution. The notation.. N (µ, σ 2 /n) is sometimes use. (iv) The multivaiate vesion of this theoem can be state as follows: Suose X 1,..., ae i.i.. k-imensional anom vaiables with mgf M X, with E fx [X i ] = µ Va fx [X i ] = Σ whee Σ is a ositive efinite, symmetic k k matix efining the vaiance-covaiance matix of the X i. Let the anom vaiable Z n be efine by Z n = n( µ) whee Then as n. Z n = 1 n X i. Z N (0, Σ) 7

8 Aenix (NOT EXAMINABLE) Poof. Relating the moes of convegence. (a) X = Suose X, an let ϵ > 0. Then as, consieing the oiginal samle sace, P [ X < ϵ ] P [ X m X < ϵ, m n ] (1) {ω : X m (ω) X(ω) < ϵ, m n} {ω : (ω) X(ω) < ϵ} But, as X, P [ X m X < ϵ, m n ] 1, as n. So, afte taking limits in equation (1), we have an so lim P [ X < ϵ ] lim P [ X m X < ϵ, m n ] = 1 lim P [ X n X < ϵ ] = 1 (b) X = Suose X, an let ϵ > 0. Then, using an agument simila to Chebychev s Lemma, E[ X ] E[ X I { Xn X >ϵ} ] ϵ P [ X > ϵ]. Taking limits as n, as n (c) X = X, E[ X ] 0 as n, so theefoe, also, as P [ X > ϵ] 0 Suose X, an let ϵ > 0. Denote, in the usual way, F Xn (x) = P [ x] an F X (x) = P [X x]. Then, by the theoem of total obability, we have two inequalities F Xn (x) = P [ x] = P [ x, X x+ϵ]+p [ x, X > x+ϵ] F X (x+ϵ)+p [ X > ϵ] F X (x ϵ) = P [X x ϵ] = P [X x ϵ, x]+p [X x ϵ, > x] F Xn (x)+p [ X > ϵ]. as A B = P (A) P (B) yiels Thus P [ x, X x + ϵ ] F X (x + ϵ) an P [ X x ϵ, x ] F Xn (x). F X (x ϵ) P [ X > ϵ] F Xn (x) F X (x + ϵ) + P [ X > ϵ] an taking limits as n (with cae; we cannot yet wite lim F Xn (x) as we o not know that this limit exists) ecalling that X, F X (x ϵ) lim inf F X n (x) lim su F Xn (x) F X (x + ϵ) Then if F X is continuous at x, F X (x ϵ) F X (x) an F X (x + ϵ) F X (x) as ϵ 0, so an thus F Xn (x) F X (x) as n. F X (x) lim inf F X n (x) lim su F Xn (x) F X (x) 8

9 Poof. (Patial conveses) (i) Let ϵ > 0. Then fo n 1, P [ X > ϵ, fo some m n ] P [ m=n { X m X > ϵ} ] P [ X m X > ϵ ] as, by elementay obability theoy, P (A B) P (A) + P (B). But, as it is the tail sum of a convegent seies (by assumtion), it follows that lim m=n P [ X m X > ϵ ] = 0. m=n Hence an lim P [ X > ϵ, fo some m n ] = 0 (ii) Ientical to at (i), an using at (b) of the evious theoem that X = 9