ST525: Advaced Statistical Theory Departmet of Statistics & Applied Probability Tuesday, September 7, 2 ST525: Advaced Statistical Theory
Lecture : The law of large umbers The Law of Large Numbers The law of large umbers cocers the limitig behavior of a sum of radom variables. The weak law of large umbers (WLLN) refers to covergece i probability. The strog law of large umbers (SLLN) refers to a.s. covergece. Kroecker s Lemma: Let x R, a R, < a a +, =, 2,, ad a. If the series = x /a coverges, the a i= x i. ST525: Advaced Statistical Theory
The Law of Large Numbers (i.i.d. case) Theorem.3 (i) Weak Law of Large Numbers: Let X, X 2,... be i.i.d. radom variables. If P( X > ), the X i E(X I { X }) p. i= (ii) Strog Law of Large Numbers: Let X, X 2,... be i.i.d. radom variables. If E X <, the ad X i a.s. EX i= c i (X i EX ) a.s. i= for ay bouded sequece of real umbers {c i }. ST525: Advaced Statistical Theory
Proof: (i) Let Y j = X j I { Xj }, T = j= X j ad Z = j= Y j. The P(T Z ) P(Y j X j ) = P( X > ). () j= By Chebyshev s iequality, for ay ɛ >, ( ) Z EZ P > ɛ Var(Z ) ɛ 2 2 which sice EY 2 = = 2 x 2 df X (x) EY 2 ɛ 2, xp( X > x)dx P( X > ). Together with (), this implies the result. ST525: Advaced Statistical Theory
Proof (cot.): Hajek-Reyi s iequality: Let Y i, i =,...,, be idepedet radom variables havig fiite variaces. The ( ) P max c l l (Y l i EY i ) > ɛ ɛ 2 ci 2 Var(Y i ). i= (ii) Let Y i = X i I { Xi i}, for iteger m >, let Z m = m i= Y i, Z i = Y i, i > m. Takig c l = l i Hajek-Reyi s iequality, we have ( ) l P max (Z m l l i EZ i ) > ɛ i=m m ɛ 2 m 2 Var(Y i ) + Var(Y i ) ɛ 2 i 2. (2) i= i= i=m+ ST525: Advaced Statistical Theory
Proof (cot.): Note that = E(Y 2 ) 2 = = = j= j= =j j= =j λ E(X 2 I {j X j}) 2 E(X 2 I {j X j}) 2 je( X I {j X j}) 2 E( X I {j X j}) = λe X, j= where λ is a costat. ST525: Advaced Statistical Theory
Proof (cot.): Lettig first ad the m i (2), we have ( { }) l P (Z i EZ i ) l > ɛ l=m i=m ( ) = lim lim P l max (Z m m l l i EZ i ) > ɛ i=m m lim m ɛ 2 m 2 Var(Y i ) =. i= This implies i= Y i a.s. EX sice EY EX ad hece (/) i= EY i EX. Furthermore P(X Y ) = = P( X > ) = = P( X > ) <. = ST525: Advaced Statistical Theory
Proof (cot.): Therefore, by Borel-Catelli s lemma, P( = m= {X Y }) =, i.e., P( = m= {X = Y }) =. For ay ω = m= {X = Y }, whe is large eough, X (ω) = Y (ω). This implies X i= Y a.s., i= which, i tur, implies i= X i a.s. EX, sice we have already show that i= Y i a.s. EX. ST525: Advaced Statistical Theory
Remarks If E X <, the a = E(X I { X }) EX ad result the WLLN is actually established i Example.28 i a much simpler way. O the other had, if E X <, the a stroger result, the SLLN, ca be obtaied. Some results for the case of E X = ca be foud i Exercise 48 ad Theorem 5.4.3 i Chug (974). The ext result is for sequeces of idepedet but ot ecessarily idetically distributed radom variables. ST525: Advaced Statistical Theory
Theorem.4 (o i.i.d. case) Let X, X 2,... be idepedet radom variables with fiite expectatios. (i) (The SLLN). If there is a costat p [, 2] such that <, the i= E X i p i p (X i EX i ) a.s.. i= (ii) (The WLLN). If there is a costat p [, 2] such that lim p i= E X i p =, the (X i EX i ) p. i= ST525: Advaced Statistical Theory
Remarks Note that the coditio for SLLN implies the coditio for WLLN (Kroecker s Lemma). A obvious sufficiet coditio for SLLN with p (, 2] is sup E X p <. The WLLN ad SLLN have may applicatios i probability ad statistics. Example.32 Let f ad g be cotiuous fuctios o [, ] satisfyig f (x) Cg(x) for all x, where C > is a costat. We ow show that lim (assumig that g(x)dx ). i= f (x i) i= g(x i) dx dx 2 dx = f (x)dx g(x)dx (3) ST525: Advaced Statistical Theory
Example.32 (cotiued) Let X, X 2,... be i.i.d. radom variables havig the uiform distributio o [, ]. The E[f (X )] = f (x)dx <, E[g(X )] = By the SLLN (Theorem.3(ii)), f (X i ) a.s. E[f (X )], i= By Theorem.(i), i= f (X i) i= g(x i) a.s. g(x)dx <. g(x i ) a.s. E[g(X )], i= E[f (X )] E[g(X )]. (4) Sice the radom variable o the left-had side of (4) is bouded by C, result (3) follows from the domiated covergece theorem ad the fact that the left-had side of (3) is the expectatio of the radom variable o the left-had side of (4). ST525: Advaced Statistical Theory
Example Let T = i= X i, where X s are idepedet radom variables satisfyig P(X = ± θ ) =.5 ad θ > is a costat. We wat to show that T / a.s. whe θ <.5. For θ <.5, EX 2 2θ 2 = 2 <. = = By Theorem.4 (i), T / a.s.. ST525: Advaced Statistical Theory
Example Let T = i= X i, where X s are idepedet radom variables satisfyig P(X = ± θ ) =.5 ad θ > is a costat. We wat to show that T / a.s. whe θ <.5. For θ <.5, EX 2 2θ 2 = 2 <. = = By Theorem.4 (i), T / a.s.. Example (Exercise 65) Let X, X 2,... be idepedet radom variables. Suppose that (X j EX j ) d N(, ), σ j= where σ 2 = Var( j= X j). ST525: Advaced Statistical Theory
We wat to show that (X j EX j ) p iff σ /. j= If σ /, the by Slutsky s theorem, (X j EX j ) = σ (X j EX j ) d. j= Assume ow σ / c (, ] but j= (X j EX j ) p. By Slutsky s theorem, σ j= σ (X j EX j ) = σ j= (X j EX j ) p. This cotradicts the fact that j= (X j EX j )/σ d N(, ). Hece, j= (X j EX j ) does ot coverge to i probability. j= ST525: Advaced Statistical Theory