5. Limit Theorems, Part II: Central Limit Theorem. ECE 302 Fall 2009 TR 3 4:15pm Purdue University, School of ECE Prof.

Transcription

1 5. Limit Theorems, Part II: Cetral Limit Theorem ECE 302 Fall 2009 TR 3 4:15pm Purdue Uiversity, School of ECE Prof. Ilya Pollak

2 WLLN ad CLT X 1,, X i.i.d. with fiite mea μ ad variace σ 2

3 WLLN ad CLT X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2

4 WLLN ad CLT X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 M = S / has variace σ 2 /

5 WLLN ad CLT X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 M = S / has variace σ 2 / Coverges i probability to μ (WLLN)

6 WLLN ad CLT X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 M = S / has variace σ 2 / Coverges i probability to μ (WLLN) S / 1/2 has costat variace σ 2

7 WLLN ad CLT X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 M = S / has variace σ 2 / Coverges i probability to μ (WLLN) S / 1/2 has costat variace σ 2 Cetral limit theorem characterizes the asymptoyc shape of the distribuyo of S / 1/2

8 Stadardized S X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2

9 Stadardized S X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 Let Z = S E[ S ] = S µ σ S σ

10 Stadardized S X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 Let Z = S E [ S ] = S µ σ S σ The E[ Z ] = 0,

11 Stadardized S X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 Let Z = S E [ S ] = S µ σ S σ The E[ Z ] = 0, var( Z ) = 1 ( σ 2 ) = 1 σ 2

12 Cetral Limit Theorem X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 Let Z = S E[ S ] = S µ σ S σ Let Z be a stadard Gaussia r.v.

13 Cetral Limit Theorem X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 Theorem: Let Z = S E[ S ] = S µ σ S σ Let Z be a stadard Gaussia r.v. ( ) = P(Z z) = Φ(z), for ay z lim P Z z ---i.e., the CDF of Z coverges to the stadard Gaussia CDF.

14 Cetral Limit Theorem X 1,, X i.i.d. with fiite mea μ ad variace σ 2 S = X X has variace σ 2 Theorem: Let Z = S E[ S ] = S µ σ S σ Let Z be a stadard Gaussia r.v. ( ) = P(Z z) = Φ(z), for ay z lim P Z z ---i.e., the CDF of Z coverges to the stadard Gaussia CDF. CauYo: this is NOT a statemet about the covergece of PDFs or PMFs

15 Gaussia ApproximaYo Treat Z as if Gaussia

16 Gaussia ApproximaYo Treat Z as if Gaussia Treat S = 1/2 σz + μ also as if Gaussia, with mea μ ad variace σ 2

17 Gaussia ApproximaYo Treat Z as if Gaussia Treat S = 1/2 σz + μ also as if Gaussia, with mea μ ad variace σ 2 Whe is moderate o theorems

18 Gaussia ApproximaYo Treat Z as if Gaussia Treat S = 1/2 σz + μ also as if Gaussia, with mea μ ad variace σ 2 Whe is moderate o theorems symmetry helps (e.g., if X i are uiform, the S 8 is very close to Gaussia)

19 Gaussia ApproximaYo Treat Z as if Gaussia Treat S = 1/2 σz + μ also as if Gaussia, with mea μ ad variace σ 2 Whe is moderate o theorems symmetry helps (e.g., if X i are uiform, the S 8 is very close to Gaussia) approximayo of P(S s) is more accurate for s E[S ]

20 Recall Ex. 5.5: Pollig Estimate Presidet Obama's approval ratig by askig persos draw at radom from the voter populatio. Let 1, if the i-th perso approves X i = 0, otherwise Model X 1, X 2,, X as idepedet Beroulli r.v.'s with mea p ad variace p(1-p), where p is the "true" approval ratig. We estimate p as the sample mea: M = X 1 + X X p(1 p) E[M ] = p, var(m ) = By WLLN, M p i probability as. By Chebyshev iequality, P( M ) Suppose we wat P M p 0.01 p(1 p) ε 2 1 4ε 2 ( ) I.e., we wat to be 95% cofidet that we are withi 0.01 of the actual approval ratig. We ca guaratee this 1 if we take 4ε = 1 = 50,000. Note: This is very coservative because the Chebyshev iequality is loose!

21 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X 1 + X X E[M ] = p, var(m ) = p(1 p)

22 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X 1 + X X p(1 p) E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia.

23 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X 1 + X X E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M ( ) p(1 p)

24 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X + X + + X 1 2 E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) p(1 p)

25 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X + X + + X 1 2 E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) = 2 1 Φ p(1 p) ε p(1 p)

26 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X 1 + X X E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) = 2 1 Φ p(1 p) ε p(1 p) Agai, p is ukow. But we kow that p(1 p) 1 4. p(1 p) 1/4 0 1/2 1 p

27 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X 1 + X X E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) = 2 1 Φ p(1 p) ε p(1 p) Agai, p is ukow. But we kow that p(1 p) 1 4. Therefore, 1 p(1 p) 2, ad

28 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X 1 + X X E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) = 2 1 Φ p(1 p) ε p(1 p) Agai, p is ukow. But we kow that p(1 p) 1 4. Therefore, 1 ε p(1 p) 2ε, p(1 p) 2, ad

29 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X + X + + X 1 2 E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) = 2 1 Φ p(1 p) ε p(1 p) Agai, p is ukow. But we kow that p(1 p) 1 4. Therefore, 1 ε p(1 p) 2ε, ad 1 Φ ε p(1 p) 1 Φ ( 2ε ). p(1 p) 2, ad

30 Ex. 5.11: Pollig X 1, X 2,, X idepedet Beroulli r.v.'s with mea p ad variace p(1-p). M = X + X + + X 1 2 E[M ] = p, var(m ) = We approximate the CDF of M as Gaussia. ( ) 2P M P M 2P ( ) M p p(1 p) ε p(1 p) = 2 1 Φ p(1 p) ε p(1 p) Agai, p is ukow. But we kow that p(1 p) 1 4. Therefore, 1 ε p(1 p) 2ε, ad 1 Φ ε p(1 p) 1 Φ ( 2ε ). So, ( ) 2 1 Φ 2ε P M ( ) p(1 p) 2, ad

31 Ex. 5.11: Pollig ( ) 2 1 Φ 2ε P M E.g., to get ( ) ( ) 0.05, set 2 2Φ P M p 0.01 ( ) 0.05

32 Ex. 5.11: Pollig ( ) 2 1 Φ 2ε P M E.g., to get P M p 0.01 ( ) ( ) 0.05, set 2 2Φ ( ) Φ(1.96) Φ 0.02 ( ) 0.05

33 Ex. 5.11: Pollig ( ) 2 1 Φ 2ε P M E.g., to get P M p 0.01 ( ) ( ) 0.05, set 2 2Φ ( ) Φ(1.96) Φ ( ) 0.05

34 Ex. 5.11: Pollig ( ) 2 1 Φ 2ε P M E.g., to get P M p 0.01 ( ) ( ) 0.05, set 2 2Φ Φ 0.02 ( ) Φ(1.96) ( ) 0.05

35 Ex. 5.11: Pollig ( ) 2 1 Φ 2ε P M E.g., to get P M p 0.01 ( ) ( ) 0.05, set 2 2Φ Φ 0.02 ( ) Φ(1.96) ( ) 0.05

36 Ex. 5.11: Pollig ( ) 2 1 Φ 2ε P M E.g., to get P M p 0.01 ( ) ( ) 0.05, set 2 2Φ Φ 0.02 ( ) Φ(1.96) ( ) 0.05 This is sigificatly beeer tha the sample size of 50,000 foud usig Chebyshev s iequality

37 Usefuless of CLT Oly meas ad variaces maeer.

38 Usefuless of CLT Oly meas ad variaces maeer. Much more accurate tha Chebyshev.

39 Usefuless of CLT Oly meas ad variaces maeer. Much more accurate tha Chebyshev. Useful computayoal shortcut, eve if we have a formula for the distribuyo of S.

40 Usefuless of CLT Oly meas ad variaces maeer. Much more accurate tha Chebyshev. Useful computayoal shortcut, eve if we have a formula for the distribuyo of S. JusYficaYo of models ivolvig Gaussia radom variables.