10/31/2018 CentralLimitTheorem

Size: px

Start display at page:

Download "10/31/2018 CentralLimitTheorem"

Victoria Hill
5 years ago
Views:

1 10/31/2018 CetralLimitTheorem 1/10

2 10/31/2018 CetralLimitTheorem Cetral Limit Theorem The Cetral Limit Theorem (hereafter, CLT) is the most importat ad most sigificat result i all of probability ad statistics. It explais the profoud importace of the ormal distributio, ad provides a foudatio, essetially, for all of elemetary statistics. Warm-up to the CLT Oe of the most sigificat aspects of the CLT is that it shows how repeated samplig of values from a radom variable X results i a radom variable whose expected value remais the same as X but with a variace which decreases liearly i the umber of samples. Formally, let X 1, X 2,, X be idepedet radom variables that are idetically distributed (i.e., all have the same probability fuctio i the discrete case or desity fuctio i the cotiuous case) ad all have the same fiite mea μ ad variace σ 2. (Note: such a collectio of RVs is said to be "Idepedet ad Idetically Distributed" or IID.) I the simple case of samplig, each X i represets oe "poke" of a sigle radom variable X, but this is ot required. Now suppose we cosider the radom variable X represetig the mea of the, i.e., First, it is hardly surprisig that the mea of X, or the "mea of the mea" remais uchaged, by virtue of the liearity of expected values, i.e., ad also for idepedet RVs X ad Y, we have Thus for "pokes" of a RV X, we have ad so, puttig these results together, we have a ot uexpected result: What about the variace of X? Does it have the same liearity property? Not quite: This is ot uexpected, sice the uits of the variace are squared! However, ote that the stadard deviatio does have a similar liearity property, except that shiftig the distributio left or right by a costat icremet b does ot affect the variace: However, we do have liearity for the sum of idepedet RVs: X X 2/10 X i X 1 + X X =. E(a X + b) = a E(X) + b E(X + Y) = E(X) + E(Y). E( ) = E(X) X 1 X 2 X X 1 + X X E ( ) = E ( ) Var(a X + b) = E [ (a X + b ) 2 ] E(a X + b) 2 1 = E(X) = E(X). = E( a 2 X 2 + a b X + b 2 ) (a E(X) + b) 2 = a 2 E( X 2 ) + a b E(X) + b 2 ) a 2 E(X ) 2 + a b E(X) + b 2 = a 2 E( X 2 ) + a b E(X) + b 2 ( a 2 E(X ) 2 + a b E(X) + b 2 ) = a 2 E( X 2 ) a 2 E(X) 2 = a 2 (E( X 2 ) E(X ) 2 ) = a 2 Var(X) σ a X+b Var(X + Y) = E [ (X + Y ) 2 ] E(X + Y) 2 = a σ X. = E( X 2 + 2XY + Y 2 ) (E(X) + E(Y) ) 2 = E( X 2 ) + 2E(X)E(Y) + E( Y 2 ) [E(X) 2 + E(X)E(Y) + E(Y ) 2 ] = E( X 2 ) + 2E(X)E(Y) + E( Y 2 ) E(X) 2 2E(X)E(Y) E(Y) 2 = E( X 2 ) E(X ) 2 + E( Y 2 ) E(Y) 2 = Var(X) + Var(Y)

3 10/31/2018 CetralLimitTheorem Fially we ca ivestigate Var( X ). ad therefore the stadard deviatio is Var X 1 + X X ( ) So the puchlie is whe we take a mea of idepedet "pokes" of a radom variable, the mea value remais the same, but the stadard deviatio gets smaller by a factor of 1. = Var( X X + ) 2 X 2 = Var( X X + ) 2 X 2 = Var( ) + Var( ) + + Var( ) X 1 X 2 X = Var(X) 2 = Var(X) σ X = Now we ca fially be precise about our experimet (from day oe!) of flippig a fair coi ad takig the average of the umber of heads! If we flip a fair coi oce, we have Beroulli(1/2) with a mea of 1/2 ad a stadard deviatio of 1/4. If we flip the coi times, it coverges to its mea value 1/2 i the very precise sese that the stadard deviatio is 1 4 σ X 2 Experimet Oe: What happes whe we keep flippig a coi? I [98]: from radom import radit = 10**6 cout = 0 for k i rage(): cout += radit(0,1) mu = cout/ prit(" = " + str() + "\tvar = " + str(0.5/) + "\tsigma = " + str(0.25 * (1/(**0.5))) + " \t result = " + str(mu) + "\t delta = " + str(mu-0.5)) = Var = 5e-07 sigma = result = delta = e-05 Experimet Two: How does variace decrease as gets larger? 3/10

4 10/31/2018 CetralLimitTheorem I [107]: import umpy as p from umpy import arage,lispace, mea, var, std, uique import matplotlib.pyplot as plt from umpy.radom import radom, radit,uiform, choice, biomial, geometric, poisso import math from collectios import Couter import padas as pd %matplotlib ilie def roud4(x): retur roud(float(x) ,4) def samplemeauiform(): X = [uiform(0,1) for i i rage()] retur sum(x)/ def display_sample_mea_uiform(,um_trials,decimals): fig, ax = plt.subplots(1,1,figsize=(12,6)) ax.set_xlim(0,1) ax.set_ylim(0,50) plt.title('uiform: Distributio of Sample Mea = ' + str()) plt.ylabel("f(x)") plt.xlabel("k i Rage(X)") # use beroulli to geerate radom samples X = [samplemeauiform() for i i rage(um_trials)] Xrouded = [roud(x,decimals) for x i X] # Now covert frequecy couts ito probabilities D = Couter( Xrouded ) Xrouded = uique(xrouded) # sorts ad removes duplicates P = [10**decimals*D[k]/um_trials for k i Xrouded] # must multiply probs by 10**decimals because plt.bar(xrouded,p,width=1/10**decimals, edgecolor='k',alpha=0.5) # bis are of width 1/10* *decimals plt.show() = 1000 # try for 1, 2, 5, 10, 30, 100, 1000 um_trials = 1000 display_sample_mea_uiform(,um_trials,3) 4/10

5 10/31/2018 CetralLimitTheorem However, we ca say much more tha this, ad that is the cotet of the ext sectio. The Cetral Limit Theorem We expect the ormal distributio to arise wheever the outcome of a situatio results from umerous small additive compoets, with o sigle or small group of effects domiat (sice all the compoets are idepedet). Hece, it occurs regularly i biostatistics (where may gees ad may evirometal factors add together to produce some result, such as itelligece or height), i errors i measuremet (where may small errors add up), ad fiace (where may small effects cotribute for example to the price of a stock). The CLT provides the formal justificatio for this pheomeo. σ To state the CLT, first, suppose we stadardize X by subtractig μ ad dividig by its stadard deviatio to obtai a ew radom variable : Z Z = X μ σ The the CLT states that as, Z coverges to the stadard ormal Z N(0, 1), that is: a 1 lim P( Z a) = dx e x 2 /2 2π A simpler versio of this theorem is the followig, which has immese cosequeces for the developmet of samplig theory (which is our ext topic i CS 237). σ As gets large, the radom variable X coverges to the distributio N ( μ, 2 ). There are several crucial thigs to remember about the CLT: 1. The mea μ of X is the same as the X i. σ 2. But the stadard deviatio gets smaller as gets larger, ad approaches 0 as approaches. 3. The distributios of the X i do NOT MATTER at all, ad as log as they have a commo mea ad stadard deviatio, they ca be completely differet distributios. Typically, however, these are separate "pokes" of the same radom variable. 4. We ca use the strog properties of the ormal distributio, such as the " rule," to quatify the radomess iheret i the samplig process. This will be the fudametal fact we will use i developig the various statistical procedures i elemetary statistics. I [6]: # Jupyter otebook specific from IPytho.display import Image from IPytho.core.display import HTML from IPytho.display import display_html from IPytho.display import display from IPytho.display import Math from IPytho.display import Latex from IPytho.display import HTML # Geeral useful imports import umpy as p from umpy import arage,lispace,roud, uique import matplotlib.pyplot as plt from umpy.radom import radom, radit,uiform, choice, biomial, geometric, poisso, ormal, expoetial from scipy.stats import orm import math from collectios import Couter import padas as pd %matplotlib ilie 5/10

6 10/31/2018 CetralLimitTheorem Experimets with the CLT We will verify the CLT with several distributios: - Uiform U ~ Uiform(0,1) mea is 0.5 std dev = sqrt( 1/ 12 ) = Beroulli X ~ Ber(0.61) mea is 0.61 std dev = (0.61)*(0.39) = Expoetial E ~ Exp(0.1) mea is 10 std dev = Normal Y ~ N(66,9) mea is 66 std dev = 3 Experimet Three: Uiform Distributio 6/10

7 10/31/2018 CetralLimitTheorem I [112]: def samplemeauiform(): X = [uiform(0,1) for i i rage()] retur sum(x)/ def display_sample_mea_uiform(,um_trials,decimals): fig, ax = plt.subplots(1,1,figsize=(12,6)) plt.title('uiform: Distributio of Sample Mea = ' + str()) plt.ylabel("f(x)") plt.xlabel("k i Rage(X)") # use beroulli to geerate radom samples X = [samplemeauiform() for i i rage(um_trials)] Xrouded = [roud(x,decimals) for x i X] # Now covert frequecy couts ito probabilities D = Couter( Xrouded ) Xrouded = uique(xrouded) # sorts ad removes duplicates P = [10**decimals*D[k]/um_trials for k i Xrouded] # must multiply probs by 10**decimals because plt.bar(xrouded,p,width=1/10**decimals, edgecolor='k',alpha=0.5) # bis are of width 1/10* *decimals # Now geerate the theoretical ormal for sample mea with std dev sigma/sqrt() mu = 0.5 sigma = ((1/12)**0.5) / **0.5 X2 = p.lispace(mu-sigma*3,mu+sigma*3,100) Y = [orm.pdf(x,mu,sigma) for x i X2] plt.plot(x2,y) plt.show() = 1000 # try for 1, 10, 30, 100 um_trials = 1000 display_sample_mea_uiform(,um_trials,3) Experimet Four: Beroulli 7/10

8 10/31/2018 CetralLimitTheorem I [117]: def extberoulli(p): if(uiform(0,1) < p): retur 1 else: retur 0 def samplemeaberoulli(p,): X = [extberoulli(p) for i i rage()] retur sum(x)/ def display_sample_mea_beroulli(p,,um_trials,decimals): fig, ax = plt.subplots(1,1,figsize=(12,6)) plt.title('beroulli Distributio of Sample Mea = ' +str() ) plt.ylabel("f(x)") plt.xlabel("k i Rage(X)") # use beroulli to geerate radom samples X = [samplemeaberoulli(p,) for i i rage(um_trials)] Xrouded = [roud(x,decimals) for x i X] # Now covert frequecy couts ito probabilities D = Couter( Xrouded ) Xrouded = uique(xrouded) # sorts ad removes duplicates P = [10**decimals*D[k]/um_trials for k i Xrouded] # must multiply probs by 10**decimals because plt.bar(xrouded,p,width=1/10**decimals, edgecolor='k',alpha=0.5) # bis are of width 1/10* *decimals # Now geerate the theoretical ormal for sample mea with std dev sigma/sqrt() mu = p sigma = (p*(1-p))**0.5/(**0.5) X2 = p.lispace(mu-sigma*3,mu+sigma*3,100) Y = [orm.pdf(x,mu,sigma) for x i X2] plt.plot(x2,y) plt.show() p = 0.8 = 1000 # try for 10, 100, 500, 1000 um_trials = display_sample_mea_beroulli(p,,um_trials,3) Experimet Five: Expoetial 8/10

10/31/2018 CetralLimitTheorem I [123]: def samplemeaexpoetial(lam,): X = [expoetial(1/lam) for i i rage()] retur sum(x)/ def display_sample_mea_expoetial(lam,,um_trials,decimals): fig, ax = plt.

9 10/31/2018 CetralLimitTheorem I [123]: def samplemeaexpoetial(lam,): X = [expoetial(1/lam) for i i rage()] retur sum(x)/ def display_sample_mea_expoetial(lam,,um_trials,decimals): fig, ax = plt.subplots(1,1,figsize=(12,6)) plt.title('exp('+str(lam)+'): Distributio of Sample Mea with = ' + str()) plt.ylabel("f(x)") plt.xlabel("k i Rage(X)") # use expoetial to geerate radom samples X = [samplemeaexpoetial(lam,) for i i rage(um_trials)] Xrouded = [roud(x,decimals) for x i X] # Now covert frequecy couts ito probabilities D = Couter( Xrouded ) Xrouded = uique(xrouded) # sorts ad removes duplicates P = [10**decimals*D[k]/um_trials for k i Xrouded] # must multiply probs by 10**decimals because plt.bar(xrouded,p,width=1/10**decimals, edgecolor='k',alpha=0.5) # bis are of width 1/10* *decimals # Now geerate the theoretical ormal for sample mea with std dev sigma/sqrt() mu = 1/lam sigma = 1/lam / (**0.5) X2 = p.lispace(mu-sigma*3,mu+sigma*3,100) Y = [orm.pdf(x,mu,sigma) for x i X2] plt.plot(x2,y) plt.show() lam = 0.1 = 100 # try for 1, 2, 5, 10, 30, 100 um_trials = display_sample_mea_expoetial(lam,,um_trials,1) Experimet Six: Normal 9/10

10/31/2018 CetralLimitTheorem I [126]: def samplemeanormal(mu,sigma,): X = ormal(mu,sigma,) retur sum(x)/ def display_sample_mea_ormal(mu,sigma,,um_trials,decimals): fig, ax = plt.

10 10/31/2018 CetralLimitTheorem I [126]: def samplemeanormal(mu,sigma,): X = ormal(mu,sigma,) retur sum(x)/ def display_sample_mea_ormal(mu,sigma,,um_trials,decimals): fig, ax = plt.subplots(1,1,figsize=(12,6)) plt.title('n('+str(mu)+','+str(sigma**2)+'): Distributio of Sample Mea with = ' + str( )) plt.ylabel("f(x)") plt.xlabel("k i Rage(X)") # use expoetial to geerate radom samples X = [samplemeanormal(mu,sigma,) for i i rage(um_trials)] Xrouded = [roud(x,decimals) for x i X] # Now covert frequecy couts ito probabilities D = Couter( Xrouded ) Xrouded = uique(xrouded) # sorts ad removes duplicates P = [10**decimals*D[k]/um_trials for k i Xrouded] # must multiply probs by 10**decimals because plt.bar(xrouded,p,width=1/10**decimals, edgecolor='k',alpha=0.5) # bis are of width 1/10* *decimals # Now geerate the theoretical ormal for sample mea with std dev sigma/sqrt() sigma = sigma / (**0.5) X2 = p.lispace(mu-sigma*3,mu+sigma*3,100) Y = [orm.pdf(x,mu,sigma) for x i X2] plt.plot(x2,y) plt.show() mu = 66 sigma = 3 = 100 # try for 1, 2, 5, 10, 30, 100 um_trials = display_sample_mea_ormal(mu,sigma,,um_trials,2) 10/10

7.1 Convergence of sequences of random variables

7.1 Convergence of sequences of random variables Chapter 7 Limit Theorems Throughout this sectio we will assume a probability space (, F, P), i which is defied a ifiite sequece of radom variables (X ) ad a radom variable X. The fact that for every ifiite