Convergence theorems. Chapter Sampling

Transcription

1 Chaper Covergece heorems We ve already discussed he difficuly i defiig he probabiliy measure i erms of a experimeal frequecy measureme. The hear of he problem lies i he defiiio of he limi, ad his was se aside i favour of he axiomaic basis. Eve, wihi he axiomaic premise, we have several impora covergece heorems relaig o a large umber of repeaed rials (samplig). The wo fudameal heorems are: he law of large umbers he ceral limi heorem These covergece heorems are he foudaios of saisics. We ca cosider he subjec of Saisics as experimeal probabiliy. Thaishecollecioadaalysisofdaa (measuremes) o deermie properies of he uderlyig heory (probabiliy). These relaios uderpi ha subjec i ha hey suppor ha he measureme process (uder cerai codiios) will give progressively beer esimaes.. Samplig To simplify he discussio, suppose we have a discree radom variable, X. This radom variable has a kow (heoreical) mea ad sadard deviaio defied by: µ E (X), σ = var (X). (.0) ad calculaed from he (kow) probabiliy mass fucio. We will refer o hese heoreical quaiies as he rue mea ad rue sadard deviaio. Now cosider a experime i which his radom variable is sampled imes uder ideical codiios. The aim of his experimeal samplig migh be o ifer or deduce properies of his heoreical disribuio. Suppose he experime ivolves samples of he radom variable. For example, rolls of a die, or idividuals chose from a populaio. Le us deoe he (radom) value of X obaied from he ih sample as, X i.theoecagaherheseoucomesogeherihefollowigway.takehesample sum of hese variables: S X + X X i + + X. (.0) We ry o esure ha he idividual samples X i are idepede. If he sample is doe correcly (ad we wo go io wha ha meas here), hese values will be idepede ad ideically disribued. If his is he case, he hey will have he same commo mea (µ) advariace(σ 2 ). The i follows ha: E (S )=E (X + X X i + + X )=E (X i )=µ, (.0) ad ha: var (S )=var(x + X X i + + X )=.var (X i )=σ 2. (.0) 69

2 70 CHAPTER. CONVERGENCE THEOREMS.2 The weak law of large umbers Le us simply sae he law as a heorem ad he prese a proof. ( ) lim S µ > ε =0. (.0) for ay ε > 0. Proof: Firsly, we kow ha he heoreical expeced value of he sample mea is he rue mea, µ, amely: ( ) S E = µ (.0) ad ha: var ( ) S = σ2. (.0) Accordig o he Chebyshev iequaliy (0.40) we have: ( ) S P µ > ε σ2 / ε 2. (.0) The, for ay fiie ε, akighelimiofaifiielylargesample,,wehave: ( ) lim P S µ > ε =0. (.0) The meaig of he heorem is ha if we ake a sufficiely large sample ( ), he he sample average (experime) coverges wih ceraiy o he rue mea (heory). Tha is, he probabiliy of he sample average S / (ha is he experime) deviaig from µ (he heory) by a give amou ε (o maer how small) iszero,ihelimiofaifiielylargesample. This elucidaes he imporace of he heorem ad moreover quaifies he samplig error. So he larger he sample size, he more accurae he sample mea approximaes he rue mea. Such a coclusio validaes ay experimeal measuremes or observaios of radomprocesses. Ofcourse,hereisa pracical problem o eforcig he power of he heorem, he abiliy o do such experimes is severely limied by irisic samplig errors (bias) ad measureme errors. These are broadly lumped ogeher as saisical errors. There is a more impressive versio of his law: he srog law of large umbers, whichhasasublebu impora differece: ( ) P lim S µ > ε =0. (.0) for ay ε > 0. The reaso i is called he srog law is ha i imposes covergece i a much sroger sese. The weak law refers o he limi of he probabiliies, while he srog law describes he probabiliy of he limi. The proof of he srog law is more complicaed - ha s he price you pay for havig a beer law. Sudyig he deails of he proof are worh he effor. bu will o be covered here. Isead we progress o he secod opic, he ceral limi heorem..3 Ceral-limi heorem As before, he heorem will be saed ad he prove. Le {X,X 2,...X } be a se of idepede, ideically-disribued variables, wih E (X) = µ ad var (X) =σ 2.The,wihS X + X X,wehave: ( ) lim P (S /) µ σ/ z = Φ(z). (.0)

3 .3. CENTRAL-LIMIT THEOREM 7 where Φ(z) ishesadardormalprobabiliydisribuiofucio: Proof Φ(z) 2π z e 2 x2 dx. (.0) Wha we aim o show is ha he sample variables have ormal disribuios as he sample size icrease,.wewilldohisidirecly,byshowighahemome-geeraig fucio of a sample variable eds o he mome-geeraig fucio of a ormal disribuio. Sice here is a uique correspodece bewee he disribuio ad is mome-geeraig fucios, his covergece implies ha he uderlyig disribuios also coverge. Firsly, recall ha he sadard ormal probabiliy desiy is give by: φ(x) = 2π e 2 x2. (.0) wih E (X) =0advar(X) =. Thehecorrespodigmome-geeraig fucio is: M X () =E ( e X) = + e x 2 x2 dx. (.0) 2π This iegral ca be evaluaed by compleig he square: x 2 x2 = (x )2, ad he chagig variable x x, ifollowshahemome-geeraigfucioforhesadard ormal disribuio is: Now urig our aeio o he experime, give ha: se: M X () =E ( e X) = e (.0) S X + X X, Z (S /) µ σ/. (.0) By his rasformaio, he sample sum S is ow covered o Z,avariablewihmeaadvariace: E (Z )=0, var (Z )=. (.0) The aim is o show ha, i he limi,oolydoeshisvariablehavehesamemeaadvariace as he sadard ormal disribuio, bu i is ideical o he sadard ormal disribuio. We have previously meioed ha all we require for his equaliy is ha heir mome-geeraig fucios are he same. So le us cosider he mome-geeraig fucio of Z.Bydefiiiowehave: M Z () =E ( ( ( )) e X) = E exp σ (X i µ) (.) i= ( ( ) ( )) = E exp σ (X µ) exp σ (X µ) Ad sice {X,...,X } are idepede variables we ca wrie: ( ( )) ( ( )) M Z () =E exp σ (X µ) E exp σ (X µ) (.). (.) Give ha he X i are ideically disribued, we have: [ ( ( ))] M Z () = E exp σ (X µ). (.)

4 72 CHAPTER. CONVERGENCE THEOREMS So he mome geeraig fucio depeds o. OecamakeaMaclauriexpasioforheexpoeial fucio i as follows: ( ( )) ( ) ( E exp σ (X µ) = E () + E σ 2 ) (X µ) + E 2! σ 2 (X µ)2 ( 3 ) +E (X 3! σ 3 µ)3 +. 3/2 Now evaluaig he expecios of each erm i ur, ad oig hae (X) = µ by defiiio, his gives rise o: ( ( )) E exp σ (X µ) = ! σ 2 σ ! σ 3 E ( (X µ) 3) +. (.-2) 3/2 his fially gives: ( ( )) ( ) E exp σ (X µ) = O 3/2. (.-2) Noe, he symbol O, ihelasermmeas ofheorder. Thaishisermhassome fiie (uspecified) coefficie bu is proporioal o 3 3/2.Ahisjucureweakehelimi,whichleadsohe resul: [ ( ( ))] ( )] lim M Z () = lim E exp σ (X µ) = lim [ O (.-2) 3/2 This culmiaes wih he followig expressio for he mome-geeraig fucio. lim M Z () =e 2 2. (.-2) Tha is he variable Z has he same mome geeraig fucio as he sadard ormal disribuio: equaio.3. Cosequely he probabiliy disribuios areideical,ieveryaspec,ihislimi. ed of proof. I summary, he disribuio of he sample of ay discree radom variable, eds owards a ormal disribuio. Cosider how his works i pracice. For ay discree radom variable X wih mea, µ, ad(fiie) sadard deviaio, σ, hesample(sum)variable: obeys he followig relaio, for large, ( (S /) µ P σ/ S X + + X, (.-2) ) z Φ(z). (.-2) Equivalely, rearragig his we ca fid he probabiliy disribuio of S : ( ) s/ µ F S (s) =P (S s) =Φ σ/. (.-2) While S is, of course a discree variable, accordig o he ceral-limi heorem, for a large sample i is ca be approximaed by a coiuous ormal disribuio. Cosequely he probabiliy desiy will have he form: f S (s) = d ds F S (s) = ( ) s/ µ σ φ σ/ ha is:, (.-2) f S (s) = [ σ 2π exp 2 ( ) ] 2 S / µ σ/. (.-2)

5 .3. CENTRAL-LIMIT THEOREM 73 This is a coiuous variable, ad o cover his o he equivale probabiliy mass for he discree variable S,weusehefacha,bydefiiio P (s S s + s) =f S (s) s, (.-2) ad whe he ierval is, s =,asiwouldforacouigvariable,forexampleheumberofheads i osses of a coi, we would ge: [ P (S = s) =f S (s) = σ 2π exp ( ) ] 2 S / µ 2 σ/, S =0,, 2,..., (.-2) for he probabiliy mass fucio for he discree variable S EXAMPLE Aceraiflighcarriesaradomumberofpassegers,X, suchhaheaverageadvariacearegive by: E (X) =50, var (X) =00. (.-2) I a sample of 20 flighs: (a) Wha is he probabiliy ha he oal umber of passegers is less ha 950? (b) Calculae he probabiliy ha he passeger oal is exacly 00. SOLUTION Alhough =20isoaverylargeumber,le suseheceral-limiheorem ayway. The he oal umber of passgers (over all 20 flighs) we will call: The we have he correspodece: The o aswer (a) we are afer S = X + X X. (.-2) =20 µ =50 σ = 00 = 0. (.-2) P (S S) wih S =950. (.-2) This is equal o, makig he same chages o boh sides of he iequaliy: ( S / µ P (S S) =P (S / µ S/ µ) =P σ/ S/ µ ) σ/ (.-2) Ad, accordig o he ceral-limi heorem: ( ) ( ) S / µ P (S S) P σ/ 950/ / / Φ 0 0/ = Φ(.8) (.-2) 0 Now, due o he reflecio symmery of he sadard ormal disribuio (a eve fucio) we have he followig ideiies Φ( z) = Φ(z) (.-2) ad for his reaso, he published ables oly eed o provide valuesforz 0. From he ables: This provides he aswer o par (a). P (S 950) = Φ(.8) (.-2) To aswer (b) we reframe he quesio as, wha is he value of: P (S 00.5) P (S 009.5)? The, usig he same argumes, we arrive a he aswers: P (S 00.5) = Φ(0.2348) = , P(S 00.5) = Φ(0.224) = (.-2) Thus P (S =00) = (.-2)

6 74 CHAPTER. CONVERGENCE THEOREMS.4 Cofidece limis ad sadard error Suppose you wish o coduc a survey, for example o deermie hepopulariyofhegovermeor he fracio of he sude populaio ha smokes. I samplig a fiie umber of people here will ieviably be uceraiy (radomess) i he oucome. Noeheless i is possible (usig he ceral-limi heorem) o esimae he error i your esimaes based o he size of he samplig. Oe ca defie cofidece limis as he probabiliy of he correcess of your aswer (wihi aceraiaccuracy)based o he size of he sample. Le us se aside, for he prese, he formidable challege of choosigwhoosampleadhowosample hem, ad focus o he size of sample required o form a opiio. Tha is a differe kid (ad more challegig problem) of addiioal saisical error. Suppose, you wish o esimae some parameer, le s call i z, wihaceraiaccuracy(error),le scalliε, wihiadegreeofceraiy. Le scallheceraiy, or more sricly he probabiliy, p. Saisicias ed o ge a bi obsessed wih saisical ess ad p- values. How may people, would you eed o sample o ge a desired accuracy? Clearly,his will deped i some way o he size of error you are willig o olerae ad he degree of ceraiy you wish o impose. A cocree example will illusrae he role of he ceral-limi heorem i providig such a esimae. We cosider a yes/o quesio ad le us say ha you are coducig a survey o, le s say he fracio of people who smoke, or he perceage of people who we o he movies i he las moh, or he fracio of he populaio ha suppor he goverme. Suppose ha he rue value (ha is if we sampled he eire populaio) of people who say hey like classical music is 0 z. We seek a value for z which is accurae o ε = ±0.05 (his is o quie he same as sayig a 5% accuracy), ad we wa (a leas) p =90%forhecofidecelimi. Oe mus be careful abou he use of he erm cofidece i his coex. This does o mea we are 90% cofide ha he esimae is correc. I simply meas ha ie imes ou of e we expec his resul. The erm cofidece limi is a uforuae choice of words i his coex. So, he survey approaches idividuals ad, of hese, s people say hey like classical music. I mahemaical erms we wish o kow he value such ha he sample fracio, s/, hasaid yes (helike classical music) is such ha he probabiliy exceeds aceraivalue,p c (he cofidece limi): ( s ) P z ε p c. (.-2) If he aswer o he quesio is simply yes/o (Beroulli variable) he he heoreical mea would be z ad he sadard deviaio for ay idividual would be: σ = z( z). Thus we seek such ha: ( ε P σ/ s/ z σ/ +ε ) σ/ p c For a large sample, we ca use he ceral-limi heorem (ormal disribuio for he sample average), so ha: ( ) ε +2Φ σ/ p c Tha is oe would require a sample size give by: ( σ ) [ ( )] 2 2 Φ pc +. (.-2) ε 2 For he example discussed above, usig ables for he sadard ormal disribuio we fid Φ (0.95).645, ad sice σ 2 4 we have: 27. So, we should sample a leas =27people,ohavea90%cofidecelimi,haourexperimeal aswer o he value of z is wihi he error Thus he sample size, is a fucio of he accuracy desired (ε) aswellashecofidecerequired(p c ), ad is proporioal o he variace (σ 2 )ofhevariable. By he same oke, for a fixed cofidece value (p c ), he error (ε) ichoosigasampleofsize would be: ε σ ( ) p + Φ. (.-2) 2

7 .5. CONVERGENCE TO THE NORMAL DISTRIBUTION 75 This depedece is kow o every saisicia, ad ideed he expressio: σ (.-2) is so widely ecouered, i is called he sadard error. However, oe mus be careful o o quoe his expressio as he error. Tha is, i would be wrog o sae ha he esimae for z is limied wihi hese bouds. The sadard error oly has a meaig i erms of heormaldisribuio. Commoly, oe seeks cofidece a he 95% level, i which case oe ca be more precise: Φ (0.975).96 ad hus: ε σ.96. (.-2) Tha is he righ-had side provides a lower boud o he error. Therefore, while our samplig error dimiishes wih ceraiy as he sample size icreases, he rae of decrease i error is frusraig slow, amely 2.Somakighesample0imeslargerolyleadso afacor3ireducioisampleerror..5 Covergece o he ormal disribuio We have used he mome-geeraig fucio o prove he ceral-limi heorem. The power of his approach is ha i works for ay disribuio fucio. We ca show explicily he covergece o a ormal disribuio direcly from he probabiliy mass. Cosider a series of rials (such as coi ossig) where W i {0, } correspods o he ih oss of he coi producig a TAILS or HEADS, respecively. Oursampleishea(large)umberofcoiosses N, ad le us ake he sample radom variable as he oal umber of HEADS. O ay give oss he probabiliy of heads is p. Thus The clearly, he W i are idepede, so ha: as is well kow for a biomial disribuio. X W + W W. (.-2) E (X) =p, var (X) =pq. (.-2) The he probabiliy mass for X (he sample) is jus he biomial disribuio f X (x) =! x!( x)! px q x. We oe oe propery of his fucio of x. For p, he fucio reaches a sigle maximum, for a paricular value of x ad he falls away rapidly as x 0. or x. This is illusraed by he bar char (figure??). The correspodece wih he ormal disribuio has bee kow for a very log ime, a leas as far back as de Moivre i 733. Le us go hrough he argume here. Firs of all, we eed o cosider a very large sample size. Thus meas evaluaig large facorials ad Sirlig s formula (see appedix for derivaio) gives us a very good approximaio for his, amely ha:! 2π e. (.-2) Tha is l! 2 l(2π)+( + 2 )l. (.-2) Now, le us defie: g(x) =lf X (x) soha,forlarge, x ad x we have: g(x) [ 2 l(2π)+( + 2 )l ] [ 2 l(2π)+(x + 2 )lx x] [ 2 l(2π)+( x + 2 )l( x) ( x)] +x l p +( x)lq. (.-3) (.-2)

8 76 CHAPTER. CONVERGENCE THEOREMS =8 =20 =40 f X (x) Figure.: The biomial disribuio for p =0.4, =8, 20, 40. Noe ha he disribuio has a sigle maximum, ear x = p, ad ha as icreases, he shape begis o resemble he bell-shaped ormal disribuio. x Now, as idicaed i figure., f(x) adheceg(x), has a maximum value a some iermediae value of x. We ca fid he locaio of he maximum by fidig he soluio of: Tha is: 2 g (x) =0 (.-2) g (x) = l x x +l( x)++ 2 +lp l q. (.-2) ( x) Now for large x, ad x he wo erms i /x ad /( x) cabeeglecedoafairapproximaio. So he value of x where he maximum occurs is approximaely he soluio of: ( ) ( x0 )p l =0 (.-2) x 0 q Tha is: x 0 = p. (.-2) Recall ha his value is he mea of he biomial disribuio. We ow see ha his becomes he mode of he sample variable, X. Now he secod derivaive a his poi, g (x 0 ), is give by: g (x 0 ) x 0 x 0 = pq. (.-2) Clearly g (x 0 ) < 0, which cofirms ha he saioary poi x 0 is a maximum. The cosider he Taylor series of g(x) iheeighbourhoodofhemaximum: The, sice g (x 0 )=0ad, g(x) g(x 0 )+(x x 0 )g (x 0 )+ 2 (x x 0) 2 g (x 0 ) (.-2) g(x 0 )= 2 l(2π)+( + 2 )l +(p + 2 )lp (q + 2 )lq + p l p + q l q. (.-2) Tha is, afer simplificaio: The: g(x 0 )= 2 l(2π) 2 l 2 l pq (.-2) g(x) 2 l(2π) 2 l 2 l pq 2pq (x p)2. (.-2)

9 .5. CONVERGENCE TO THE NORMAL DISTRIBUTION 77 I follows ha, i he limi of a large sample, we have he approximaio: f X (x) ) (x p)2 exp ( 2π pq 2pq. (.-2) So we fid ha X has a ormal disribuio wih mea give by µ = p ad variace pq. We see ha he resul cocurs wih he ceral-limi heorem, sice for he geomeric variable The accordig o he ceral-limi heorem, i he form (.3) µ = p, σ = pq (.-2) P (S = s) 2πpq e (s p)2 2pq. (.-2) which agrees wih (.5).