Lecture 7 Limit Theorems: Law of Large Numbers and the Central Limit Theorem

Transcription

1 9.07 Itroductio to Statistics for Brai ad Cogitive Scieces Emery N. Brow Lecture 7 Limit Theorems: Law of Large Numbers ad the Cetral Limit Theorem I. Objectives Uderstad the logic behid the Law of Large Numbers ad its relatio to the frequecy iterpretatio of probability theory Uderstad how to prove the Weak Law of Large Numbers Uderstad the logic behid the Cetral Limit Theorem Uderstad how to prove a versio of the Cetral Limit Theorem Uderstad how to costruct approximatios to the distributio of the sums of radom variables usig the Cetral Limit Theorem Uderstad the coditios uder which the Gaussia distributio ca be used to approximate biomial ad Poisso probabilities II. Law of Large Numbers To motivate our study of the Law of Large Numbers we begi with a example. Example 7.. A Opiio Poll. A electio is to be coducted for Presidet of the Society for Neurosciece ad there are two cadidates for the office: Cadidate A ad Cadidate B. If we poll voters ad we assume that each idividual polled reports truthfully his/her votig itetio the how sure ca we be that the fractio of people who report that they will vote for Cadidate A is a good guess (estimate) of the umber who will actually vote for Cadidate A? Our ituitio is that the fractio of people who say they are goig to vote for Cadidate A should be a good guess of the proportio of votes that Cadidate A will get i the electio. Furthermore, the more people we poll, the better our guess should be. With what we kow already, we ca formalize our ituitio. If we let p deote the fractio of people who will vote for Cadidate A, we ca defie X to be the Beroulli radom variable which is if perso i is goig to vote for Cadidate A ad 0 otherwise. Let i i. i X X X is the fractio of people polled who vote for Cadidate A. Our ituitio is that X is a good guess for p. This ituitio is what we colloquially call the Law of Averages. To begi thikig formally about this problem we recall that a statistic is ay fuctio of a set of data. Because experimetal data are radom ad the statistic is a fuctio of the data, the statistic is also a radom variable. It therefore, has a probability desity or probability mass fuctio. If we observe radom variables X, X,..., X such that the X i 's are idepedet ad all have the same probability distributio, the the collectio X, X,..., X is called a radom sample or simply a sample. The sample mea is oe of the most basic statistics ad we recall that it is defied as

2 Remark 7.. If the mea ad variace of the Propositios 6. ad 6. we have X Xi. i (7.) X 's are respectively ad variace i, the by E( X ) E( Xi ) (7.) i Var( X ) Var( Xi ) (7.3) i X. (7.4) The Law of Large Numbers tells us that as the sample size icreases the probability that i i X X is close to the mea approaches. To prove this, we establish first two techical results. A. Some Techical Results Propositio 7. (Markov Iequality). If X is a radom variable that takes o oly oegative values, the for ay a 0 [ ] Pr( X a) EX. (7.5) a Proof: To see this ote that E[ X ] xf ( x) dx xf ( x) dx xf ( x) dx 0 0 a xf ( x) dx a f ( x) dx a Pr( x a). a Propositio 7. (Chebyshev s Iequality). If X is a radom variable with mea ad variace ad k 0, the Proof: Let Y X the Y 0 ad by the Markov Iequality a a (7.6) Pr{ X k}. (7.7) k EY ( ) (7.8) Pr{ X k} Pr{ Y k} Pr{ Y k } Havig established Chebyshev s Iequality we ca prove the Weak Law of Large Numbers. k k

3 B. The Law of Large Numbers Propositio 7.3 Law of Large Numbers (Weak). Let X, X,..., X be a idepedet, idetically distributed (i.i.d.) sample from a populatio with mea ad variace. The the probability that the differece betwee the sample mea ad true mea remais greater tha ay fiite amout goes to zero as the sample size goes to ifiity. Proof: If we pick 0, the we ca Usig Chebyshev s Iequality, for 0, Pr( X ) Pr( X ). (7.9) Var( X ) Pr( X ) 0 (7.0) as goes to ifiity. Hece, Pr( X ) Pr( X ). Example 7. The Law of Large Numbers ad the Behavior of the Sample Mea. To gai ituitio about what is occurrig with the Law of Large Numbers, we cosider (Figure 7A) the mea of observatios from a gamma distributio with ad (Figure 7A,colum ) ad a arbitrary distributio (Figure 7A, colum ) for,, 4,6 ad 400. We ca see that i the case of the gamma distributio (Figure 7A, colum ) as grows, the distributio of the sample mea gets more ad more tightly clustered aroud the true mea. Similarly, for the arbitrary distributio a aalogous pheomeo is observed (Figure 7A, colum ). 3

4 Figure 7A. Illustratio of the Law of Large Number by averagig observatios from a gamma distributio with ad (colum ) ad a bimodal distributio (colum ) for,,4,6 ad 400. Reproduced from Dekkig et al. (00). Example 7. (cotiued). Ca we use Chebyshev s Iequality to figure out how large should be such that we are 95% certai that the differece betwee X ad is less tha 0.005? We will aswer this questio i Homework Assigmet 6. Remark 7.. From the result i Propositio 7.3, we say that X coverges i probability to. The best result kow is the Strog Law of Large Numbers. It states that Pr( X 0). (7.) as log as exists. This meas X is guarateed to coverge to i the usual umerical sese. Therefore, we are correct i thikig that a observed X is close to the true mea. This is ot guarateed by covergece i probability. The statemet, the sample mea is close to the true mea does ot apply to a particular realizatio. I cotrast, the Strog Law of Large Numbers addresses what occurs to the result of a sigle realizatio of the data (Pawita, 00, pp. 3-3). Remark 7.3. What is apparet from Figure 7A is that the sample mea is gettig closer to the true mea. This is a first-order effect because it ivolves the mea (first momet). As the umber of samples became large i Figure 7A, the shape of the distributio became more symmetric or Gaussia-like. We might thik of this effect as a secod-order effect because it tells us somethig about the variability (secod momet) of the statistic or the shape of its distributio. The secod-order effect addresses the questio of what is the behavior of the radom variable X as it approaches its mea? That is, ca we characterize its probability desity? The Cetral Limit Theorem allows us to aswer these questios. III. The Cetral Limit Theorem A. Motivatio The Cetral Limit Theorem is the most importat theorem i probability theory ad cocers the behavior of the sample mea as grows idefiitely large. Before statig the Cetral Limit Theorem, we give a example to motivate its derivatio alog with some techical results we will eed to prove it. Example 7.3 A Fast Spikig Neuro. Cosider a euro with a costat rate that obeys a Poisso probability law. Assume that = 40 Hz. I a -secod iterval how likely are we to observe 40, 4, or 4 spikes? We first develop a Gaussia approximatio to compute this result the we compare it with the exact result computed directly from the Poisso probability mass fuctio with = 40 Hz. To costruct a Gaussia approximatio we study the behavior of a sequece of Poisso radom variables such that X Poisso( ). ad. We have that E( X) Var( X), ad hece, EX ( ) ad Var( X). Therefore, we must stadardize X to get a limitig distributio. Let 4

5 Z X E( X) (7.) [ Var( X )] Z X (7.3) We have EZ ( ) 0 ad Var( Z). We show that M () z t e which by the Cotiuity Theorem (Propositio 7.4, stated below) meas that F ( z ) F( z) ( z ). Recall from Lecture 6 (Result 6.7 ad Remark 6.0) that if X has mgf Mx () t ad Y a bx, the M y( t) e M x( bt). We have t ( e ) M () t e ad hece, x z t at a ad b X, Z t M z ( t) e M x ( t) t t ( e ) e e. (7.4) We have that t log M ( t) t ( e ). (7.5) z Recall that e x k k k0 3 x x x x......!! 3! (7.6) so that Eq. 7.5 ca be writte as t t t log M z ( t) t ( ) 3 3! t t t t ! t t (7.7) 5

6 t As, log Mz ( t) ad thus coclude that Fz ( z) Fz ( z) ( z ). t M ( t) e. By the Cotiuity Theorem stated below, we z I Homework Assigmet 6, we preset a alterative derivatio of this result by showig directly that as the Poisso pmf Pz ( z ) ( ) z the pdf of a stadard Gaussia radom variable. Applyig this result to our fast spikig euro i Example 7.3 we have Gaussia Approximatio 39.5 X 4.5 Pr(40 X 4) Pr Pr z (0.395) ( 0.079) (7.8) , where we have used the cotiuity correctio defied below i sectio III E. Exact Poisso Solutio 40, 4 or 4 k e (7.9) k40 Pr( X ) k! We see that i this case, the approximatio is very reasoable. B. Proof of the Cetral Limit Theorem To prove the Cetral Limit Theorem, we eed to state first a defiitio, a theorem ad a techical result. Defiitio 7.. If X,..., X is a sequece of radom variables with cumulative distributio fuctios F, F,... ad let X be a radom variable with distributio fuctio F. The sequece X coverges i distributio to X if lim F( x) lim Pr( X x) Pr( X x) F( x). (7.0) 6

7 Propositio 7.4 (Cotiuity Theorem). Let F be a sequece of cumulative distributio fuctios with correspodig momet geeratig fuctios M( t ). Let F( X ) be a cumulative distributio fuctio with momet geeratig fuctio Mt ( ). If M( t) M( t) for all t i a ope iterval cotaiig zero, the F ( X ) F( X ) at all cotiuity poits of F( X ). Result 7.. If a a, the lim ( a ) a e. (7.) This limit is established i advaced calculus courses. Propositio 7.5. Cetral Limit Theorem: Let X, X,..., X be a sequece of idepedet radom variables havig mea ad variace, commo distributio fuctio Fx ( ) ad momet geeratig fuctio Mt () defied i a eighborhood of zero. Let i i X X as ( X ) Pr( x) ( x). (7.) Proof: To simplify the calculatios, we assume 0. Let ad S Xi i Z S. (7.3) Because the X,..., X are idepedet, the mgf of S is ad by Result 6.7 ad Remark 6.0 M ( t) [ M ( t)] s t M z ( t) [ M ( )]. (7.4) The Taylor series expasio of M() s about 0 is 3 sm'''(0) M ( s) M (0) sm '(0) s M ''(0) (7.5) 3 Now M(0), E( X ) 0, M '(0) 0 ad M ''(0). We have 7

8 3 t t t M ( ) [ ( ) ( ) 3 t t (7.6) Note that t t 3 t (7.7) as. The same is true for ay term larger tha t 3. Hece, for large. By Result 7. ad Propositio 7.4 t Mz ( t) [ ] (7.8) t M () t e (7.9) z which is the mgf of a stadard Gaussia radom variable. C. Applicatios of the Cetral Limit Theorem Example 7.3 (cotiued). The Gaussia Approximatio to the Poisso Distributio. I the case of the Poisso distributio with parameter we take the Gaussia mea ad stadard deviatio to be ad respectively. This approximatio is cosidered to be quite accurate accordig to Port (994, p. 685) for 5. That is, we must have o average 5 evets per uit time for the Gaussia distributio to be a good approximatio to the Poisso distributio. Kass (006) says the approximatio is good for 5. We will ivestigate which is correct i Homework Assigmet 6. Hece cosider a oe secod time iterval ad a euro whose spike rate is o the order of 5 Hz. If its spikig activity obeys a Poisso process, the the distributio of the spikig activity o the oe secod iterval could be approximated by the Gaussia distributio. We see already that the approximatio is good for 40. What are the implicatios of these observatios for the recet paper by Wu et al. Neural Computatio (006) where Gaussia distributios are used to model the probability distributios of MI spikig activity? Example 7.4 The Gaussia Approximatio to the Biomial. If is large ad p is ot too close to either 0 or, the we ca approximate the cumulative distributio fuctio for the Biomial pmf as 8

9 k Pr( X k) ( ) (7.30) where p ad [ p( p)]. A commoly used rule of thumb that is somewhat coservative at least for 0. p 0.8 is that it is reasoably accurate provided p 5 ad ( p ) 5. How does this compare to the rule for the Poisso approximatio to the biomial? Example. (cotiued). Let us retur to the Graybiel example to illustrate the effect of the CLT i the case of Beroulli trials (which ultimately provides the Gaussia approximatio to the biomial distributio). For 40 trials we have 40 values of 0 (icorrect resposes) ad (correct resposes). Suppose p 0.55 the we would expect i today s experimet there would be correct resposes. However, whe we examied the distributio of the sample mea (the proportio of correct resposes) it tured out to appear very close to Gaussia. See Figure D. We ca study the cetral limit theorem by simulatio usig the biomial distributio. Let us pick p 0., sice for p 0.55, the biomial pmf i Figure D looked very Gaussia. From Example., this distributio could be motivated by the observatio that o the first day of the experimet, the aimal had 8 correct out of 40. Therefore, we might assume that o the secod day, the probability of a correct respose is 0. for ay trial. We begi with Xi ' s beig equal to either 0 or. For example, with 40, we would have 40 values of 0 or. Suppose that p 0., so that we would expect, o average, 8 of the 40 values to be s. Imagie a histogram of these 0 s ad s. Certaily with oly two bis, these 40 values would ot look Gaussia. However, whe we look at the distributio of the sample mea, it turs out to look very close to Gaussia. Here, from our iitial 40 0 s ad s, we would obtai oe value of the sample mea, which would be the proportio of s i the sample. Now we have to imagie havig a large umber of such samples, say 00 of them. That is, we take 00 samples of size 40 ad for each sample we compute the mea of the 40 observatios. If we made a histogram of those 00 sample meas we would get somethig icely approximatig a Gaussia distributio. We ca study the theoretical distributio of the sample mea. For istace, for 4 the possible values of X are 0, , 0.75 ad ad ca be plot p( X 0), p( X 0.5),..., p( X ) versus 0, , 0.75 ad. This is show i the first plot i Figure 7B. This plot essetially tells us what the heights of the histogram bar would be if we did have a histogram of the 00 sample meas for a sample size of 4. The distributio of X does ot look very close to Gaussia. However, as the rest of Figure 7B shows, as icreases, the distributio of X looks more ad more Gaussia. What we have just doe is study the distributio of X for Beroulli trials for several values of with p 0.0. The distributio of S x is Biomial ad the picture of its distributios would look just like the pictures we had for the distributio of X except the x-axis would be multiplied by. I particular, as gets large, we see the distributio looks Gaussia. This effect of the CLT may be cosidered as a explaatio for the Gaussia approximatio to the biomial. i i 9

10 These results have allowed us to obtai a approximate distributio for oe of our most importat statistics, amely the sample mea. Ca we use the CLT to predict a reasoable (most probable) rage for the umber of correct resposes we ca expect to see i today s experimet if the aimal does ot lear? Alteratively, ca we combie the CLT with the rule to costruct a predictive cofidece iterval for p i today s experimet? How well does the rule for assessig the accuracy of the Gaussia approximatio to the biomial apply? That is, based o the rule for applicatio of the Gaussia approximatio to the biomial, would we have predicted that the Gaussia approximatio would have worked well for p 0.55 ad 40? How good is the approximatio? Ca we quatify the degree of agreemet or goodessof-fit? We will aswer these questios over the course of the ext couple of weeks. Figure 7B. Cetral Limit Theorem illustrated for the biomial 4,0, 5,00 ad p 0.. D. Normalizatio i the Cetral Limit Theorem The ormalizatio for the sum of the radom variables i the cetral limit theorem is. To illustrate the importace of the choice of ormalizatio, we cosider (Figure 7C) sums of radom variables from a gamma distributio with ad, ad a ormalizatio of (colum ), of (colum ) ad of (colum 3) for,, 4,6 ad 00. Whe the 4 ormalizatio is (Figure 7C, colum ), we see that while the distributio is fairly symmetric by the time 6 the value of the distributio at the mode is tedig to ifiity. At the other extreme whe the ormalizatio is (Figure 7C, colum 3), the radom behavior i the process is completely squashed by the time 6. I cotrast, whe the ormalizatio is (Figure 7C, colum ), we see that the behavior of the process cotiues to become symmetric ad it remais radom as, suggestig that is the correct ormalizatio. 4 0

11 Figure 7C. Rate of Covergece for the average of radom draws from a gamma distributio with ad ad a ormalizatio of Dekkig et al. (00). 4 (colum ), of (colum ) ad of (colum 3). Reproduced from E. Cotiuity Correctio Suppose that X is a discrete radom variable takig o oly iteger values ad that the pmf of X is approximated as a Gaussia radom variable with mea ad variace. The if a ad b are itegers, the correctio of cotiuity to use the Gaussia distributio is Pr( X b) ([ b / ]/ ) Pr( a X ) ([ a / ]/ ) (7.3) Pr( a X b) ([ b / ]/ ) ([ a / ]/ ). istead of Pr( X b) ([ b ]/ ) Pr( a X ) ([ a ]/ ) (7.3) Pr( a X b) ([ b ]/ ) ([ a ]/ ).

12 respectively. We applied the cotiuity correctio i the Gaussia approximatio to the Poisso distributio i Example 7.3. F. A More Geeral Cetral Limit Theorem ad a Example of Cetral Limit Behavior i a Physical System We ca state a more geeral versio of the Cetral Limit Theorem. Propositio 7.6. Cetral Limit Theorem (Geeral). If X, X,..., X are idepedet radom variables, possibly havig differet distributios but with o idividual X makig a domiat cotributio to the mea X, the for sufficietly large, the distributio of X is approximately Gaussia with mea X ad stadard deviatio Var( X ). This versio of the Cetral Limit Theorem helps to explai why the Gaussia distributio arises so ofte i statistical theory, ad also why it seems to fit, at least roughly, so may observed pheomea. It says that wheever we average a large umber of small idepedet effects, the result will be distributed as a Gaussia radom variable. While the Cetral Limit Theorem ivolves the sample mea, it drives the large-sample behavior of most statistics: a statistic derived from a sample may usually be writte, approximately, as some fuctio (possibly a complicated fuctio) of the sample mea, ad that usually produces approximate Gaussia ature of the statistic itself. A importat example is the maximum likelihood estimator, which is very widely used, ad which we will discuss i Lecture 9. Example 3. Magetoecephalography (cotiued). Why should the MEG measuremets i Figure 3E be distributed as a Gaussia radom variable? It is a empirical observatio that whe the magetic field iside the shielded eviromet is measured at a give SQUID sesor the distributio of the backgroud measuremets is Gaussia. The cetral limit theorem offers a theoretical reaso for this observatio. To see this, we ote that MEG measures the magetic fields i a give locatio. Aywhere there are curret dipoles magetic fields are geerated. The curret dipoles that we are iterested i are those emaatig from the brai. Remember that magetic fields or forces are a byproduct of electric currets or movig charged particles. I ay eviromet there are always electric currets flowig aroud. The SQUID sesors are 5 exquisitely sesitive. The fields they detect are o the order of 0 Tesla (femtotesla). Therefore, if you imagie the local magetic fields beig recorded ear a give SQUID sesor i a room desiged to have a homogeeous magetic field, there are a lot of local curret fluctuatios givig rise to local magetic field fluctuatios. If the eviromet is homogeeous, the o oe local fluctuatio will domiate (Lideberg-Feller coditio) ad the local magetic field is simply the vector sum of the local fluctuatios. This later poit gives the liear superpositio. Hece, Gaussia structure is ot surprisig. Moreover, ote that the observatios are ot idepedet (See Homework Assigmet 6). IV. Summary The Law of Large Numbers states that as the umber of observatios i a sample of data icreases the sample mea coverges to the populatio mea whereas the Cetral Limit Theorem tells us that sums of radom variables properly ormalized ca be approximated as a Gaussia distributio. This completes our sectio o probability theory. I the ext half of the course we will study statistical theory ad its applicatio to several problems i brai ad cogitive scieces makig uses of the several results i probability theory we have developed. i

13 Ackowledgmets I am grateful to Paymo Hosseii for makig Figures 7A ad 7C ad to Uri Ede for makig Figure 7B, to Julie Scott ad Riccardo Barbieri for techical assistace i preparig this lecture ad to Jim Mutch for very helpful proofreadig ad commets. Sectios of this lecture have bee take with permissio from the class otes for Statistical Methods i Neurosciece ad Psychology writte by Robert Kass i the Departmet of Statistics at Caregie Mello Uiversity. Figures 7A ad 7C are reproduced from Dekkig et al. (00). Refereces DeGroot MH, Schervish MJ. Probability ad Statistics, 3rd editio. Bosto, MA: Addiso Wesley, 00. Dekkig FM, Kraaikamp C, Lopuhaa HP, Meester LE. A Moder Itroductio to Probability ad Statistics. Lodo: Spriger-Verlag, 00. Pawita Y. I All Likelihood: Statistical Modelig ad Iferece Usig Likelihood. Lodo: Oxford, 00. Port SC. Theoretical Probability for Applicatios. New York: Joh Wiley & Sos, 994. Rice JA. Mathematical Statistics ad Data Aalysis, 3 rd editio. Bosto, MA, 007. Literature Referece Wu W, Gao Y, Bieestock E, Dooghue JP, Black MJ. Bayesia populatio decodig of motor cortical activity usig a Kalma filter. Neural Computatio 8():80-8, 006. Website Referece Kass RE, Chapter : Probability ad Radom Variables 3

14 MIT OpeCourseWare Statistics for Brai ad Cogitive Sciece Fall 06 For iformatio about citig these materials or our Terms of Use, visit: