2 Definition of Variance and the obvious guess

Transcription

1 1 Estimatig Variace Statistics - Math 410, 11/7/011 Oe of the mai themes of this course is to estimate the mea µ of some variable X of a populatio. We typically do this by collectig a sample of idividuals out of the populatio, {x 1,..., x } ad usig the sample mea x = x x as our estmate for µ. But i order to uderstad how good this estimate is, we also eed to be able to make some estimatio of the variace of X (occasioally we kow the variace of X but ot the mea µ, but most of the time, if we do t kow µ, we also do t kow the variace). Defiitio of Variace ad the obvious guess Recall that the defiitio of the variace of X: Let the size of the populatio be N (typically much larger tha our sample size). Label the value of X o the idividuals i the populatio y 1,..., y N (I m usig y so as ot to cofuse with the idividuals i the sample as described above). Let the mea of X (calculated from the etire populatio) be µ. The σ X = Var(X) = N (y i µ). (1) Of course makig this calculatio ivolves examiig the etire populatio (as does computig µ), which is ofte impossible or impractical. The obvious guess for how to estimate Var(X) is to use formula (1) replacig the etire populatio with the sample data {x 1,..., x }, replacig N with ad replacig µ with x. I other words, we might guess that s = (x i x) () is the best way to estimate σx from our data. I fact this guess is wrog. It is what is called biased ad will cosistetly uder estimate the variace of X.

2 The right way to estimate the variace from a sample is s = (x i x) 1. (3) This is of course still a estimate, but it is cetered aroud the actual populatio variace, ulike the estimate (). 3 What s wrog with the biased estimate ()? The easiest way to see that there is somethig wrog with () is to look at examples for small values of. 1. = 1. Here we take a sample of size 1. The x = x 1 ad () is always 0. So i this case, it cosistetly uderestimates the variace (well, uless the variable X happes to be costat). It is easy to see that the problem is caused by the fact that we had to use the sample mea istead of the populatio mea.. Here is a example with =. Let the radom variable X be the result of a fair toi coss, that is a Beroulli R.V. with π =.5. If you like, the populatio here is all coi tosses, ad X is 1 if the toss results i a head, ad 0 if tails. We kow the expected value of X is µ = E(X) =.5. We also kow (from a easy calculatio) that Var(X) =.5. So let s calculate the possibilities for s by lookig at all possible samples of two coi tosses. Our possible samples are {0, 0}, {0, 1}, {1, 0}, {1, 1}, each of which occurs with probability 1/4. {0,0}: I this case, x 1 = x = 0. so x is also 0, ad s = (0 0) + (0 0) = 0. {0,1}: I this case x =.5, so s = (0.5) + (1.5) =.5 {1,0}: As i the previous case, s =.5. {1,1}: As i the first case, s = 0.

3 So if we let S be the radom variable defied o all possible samples of size by formula (), we otice from the 4 cases cosidered above that S has two equally likely values: 0 ad.5. So E(S ) =.15. Note that this is half of Var(X), so is a rather bad estimate. (Oe ca repeat this with a Beroulli variable with some other value of π. The Var(X) = π(1 π). The four calculatios of s are the same, but they occur with probabilities (1 π), π(1 π), π(1 π) ad π respectively. So E(S ) =.5(π)(1 π) Agai, this is half of the actual value of Var(X), so a rather poor estimate.) 3. = 3. If you foud the previous example covicig, you may wat to skip this oe. Agai take X Beroulli with π =.5. As before E(X) =.5, Var(X) =.5. Now take three idividuals i your sample. The possible samples are {0, 0, 0}, {0, 0, 1}, {0, 1, 0}, {0, 1, 1}, {1, 0, 0}, {1, 0, 1}, {1, 1, 0}, {1, 1, 1}. The values oe gets for s are 0, 9, 9, 9, 9, 9, 9, 9, 0. Each of these outcomes are equally likely, so E(S 3) = = 1 6. Agai, E(S 3 ) < Var(X). The problem that comes up i all of these examples is caused by the eed to use the sample mea rather tha the populatio mea. The distace (i the least squares sese) from the sample data to the sample mea will be less tha the distace to the populatio mea. There is a patter to be observed i the above examples. I each case (there are oly 3), E(S) = 1 Var(X). (4)

4 This is suggestive. (Here S = (X i X) where X i is the variable (defied o samples of size ) which is the value of X o the ith idividual i the sample, ad X is the mea of the X i.) 4 The ubiased estimator for Var(X) Our guess from equatio (4) is that 1 E(S ) = Var(X). I other words, if we defie S = 1 S = (X i X) 1 (5) the we expect E(S ) = Var(X). This is ot especially difficult to prove. Theorem 1. Let S be defied as i (5). The E(S ) = Var(X). I other words, S is a ubiased estimator for Var(X). Proof. We ll eed to use: E(aX + by ) = ae(x) + be(y ) Var(aX) = a Var(X) Var(X 1 + X ) = Var(X 1 ) + Var(X ) (assumig of course that X 1 ad X are idepedet). We also use that µ = E(X) = E(X). The E(S ) = E ( 1 1 ) (X i X) ( ) = 1 1 E (Xi X i X + X )

5 We use the fact that X i = X, ad the last expressio above is the equal to [ 1 ] 1 E( Xi ) E(X )+E(X ) = 1 E(Xi ) E(X ) 1 = 1 [ ] E(X ) E(X ) = 1 1 (E(X ) E(X )) Now we remember that Var(X) = E(X ) E(X) so usig that i the expressio immediately above gives = [ Var(X) + E(X) Var(X) E(X) ] 1 = [Var(X) + µ 1 ] 1 Var(X X ) µ = 1 [Var(X) Var(X) ] = [ ] 1 1 Var(X) = Var(X). 5 Summary Here is what to keep i mid i applicatios. 1. If you are lucky eough to already kow the populatio variace for some reaso, the you do t eed to approximate it.. If you kow the populatio mea, µ, but ot the variace, the (x i µ) is your best ubiased approximatio to the populatio variace. 3. If you do t kow the populatio mea or the populatio variace, the the sample variace s = (x i x) 1 is your best ubiased approximatio to the populatio variace.

6 4. If you forget, ad use (x i x) = 1 s istead of s, it is ot likely to make much differece if is large, sice 1 is the close to 1. But you will be usig a estimate that teds to uderestimate the populatio variace.