Econ 325/327 Notes on Sample Mean, Sample Proportion, Central Limit Theorem, Chi-square Distribution, Student s t distribution 1.

Transcription

1 Eco 325/327 Notes o Sample Mea, Sample Proportio, Cetral Limit Theorem, Chi-square Distributio, Studet s t distributio 1 Sample Mea By Hiro Kasahara We cosider a radom sample from a populatio. Defiitio 1. A radom sample of size is a sequece X 1,..., X of radom variables which are i.i.d., i.e. the X i s are idepedet ad have same probability mass fuctio (p.m.f) f X (x) if they are discrete or probability desity fuctio (p.d.f) f X (x) if they are cotiuous. I a radom sample of observatios, each of observatios is selected radomly from a populatio distributio. Suppose that {X 1, X 2,..., X } is a radom sample from a populatio, where E[X i ] = µ ad Var[X i ] =. We do ot assume ormality, amely, X i is idepedetly draw from some populatio distributio fuctio of which exact form is ot kow to us but we kow that E[X i ] = µ ad Var[X i ] =. The sample mea ad the sample variace are defied as X = 1 X i, s 2 = 1 1 m (X i X ) 2, respectively. Here, the subscript i X ad s 2 idicates that they are computed usig observatios. Note that both X ad s 2 are radom variables because X i s are radom variables. The expected value of X is E( X 1 ) = E X i = 1 E(X i ) = 1 µ = 1 (1) µ = µ, Therefore, X is a ubiased estimator of µ. 1 c Hiroyuki Kasahara. Not to be copied, used, revised, or distributed without explicit permissio of copyright ower. 1

2 The variace of X is (( X = Var( X ) = E 1 ) ) 2 ( X i µ = E 1 ( = E 1 2 ) 2 (X i µ) X i ) 2 (Defie X i = X i µ) = 1 2 E ( = 1 2 { E = 1 2 { E 1 X i X2 i j=i X2 i + 0 j=i+1 } X i Xj ) } E Xi Xj (because X i ad X j are idepedet) (2) = 1 2 (because E X2 i = Var(X i )) = ) = 1 2 σ2 = σ2. Note that E Xi Xj = E ((X i µ)(x j µ)) = Cov(X i, X j ) = 0 because X i ad X j are radomly draw ad, therefore, idepedet. The stadard deviatio of X is σ 2 σ X = = σ. This implies that, as the sample size icreases, the variace ad the stadard deviatio of X decreases. Cosequetly, the distributio of X will put more ad more probability mass aroud its mea µ as icreases ad, evetually, the variace of X shriks to zero as log as < ad the distributio of X will be degeerated at µ as goes to ifiity. This result is called the law of large umbers. See the ext sectio for details. It is importat to emphasize that we have E[ X ] = µ ad Var( X ) = σ2 eve whe the populatio distributio is ot ormal. However, without kowig the exact form of populatio distributio where X i is draw from, we do ot kow the distributio fuctio of X beyod E[ X ] = µ ad Var( X ) = σ ; i particular, X is ot ormally distributed i geeral whe is fiite. For example, if X i is draw from Berouilli distributio with X i = 1 with probability p ad X i = 0 with probability 1 p, the we still have E[ X ] = p ad Var( X ) = σ2 = p(1 p) but, give fiite, we do ot expect that X N(µ, /). O the other had, if we are willig to assume that the populatio distributio is ormal, i.e., X i N(µ, ), the we have that X N(µ, /). This is because that the average of idepedetly ad idetically distributed ormal radom variables is also a ormal radom variable. 2

3 The Law of Large Numbers The formal defiitio of the law of large umbers is as follows. Theorem 1 (The Law of Large Numbers). Let X = (1/) X i, where X i is idepedetly draw from the idetical distributio with fiite mea ad fiite variace. The, for every ɛ > 0, lim P ( X µ < ɛ) = 1. We say that X coverges i probability to µ, which is deoted as X p µ. That is, as the sample size icreases to ifiity, the probability that the distace betwee X ad µ is larger tha ɛ approaches zero for ay ɛ > 0 regardless of how small the value of ɛ is. I other words, the relative frequecy that X falls withi ɛ distace of µ is arbitrary close to oe whe the sample size is large eough. 2 This result is ituitive give that Var( X ) = / so that the variace of X shriks to zero as. The proof of the law of large umber uses Chebyshev s iequality. Chebyshev s iequality: Give a radom variable X with fiite mea ad fiite variace, for every ɛ > 0, we have P ( X E(X) ɛ) V ar(x). ɛ 2 Proof of the Law of Large Numbers: By choosig X = X i Chebyshev s iequality above, we have P ( X E( X ) ɛ) V ar( X ). Substitutig E( X ɛ 2 ) = µ ad V ar( X ) = /, we have P ( X µ ɛ) σ2 for every = 1, 2,... Because σ2 0 as ɛ 2 ɛ 2 for every ɛ > 0, we have lim P ( X µ ɛ) = 0 for every ɛ > 0. Therefore, lim P ( X µ > ɛ) = 1 lim P ( X µ ɛ) = 1 0 = 1. The Cetral Limit Theorem Cosider a radom variable Z defied as Z = X µ σ/. (3) This is the stadardized radom variable of X because E( X ) = µ ad V ar( X ) = /. The, it is easy to prove (try to prove yourself) that E(Z ) = 0 ad Var(Z ) = 1 for every = 1, 2,... Therefore, while X teds to degeerate to µ as, the stadardized variable Z does ot degeerate eve whe. Give fiite, the exact form of distributio of X, or 2 We may iterpret lim P ( X µ < ɛ) = 1 as follows. First, cosider a radom variable defied by X µ for each. Give some positive costat ɛ, we may evaluate the probability that this radom variable X µ is smaller tha ɛ, i.e., P ( X µ < ɛ). This is some umber betwee 0 ad 1. By cosiderig the case of = 1, 2, 3, ad so o, we have a sequece of umbers P ( X 1 µ < ɛ), P ( X 2 µ < ɛ), P ( X 3 µ < ɛ), ad so o. The Law of Large Number states that this sequece of umbers {P ( X µ < ɛ) : = 1, 2,...} coverges to 1 as for ay ɛ > 0, however small ɛ is, i.e., the probability that the radom variable X µ is smaller tha ay positive costat ɛ approaches oe. Therefore, for sufficietly large, almost all realized values of X are arbitrary close to µ. 3

4 Z, is ot kow uless we assume that X i is ormally distributed. What is the distributio of Z whe? The Cetral Limit Theorem provides the aswer. Theorem 2 (The Cetral Limit Theorem). If {X 1, X 2,..., X } is a radom sample with fiite mea µ ad fiite positive variace (i.e., < µ < ad 0 < < ), the the distributio of Z defied i (3) is N(0, 1) i the limit as, i.e., for ay fixed umber x, ( lim P X µ) σ/ x = Φ(x), where Φ(x) = x 1 2π e z2 /2 dz. The proof is beyod the scope of this course. We say that X µ σ/ to a ormal with mea 0 ad variace 1, which is deoted as X µ σ/ d N(0, 1). coverges i distributio Cosider a radom variable ( X µ) = σ X µ σ/, where the variace of ( X µ) is. Therefore, we ca equivaletly state that ( X µ) coverges i distributio to a ormal with mea 0 ad variace, i.e., ( X µ) d N(0, ). The cetral limit theorem is a importat result because may radom variables i empirical applicatios ca be modeled as the sums or the meas of idepedet radom variables. I practice, the cetral limit theorem ca be used to approximate the cdf of the meas of radom variable by the ormal distributio whe the sample size is sufficietly large. The cetral limit theorem applies to the case whe X i has discrete value, for example, X i is a Beroulli radom variable with its support {0, 1}. If Z is approximately distributed as N(0, 1) whe is large, the X should be approximately distributed as N(µ, σ/ ). Where does 1/4 ( X µ) coverge? Note that V ar( 1/4 ( X µ)) = ( 1/4 ) 2 V ar( X µ) = 1/2 ( /) = /. Therefore, the variace of 1/4 ( X µ) coverges to zero as. As a result, 1/4 ( X µ) coverges i probability to zero. Where does ( X µ) coverge? I this case, V ar(( X µ)) = 2 ( /) = so that V ar(( X µ)) diverges to. As a result, ( X µ) diverges to or. Multiplyig ( X µ) by, we have a radom variable that either degeerates to a poit or diverges to. 4

5 Sample Proportio Suppose that {X 1, X 2,..., X } is a radom sample, where X i takes a value of zero or oe with probability 1 p ad p, respectively. That is, X i = { 0 with prob. 1 p 1 with prob. p (4) The, from (1)-(2), the expected value ad the variace of the sample mea ˆp := X = (1/) X i are give by E( X ) = E(X) = xp x (1 p) 1 x = (0)(1 p) + (1)(p) = p, Var( X ) = Var(X) x=0,1 = p(1 p), where the last equality uses Var(X) = x=0,1 (x p)2 p x (1 p) 1 x = p(1 p). By the cetral limit theorem, the distributio of X is approximately ormal for large sample sizes ad the stadardized variable Z := X E( X ) Var( X ) = X p p(1 p)/ is approximately distributed as N(0, 1). Sample Variace ad Chi-square distributio Suppose that {X 1, X 2,..., X } is a radom sample from a populatio, where E[X i ] = µ ad Var[X i ] =. The sample variace are defied as s 2 = 1 1 (X i X ) 2, (5) respectively. Note that s 2 is a radom variable because X i s ad X are radom variables. As show i the Appedix of Chapter 6 i Newbold, Carlso, ad Thore, the expected value of the sample variace s 2 is E[s 2 ] = (6) so that the sample variace s 2 is a ubiased estimator of. The result that E[s 2 ] = does ot require the ormality assumptio, i.e., X i is ot ecessarily ormally distributed. I the defiitio of (5), we divide the sum of (X i X ) 2 by ( 1) rather tha. We may alteratively cosider the followig estimator of the populatio variace of X i : ˆ = 1 (X i X ) 2. What is the expected value of ˆ? Because ˆ = 1 (X i X ) 2 = (X i X ) 2 = 1 s2, the expected value of ˆ is give by E[ˆ ] = 1 E[s2 ] = 1 σ2, which is ot 5

6 equal to but strictly smaller tha. Therefore, ˆ is a (dowward) biased estimator of. O the other had, as, 1 1 so that the bias of ˆσ2 will disappear as ad, hece, we may use ˆ i place of s 2 whe is large. While E[s 2 ] = holds without assumig that X i s are draw from ormal distributio, it is ot possible to kow the exact form of the distributio of radom variable s 2 i geeral whe is fiite. Cosider a trasformatio of s 2 by multiplyig by 1 ad divide by : ( 1)s 2 = (X i X ) 2, (7) where the right had side is obtaied by pluggig s 2 = 1 1 (X i X ) 2 ito the left had side. This is a radom variable because X i s ad X are radom. If we are willig to assume that X i is draw from the ormal distributio with mea µ ad variace, the we may show that the radom variable ( 1)s2 has a distributio kow as the chi-square distributio with 1 degree of freedom which we deote by χ 2 1, i.e., ( 1)s 2 = χ 2 1. (8) The chi-square distributio with the r degree of freedom, deoted by χ 2 r, is characterized by the sum of r idepedet stadard ormally distributed radom variables Z1, 2 Z2, 2..., Zr 2, where Z i N(0, 1) ad Z i ad Z j are idepedet if i j for i, j = 1,..., r. Namely, W = Z1 2 + Z Zr 2 has a distributio that is χ 2 r. The proof for this is beyod the scope of this class but is available i Chapter 5.4 of Hogg, Tais, ad Zimmerma. I view of this characterizatio, if we cosider a versio of (7) by replacig X with µ, the (X i µ) 2 2 Xi µ = = Z i 2 σ where Z i N(0, 1) ad Z i ad Z j are idepedet if i j, ad therefore, is the sum of idepedet stadard ormally distributed radom variables which is χ 2. This is slightly differet from (8) because we replace X with µ i the defiitio of ( 1)s2. Whe we replace X with µ, oe degree of freedom is lost ad, as a result, ( 1)s2 = is distributed as χ 2 1 rather tha χ 2. For example, cosider the case of = 2. The, with X = (1/2)(X 1 + X 2 ), ( 1)s 2 = 2 (X i X ) 2 2 X1 X 2 =, 2σ (X i µ) 2 (X i X ) 2 where X 2σ 1 X 2 is a stadard ormal radom variable so that ( 1)s2 χ The last equality follows from 2 (X i X ) 2 = (X 1 X 2 + (X 2 X ) 2 = 2 X 1 X1+X ( X2 X1+X2 2 = X1 X 2 ) 2 ( 2 + X2 X 1 ) 2 ( 2 = 2 X1 X 2 ) 2 2 = X 2. 1 X 2 2 Fially, whe both X1 ad X 2 are idepedetly draw from N(µ, ), X 1 X 2 are ormally distributed with mea E(X 1 X 2 ) = µ µ = 0 ad variace Var(X 1 X 2 ) = Var(X 1 ) + Var(X 2 ) = 2 so that, by stadardizig X 1 X 2, we have X 2σ 1 X 2 N(0, 1). 6

7 Example 1 (Chi-square distributio with the degree of freedom 1 ad stadard ormal distributio). If Z is N(0, 1), the P ( Z < 1.96) = Usig the fact that Z 2 is the chi-square distributed with the degree of freedom 1, i.e., Z 2 = χ 2 1, what is the value of a i the followig equatio? P (χ 2 1 < a) = To aswer this, ote that P (χ 2 1 < a) = P (Z 2 < (1.96) 2 ) = P ( Z < 1.96) = Therefore, a = (1.96) 2 = Checkig the Chi-square table whe the degree of freedom equal to 1 cofirms this result. Questio: what is the value of b such that P (χ 2 1 < b) = 0.9? Try to aswer this usig the Stadard ormal table ad check the result with the Chi-square table. We may also derive the cumulative distributio fuctio ad the probability desity fuctio of chi-square radom variable with the degree of freedom 1 from the stadard ormal probability desity fuctio φ(z) = 1 2π e z2 /2 as follows. Let Z N(0, 1). The, the cumulative distributio fuctio of chi-square radom variable with the degree of freedom 1 is F χ(1) (a) = P (χ 2 1 < a) = P (Z 2 < a) = P ( a Z a) = a The probability desity fuctio ca be obtaied by differetiatig F χ(1) (a) as a φ(z)dz. f χ(1) (a) = df χ(1)(a) da = 1 2 a 1/2 φ( a) a 1/2 φ( a) = a 1/2 φ( a) = a 1/2 2π e a/2. I particular, f χ(1) (a) as a 0. Studet s t distributio Suppose that X 1, X 2,..., X are radomly sampled from N(µ, ). The for ay 2, the sample mea X = 1 X i is ormally distributed, X N(µ, /). If we stadardize X by subtractig mea µ ad dividig by variace /, we have stadard ormal variable, i.e., X µ σ/ N(0, 1). I practice, we do ot kow the variace of X i. I such a case, we might wat to use the sample variace s 2 = 1 1 (X i X ) 2 i place of the populatio variace to stadardize X as t = X µ s 2 /. (9) X This is called t-statistic. Note that the distributio of µ s 2 / is differet tha that of X µ σ/ because s 2 is a radom variable while is costat so that X µ cotais the additioal s 2 / source of radomess from s 2 X. I fact, the variace of µ s 2 / is larger tha that of X µ σ/. The t-statistic defied i (9) has the kow distributio called Studet s t distributio with ( 1) degrees of freedom. 7

8 Let Z N(0, 1) ad χ 2 v χ 2 (v) with v degrees of freedom, where Z ad χ 2 v are idepedet. The, a radom variable from Studet s t distributio with v degrees of freedom ca be costructed as t v = Z χ 2 v /v. (10) To see the coectio betwee (10) ad t-statistic defied i (9), divide both the umerator ad the deomiator of (9) by σ/ ad rearrage the terms to obtai t = X µ σ/ ( 1)s 2 /( 1). Note that X µ σ/ N(0, 1) while ( 1)s2 follows the chi-square distributio with ( 1) degrees of freedom. Therefore, by lettig Z = X µ σ/, χ2 1 = ( 1)s2, ad v = 1 i (10), we have that t = X µ s 2 / follows the Studet s t distributio with ( 1) degrees of freedom. A few commets: The importat assumptio to obtai Studet s t distributio is that X 1, X 2,..., X are radomly draw from N(µ, ). For example, if X i is a Beroulli radom variable with P (X i = 1) = p ad P (X i = 0) = 1 p, the the distributio of is ot Studet s X µ s 2 / t distributio because either X µ σ/ N(0, 1) or ( 1)s2 χ 2 1. We caot use Studet s t distributio whe X i is ot ormally distributed. As, we have s 2 p. This suggests that a t-statistic coverges i distributio to the stadard ormal distributio as. 8