Colin Cameron: Asymptotic Theory for OLS

Transcription

1 Colin Cameron: Asymtotic Theory for OLS. OLS Estimator Proerties an Samling Schemes.. A Roama Consier the OLS moel with just one regressor y i = βx i + u i. The OLS estimator b β = ³ P P i= x iy i canbewrittenas P i= bβ = β + x iu i P. Then uner assumtions given below (incluing E[u i x i ]=0) bβ β + lim P i= x iu i lim P 0 = β + lim P = β. It follows that β b is consistent for β. An uner assumtions given below (incluing E[u i x i ]=0an V[u i x i ]=σ 2 ) P i ( β β) b = i= x iu i P h0,σ 2 ³ lim P lim P It follows that b β is asymtotically normal istribute with b β a µ "0,σ 2 lim X #. ³ P 0,σ Samling Schemes an Error Assumtions The key for consistency is obtaining the robability of the two averages (of x i u i an of x 2 i ), by use of laws of large numbers (LL). An for asymtotic normality the key is the limit istribution of the average of x i u i, obtaine by a central limit theorem (CLT). Different assumtions about the stochastic roerties of x i an u i lea to ifferent roerties of x 2 i an x iu i an hence ifferent LL an CLT. For the ata ifferent samling schemes assumtions inclue:. Simle Ranom Samling (SRS). SRSiswhenweranomlyraw(y i,x i ) from the oulation. Then x i are ii. So x 2 i are ii, an x iu i are ii if the errors u i are ii.

2 2. Fixe regressors. This occurs in an exeriment where we fixthex i an observe the resulting ranom y i.givenx i fixe an u i ii it follows that x i u i are ini (even if u i are ii), while x 2 i are nonstochastic. 3. Exogenous Stratifie Samling This occurs when oversamle some values of x an unersamle others. Then x i are ini, so x i u i are ini (even if u i are ii) an x 2 i are ini. The simlest results assume u i are ii. In ractice for cross-section ata the errors may be ini ue to conitional heteroskeasticity, with V[u i x i ]=σ 2 i varying with i. 2. Asymtotic Theory for Consistency Consier the limit behavior of a sequence of ranom variables b as.this is a stochastic extension of a sequence of real numbers, such as a =2+(3/). Examles inclue: () b is an estimator, say b θ; (2) b is a comonent of an estimator, such as P i x iu i ;(3)b is a test statistic. 2.. Convergence in Probability, Consistency, Transformations Due to samling ranomness we can never be certain that a ranom sequence b,such as an estimator b θ, will be within a given small istance of its limit, even if the samle is infinitely large. But we can be almost certain. Different ways of exressing this almost certainty correson to ifferent tyes of convergence of a sequence of ranom variables to a limit. The one most use in econometrics is convergence in robability. Recall that a sequence of nonstochastic real numbers {a } converges to a if for any ε>0, there exists = (ε) such that for all >, a a <ε. e.g. if a =2+3/, then the limit a =2since a a = 2+3/ 2 = 3/ <ε for all > =3/ε. For a sequence of r.v. s we cannot be certain that b b <ε,evenforlarge, ue to the ranomness. Instea, we require that the robability of being within ε is arbitrarily close to one. Thus {b } converges in robability to b if lim Pr[ b b <ε]=, for any ε>0. A formal efinition is the following. 2

3 Definition A: (Convergence in Probability) A sequence of ranom variables {b } converges in robability to b if for any ε>0 an δ>0, there exists = (ε, δ) such that for all >, Pr[ b b <ε] > δ. We write lim b = b, where lim is short-han for robability limit, orb b. The limit b may be a constant or a ranom variable. The usual efinition of convergence for a sequence of real variables is a secial case of A. For vector ranom variables we can aly the theory for each element of b. [Alternatively relace b b by the scalar (b b) 0 (b b) =(b b ) (b K b K ) 2 or its square root b b.] ow consier {b } to be a sequence of arameter estimates b θ. Definition A2: (Consistency) An estimator b θ is consistent for θ 0 if lim b θ = θ 0. Unbiaseness ; consistency. Unbiaseness states E[ b θ]=θ 0. Unbiaseness ermits variability aroun θ 0 that nee not isaear as the samle size goes to infinity. Consistency ; unbiaseness. e.g. a / to an unbiase an consistent estimator - now biase but still consistent. A useful roerty of lim is that it can aly to transformations of ranom variables. Theorem A3: (Probability Limit Continuity). Let b be a finite-imensional vector of ranom variables, an g( ) be a real-value function continuous at a constant vector oint b. Then b g(b ) g(b). b This theorem is often referre to as Slutsky s Theorem. We instea call Theorem A2 Slutsky s theorem. Theorem A3 is one of the major reasons for the revalence of asymtotic results versus finite samle results in econometrics. It states a very convenient roerty that oes not hol for exectations. For examle, lim(a,b ) = (a, b) imlies lim(a b ) = ab, whereas E[a b ] generally iffers from E[a]E[b]. Similarly lim[a /b ]=a/b rovie b 6= 0. 3

4 2.2. Alternative Moes of Convergence It is often easier to establish alternative moes of convergence, which in turn imly convergence in robability. [However, laws of large numbers, given in the next section, areusemuchmoreoften.] Definition A4: (Mean Square Convergence) A sequence of ranom variables {b } is sai to converge in mean square to a ranom variable b if lim E[(b b) 2 ]=0. We write b m b. Convergence in mean square is useful as b m b imlies b b. Another result that can be use to show convergence in robability is Chebychev s inequality. Theorem A5: (Chebyshev s Inequality) For any ranom variable Z with mean µ an variance σ 2 Pr[(Z µ) 2 >k] σ 2 /k, for any k>0. A final tye of convergence is almost sure convergence (enote as ). This is concetually ifficult an often har to rove. So ski. Almost sure convergence (or strong convergence) imlies convergence in robability (or weak convergence) Laws of Large umbers Laws of large numbers are theorems for convergence in robability (or almost surely) in the secial case where the sequence {b } is a samle average, i.e. b = X where X = X X i. i= ote that X i here is general notation for a ranom variable, an in the regression context oes not necessarily enote the regressor variables. For examle X i = x i u i. A LL is much easier way to get the lim than use of Definition A or Theorems A4 or A5. Wiely use in econometrics because the estimators involve averages. Definition A7: (Law of Large umbers) Aweak law of large numbers (LL) secifies conitions on the iniviual terms X i in X uner which ( X E[ X ]) 0. 4

5 For a strong law of large numbers the convergence is instea almost surely. If a LL can be alie then lim X = lim E[ X ] in general = lim P i= E[X i] if X i ineenent over i = µ if X i ii. Leaing examles of laws of large numbers follow. Theorem A8: (Kolmogorov LL )Let{X i } be ii (ineenent an ientically istribute).ifanonlyife[x i ]=µ exists an E[ X i ] <, then( X µ) as 0. Theorem A9: (Markov LL )Let{X i } be ini (ineenent but not ientically istribute) with E[X i ]=µ i an V[X i ]=σ 2 i. If P i= (E[ X i µ i +δ ]/i +δ ) <, for some δ>0, then( X P i= E[X i]) as 0. The Markov LL allows nonientical istribution, at exense of require existence of an absolute moment beyon the first. The rest of the sie-conition is likely to hol with cross-section ata. e.g. if set δ =,thenneevariancelus P i= (σ2 i /i2 ) < which haens if σ 2 i is boune. Kolmogorov LL gives almost sure convergence. Usually convergence in robability is enough an we can use the weaker Khinchine s Theorem. Theorem A8b: (Khinchine s Theorem) Let{X i } be ii (ineenent an ientically istribute). If an only if E[X i ]=µ exists, then ( X µ) 0. Which LL shoul I use in regression alications? It eens on the samling scheme. 3. Consistency of OLS Estimator Obtain robability limit of b β = β +[ P i= x iu i ]/[ P ]. 3.. Simle Ranom Samling (SRS) with ii errors Assume x i ii with mean µ x an u i ii with mean 0. As x i u i are ii, aly Khinchine s Theorem yieling P i x iu i E[xu] =E[x] E[u] = 0. As x 2 i are ii, aly Khinchine s Theorem yieling P i x2 i E[x 2 ] which we assume exists. 5

6 By Theorem A3 (Probability Limit Continuity) lim[a /b ]=a/b if b 6= 0.Then lim β b = β + lim P i= x iu i lim P = β + 0 E[x 2 ] = β Fixe Regressors with ii errors Assume x i fixe an that u i ii with mean 0 an variance σ 2. Then x i u i areiniwithmeane[x i u i ]=x i E[u i ]=0an variance V[x i u i ]=x 2 i σ2. Aly Markov LL yieling P i x iu i P i E[x iu i ] 0, so P i x iu i 0. The sie-conition with δ =is P σ2 /i 2 which is satisfie if x i is boune. We also assume lim P i x2 i exists. By Theorem A3 (Probability Limit Continuity) lim[a /b ]=a/b if b 6= 0.Then lim β b = β + lim P i= x iu i lim P 0 = β + lim P = β Exogenous Stratifie Samling with ii errors Assume x i ini with mean E[x i ] an variance V[x i ] an u i ii with mean 0. ow x i u i areiniwithmeane[x i u i ]=E[x i ]E[u i ]=0an variance V[x i u i ]=E[x 2 i ]σ2, so nee Markov LL. This yiels P i x iu i 0, with the sie-conition satisfie if E[x 2 i ] is boune. An x 2 i are ini, so nee Markov LL with sie-conition that requires e.g. existence an bouneness of E[x 4 i ]. Combining again get lim β b = β. 4. Asymtotic Theory for Asymtotic ormality Given consistency, the estimator b θ has a egenerate istribution that collases on θ 0 as. Socannotostatisticalinference. [Ineethereisnoreasontooitif.] ee to magnify or rescale b θ to obtain a ranom variable with nonegenerate istribution as. 4.. Convergence in Distribution, Transformation Often the aroriate scale factor is, so consier b = ( b θ θ 0 ). b has an extremely comlicate cumulative istribution function (cf) F. But like any other function F it may have a limit function, where convergence is in the usual (nonstochastic) mathematical sense. 6

7 Definition A0: (Convergence in Distribution) A sequence of ranom variables {b } is sai to converge in istribution to a ranom variable b if lim F = F, ateverycontinuityointoff,wheref is the istribution of b, F is the istribution of b, an convergence is in the usual mathematical sense. We write b b, ancallf the limit istribution of {b }. b b imlies b b. In general, the reverse is not true. But if b is a constant then b b imlies b b. To exten limit istribution to vector ranom variables simly efine F an F to be the resective cf s of vectors b an b. A useful roerty of convergence in istribution is that it can aly to transformations of ranom variables. Theorem A: (Limit Distribution Continuity). Letb be a finite-imensional vector of ranom variables, an g( ) be a continuous real-value function. Then b b g(b ) g(b). (4.) This result is also calle the Continuous Maing Theorem. Theorem A2: (Slutsky s Theorem) Ifa a an b b, where a is a ranom variable an b is a constant, then (4.2) (i) a + b a + b (ii) a b ab (iii) a /b a/b, rovie Pr[b =0]=0. Theorem A2 (also calle Cramer s Theorem) ermits one to searately fin the limit istribution of a an the robability limit of b, rather than having to consier the joint behavior of a an b. Result (ii) is esecially useful an is sometimes calle the Prouct Rule Central Limit Theorems Central limit theorems give convergence in istribution when the sequence {b } is a samle average. A CLT is much easier way to get the lim than e.g. use of Definition A0. 7

8 By a LL X has a egenerate istribution as it converges to a constant, lime[ X ]. So scale ( X E[ X ]) by its stanar eviation to construct a ranom variable with unit variance that will have a nonegenerate istribution. Definition A3: (Central Limit Theorem) Let Z = X E[ X ], V[ X ] where X is a samle average. A central limit theorem (CLT) secifies the conitions on the iniviual terms X i in X uner which Z [0, ], i.e. Z converges in istribution to a stanar normal ranom variable. ote that Z =( X E[ X ])/ V[ X ] in general = P q i= (X P i E[X i ])/ i= V[X i] if X i ineenent over i = ( X µ)/σ if X i ii. If X satisfies a central limit theorem, then so too oes h() X for functions h( ) such as h() =,since Z = h() X E[h() X ]. V[h() X ] Often aly the CLT to the normalization X = /2 P i= X i,sincev[ X ] is finite. Examles of central limit theorems inclue the following. Theorem A4: (Lineberg-Levy CLT )Let{X i } be ii with E[X i ]=µ an V[X i ]=σ 2. Then Z = ( X µ)/σ [0, ]. Theorem A5: (Liaounov CLT )Let{X i } be ineenent with E[X i ] = µ i an ³ P ³ V[X i ]=σ 2 i. If lim i= E[ X P (2+δ)/2 i µ i 2+δ ] / i= σ2 i =0, for some choice of δ>0, thenz = P i= (X i µ i )/ q P i= σ2 i [0, ]. Linberg-Levy is the CLT in introuctory statistics. For the ii case the LL require µ exists, while CLT also requires σ 2 exists. 8

9 For ini ata the Liaounov CLT aitionally requires existence of an absolute moment of higher orer than two. Which CLT shoul I use in regression alications? It eens on the samling scheme. 5. Limit Distribution of OLS Estimator Obtain limit istribution of ( b β β) =[ P i= x iu i ]/[ P ]. 5.. Simle Ranom Samling (SRS) with ii errors Assume x i ii with mean µ x anseconmomente[x 2 ], an assume u i ii with mean 0 an variance σ 2. Then x i u i are ii, with mean 0 an variance σ 2 E[x 2 ].[Proofforvariance:V x,u [xu] = E x [V[xu x]]+v x [E[xu x]] = E x [x 2 σ 2 ]+0 = σ 2 E x [x 2 ]. Aly Lineberg-Levy CLT yieling à P! i= x iu i 0 = σ 2 E[x 2 ] P i= x iu i σ 2 E[x 2 ] [0, ]. Using Slutsky s theorem that a b a b (for a a an b b), this imlies X i= x iu i [0,σ 2 E[x 2 ]]. Then using Slutsky s theorem that a /b a/b (for a a an b b) ( b β β) = P i= x iu i P 0,σ 2 E[x 2 ] lim P 0,σ 2 E[x 2 ] E[x 2 ] where we use result from consistency roof that lim P =E[x2 ] Fixe Regressors with ii errors h 0,σ 2 E[x 2 ] i, Assume x i fixe an an u i ii with mean 0 an variance σ 2. Then x i u i areiniwithmean0 an variance V[x i u i ]=x 2 i σ2. Aly Liaounov LL yieling P i= x iu i 0 q lim P σ2 = 9 P i= x iu i q σ 2 lim P [0, ].

10 Using Slutsky s theorem that a b a b (for a a an b b), this imlies X i= x iu i [0,σ 2 lim X ]. Then using Slutsky s theorem that a /b a/b (for a a an b b) P i ( β β) b = i= x iu i P h 0,σ 2 lim P lim P 5.3. Exogenous Stratifie Samling with ii errors Assume x i ini with mean E[x i ] an variance V[x i ] an u i ii with mean 0. Similar to fixe regressors will nee to use Liaounov CLT. We will get µ ( bβ β) "0,σ 2 lim X #. µ "0,σ 2 lim X #. 6. Asymtotic Distribution of OLS Estimator From consistency we have that β b has a egenerate istribution with all mass at β, while ( β b β) has a limit normal istribution. For formal asymtotic theory, such as eriving hyothesis tests, we work with this limit istribution. But for exosition it is convenient to think of the istribution of β b rather than ( β b β). We o this by introucing the artifice of "asymtotic istribution". Secifically we consier large but not infinite, an ro the robability limit in the receing result, so that " µ ( β b X # β) 0,σ 2. It follows that the asymtotic istribution of β b is " µ X # bβ a β,σ 2. ote that this is exactly the same result as we woul have got if y i = βx i + u i with u i [0,σ 2 ]. 0

11 7. Multivariate ormal Limit Theorems The receing CLTs were for scalar ranom variables. Definition A6a: (Multivariate Central Limit Theorem)Letµ = E[ X ] an V = V[ X ]. A multivariate central limit theorem (CLT) secifies the conitions on the iniviual terms X i in X uner which V /2 (b µ ) [0, I]. This is formally establishe using the following result. Theorem A6: (Cramer-Wol Device)Let{b } be a sequence of ranom k vectors. If λ 0 b converges to a normal ranom variable for every k constant non-zero vector λ, thenb converges to a multivariate normal ranom variable. The avantage of this result is that if b = X,thenλ 0 b = λ X + +λ k X k will be a scalar average an we can aly a scalar CLT, yieling λ 0 X λ 0 µ λ 0 V λ [0, ], an hence V /2 (b µ ) [0, I]. Microeconometric estimators can often be exresse as ( b θ θ0 )=H a, where lim H exists an a has a limit normal istribution. The istribution of this rouct can be obtaine irectly from art (ii) of Theorem A2 (Slutsky s theorem). We restate it in a form that arises for many estimators. Theorem A7: (Limit ormal Prouct Rule) Ifavectora H, where H is ositive efinite, then H [µ, A] an a matrix H a [Hµ, HAH 0 ]. For examle, the OLS estimator ³ bβ β0 = µ X0 X X 0 u, is H =( X 0 X) times a = /2 X 0 u an we fin the lim of H an the limit istribution of a.

12 Theorem A7 also justifies relacement of a limit istribution variance matrix by a consistent estimate without changing the limit istribution. Given ³ bθ θ0 [0, B], then it follows by Theorem A7 that B /2 ³ b θ θ 0 [0, I] for any B that is a consistent estimate for B an is ositive efinite. A formal multivariate CLT yiels V /2 (b µ ) [0, I]. Premultily by V /2 an aly Theorem A7, giving simler form b µ [0, V], where V = lim V an we assume b an V are aroriately scale so that V exists an is ositive efinite. Different authors exress the limit variance matrix V in ifferent ways.. General form: V = lim V. With fixe regressors V =limv. 2. Stratifie samling or fixe regressors: Often V is a matrix average, say V = P i= S i,where S i is a square matrix. A LL gives V E[V ] 0. Then V = lime[v ] = lim P i= E[S i]. 3. Simle ranom samling: S i are ii, E[S i ]=E[S], sov =E[S]. As an examle, lim P i x ix 0 i = lim P i E[x ix 0 i ] if LL alies an =E[xx0 ] uner simle ranom samling. 8. Asymtotic ormality It can be convenient to re-exress results in terms of b θ rather than ( b θ θ 0 ). Definition A8: (Asymtotic Distribution of θ)if b ( θ b θ0 ) [0, B], then we say that in large samles b θ is asymtotically normally istribute with bθ a θ 0, B, 2

13 where the term in large samles means that is large enough for goo aroximation but not so large that the variance B goes to zero. A more shorthan notation is to imlicitly resume asymtotic normality an use the following terminology. Definition A9: (Asymtotic Variance of b θ)if(??) hols then we say that the asymtotic variance matrix of b θ is V[ b θ]= B. (8.) Definition A20: (Estimate Asymtotic Variance of b θ)if(??) holsthenwesay that the estimate asymtotic variance matrix of b θ is where bb is a consistent estimate of B. bv[ b θ]= bb. (8.2) Some authors use the Avar[ b θ] an [Avar[ b θ] in efinitions A9 an A20 to avoi otential confusion with the variance oerator V[ ]. It shoul be clear that here V[ b θ] means asymtotic variance of an estimator since few estimators have close form exressions for the finite samle variance. As an examle of efinitions 8-20, if {X i } are ii [µ, σ 2 ] then ( X µ)/σ [0, ], or equivalently that X [µ, σ 2 a ].Then X µ,σ 2 / ;theasymtotic variance of X is σ 2 / ; an the estimate asymtotic variance of X is s 2 /, where s 2 is a consistent estimator of σ 2 such as s 2 = P 2 i Xi X /( ). Definition A2: (Asymtotic Efficiency) A consistent asymtotically normal estimator θ b of θ is sai to be asymtotically efficient if it has an asymtotic variancecovariance matrix equal to the Cramer-Rao lower boun à " 2 ln L E / θ θ θ0#! OLS Estimator with Matrix Algebra ow consier b β =(X 0 X) X 0 y with y = Xβ + u, so bβ = β +(X 0 X) X 0 u. ote that the k k matrix X 0 X = P i x ix 0 i where x i is a k vector of regressors for the i th observation. 3

14 9.. Consistency of OLS To rove consistency we rewrite this as bβ = β + X 0 X X 0 u. The reason for renormalization in the right-han sie is that X 0 X = P i x ix 0 i is an average that converges in robability to a finite nonzero matrix if x i satisfies assumtionsthatermitalltobealietox i x 0 i. Then lim b β = β + lim X 0 X lim X 0 u, using Slutsky s Theorem (Theorem A.3). The OLS estimator is therefore consistent for β (i.e., lim b β OLS = β) if lim X 0 u = 0. If a law of LL can be alie to the average X 0 u = P i x iu i then a necessary conition for this to hol is that E[x i u i ]=0. The funamental conition for consistency of OLS is that E[u i x i ]=0 so that E[x i u i ]= Limit Distribution of OLS Given consistency, the limit istribution of β b is egenerate with all the mass at β. To obtain a limit istribution we multily β b OLS by,so ( β b β) = X 0 X /2 X 0 u. We know lim X 0 X exists an is finite an nonzero from the roof of consistency. For ii errors, E[uu 0 X] =σ 2 I an V[X 0 u X] =E[X 0 uu 0 X 0 X] =σ 2 X 0 X we assume that a CLT can be alie to yiel /2 X 0 u [0,σ 2 lim X 0 X ]. Then by Theorem A7: (Limit ormal Prouct Rule) ( β b β) lim X 0 X [0,σ 2 lim X 0 X ] [0,σ 2 lim X 0 X ] Asymtotic Distribution of OLS Then roing the limits ( β b β) [0,σ 2 X 0 X ], 4

15 so bβ a [β,σ 2 (X 0 X) ]. The asymtotic variance matrix is V[ b β]=σ 2 (X 0 X), an is consistently estimate by the estimate variance matrix bv[ b β]=s 2 (X 0 X), where s 2 is consistent for σ 2. For examle, s 2 = bu 0 bu/( k) or s 2 = bu 0 bu/ OLS with Heteroskeatic Errors What if the errors are heteroskeastic? If E[uu 0 X] =Σ = Diag[σ 2 i ] then V[X0 u X] = E[X 0 uu 0 X 0 X] =X 0 ΣX = P i= σ2 i x ix 0 i.acltgives leaing to /2 X 0 u [0, lim X 0 ΣX], ( b β β) lim X 0 X [0, lim X 0 ΣX] [0,σ 2 lim X 0 X lim X 0 ΣX lim X 0 X ]. Then roing the limits etcetera bβ a [β, (X 0 X) X 0 ΣX(X 0 X) ]. The asymtotic variance matrix is V[ β]=(x b 0 X) X 0 ΣX(X 0 X). White (980) showe that this can be consistently estimate by bv[ β]=(x b 0 X) ( X bu i 2 x i x 0 i= i)(x 0 X), even though bu 2 i is not consistent for σ 2 i. 5