Consistency and asymptotic normality

Transcription

1 Consistency an ymtotic normality Cls notes for Econ 842 Robert e Jong Aril Stochtic convergence The ymtotic theory of minimization estimators relies on various theorems from mathematical statistics. The objective of this section is to exlain the main theorems that unerin the ymtotic theory for minimization estimators. 1.1 Infimum an suremum, an the limit oerations Suremum an infimum The suremum of a set A, if it exists, is the smallest number y for which x y for every x A. Analogously, the infimum of a set A, if it exists, is the largest number y for which x y for every x A. For an oen set such A = (0, 1), the suremum su A an the infimum inf A are well-efine 1 an 0, resectively. If the suremum is attaine for some element in A, the suremum is also calle the maximum; analogously, the minimum of a set is efine the infimum if it is attaine for some element of A. Therefore, the maximum an minimum of A = (0, 1) are not efine, because they are attaine at 0 an 1, which are not insie (0, 1). If we efine the extene real line R {, + }, the su an inf are always efine on the extene real line. Deartment of Economics, Ohio State University, 429 Ars Hall, Columbus, OH 43210, ejong@econ.ohio-state.eu 1

2 1.1.2 The limit oeration For a eterministic sequence x n, we say that it converges to x, notation lim n x n = x, if for all ε > 0, there exists an N ε such that x n x < ε for all n N ε. The limsu (suerior limit, limes suerior in Latin) is efine lim su x n = lim n x n = inf su x m. n n 1 m n The limsu is the eventual uer limit of a sequence. For examle, consier x n = n 1. Then lim su x n = inf su m 1 = inf n n 1 m n n 1 n 1 = 0, just one woul exect. From the earlier iscussion, it is now clear that the limsu is always efine on the extene real line. The limsu of a sequence is the eventual uer boun of a sequence. For examle, lim su n sin(n) = 1, while the limit is obviously unefine. Analogously, we efine lim inf n x n = lim n x n = su n 1 For x n = n 1, we obtain lim inf n x n = su n 1 inf x m. m n inf x m = su 0 = 0, m n n 1 one woul exect from an eventual lower limit. If an only if the limsu an liminf of a sequence are equal, then the limit of the sequence exists. If, after noting that log(1 + x) x, we rea something like lim n n2 log(φ(n)) lim n 2 (Φ(n) 1) = 0 n where Φ( ) enotes the stanarnormal istribution function, we shoul realize the following. In rincile, when writing own the left-han sie exression, it is ossible for this exression to be unefine. The correct interretation of the above equation is that the inequality fins an uer boun if the exression exists; but formally the existence of the left-han sie remains unroven. The sertion that the lim su is less than or equal to zero is always true however if the limsu is efine on the extene real line. 2

3 1.2 Almost sure convergence, convergence in robability, an convergence in istribution Definitions of stochtic convergence concets While for a eterministic sequence X n it is relatively straightforwar to efine convergence of X n to a number X, the situation becomes more comlicate in the ce where X n an/or X is allowe to be a ranom variable. There are four tyes of convergence for ranom variables that are use frequently in the econometric literature. They are convergence in istribution, convergence in robability, convergence in L, an almost sure convergence. We say that X n converges to X in istribution if lim P (X n y) = P (X y) n for all y for which P (X y) is continuous in y. To see why we only want this requirement to hol only for continuity oints of P (X y), consier X n = n 1. This is a eterministic sequence an we shoul hoe that it converges in istribution to X = 0. However, setting y = 0, P (X n 0) = P (n 1 0) = 0 P (X 0) = 1. Because the convergence in istribution requirement exclues iscontinuity oints however, X n = n 1 oes converge in istribution to zero. X n converges in robability to X if for all δ > 0, lim P ( X n X > δ) = 0. n We tyically write X n X or lim n X n = X to enote convergence in robability. X n is sai to converge almost surely to X if P ( lim n X n = X) = 1 which is equivalent to the conition that for all δ > 0, lim P (su X m X > δ) = 0. n m n We then write X n X or lim n X n = X almost surely. Convergence in L of X n to X for 1 hols if lim E X n X = 0. n 3

4 1.2.2 Differences between stochtic convergence concets There are subtle ifferences among these convergence concets; however, convergence in istribution is in some ways in a ifferent category from the other stochtic convergence concets. The imortant thing to realize about convergence in istribution is that if X n converges to X, an X is a stanarnormally istribute ranom variable, then X n also converges to Y, which is a ranom variable that is also stanarnormally istribute. For examle, a sequence X n of i.i.. N(0, 1) ranom variables converges in istribution to a N(0, 1) ranom variable n. This may be counterintuitive, because X n will never get closer to any articular ranom variable; but the imortant thing to realize is that only the istribution of X n nees to converge for convergence in istribution. While convergence in istribution only says something about istributions, convergence in robability, almost sure convergence an convergence in L are statements imlying that X n an X are ymtotically close. Almost sure convergence imlies convergence in robability, can be eily seen from my efinitions above. Convergence in L imlies convergence in robability. This follows from the Markov inequality P (Y > δ) δ 1 E Y, because P ( X n X > δ) = P ( X n X > δ ) δ E X n X. Convergence in robability imlies convergence in istribution, an convergence in istribution to a constant imlies convergence in robability to that constant; I will leave this unroven here. No other relations between concets hol without further sumtions An o examle A clsical examle of a sequence X n that converges in robability to zero but not in L is the following. Let X n be a ranom variable that takes the value ex(n) with robability 1/n an zero otherwise. Then for all δ > 0, lim P ( X n > δ) lim P (X n = ex(n)) = lim n 1 = 0, n n n so X n converges in robability to 0. However, for any 1, E X n = ex(n)p (X n = ex(n)) = n 1 ex(n), so X n oes not converge in L. Another examle where convergence in robability hols but convergence in L oes not is the ce where X n = Y/n where Y is Cauchy. In that ce, we have convergence in robability, but not in L, because moments of orer 1 an higher of the Cauchy istribution o not exist. 4

5 If in aition X n is ineenent across n, we also have an examle of a sequence that converges in robability, but not almost surely. This is because while X n 0, su m n X m oes not converge to zero in robability. After all, su m n X m ex(n) if one of the X m for which m n is nonzero; that is, su m n X m 1 only if X m = 0 m n. Therefore, for all 0 < δ < 1, P (su X m δ) P ( m n : X m = 0) m n = P (X m = 0) = m=n = ex( (1 1/m) m=n log(1 1/m)) = 0. m=n To show that lt result, note that because log(1 + x) x for x > 1, lim ex( K log(1 1/m)) lim ex( K (1/m)) = 0 K K m=n m=n because m 1 is not summable from 1 to infinity. This imlies that X n converges in robability to zero, but not almost surely Averages Less exotic statistics that we will encounter in econometrics are averages of i.i.. or weakly eenent ranom variables. Consier the following simle examle. Let X n = n 1/2 (z i Ez i ), where the z i are i.i.. an Ezi 2 <. Then X n converges in istribution to a normally istribute ranom variable. For each eterministic sequence a n such that lim n a n = 0 we have a n X n 0. However, it turns out that a n X n 0 is only true if a n (log log n) 1/2 0; this follows from the so-calle law of the iterate logarithm. 5

6 1.3 The law of large numbers Statement of the law of large numbers Consistency of estimators is erive using a law of large numbers (LLN). The LLN serts that n 1 (z i Ez i ) converges to zero. It can be shown that if E z i < an z i is i.i.. (ineenent an ientically istribute), then the above exression converges to zero almost surely (an therefore also in robability; the almost sure version is Kolmogorov s strong law of large numbers). If E z i is not finite, the LLN oes not nee to hol; a counterexamle woul be the Cauchy istribution, for which z n is istribute ientically to z i. If convergence to zero hols almost surely, we talk about the strong law of large numbers, while if the converge to zero is in robability, we refer to the result a weak law of large numbers Weak LLN uner ineenence If z i h mean zero, is i.i.. an Ezi 2 <, then V (n 1 z i ) = n 2 V (z i ) = n 1 V (z i ) 0, so a weak law of large numbers will hol for z i in this ce. The conition Ezi 2 < can be weakene to E z i < by a truncation argument. This can be one follows. Define zi K = z i I( z i > K) an z ik = z i I( z i K). Then E n 1 z i E n 1 (z ik Ez ik ) + E n 1 (zi K Ezi K ) (E n 1 (z ik Ez ik ) 2 ) 1/2 + 2n 1 E zi K (n 1 V (z ik )) 1/2 + 2E z i I( z i > K) (n 1 K 2 ) 1/2 + 2 z F (z), z >K an by taking first the limit n an then the limit K, the convergence in L 1 follows. To rove Kolmogorov s strong law of large numbers is much more involve; this tyically requires a result known the three series theorem. I will not ursue that here. 6

7 1.3.3 Discussion There exist many formulations of the LLN that are ifferent from the above. For examle, if z i is uncorrelate, Ez 2 i < an Var(n 1 (z i Ez i ) = n 2 Var(z i ) 0, then the weak LLN will also hol, in site of the fact that the i.i.. sumtion h not been mae. Also, the ineenence sumtion can be relaxe further to allow for ranom variables z i that have faing memory roerties. To illustrate this, note that in general Var(n 1 (z i Ez i )) = n 2 cov(z i, z s ) s=1 2n 2 2n 2 cov(z i, z s ) s=i cov(z i, z i+m ), m=0 an if the lt exression converges to zero somehow, then a weak LLN can be obtaine well. Often, alternatives to the clsical LLN for i.i.. variables involve sumtions on the existence of moments E z i 1+δ for some δ > The central limit theorem Statement of the central limit theorem The central limit theorem (CLT) states that n 1/2 (z i Ez i ) N(0, Var(z i )). If z i is i.i.. an Ez 2 i <, then the above result hols. 7

8 1.4.2 CLT an characteristic function The characteristic function of a ranom variable X is efine ψ(r) = E ex(irx), where i satisfies i 2 = 1. There is a one-to-one corresonence between characteristic functions an istribution functions, an because ψ( ) 1, showing that the characteristic function converges to the characteristic function of a normal istribution suffices to show convergence in istribution to a normal. The reoning for roving a CLT for a mean zero, i.i.. sequence z i with a variance σ 2 then starts by observing that E ex(irn 1/2 z i ) = n E ex(irn 1/2 z i ) by ineenence. By the Taylor series exansion, ex(x) 1 + x + 0.5x 2 an log(1 + x) x, an therefore n E ex(irn 1/2 z i ) E(1 + (irn 1/2 z i ) + 0.5(irn 1/2 z i ) 2 ) = n (1 0.5r 2 n 1 σ 2 ) = ex( log(1 0.5r 2 n 1 σ 2 )) ex( 0.5r 2 σ 2 ), which is the characteristic function of the N(0, σ 2 ) istribution. Of course, the analytical ifficulty is to justify the signs in the above reoning Discussion For the CLT, lots of alternative secifications are also ossible. It is ossible to allow for some heterogeneity of the istributions of z i, an also the requirement of ineenence can be relaxe to some tye of faing memory sumtion. Heterogeneity of the istributions of z i refers to the fact that the istributions F i (x) of z i are not ientical for all i. 8

9 1.5 Moment conitions Many articles an books in econometrics contain so-calle moment conitions of the tye E z i <, or lim su n 1 n or sume that for some δ > 0, E z i 2+δ <. E z i 2 <, By now, you robably realize that these conitions tyically originate from the verification of conitions necessary for obtaining a law of large numbers or central limit theorem. For examle, n 1 n (z i Ez i ) 0 if the z i are i.i.. an E z i <. So, if somewhere along the line we nee the result n 1 n (z i Ez i ) 0, we simly imose E z i < a regularity conition. The same hols for the central limit theorem. Finally, note that E z i < is imlie by E z i q < for some q > ; this is because for 1 q, (E X ) 1/ (E X q ) 1/q. An examle of a istribution for which moment conitions can fail to hol is the Stuent istribution. If z i h a Stuent istribution with k egrees of freeom, then E z i k oes not exist; remember that a t istribution with one egree of freeom is the Cauchy istribution. 1.6 The ominate convergence theorem A tool that is frequently use in ymtotic theory in econometrics is the ominate convergence theorem. Theorem 1 If X n then lim EX n = EX. n X an X n Y a.s. for some ranom variable Y an EY <, This result is wiely use in econometrics; notably, it is use for ymtotic normality of minimization estimators when the mean value in the Taylor series exansion argument is relace by its limit. 9

10 1.7 A joint convergence result Justification of the elta metho The following result lays a key role in ymtotic theory for justifying variance estimation: Theorem 2 If X n X an Y n both arguments, then f(x n, Y n ) c for some constant c an f(x, y) is continuous in f(x, c). Using the above theorem, we can reon follows. If n 1 n z2 i N(0, σ 2 ), then n 1/2 n z i n 1 n z2 i N(0, 1). σ 2 an n 1/2 n z i Note that this imlies that f(x, y) = x/ y, which is not continuous for y near 0, leaving a small roblem in our argument. This roblem can be formally fixe, using the sumtion that σ 2 > A beginner s mistake However, it is not allowe to reon n 1/2 n z i n 1 n z2 i = n 1/2 n 1 n z i n 1 n z2 i n 1/2 0 σ 2 = 0 many beginners ten to o. You cannot fin the continuous function f(.,.) that will allow you to o this trick; you woul nee f(x, y) = n 1/2 x/ y, but eenence of f(, ) on n is not allowe. The correct reoning islaye reviously sume f(x, y) = x/ y an clearly, this is allowe. 1.8 The multivariate ce Often when convergence in istribution of a ranom vector is shown, a result know the Cramèr-Wol evice is use. The Cramèr-Wol evice is a tool that can be use to simlify the roblem of showing convergence in istribution of a vector to convergence in istribution X if an only if for all of scalars. It serts that for a vector X n of imension k, X n k-vectors ξ, ξ X n ξ X. This imlies that using the Cramèr-Wol evice, the roblem of showing ymtotic normality of a vector can be translate into a roblem involving an alication of the CLT for scalars. 10

11 1.9 Why ymtotic normality? The reon econometricians are intereste in establishing ymtotic normality of estimators is that this roerty allows us to justify inference roceures such t-tests an F -tests. Of course, t-tests that are justifie be on ymtotic theory are really ymtotically stanarnormal, an F -tests that are justifie be on ymtotic theory are really ymtotically chi-square. Asymtotic normality of a k-vector X n, i.e. X n N(0, V ), imlies that if ˆV V, X nj ˆVjj N(0, 1), which will imly valiity of t-values. A similar argument can be use to show that F -tests are ymtotically vali if we have ymtotic normality of our estimator The argmin functional The argmin is efine a value at which the function over which the argmin is takes is minimize. For examle, argmin x R (1 + x 2 ) = 0. Note that for two functions that are arbitrarily close, the argmin functional can be very ifferent. This will be a source of comlications when we will rove consistency of estimators in situations when we cannot exlicitly write ˆθ n an exlicit an relatively simle function of the (y i, x i ), i = 1,..., n. Consier for examle, for θ [0, 1], an f n (θ) = θn 1 g n = (1 θ)n 1. Then su θ Θ f n (θ) g n (θ) 0, while argmin θ [0,1] f n (θ) = 0 an argmin θ [0,1] g n (θ) = 1. Obviously, both argmins o not get close ymtotically; however, also note that this is because the limit function 0 that both f n ( ) an g n ( ) converge to oes not have a unique minimum. The argmax is efine analogously to the argmin, the value where at which the function over which the argmax is takes is maximize. In rincile, the argmin of a function Q n (θ) can be attaine at multile values in Θ, which woul make the statement ˆθ n = argmin θ Θ Q n (θ) unefine; but this ossibility is tyically isregare in econometric rose. 11

12 1.11 Exercises 1. Assume that x n is a nonecreing, boune eterministic sequence of scalars. Show that lim n x n exists. 2. Assume that X n is an increing sequence of ranom variables such that X n X. Show that in this ce, X n X also. 3. Assume that X n an Y n are sequences of ranom variables such that X n X an Y n Y. Show that Y n + X n X + Y. 4. Assume that X n an Y n are sequences of ranom variables such that X n X an Y n Y. Fin an examle that shows that Y n + X n X + Y oes not nee to hol. 5. Assume that X n is a sequence of ranom variables such that X n c for a constant c. Show that X n c also. 6. In section 1.2.3, what haens to the o examle if the n 1 in its efinition is relace by n 2? 7. Let x i be i.i.. an N(0, 1) istribute. Using characteristic functions, show that n 1/2 n x i is also N(0, 1) istribute. 8. Formally rove the ominate convergence theorem. 12

13 2 Asymtotics of the linear moel: the scalar ce 2.1 The consistency argument In this section we will investigate the ymtotic behavior of let squares estimators. The urose is only to serve an introuction to the more general ce, where we will only sume that our estimator is some minimization estimator, for which no close form solution is available. To unerstan the rinciles involve better, we will focus on the ce of a scalar regressor x i in this section. In the ce of the simle linear moel y i = θ 0 x i + ε i, where x i R, the close form solution for the let squares estimator is ˆθ n = argmin θ Θ n 1 (y i θx i ) 2 = n y ix i n x2 i if the arameter sace Θ is equal to R. Parameter saces other than R can sometimes be natural; for examle, in an ARCH(1) moel, a arameter sace of [0, 1] for the α arameter may be referable. Later, we will see that the general consistency roof forces an sumtion of comactness of the arameter sace on us. Note that this solution for ˆθ n w obtaine by ifferentiation with resect to θ. Next, we note that - if the true moel is correct - it is ossible to write n ˆθ n = θ 0 + ε ix i n. x2 i After this ste, we have to make some sumtion on the behavior of x i : is it a eterministic or a stochtic sequence? Consistency argument: the eterministic regressor ce First, we sume that x i is eterministic. Then note that n ε ix i n = n 1 n ε ix i x2 i n 1 n, an therefore ˆθ n θ 0 if x2 i 13

14 1. n 1 n ε ix i 0; 2. n 1 n x2 i Q, where Q > Consistency argument: the ranom regressor ce Uner the sumtion that x i is ranom, sufficient conitions for ˆθ n 1. n 1 n ε ix i 0; θ 0 are 2. n 1 n x2 i Q, where Q > 0. Both conitions follow, by Kolmogorov s law of large numbers, from the conitions 1. (x i, ε i ) is i.i..; 2. Ex 2 i < ; 3. Eε i x i = 0; 4. E ε i x i <. In the ce that ε i an x i are ineenent of each other, conitions 1-4 are imlie by 1. (x i, ε i ) is i.i..; 2. Ex 2 i < ; 3. Eε i = 0; 4. E ε i <. 2.2 The ymtotic normality argument Fining the limit istribution of ˆθ n requires some more work than the simle consistency result. If we sume that the ε i are i.i.. normal with variance σ 2 an x i is eterministic, then n ε ix i n x2 i is istribute N(0, σ 2 ( n x2 i ) 1 ) for any samle size. Therefore, if n 1 n x2 i Q, we get n 1/2 (ˆθ n θ 0 ) N(0, σ 2 Q 1 ). The ymtotic version of this argument simly relaces the exact normality by ymtotic normality, an uses a CLT a relacement for the exact normality sumtion. 14

15 2.2.1 Asymtotic normality argument: the eterministic regressor ce Sufficient conitions for the ymtotic normality result n 1/2 (ˆθ n θ 0 ) 1. n 1/2 n ε ix i N(0, σ 2 Q); 2. n 1 n x2 i Q where Q > 0. N(0, σ 2 Q 1 ) are It is ossible to use a central limit theorem for ineenent, but not ientically istribute ranom variables to establish Asymtotic normality argument: the ranom regressor ce At this oint, our interest will be in relaxing the sumtion that the x i are eterministic. If we sume 1. (x i, ε i ) is i.i..; 2. Eε i x i = 0; 3. Ex 2 i < an Eε 2 i x 2 i < ; then by the central limit theorem, n 1/2 x i ε i N(0, Eε 2 i x 2 i ), an in aition, n 1 x 2 i Q, an therefore by Theorem 2, Therefore, n 1/2 n x iε i n 1 n x2 i n 1/2 (ˆθ n θ 0 ) where P = Eε 2 i x 2 i. N(0, Q 2 Eε 2 i x 2 i ). N(0, Q 2 P ), 15

16 2.2.3 Heterosketicity If we sume in aition that ε i an x i are ineenent, then P = Eε 2 i x 2 i = Eε 2 i Ex 2 i = σ 2 Q, an then our result becomes n 1/2 (ˆθ n θ 0 ) N(0, σ 2 Q 1 ). The above result also follows from suming that E(ε 2 i x i ) = σ 2 (a homosketicity sumtion). However, lee note that ymtotic normality is also obtaine without homosketicity sumtion. We will iscuss this oint further when we iscuss the multivariate ce. In the sections that follow, we will always sume that the x i sequence is stochtic. 16

17 3 Missecifie linear moels: the scalar ce Unfortunately, in ractice, we never know whether the moel that we secifie is true or not. Therefore, we may woner what haens to our estimator if the moel estimate is missecifie. For examle, if y i = x 2 i + ε i where (ε i, x i ) is i.i.. an ε i is ineenent of x i, is the true moel instea of the linear moel from the revious section, what will our OLS estimator o? Will it waner off to infinity, or will it have a limit istribution? 3.1 Missecifie linear moels: consistency In orer to she some light on this issue, note that for the OLS estimator from the revious section we know that ˆθ n = n y ix i n x2 i = n x3 i n x2 i + = n 1 n x3 i n 1 n x2 i n ε ix i n x2 i + n 1 n ε ix i n 1 n. x2 i By the law of large numbers, if E x i 3 < an E ε i x i <, Ex 3 i, an n 1 n 1 n 1 Therefore, ˆθ n x 3 i ε i x i 0, x 2 i Ex 3 i /Ex 2 i. Q = Ex 2 i. The value θ = Ex 3 i /Ex 2 i is often referre to the seuo-true value. 17

18 3.2 Missecifie linear moels: ymtotic normality To see what haens to the limit istribution of our missecifie estimator, efine ˆQ n = n 1 n x2 i an note that n 1/2 (ˆθ n Ex 3 i /Q) = n 1/2 n ε ix i n 1 n x2 i = Qn 1/2 n ε ix i Q ˆQ n + n 1/2 n x3 i n 1 n x2 i n 1/2 n Ex3 i Q n + Qn 1/2 (x3 i Ex 3 i ) Q ˆQ n1/2 ( ˆQ n Q)Ex 3 i. n ˆQ n Q Therefore, by Theorem 2, n 1/2 (ˆθ n Ex 3 i /Q) h the same limit istribution Qn 1/2 n ε ix i + Qn 1/2 n (x3 i Ex 3 i ) n 1/2 ( ˆQ n Q)Ex 3 i. Q 2 In general, (n 1/2 ε i x i, n 1/2 (x 3 i Ex 3 i ), n 1/2 (x 2 i Ex 2 i )) is an ymtotically normal vector; therefore, n 1/2 (ˆθ n Ex 3 i /Q) is ymtotically normally istribute. 3.3 Discussion The striking thing about this result is that ˆθ n is still both converging at rate n 1/2 an ymtotically normally istribute, in site of the obvious missecification of the moel that w estimate. If we efine θ = Ex 3 i /Q, then our result can be rewritten n 1/2 (ˆθ n θ ) N(0, V ). The value for θ coul also be obtaine along a ifferent route. Since the OLS estimator is the let squares estimator, one might exect that θ minimizes E(y i θx i ) 2. This is inee the ce, since E(y i θx i ) 2 = E(ε i + x 2 i θx i ) 2 = σ 2 + Ex 4 i 2θEx 3 i + θ 2 Ex 2 i 2Eε i x 2 i 2θEε i x i ; Differentiation with resect to θ then learns that θ is the minimizer of E(y i θx i ) 2 if Eε i x i = 0. Our conclusion therefore is that if the oulation objective function is uniquely minimize at some value, we will get consistency for that value. This will turn out to be true in more general settings well. 18

19 4 Asymtotics of the linear moel: multivariate ce Now let us consier the well-known moel y i = θ 0x i + ε i where θ, x i R k. The OLS estimator now is ˆθ n = (X X) 1 X y where X = (x 1, x 2,..., x n ) an y = (y 1,..., y n ) an ε = (ε 1,..., ε n ). If the moel is true, ˆθ n = (X X) 1 X y = θ 0 + (X X) 1 X ε. 4.1 Consistency of the linear moel: multivariate ce Since all elements of X X are summations over n elements, it is not unreonable to sume that X X/n Q, where Q enotes some k k matrix that is sume to be invertible, an that X ε 0. These results can be erive by alying the LLN k 2 an k times, resectively. To see why X X an X y are summations over n elements, the following trick is useful. Let s i enote a vector of all zero elements, excet for an element of 1 on sot i. Then I n = n s is i, y i = s iy an x i = X s i. Therefore, X X = X s i s ix = X s i (X s i ) = x i x i, an similarly, X y = X s i s iy = X s i y i = x i y i. Therefore, because ˆθ = θ 0 + (n 1 x i x i) 1 n 1 x i ε i an letting. enote the Eucliean norm, consistency of ˆθ n follows if 1. (ε i, x i ) is i.i..; 2. Eε i x i = 0; 3. E x i 2 < ; 4. E ε i x i <. 19

20 4.2 Asymtotic normality of the linear moel: multivariate ce Also, by the multivariate central limit theorem, usually n 1/2 X ε N(0, E(X εε X)), an therefore, enoting P = E(X εε X/n) = Eε 2 i x i x i, n 1/2 (ˆθ n θ 0 ) N(0, Q 1 P Q 1 )). If in aition we sume that E(ε 2 i x i ) = σ 2 (i.e., no heterosketicity), then P = E(X εε X/n) = E(X E(εε X)X/n) = σ 2 E(X I n X/n) = σ 2 Q, where I n enotes the n n ientity matrix. Then we obtain the more familiar result n 1/2 (ˆθ n θ 0 ) Writing n 1/2 (ˆθ n θ 0 ) = (n 1 N(0, σ 2 Q 1 ). x i x i) 1 n 1/2 ε i x i, an again letting. enote the Eucliean norm, we can now conclue that we obtain the ymtotic normality result n 1/2 (ˆθ n θ 0 ) N(0, Q 1 P Q 1 ) if 1. (ε i, x i ) is i.i..; 2. E x i 2 < ; 3. Eε i x i = 0; 4. Eε 2 i x i 2 <. Note that it is ossible to estimate Q 1 P Q 1 also. estimate for Q an using ˆP n = n 1 x i x iê 2 i, t=1 This is one by using ˆQ n an where ê i is the regression resiual from the regression of y i on x i. If t-values are calculate in this way, we usually say that White s t-values, t-values accoring to the metho of White, or heterosceticity-consistent t-values are calculate. 20

21 5 Consistency of extremum estimators: bic ce The next thing that we will set out to o is the consieration of general minimization estimators for which no close form solution is available. Estimators are usually roose solutions of some minimization roblem; maximum likelihoo estimators an (non)linear let squares estimators are examles of this. Close form solutions for ˆθ n are rarely available, an therefore it is of great interest to obtain general results regaring consistency an ymtotic normality of such estimators. Let Usually, ˆθ n = argmin θ Θ Q n (θ). Q n (θ) = n 1 ψ(z i, θ), but we will not force this structure at first. Note that GMM estimators are not containe in the average framework. That is to say, our minimization estimator is sume to minimize a ranom function Q n (θ), an we will not be exlicit about the exact form of Q n ( ). The Θ is the arameter sace, the sace over which the objective function is minimize. Next, we nee to acquire an unerstaning of a key ect of the consistency roof: the uniform law of large numbers. 5.1 The uniform law of large numbers From the law of large numbers, we exect that n 1 (ψ(z i, θ) Eψ(z i, θ)) 0 for all θ Θ, where Θ enotes some arameter sace (that in most ces is referably chosen R k ). Therefore, one might susect that the conition su θ Θ n 1 (ψ(z i, θ) Eψ(z i, θ)) 0. is (nearly) always satisfie. This is not true, however; there is a ifference between the ointwise LLN an the uniform LLN. As an examle of the roblem involve, consier Θ = [0, 1] an f n (θ) = I(0 < θ 1/n). 21

22 Then, for all θ, lim n f n (θ) = 0, while still su θ Θ f n (θ) = 1 for all n! One might argue that this examle is nonstochtic an therefore robably not comletely satisfactory. The sace θ = [0, 1] is comact however, an the f n ( ) function can also be moifie to such to give an examle of a continuous function with similar behavior. If stochtic convergence is introuce, comlications can only multily. Therefore, the conclusion h to be that uniform convergence an ointwise convergence are two ifferent concets. Obviously, uniform convergence imlies ointwise convergence. 5.2 Consistency: stanar sumtions For consistency of general minimization estimators, the following sumtions suffice. Notice that the sumtion of comactness of Θ h not (yet) been mae: Assumtion 1 1. Q n (θ) an ˆθ n = argmin θ Θ Q n ( ) are a well-behave ranom functions of θ. 2. Uniform convergence: su θ Θ Q n (θ) Q(θ) 0, for a eterministic function Q( ) an Q( ) is continuous on Θ. 3. Ientification (art 1): there exists a unique θ such that Q(θ ) = min θ Θ Q(θ). 4. Ientification (art 2): For all δ > 0, inf θ Θ; θ θ >δ Q(θ) > Q(θ ). Assumtion 1.1 efines ˆθ n an rules out technical roblems involving the existence of ranom variables (it rules out meurability roblems). Assumtions 1.3 an 1.4 are regularity sumtions that nee to ensure that there is one single minimizer of the limit objective function that is well-behave. These conitions are ientifiability conitions an will fail to hol, for examle, if we have multicollinearity in our regressors. Note that in the ce of exact multicollinearity, Q n ( ) oes not have a unique minimizer, while the conition above says that the limit objective function shoul not have multile minimizers. Assumtion 1.4 hols automatically if Θ is comact an 1.3 hols. Assumtion 1.2 is a uniform law of large numbers. Using the above sumtions, the following theorem can be establishe : Theorem 3 Assume ˆθ n exists a ranom variable for all n. Then Assumtions 1.1, 1.2 an 1.3 imly that ˆθ n θ almost surely. 22

23 Proof: We will show that, for n large enough, ˆθ n cannot be further from θ than δ, say. The result then follows since δ w arbitrary. The iea of the roof is to show that Q n (θ) Q n (θ ) excees zero if θ θ > δ an n is large, an therefore, for n large enough, ˆθ n θ δ. We show this by efining ε > 0 inf Q(θ) = Q(θ ) + ε θ θ >δ an noting that for n large enough Therefore, su Q n (θ) Q(θ) < ε/2. θ Θ inf Q n (θ) Q n (θ ) θ θ >δ = inf Q n (θ) Q(θ) + Q(θ) θ θ >δ +Q n (θ ) Q(θ ) + Q(θ ) inf θ θ >δ Q(θ) Q(θ ) su Q n (θ) Q(θ) θ Θ Q n (θ ) Q(θ ) > ε ε/2 ε/2 = 0. Therefore, for n large enough, ˆθ n θ δ since the criterion function is too large for other values of ˆθ n. Therefore, lim n ˆθn = θ almost surely. An alternative line of reoning is follows. Note that, since θ minimizes Eψ(z i, θ), 0 Q(ˆθ n ) Q(θ ). Next, note that 0 Q(ˆθ n ) Q(θ ) = Q(ˆθ n ) Q n (ˆθ n ) + Q n (ˆθ) Q(θ ) Q(ˆθ n ) Q n (ˆθ n ) + Q n (θ ) Q(θ ) 23

24 2 su Q n (θ) Q(θ) 0. θ Θ The secon inequality follows because ˆθ n minimizes Q n (θ) over θ Θ. So, lim Q(ˆθ n ) = Q(θ ), n an therefore by the uniqueness sumtions of Assumtion 1 an Assumtion 2 that ensure that ˆθ n will not woner off, ˆθ n θ. 5.3 The uniform convergence sumtion Consier the following examle. If we set Q n (θ) = n 1 (y i θ) 2, then an Q(θ) = E(y i θ) 2 Q n (θ) Q(θ) = n 1 (yi 2 Eyi 2 ) 2θn 1 (y i Ey i ). This imlies that su Q n (θ) Q(θ) =, θ R an therefore the uniform convergence conition fails if we set Θ = R. If Θ = [ 1, 1], say, the suremum will be attaine at either θ = 1 or θ = 1, an uniform convergence will hol uner stanar regularity conitions. As it turns out, the uniform convergence conition is nonlinear settings is ey to show uner some conitions if the arameter sace is comact. It is not clear what this imlies for emirical ractice. Formally, the stanar arguments will be exlaine below only hol if the researcher romises to never let the arameter outsie of an a riori ientifie arameter sace. 24

25 It is ossible to relax the comactness sumtion in the ce of a globally convex or concave objective function (such in the ce of let squares an ML estimation of robit, logit, an (after rearametrizing) Tobit moels. In general however, stanar theory will force us to make the comactness sumtion. It is ossible to show examles of inconsistent estimators in ces where all the sumtions above hol, excet that uniform convergence is weakene to ointwise convergence. 25