Least Squares Estimation- Large-Sample Properties

Transcription

1 Least Squares Estimatio- Large-Sample Properties Pig Yu School of Ecoomics ad Fiace The Uiversity of Hog Kog Pig Yu (HKU) Large-Sample 1 / 63

2 1 Asymptotics for the LSE 2 Covariace Matrix Estimators 3 Fuctios of Parameters 4 The t Test 5 p-value 6 Cofidece Iterval 7 The Wald Test Cofidece Regio 8 Problems with Tests of Noliear Hypotheses 9 Test Cosistecy 10 Asymptotic Local Power Pig Yu (HKU) Large-Sample 2 / 63

3 Itroductio If ujx N(0,σ 2 ), we have show that βjx b N β,σ 2 (X 0 X) 1. I geeral the distributio of ujx is ukow. Eve if it is kow, the ucoditioal distributio of b β is hard to derive sice bβ = (X 0 X) 1 X 0 y is a complicated fuctio of fx i g i=1. The asymptotic (or large sample) method approximates (ucoditioal) samplig distributios based o the limitig experimet that the sample size teds to ifiity. It does ot require ay assumptio o the distributio of ujx, ad oly some momets restrictios are imposed. Three steps: cosistecy, asymptotic ormality ad estimatio of the covariace matrix. Pig Yu (HKU) Large-Sample 2 / 63

4 Asymptotics for the LSE Asymptotics for the LSE Pig Yu (HKU) Large-Sample 3 / 63

5 Asymptotics for the LSE Cosistecy Express b β as bβ = (X 0 X) 1 X 0 y = (X 0 X) 1 X 0 (Xβ + u) = β +(X 0 X) 1 X 0 u. (1) To show b β is cosistet, we impose the followig additioal assumptios. Assumptio OLS.1 0 : rak(e[xx 0 ]) = k. Assumptio OLS.2 0 : y = x 0 β + u with E[xu] = 0. h Assumptio OLS.1 0 implicitly assumes that E kxk 2i <. Assumptio OLS.1 0 is the large-sample couterpart of Assumptio OLS.1. Assumptio OLS.2 0 is weaker tha Assumptio OLS.2. Pig Yu (HKU) Large-Sample 4 / 63

6 Asymptotics for the LSE Theorem Uder Assumptios OLS.0, OLS.1 0, OLS.2 0 ad OLS.3, b β p! β. Proof. From (1), to show β b p! β, we eed oly to show that (X 0 X) 1 X 0 u p! 0. Note that! (X 0 X) 1 X 0 1! 1 u = x i x 0 1 i x i u i i=1 i=1! = g 1 x i x 0 i, 1 p x i u i! E[xi x 0 i ] 1 E[x i u i ] = 0. i=1 i=1 Here, the covergece i probability is from (I) the WLLN which implies 1 x i x 0 i i=1 p! E[xi x 0 i ] ad 1 p x i u i! E[xi u i ]; (2) i=1 (II) the fact that g(a,b) = A 1 b is a cotiuous fuctio at E[x i x 0 i ],E[x iu i ]. The last equality is from Assumptio OLS.2 0. Pig Yu (HKU) Large-Sample 5 / 63

7 Asymptotics for the LSE Proof. [Proof cotiue] (I) To apply the WLLN, we require (i) x i x 0 i ad x i u i are i.i.d., which is implied h by Assumptio OLS.0 ad that fuctios of i.i.d. data are also i.i.d.; (ii) E kxk 2i < (OLS.1 0 ) ad E[kxuk] <. E[kxuk] < is implied by the Cauchy-Schwarz iequality, a E[kxuk] E hkxk 2i 1/2 h E juj 2i 1/2, which is fiite by Assumptio OLS.1 0 ad OLS.3. (II) To guaratee A 1 b to be a cotiuous fuctio at E[x i x 0 i ],E[x iu i ], we must assume that E[x i x 0 i ] 1 exists which is implied by Assumptio OLS.1 0. b a Cauchy-Schwarz iequality: For ay radom m matrices X ad Y, E [kx 0 Yk] E hkxk 2i 1/2 h E kyk 2i 1/2, where the ier product is defied as hx,yi = E [kx 0 Yk], ad for a m matrix A, 1/2 kak = m i=1 j=1 a2 ij = [trace(a 0 A)] 1/2. b If x i 2 R, E[x i xi 0] 1 = E[xi 2 ] 1 is the reciprocal of E[xi 2 ] which is a cotiuous fuctio of E[xi 2 ] oly if E[xi 2 ] 6= 0. Pig Yu (HKU) Large-Sample 6 / 63

8 Asymptotics for the LSE Cosistecy of bσ 2 ad s 2 Theorem Uder the assumptios of Theorem 1, bσ 2 p! σ 2 ad s 2 p! σ 2. Proof. Note that bu i = y i x 0 i b β = u i + x 0 i β x0 b iβ = u i x 0 bβ i β. Thus bu 2 i = u 2 i 2u i x 0 i bβ 0 β + bβ β xi x 0 bβ i β (3) ad bσ 2 = 1 bu i 2 i=1 Pig Yu (HKU) Large-Sample 7 / 63

9 Asymptotics for the LSE Proof. [Proof cotiue] = 1 ui u i x 0 i i=1 i=1 p! σ 2,! bβ 0 β + bβ 1 β i=1 where the last lie uses the WLLN, (2), Theorem 1 ad the CMT. Fially, sice /( k)! 1, it follows that by the CMT. s 2 = k bσ 2 p! σ 2 x i x 0 i! bβ β Oe implicatio of this theorem is that multiple estimators ca be cosistet for the populatio parameter. While bσ 2 ad s 2 are uequal i ay give applicatio, they are close i value whe is very large. Pig Yu (HKU) Large-Sample 8 / 63

10 Asymptotics for the LSE Asymptotic Normality To study the asymptotic ormality of β, b we impose the followig additioal assumptio. h Assumptio OLS.5: E[u 4 ] < ad E kxk 4i <. Theorem Uder Assumptios OLS.0, OLS.1 0, OLS.2 0, OLS.3 ad OLS.5, p bβ β d! N(0,V), where V = Q 1 ΩQ 1 with Q = E x i x 0 h i i ad Ω = E x i x 0 i u2 i. Proof. From (1), p bβ! β = 1 1 x i x 0 i i=1 1 p x i u i!. i=1 Pig Yu (HKU) Large-Sample 9 / 63

11 Asymptotics for the LSE Proof. [Proof cotiue] Note first that h xi E x 0 i u2 i h xi i E x 0 2i 1/2 h i 1/2 i E ui 4 E hkx i k 4i 1/2 h i 1/2 E ui 4 <, (4) where the first iequality is from the Cauchy-Schwarz iequality, the secod iequality is from the Schwarz matrix iequality, a ad the last iequality is from Assumptio OLS.5. So by the CLT, 1 d p x i u i! N (0,Ω). i=1 Give that 1 i=1 x ix 0 i p! Q, p bβ β d! Q 1 N (0,Ω) = N(0,V) by Slutsky s theorem. a Schwarz matrix iequality: For ay radom m matrices X ad Y, kx 0 Yk kxkkyk. This is a special form of the Cauchy-Schwarz iequality, where the ier product is defied as hx,yi = kx 0 Yk. I the homoskedastic model, V reduces to V 0 = σ 2 Q 1. We call V 0 the homoskedastic covariace matrix. Pig Yu (HKU) Large-Sample 10 / 63

12 Asymptotics for the LSE Partitioed Formula of V 0 Sometimes, to state the asymptotic distributio of part of β b as i the residual regressio, we partitio Q ad Ω as Q11 Q Q = 12 Ω11 Ω,Ω = 12. Q 21 Q 22 Ω 21 Ω 22 Recall from the proof of the FWL theorem, Q 1 = Q Q Q 12Q 1 22 Q Q 21Q 1 11 Q !, where Q 11.2 = Q 11 Q 12 Q 1 22 Q 21 ad Q 22.1 = Q 22 Q 21 Q 1 11 Q 12. Thus whe the error is homoskedastic, AVar bβ 1 = σ 2 Q , ad ACov bβ 1, β b 2 = σ 2 Q Q 12Q We ca also derive the geeral formulas i the heteroskedastic case, but these formulas are ot easily iterpretable ad so less useful. Pig Yu (HKU) Large-Sample 11 / 63

13 Asymptotics for the LSE LSE as a MoM Estimator The LSE is a MoM estimator, ad the momet coditios are E [xu] = 0 with u = y x 0 β. The sample aalog is the ormal equatio 1 x i y i x 0 i β = 0, i=1 the solutio of which is exactly the LSE. M = E x i x 0 h i i = Q, ad Ω = E x i x 0 i u2 i, so p bβ β d! N 0,Q 1 ΩQ 1 = N (0,V). Note that the asymptotic variace V takes the sadwich form. The larger the E x i x 0 i, the smaller the V. Although the LSE is a MoM estimator, it is a special MoM estimator because it ca be treated as a "projectio" estimator. Pig Yu (HKU) Large-Sample 12 / 63

14 Asymptotics for the LSE Ituitio Cosider a simple liear regressio model y i = βx i + u i, where E[x i ] is ormalized to be 0. From itroductory ecoometrics courses, ad uder homoskedasticity, x i y i i=1 bβ = xi 2 i=1 = d Cov(x,y) dvar(x), AVar bβ = σ 2 Var(x). So the larger the Var(x), the smaller the AVar bβ. Actually, Var(x) = E[xu] β. Pig Yu (HKU) Large-Sample 13 / 63

15 Asymptotics for the LSE Asymptotics for the Weighted Least Squares (WLS) Estimator The WLS estimator is a special GLS estimator with a diagoal weight matrix. Recall that which reduces to bβ WLS = whe W = diagfw 1,,w g. bβ GLS = (X 0 WX) 1 X 0 Wy,! 1 w i x i x 0 i i=1! w i x i y i i=1 Note that this estimator is a MoM estimator uder the momet coditio (check!) so p bβ WLS where V W = E w i x i x 0 h 1 i E wi 2 x i x 0 i u2 i E [w i x i u i ] = 0, d! β N (0,VW ), i E w i x i x 0 i 1. Pig Yu (HKU) Large-Sample 14 / 63

16 Covariace Matrix Estimators Covariace Matrix Estimators Pig Yu (HKU) Large-Sample 15 / 63

17 Covariace Matrix Estimators Sample Aalogs Sice Q = E x i x 0 h i i ad Ω = E x i x 0 i u2 i, ad bω = 1 x i x 0 i bu2 i i=1 bq = 1 x i x 0 i = 1 X0 X, i=1 = 1 o X0 diag bu 1 2,, bu2 X 1 X0 DX b (5) are the MoM estimators (exercise) for Q ad Ω, where bu i residuals. Give that V = Q 1 ΩQ 1, it ca be estimated by ad AVar( b β ) is estimated by bv = b Q 1 b Ω b Q 1, bv/ = X 0 X 1 X 0 b DX X 0 X 1. i=1 are the OLS Pig Yu (HKU) Large-Sample 16 / 63

18 Covariace Matrix Estimators History ad Limitatio h i People thought Ω =E x i x 0 i σ 2 i whose estimatio requires estimatig coditioal variaces. bv appeared first i the statistical literature Eicker (1967) ad Huber (1967), ad was itroduced ito ecoometrics by White (1980c). So this estimator is ofte called the "Eicker-Huber-White formula" or somethig of the kid. Other popular ames for this estimator iclude the "heteroskedasticity-cosistet (or robust) covariace matrix estimator" or the "sadwich-form covariace matrix estimator" I the homoskedastic case, we ca estimate V by b V 0 = bσ 2 b Q 1. Practical Suggestio: Use V b rather tha V b0 wheever possible. [AVar bβ why? j = i=1 w ij bu2 homo i /SSR j! [AVar bβ j = 1 i=1 bu2 i /SSR j. It is hard to judge which formula, homoskedasticity-oly or heteroskedasticity-robust, is larger (why?). Although either way is possible i theory, the heteroskedasticity-robust formula is usually larger tha the homoskedasticity-oly oe i practice. It ca be show that the former is actually more variable tha the latter, which is the price paid for robustess. Pig Yu (HKU) Large-Sample 17 / 63

19 Covariace Matrix Estimators Cosistecy of b V Theorem Uder the assumptios of Theorem 3, b V p! V. Proof. From the WLLN, b Q is cosistet. As log as we ca show b Ω is cosistet, by the CMT bv is cosistet. Usig (3) bω = 1 x i x 0 i bu2 i i=1 = 1 x i x i u2 i x i x 0 bβ i β xi u i + 1 bβ 0 2 x i x 0 i β xi. i=1 i=1 i=1 h xi From (4), E x 0 i u2 i i <, so by the WLLN, 1 i=1 x ix 0 i u2 p i! Ω. We eed oly to prove the remaiig two terms are o p (1). Pig Yu (HKU) Large-Sample 18 / 63

20 Covariace Matrix Estimators Proof. The secod term satisfies 2 0 x i x 0 bβ i β xi u i 2 0 x ix 0 bβ i β xi u i i=1 i=1 2 x i x 0 0 bβ i β xi ju i j i=1 2 kx i k 3 ju i j! β b, β i=1 where the first iequality is from the triagle iequality, a ad the secod ad third iequalities are from the Schwarz matrix iequality. a Triagle iequality: For ay m matrices X ad Y, kx + Yk kxk + kyk. Pig Yu (HKU) Large-Sample 19 / 63

21 Covariace Matrix Estimators Proof. [Proof cotiue] By Hölder s iequality, a so by the WLLN, 1 i=1 kx ik 3 ju i j h i E kx i k 3 ju i j E hkx i k 4i 3/4 h E ju i j 4i 1/4 <, h i p! E kx i k 3 ju i j <. Give that β b β = o p (1), the secod term is o p (1)O p (1) = o p (1). The third term satisfies 1 bβ 0 2 x i x 0 i β xi 1 xi x 0 bβ 0 2 i β xi i=1 i=1 1 kx i k 4 β b 2 β = op (1), i=1 where the steps follow from similar argumets as i the secod term. a Hölder s iequality: If p > 1 ad q > 1 ad 1 p + 1 q = 1, the for ay radom m matrices X ad Y, E [kx 0 Yk] E kxk p 1/p E kyk q 1/q. Pig Yu (HKU) Large-Sample 20 / 63

22 Fuctios of Parameters Fuctios of Parameters Pig Yu (HKU) Large-Sample 21 / 63

23 Fuctios of Parameters Fuctios of Parameters Let θ = r(β ) deote the parameter of iterest, where r: R k! R q. Assumptio RLS.1 0 : r() is cotiuously differetiable at the true value β ad R = β r(β )0 has rak q. Theorem Uder the assumptios of Theorem 3 ad Assumptio RLS.1 0, where θ b = r bβ, ad V θ = R 0 VR. p bθ θ d! N (0,Vθ ), Proof. By the CMT, θ b is cosistet for θ. By the Delta method, if r() is differetiable at the true value β, p bθ θ = p r bβ d! r(β ) R 0 N (0,V) = N (0,V θ ) where V θ = R 0 VR > 0 if R has full rak q. Pig Yu (HKU) Large-Sample 22 / 63

24 Fuctios of Parameters Estimatio of V θ A atural estimator of V θ is bv θ = b R 0 b V b R, (6) where b R = r bβ 0 / β. If r() is a C (1) fuctio, the by the CMT, b V θ p! Vθ (why?). If r(β ) is liear: r(β ) = R 0 β for some k q matrix R. I this case, β r(β )0 = R ad R b = R, so V b θ = R 0b VR. For example, if R is a "selector matrix" I R = qq, 0 (k q)q so that if β = (β 0 1,β 0 2 )0, the θ = R 0 β = β 1 ad the upper-left block of b V. bv θ = (I,0) b V I 0 = b V 11, Pig Yu (HKU) Large-Sample 23 / 63

25 The t Test The t Test Pig Yu (HKU) Large-Sample 24 / 63

26 The t Test The Studetized Statistic Whe q = 1 (so r(β ) is real-valued), the stadard error 1 for θ b is the square root of 1 V b θ, that is, s bθ = 1/2p R b0b VR. b The studetized statistic θ t (θ) = b θ. s bθ Sice p bθ θ d! N(0,Vθ ) ad p s bθ p! p Vθ, by Slutsky s theorem, we have Theorem Uder the assumptios of Theorem 5, t (θ) d! N(0,1). Thus the asymptotic distributio of the t-ratio t (θ) is the stadard ormal. Sice the stadard ormal distributio does ot deped o the parameters, we say that t (θ) is asymptotically pivotal. 1 A stadard error for a estimator is a estimate of the stadard deviatio of that estimator Pig Yu (HKU) Large-Sample 25 / 63

27 The t Test The t Test The most commo oe-dimesioal hypotheses are the ull agaist the alterative where θ 0 is some pre-specified value. H 0 : θ = θ 0, (7) H 1 : θ 6= θ 0, (8) The stadard test for H 0 agaist H 1 is based o the absolute value of the t-statistic, θ t = t (θ 0 ) = b θ 0. s bθ d d Uder H 0, t! Z N(0,1), so jt j! jz j by the CMT. G(u) = P(jZ j u) = Φ(u) (1 Φ(u)) = 2Φ(u) 1 Φ(u) is called the asymptotic ull distributio. Pig Yu (HKU) Large-Sample 26 / 63

28 The t Test Asymptotic Size ad Asymptotic Critical Value The asymptotic size of the test is defied as the asymptotic probability of a Type I error: lim P(jt j > cjh 0 true) = P(jZ j > c) = 1 Φ(c).! The asymptotic size of the test is a simple fuctio of the asymptotic ull distributio G ad the critical value c. c is called the asymptotic critical value because it has bee selected from the asymptotic ull distributio. Let z α/2 be the upper α/2 quatile of the stadard ormal distributio. That is, if Z N(0,1), the P(Z > z α/2 ) = α/2 ad P(jZ j > z α/2 ) = α. For example, z.025 = 1.96 ad z.05 = A test of asymptotic sigificace α rejects H 0 if jt j > z α/2. Otherwise the test does ot reject, or "accepts" H 0. Pig Yu (HKU) Large-Sample 27 / 63

29 The t Test Oe-Sided Alterative The alterative hypothesis (8) is called a two-sided alterative. Oe-sided alteratives: or H 1 : θ > θ 0 (9) H 1 : θ < θ 0. (10) Tests of (7) agaist (9) or (10) are based o the siged t-statistic t. The hypothesis (7) is rejected i favor of (9) if t > c (why?) where c satisfies α = 1 Φ(c). The critical values are smaller tha for two-sided tests (why?). Specifically, the asymptotic 5% critical value is c = Thus, we reject (7) i favor of (9) if t > Should we use the two-sided critical value 1.96 or the oe-sided critical value 1.645? The aswer is that we should use oe-sided tests ad critical values oly whe the parameter space is kow to satisfy a oe-sided restrictio such as θ θ 0. Sice liear regressio coefficiets typically do ot have a priori sig restrictios, we coclude that two-sided tests are geerally appropriate. Pig Yu (HKU) Large-Sample 28 / 63

30 p-value p-value Pig Yu (HKU) Large-Sample 29 / 63

31 p-value p-value The rejectio/acceptace dichotomy is associated with the Neyma-Pearso approach to hypothesis testig; p-value is associated with R.A. Fisher. Defie the tail probability, or asymptotic p-value fuctio p(t) = P (jz j > t) = 1 G(t) = 2(1 Φ(t)), where G() is the cdf of jz j. The the asymptotic p-value of the statistic jt j is p = p(jt j). So the p-value is the probability of obtaiig a test statistic result at least as extreme as the oe that was actually observed or the smallest sigificace level at which the ull would be rejected, assumig that the ull is true. Sice the distributio fuctio G is mootoically icreasig, the p-value is a mootoically decreasig fuctio of t ad is a equivalet test statistic. [Figure 1] Caveat: the p-value p should ot be iterpreted as the probability that either hypothesis is true. For example, p is NOT the probability that the ull hypothesis is false. Rather, p is a measure of the stregth of iformatio agaist the ull hypothesis. Pig Yu (HKU) Large-Sample 30 / 63

32 p-value Figure: Obtaiig the p-value i a Two-Sided t-test: jt j = 1.85 Pig Yu (HKU) Large-Sample 31 / 63

33 p-value cotiue... A equivalet statemet of a Neyma-Pearso test is to reject at the α level if ad oly if p < α. I this sese, p-values ad hypothesis tests are equivalet sice p < α if ad oly if jt j > z α/2. The p-value is more geeral, however, i that the reader is allowed to pick the level of sigificace α, i cotrast to Neyma-Pearso rejectio/acceptace reportig where the researcher picks the level. The p-value fuctio has simply made a uit-free trasformatio of the test d statistic. That is, uder H 0, p! U[0,1], regardless of the complicatio of the distributio of the origial test statistic. Why? The asymptotic distributio of jt j is G(x) = 1 p(x). Thus P (1 p u) = P (1 p(jt j) u) = P (G(jt j) u) = P jt j G 1 (u)! G(G 1 (u)) = u, establishig that 1 p d! U[0,1], from which it follows that p d! U[0,1]. Pig Yu (HKU) Large-Sample 32 / 63

34 Cofidece Iterval Cofidece Iterval Pig Yu (HKU) Large-Sample 33 / 63

35 Cofidece Iterval Cofidece Iterval A cofidece iterval (CI) C is a iterval estimate of θ 2 R which is assumed to be fixed. It is a fuctio of the data ad hece is radom. So it is ot correct to say that "θ will fall i C with high probability", rather, C is desiged to cover θ with high probability. Either θ 2 C or θ /2 C. The coverage probability is P(θ 2 C ). We typically caot calculate the exact coverage probability P(θ 2 C ). However we ofte ca calculate the asymptotic coverage probability lim! P(θ 2 C ). We say that C has asymptotic (1 α) coverage for θ if P(θ 2 C )! 1 α as!. Pig Yu (HKU) Large-Sample 34 / 63

36 Cofidece Iterval Test Statistic Iversio test statistic iversio method: collectig parameter values which are ot rejected by a statistical test. The t-test rejects H 0 : θ = θ 0 if jt (θ 0 )j > z α/2. A CI is the costructed usig the values for which this test does ot reject [Figure 2]: 8 C = < θjjt (θ)j z α/2 = : θ θ z α/2 b θ 9= z α/2 s bθ ; h = bθ zα/2 s bθ, θ b i + z α/2 s bθ. The most commo professioal hchoice for 1 i α is h95%, or α i =.05. This correspods to selectig the CI bθ 1.96s bθ bθ 2s bθ. The iterval has bee costructed so that as!, P (θ 2 C ) = P (jt (θ)j z α/2 )! P (jz j z α/2 ) = 1 α, so C is ideed a asymptotic (1 α) CI. Pig Yu (HKU) Large-Sample 35 / 63

37 Cofidece Iterval Figure: Test Statistic Iversio: acceptace regio for θ b h at θ is θ i z α/2 s bθ,θ + z α/2 s bθ Pig Yu (HKU) Large-Sample 36 / 63

38 The Wald Test The Wald Test Pig Yu (HKU) Large-Sample 37 / 63

39 The Wald Test The Wald Test Whe θ = r(β ) is a q 1 vector, it is desired to test the joit restrictios simultaeously. I this case the t-statistic approach does ot work. We have the ull ad alterative H 0 : θ = θ 0 vs H 1 : θ 6= θ 0. The Wald statistic for H 0 agaist H 1 is 0 W = bθ θ 0 bv 1 bθ θ θ 0. We have kow that p bθ θ 0 d! N (0,Vθ ), ad b V θ p! Vθ uder H 0. So W d! χ 2 q uder the ull (why?). Whe q = 1, W = t 2 (check). Correspodigly, the asymptotic distributio χ 2 1 = N(0,1)2. A asymptotic Wald test rejects H 0 i favor of H 1 if W exceeds χ 2 q,α, the upper-α quatile of the χ 2 q distributio. For example, χ2 1,.05 = 3.84 = z The asymptotic p-value for W is p = p(w ), where p(x) = P(χ 2 q x) is the tail probability fuctio of the χ 2 q distributio. As before, the test rejects at the α level if p < α, ad p is asymptotically U[0,1] uder H 0. Pig Yu (HKU) Large-Sample 38 / 63

40 The Wald Test Cofidece Regio Cofidece Regio As CIs, we ca costruct cofidece regios for multiple parameters, e.g., θ = r(β ) 2 R q. By the test statistic iversio method, a asymptotic (1 θ is o C = θjw (θ) χ 2 q (α), where W (θ) = plae. 0 bθ θ bv 1 bθ θ α) cofidece regio for θ. Sice b V θ > 0, C is a ellipsoid i the θ Assume q = 2 ad θ = (β 1,β 2 ) 0 ; the C is a ellipse i the (β 1,β 2 ) plae as show i Figure 3. C 0 CI(β 1 )CI(β 2 ) does ot work! that is, P((β 1,β 2 ) 2 C 0 ) 6= 1 α i geeral. Pig Yu (HKU) Large-Sample 39 / 63

41 The Wald Test Cofidece Regio 0 Figure: Cofidece Regio for (β 1,β 2 ) Pig Yu (HKU) Large-Sample 40 / 63

42 Problems with Tests of Noliear Hypotheses Problems with Tests of Noliear Hypotheses Pig Yu (HKU) Large-Sample 41 / 63

43 Problems with Tests of Noliear Hypotheses Problems with Tests of Noliear Hypotheses Take the model y i = β + u i,u i N(0,σ 2 ) ad cosider the hypothesis H 0 : β = 1. Let β b ad bσ 2 be the sample mea ad variace of y i. The stadard Wald test for H 0 is 2 bβ 1 W = bσ 2. Note that H 0 is equivalet to the hypothesis H 0 (s): β s = 1, for ay positive iteger s. Lettig r(β ) = β s, ad otig R = sβ s 1, we fid that the stadard Wald test for H 0 (s) is bβ s 2 1 W (s) = bσ 2 s 2b β 2s 2. While the hypothesis β s = 1 is uaffected by the choice of s, the statistic W (s) varies with s. Pig Yu (HKU) Large-Sample 42 / 63

44 Problems with Tests of Noliear Hypotheses Figure: Wald Statistic as a fuctio of s: /bσ 2 = 10 I each case there are values of s for which the test statistic is sigificat relative to asymptotic critical values, while there are other values of s for which the test statistic is isigificat. Pig Yu (HKU) Large-Sample 43 / 63

45 Problems with Tests of Noliear Hypotheses Fiite-Sample Distributio The first-order asymptotic theory is ot useful to help pick s, as W (s) d! χ 2 1 uder H 0 for ay s, while Mote Carlo simulatio ca be quite useful as a tool to study ad compare the exact distributios of statistical procedures i fiite samples. The method uses radom simulatio to create artificial datasets, to which we apply the statistical tools of iterest. This produces radom draws from the statistic s samplig distributio. Through repetitio, features of this distributio ca be calculated. Let s focus o the Type I error of the test usig the asymptotic 5% critical value the probability of a false rejectio, P(W (s) > 3.84jβ = 1). This probability depeds oly o s,, ad σ 2 i this simple model. Table 1 reports the simulatio estimate of the Type I error probability from 50,000 radom samples. The probabilities i Table 1 are calculated as the percetage of the 50,000 simulated Wald statistics W (s) which are larger tha The ull hypothesis β s = 1 is true, so these probabilities are Type I error. Pig Yu (HKU) Large-Sample 44 / 63

46 Problems with Tests of Noliear Hypotheses σ = 1 σ = 3 s = 20 = 100 = 500 = 20 = 100 = Table 1: Type I Error Probability of Asymptotic 5% W (s) Test Note: Rejectio frequecies from 50,000 simulated radom samples Pig Yu (HKU) Large-Sample 45 / 63

47 Problems with Tests of Noliear Hypotheses Results from Table 1 The ideal Type I error probability is 5%(.05) with deviatios idicatig distortio. Type I error rates betwee 3% ad 8% are cosidered reasoable. Error rates above 10% are cosidered excessive. Rates above 20% are uacceptable. Whe comparig statistical procedures, we compare the rates row by row, lookig for tests for which rejectio rates are close to 5% ad rarely fall outside of the 3% 8% rage. For this particular example the oly test which meets this criterio is the covetioal W = W (1) test. Ay other choice of s leads to a test with uacceptable Type I error probabilities; as s icreases test performace deteriorates. Impact of variatio i sample size : i each case, the Type I error probability improves towards 5% as the sample size icreases. There is, however, o magic choice of for which all tests perform uiformly well. Pig Yu (HKU) Large-Sample 46 / 63

48 Problems with Tests of Noliear Hypotheses A More Complicated Example Take the model ad the hypothesis y i = β 0 + x 1i β 1 + x 2i β 2 + u i,e[x i u i ] = 0 (11) H 0 : θ β 1 β 2 = θ 0. bβ = bβ 0, b β 1, b β 2 0, b Vbβ = b V/, ad b θ = b β 1 / b β 2. A t-statistic for H 0 is β t 1 = b 1 / β b 2 θ 0 s( θ) b. where s( θ) b = br 0 b b 1/2 0 1Vbβ R 1 with R1 b = 0, b 1 bβ, 1. β 2 A equivalet ull hypothesis is bβ 2 2 H 0 : β 1 θ 0 β 2 = 0. Pig Yu (HKU) Large-Sample 47 / 63

49 Problems with Tests of Noliear Hypotheses cotiue... A t-statistic based o this formulatio of the hypothesis is t 2 = b β 1 θ 0 b β 2 R 0 2 b V bβ R 2 1/2, where R 2 = (0,1, θ 0 ) 0. Mote Carlo Simulatio: let x 1i ad x 2i be mutually idepedet N(0,1) variables, u i be a idepedet N(0,σ 2 ) draw with σ = 3, ad ormalize β 0 = 1 ad β 1 = 1. This leaves β 2 as a free parameter, alog with sample size. Ideally, the etries i Table 2 should be However, the rejectio rates for the t 1 statistic diverge greatly from this value, especially for small values of β 2. The left tail probabilities P(t 1 < 1.645) greatly exceed 5%, while the right tail probabilities P(t 1 > 1.645) are close to zero i most cases. I cotrast, the rejectio rates for the liear t 2 statistic are ivariat to the value of β 2, ad are close to the ideal 5% rate for both sample sizes. The implicatio of Table 2 is that the two t-ratios have dramatically differet samplig behaviors. Pig Yu (HKU) Large-Sample 48 / 63

50 Problems with Tests of Noliear Hypotheses = 100 = 500 P(t < 1.645) P(t > 1.645) P(t > 1.645) P(t > 1.645) β 2 t 1 t 2 t 1 t 2 t 1 t 2 t 1 t Table 2: Type I Error Probability of Asymptotic 5% t-tests Pig Yu (HKU) Large-Sample 49 / 63

51 Problems with Tests of Noliear Hypotheses Solutios The commo message from both examples is that Wald statistics are sesitive to the algebraic formulatio of the ull hypothesis. Solutio I: If the hypothesis ca be expressed as a liear restrictio o the model parameters, this formulatio should be used. If o liear formulatio is feasible, the the "most liear" formulatio should be selected, ad alteratives to asymptotic critical values should be cosidered. Solutio II: Cosider alterative tests to the Wald statistic, such as the miimum distace statistic. Pig Yu (HKU) Large-Sample 50 / 63

52 Test Cosistecy Test Cosistecy Pig Yu (HKU) Large-Sample 51 / 63

53 Test Cosistecy Test Cosistecy Defiitio A test of H 0 : θ 2 Θ 0 is cosistet agaist fixed alteratives if for all θ 2 Θ 1, P (Reject H 0 jθ)! 1 as!. Suppose that y i is i.i.d. N(µ,1). Cosider the t-statistic t (µ) = p (y tests of H 0 : µ = 0 agaist H 1 : µ > 0. We reject H 0 if t = t (0) > c. Note that t = t (µ) + p µ µ), ad ad t (µ) Z has a exact N(0,1) distributio. This is because t (µ) is cetered at the true mea µ, while the test statistic t (0) is cetered at the (false) hypothesized mea of 0. The power of the test is P (t > cjθ) = P Z + p µ > c p = 1 Φ c µ. This fuctio is mootoically icreasig i µ ad, ad decreasig i c. For ay c ad µ 6= 0, the power icreases to 1 as!. This meas that for µ 2 H 1, the test will reject H 0 with probability approachig 1 as the sample size gets large. Pig Yu (HKU) Large-Sample 52 / 63

54 Test Cosistecy cotiue... For tests of the form Reject H 0 if T > c, a sufficiet coditio for test cosistecy is that T! with probability oe for all θ 2 Θ 1. I geeral, the t-test ad Wald test are cosistet agaist fixed alteratives. For example, i testig H 0 : θ = θ 0, q sice s bθ = bv θ /. θ t = b θ 0 θ = b θ + s bθ s bθ p (θ θ 0 ) qb Vθ Term I + Term II (12) d Term I! N(0,1). Term II = 0 if θ = θ 0, ad p! + if θ > θ 0, ad p! if θ < θ 0. Thus both the two-sided t-test ad oe-sided t-test are cosistet. For aother example, The Wald statistic for H 0 : θ = r(β ) = θ 0 agaist H 1 : θ 6= θ 0 is 0 W = bθ θ 0 bv 1 bθ θ θ 0. Uder H 1, θ b p! θ 6= θ 0. Thus 0 bθ θ 0 bv 1 bθ p! θ θ 0 (θ θ 0 ) 0 Vθ 1 (θ θ 0 ) > 0. Hece uder H 1, p W!. This implies that Wald tests are cosistet. Pig Yu (HKU) Large-Sample 53 / 63

55 Asymptotic Local Power Asymptotic Local Power Pig Yu (HKU) Large-Sample 54 / 63

56 Asymptotic Local Power Local Alteratives Cosistecy is a good property for a test, but does ot give a useful approximatio to the power of a test. To approximate the power fuctio we use local alteratives. This is similar to our aalysis of restrictio estimatio uder misspecificatio. The techique is to idex the parameter by sample size so that the asymptotic distributio of the statistic is cotiuous i a localizig parameter. I the t-test, we cosider parameter vectors β which are idexed by sample size ad satisfy the real-valued relatioship θ = r(β ) = θ 0 + 1/2 h, (13) where the scalar h is called a localizig parameter, ad the sequece of local alteratives θ is called a Pitma drift or a Pitma sequece. We idex β ad θ by sample size to idicate their depedece o. The way to thik of (13) is that the true value of the parameters are β ad θ. The parameter θ is close to the hypothesized value θ 0, with deviatio 1/2 h. Pig Yu (HKU) Large-Sample 55 / 63

57 Asymptotic Local Power How to Uderstad Local Alteratives? The specificatio (13) states that for ay fixed h, θ approaches θ 0 as gets large. Thus θ is close or local to θ 0. For a fixed alterative, the power will coverge to 1 as!. To offset the effect of icreasig, we make the alterative harder to distiguish from H 0 as gets larger. The rate 1/2 is the correct balace betwee these two forces. The cocept of a localizig sequece (13) might seem odd at first as i the actual world the sample size caot mechaically affect the value of the parameter. Thus (13) should ot be iterpreted literally. Istead, it should be iterpreted as a techical device which allows the asymptotic distributio of the test statistic to be cotiuous i the alterative hypothesis. Pig Yu (HKU) Large-Sample 56 / 63

58 Asymptotic Local Power Local Power Fuctio Similarly as i (12), θ t = b θ 0 θ = b θ + s bθ s bθ p (θ θ 0 ) q bv θ d! Z + δ uder the local alterative, where Z N(0,1) ad δ = h/ p V θ. I testig the oe-sided alterative H 1 : θ > θ 0, a t-test rejects H 0 for t > z α. The asymptotic local power of this test is the limit of the rejectio probability uder the local alterative, lim P (Reject H 0jθ = θ )! = lim P (t > z α jθ = θ )! = P (Z + δ > z α ) = 1 Φ(z α δ ) = Φ(δ z α ) π α (δ ). We call π α (δ ) the local power fuctio. Pig Yu (HKU) Large-Sample 57 / 63

59 Asymptotic Local Power Figure: Asymptotic Local Power Fuctio of Oe-Sided t-test with Differet Asymptotic Sizes Pig Yu (HKU) Large-Sample 58 / 63

60 Asymptotic Local Power How to Read the Local Power Fuctio? We do ot cosider δ < 0 sice θ should be greater tha θ 0. δ = 0 correspods to the ull hypothesis so π α (0) = α. The power fuctios are mootoically icreasig i both δ ad α. The mootoicity with respect to α is due to the iheret trade-off betwee size ad power. The coefficiet δ ca be iterpreted as the parameter deviatio measured as a multiple of the stadard error s bθ. q Why? s bθ = 1/2 bv θ 1/2p V θ, so δ = p h 1/2 h = θ θ 0 Vθ s bθ s bθ Thus i the figure, we ca iterpret the power fuctio at δ = 1 (e.g., 26% for a 5% size test) as the power whe the parameter θ is oe stadard error above the hypothesized value. Pig Yu (HKU) Large-Sample 59 / 63

61 Asymptotic Local Power Read Vertically or Horizotally? I the figure there is a vertical dotted lie at δ = 1, showig that the asymptotic local power π α (1) equals 39% for α = 0.10, equals 26% for α = 0.05 ad equals 9% for α = This is the differece i power across tests of differig sizes, holdig fixed the parameter i the alterative. I the figure there is also a horizotal dotted lie at 50% power. 50% power is a useful bechmark, as it is the poit where the test has equal odds of rejectio ad acceptace. The dotted lie crosses the three power curves at δ = 1.29 (α = 0.10), δ = 1.65 (α = 0.05), ad δ = 2.33 (α = 0.01). This meas that the parameter θ must be at least 1.65 stadard errors above the hypothesized value for the oe-sided test to have 50% (approximate) power. The ratio of these values (e.g., 1.65/1.29 = 1.28 for the asymptotic 5% versus 10% tests) measures the relative parameter magitude eeded to achieve the same power. (Thus, for a 5% size test to achieve 50% power, the parameter must be 28% larger tha for a 10% size test.) The square of this ratio (e.g., (1.65/1.29) 2 = 1.64) ca be iterpreted as the icrease i sample size eeded to achieve the same power uder fixed parameters. That is, to achieve 50% power, a 5% size test eeds 64% more observatios tha a 10% size test. Why? δ = h/ p V θ = p (θ θ 0 ) / p V θ. Holdig θ ad V θ fixed, δ 2. Pig Yu (HKU) Large-Sample 60 / 63

62 Asymptotic Local Power Local Power i the Wald Test Local parametrizatio: where h is a q 1 vector. Uder (14), p bθ Applied to the Wald statistic, W = θ = r(β ) = θ 0 + 1/2 h, (14) θ 0 = p bθ bθ θ 0 0 bv 1 θ θ + h d! Z h N(h,V θ ). bθ d! θ 0 Z 0 h Vθ 1 Z h χ 2 q (λ ). where χ 2 q (λ ) is a o-cetral chi-square distributio with q degrees of freedom ad o-cetral parameter λ = h 0 Vθ 1 h. Uder the ull, h = 0, ad the χ 2 q (λ ) distributio the degeerates to the usual χ2 q distributio. I the case of q = 1, jz + δj 2 χ 2 1 (λ ) with λ = δ 2. The asymptotic local power of the Wald test at the level α is P χ 2 q (λ ) > χ2 q,α π q,α (λ ). Pig Yu (HKU) Large-Sample 61 / 63

63 Asymptotic Local Power Figure: Asymptotic Local Power Fuctio of the Wald Test Pig Yu (HKU) Large-Sample 62 / 63

64 Asymptotic Local Power Read the Local Power Fuctio The power fuctios are mootoically icreasig i λ ad asymptote to oe. The figure also shows the power loss for fixed o-cetrality parameter λ as the dimesioality of the test icreases. The power curves shift to the right as q icreases, resultig i a decrease i power. This is illustrated by the dotted lie at 50% power. The dotted lie crosses the three power curves at λ = 3.85 (q = 1), λ = 4.96 (q = 2), ad λ = 5.77 (q = 3). The ratio of these λ values correspod to the relative sample sizes eeded to obtai the same power (why?). Thus icreasig the dimesio of the test from q = 1 to q = 2 requires a 28% icrease i sample size, or a icrease from q = 1 to q = 3 requires a 50% icrease i sample size, to obtai a test with 50% power. Ituitio: whe testig more restrictios, we eed more deviatio from the ull (or equivaletly, more data poits) to achieve the same power. Pig Yu (HKU) Large-Sample 63 / 63