GENERALISED LEAST SQUARES AND RELATED TOPICS

Transcription

1 GENERALISED LEAST SQUARES AND RELATED TOPICS Haris Psaradakis Birkbeck, University of London Nonspherical Errors Consider the model y = Xβ + u, E(u) =0, E(uu 0 )=σ 2 Ω, where Ω is a symmetric and positive definite n n matrix (and, for simplicity, X is nonstochastic) This model allows the errors to be heteroskedastic and/or autocorrelated, and is often referred to as the generalised linear regression model If ˆβ OLS =(X 0 X) X 0 y is the OLS estimator of β, itcanbeshownthat: E(ˆβ OLS )=β +(X 0 X) X 0 E(u) E( β) =β var(ˆβ OLS )=E[(X 0 X) X 0 uu 0 X(X 0 X) ]=σ 2 (X 0 X) X 0 ΩX(X 0 X) Crucially, since var(ˆβ OLS ) 6= σ 2 (X 0 X), statistical inference based on ˆσ 2 OLS(X 0 X),with ˆσ 2 OLS =(y ˆβ OLS X) 0 (y ˆβ OLS X)/(n k), is invalid and likely to be unreliable plim ˆβ OLS = β if the largest eigenvalue of Ω is bounded for all n and the largest eigenvalue of (X 0 X) tends to zero as n ˆσ 2 OLS is, in general, a biased and inconsistent estimator of σ 2

2 2 GLS Estimator Assume that Ω is known Since Ω is a (real) symmetric matrix with eigenvalues λ,,λ n (say), it can be diagonalised as S 0 ΩS = Λ, where Λ =diag(λ,, λ n ) and S is an orthogonal matrix (S 0 = S ) with the eigenvectors of Ω as columns The diagonal matrix Λ is positive definite and may be factored as Λ = Λ /2 Λ /2,where ³p Λ /2 =diag λ,, p λ n Letting P = SΛ /2,wehave S 0 ΩS = Λ Ω = SΛS 0 Ω =(S 0 ) Λ S Ω = SΛ /2 Λ /2 S Ω = PP 0, and Ω =(P 0 ) P P 0 ΩP = I n By premultiplying the model by P 0, we can transform it into a specification which satisfies the usual Gauss Markov assumptions More specifically, we have P 0 y = P 0 Xβ + P 0 u, or y 0 = X 0 β + u 0 Notice that E(u 0 )=E(P 0 u)=0and E(u 0 u 0 0)=E(P 0 uu 0 P )=σ 2 P 0 ΩP = σ 2 I n Since the transformed model satisfies the classical assumptions, efficient estimation of β canbeachievedbyols: ˆβ = (X 0 0X 0 ) X 0 0y 0 =(X 0 PP 0 X) X 0 PP 0 y ˆβ =(X 0 Ω X) X 0 Ω y This estimator is the generalised least squares (GLS) estimator of β It can also be obtained as the solution to the problem: min β (y Xβ) 0 Ω (y Xβ) 2

3 An unbiased estimator of σ 2 is ˆσ 2 = (n k) (y 0 X 0ˆβ) 0 (y 0 X 0ˆβ) = (n k) (P 0 y P 0 X ˆβ) 0 (P 0 y P 0 X ˆβ) = (n k) (y X ˆβ) 0 PP 0 (y X ˆβ) = (n k) (y X ˆβ) 0 Ω (y X ˆβ) Properties of the GLS estimator: E(ˆβ) =β +(X 0 0X 0 ) X 0 0E(u 0 )=β var(ˆβ) =σ 2 (X 0 0X 0 ) = σ 2 (X 0 PP 0 X) = σ 2 (X 0 ΩX) Since the transformed model satisfies the assumptions of the Gauss Markov theorem, it follows that the GLS estimator (which is obtained by applying OLS to the transformed model) is the BLUE of β ThisresultisknownasAitken stheorem If u N(0,σ 2 Ω), then ˆβ N(β,σ 2 (X 0 Ω X) ) Linear restrictions of the form H 0 : Rβ r =0(R being q k with full rank q k) can be tested using the F -statistic F = (Rˆβ r) 0 [R(X 0 Ω X) R 0 ] (Rˆβ r) qˆσ 2 H 0 F (q, n k) If lim (n X 0 0X 0 )= lim(n X 0 Ω X)=Q 0 is finite and positive definite, then ˆβ p β; ifn /2 X0u 0 0 = n /2 X 0 Ω u d N(0,σ 2 Q 0 ),then n(ˆβ β) d N(0,σ 2 Q 0 ) as n ˆσ 2 is unbiased and consistent 3 Feasible GLS In practice Ω istypicallyunknown,butitisoftenassumedtodependinaknownwayona vector of unknown parameters α If ˆα is a consistent estimator of α, we can use ˆΩ = Ω(ˆα) 3

4 in lieu of Ω to obtain the feasible GLS (FGLS) estimator β =(X 0 ˆΩ X 0 ) X 0 ˆΩ y The FGLS β is consistent for β if µ plim n X0 ˆΩ X is finite and nonsingular and µ plim n X0 ˆΩ u =0 The FGLS β is asymptotically equivalent to the GLS ˆβ (so β and ˆβ have the same asymptotic distribution) if and µ plim n X0 ˆΩ X µ plim n X 0 ˆΩ u µ = plim n X0 Ω X µ = plim n X 0 Ω u Except for some simple cases, the finite-sample properties and exact distribution of FGLS estimators are unknown We note that FGLS is possible only when some structure is imposed on Ω With an unrestricted Ω, therearen(n +)/2 parameters in σ 2 Ω, which cannot be estimated from asampleofsizen 4 Maximum Likelihood In the generalised linear regression model with u N(0,σ 2 Ω) and known Ω, thegls estimator is the MLE To verify this, note that y N(Xβ,σ 2 Ω) and so the likelihood function is L(β,σ 2 )=(2π) n/2 [det(σ 2 Ω)] /2 exp 2 (y Xβ)0 (σ 2 Ω) (y Xβ) 4

5 with log L = n 2 log 2π 2 log det(σ2 Ω) 2 (y Xβ)0 (σ 2 Ω) (y Xβ) Noting that = n 2 log 2π n 2 log σ2 2 log det Ω 2σ 2 (y Xβ)0 Ω (y Xβ) (y Xβ) 0 Ω (y Xβ)=y 0 Ω y 2β 0 X 0 Ω y + β 0 X 0 Ω Xβ, the necessary conditions for maximisation of log L are: log L β = 2X 0 Ω y +2X 0 Ω Xβ =0, log L σ 2 The solution of the likelihood equations is: = n 2σ 2 + 2σ 4 (y Xβ)0 Ω (y Xβ)=0 ˆβ ML =(X 0 Ω X) X 0 Ω y = ˆβ, We also have ˆσ 2 ML = n (y X ˆβ) 0 Ω (y X ˆβ) 2 log L β β 0 = σ 2 X0 Ω X, log L β σ 2 = 2σ 4 X0 Ω (y Xβ), log L = n σ 4 2σ 4 2σ (y 6 Xβ)0 Ω (y Xβ), and so the Fisher information matrix for (β,σ 2 ) can be shown to be X 0 Ω X σ n 2σ 4 Finally, n(ˆβml β) n(ˆσ 2 ML σ 2 ) where Q 0 = lim (n X 0 Ω X) d N 0 0 5, σ2 Q σ 4,

6 5 Heteroskedasticity Consider the following model with heteroskedastic errors: y = Xβ + u, E(u) =0, E(uu 0 )=σ 2 Ω, Ω =diag(ω 2,, ω 2 n) This is a special case of the generalised linear regression model 5 Hetroskedasticity-Robust Inference As discussed earlier, the OLS estimator ˆβ OLS =(X 0 X) X 0 y of β is still unbiased and consistent (although not BLUE) in the presence of heteroskedasticity Furthermore, it can be shown that, under fairly mild regularity conditions, ˆβ OLS a N(β,n σ 2 Q MQ ), where µ Q = plim n X0 X and and Q and M are assumed to be finite and positive definite µ M = plim n X0 ΩX, If Ω is known, efficient estimates can be obtained by means of GLS or ML If Ω is unknown, one may specify a model for ω 2 t, estimate this model (and hence Ω), and apply FGLS However, efficiency gains from FGLS are guaranteed only if the model for the error variances is correct For this reason, it has become popular to estimate β by OLS even when heteroskedasticity is suspected but to adjust the standard errors and related test statistics so that they are valid in the presence of arbitrary heteroskedasticity Halbert White showed that it is possible to construct an estimator of the asymptotic covariance matrix of ˆβ OLS which is consistent in the presence of heteroskedasticity of unknown form Such an estimator is called a heteroskedasticity-consistent covariance matrix estimator Recall that the asymptotic covariance matrix of ˆβ OLS is µ n plim n X0 X µ σ 2 plim n X0 ΩX plim µ n X0 X 6

7 In general, it is impossible to estimate Ω consistently since it has n elements However, n σ 2 X 0 ΩX = n σ P 2 n t= ω2 t x t x 0 t (x 0 t is the t th row of X) has only k 2 elements and can be estimated consistently by n X 0 ˆΩX, whereˆω may be one of several inconsistent estimator of σ 2 Ω The heteroskedasticity-consistent (or heteroskedasticity-robust) estimator of the asymptotic covariance matrix of ˆβ OLS is then given by Ĥ =(X 0 X) (X 0 ˆΩX)(X 0 X) White proposed using ˆΩ = diag(û 2,, û 2 n), where û t are OLS residuals Alternative estimators can be obtained by replacing û 2 t in the main diagonal of ˆΩ with one of the following: µ n û 2 t, n k û 2 t h t, or û 2 t ( h t ) 2, where h t is t th diagonal element of X(X 0 X) X 0 Such estimators tend to have better finite-sample properties than the original White estimator that uses û 2 t The square roots of the elements on the main diagonal of Ĥ are often referred to as heteroskedasticityconsistent (or Eicker White) standard errors When testing hypotheses about β, tests which are asymptotically robust to heteroskedasticity of unknown form can be constructed by using a heteroskedasticity-consistent estimator of the asymptotic covariance matrix of ˆβ OLS instead of the usual OLS covariance matrix estimator 52 Testing for Heteroskedasticity 52 The Breusch Pagan Godfrey Koenker Test The Breusch Pagan Godfrey Koenker LM test for heteroskedasticity tests H 0 : E(u 2 t )= σ 2 against H : E(u 2 t ) = σ 2 h(ztα), 0 where h( ) is an arbitrary positive function with h(0) = and z t is a vector of independent variables (in Koenker s version of the test, z t = x t ) The null hypothesis of homoskedasticity is equivalent to α =0AnLMtestof this hypothesis can be carried out by obtaining the R 2 in the auxiliary regression of the squared OLS residuals from the model (û 2 t ) on a constant term and the variables in z t The 7

8 Breusch Pagan Godfrey Koenker test statistic is H BPK = nr 2, which is asymptotically χ 2 -distributed under H 0 with degrees of freedom equal to the dimension of z t 522 The White Test The White LM test tests the null hypothesis of homoskedasticity against the alternative of heteroskedasticity of unspecified form The test can be carried out by obtaining the R 2 in the auxiliary regression of the squared OLS residuals (û 2 t ) on a constant, all the original regressors in X, their squares and their cross-products The White LM test statistic is H W = nr 2,whichhasanasymptoticχ 2 (p) distribution under the null, where p is the number of regressors in the auxiliary regression, excluding the intercept An alternative version of the test is based on the standard F -testforthehypothesisthatallcoefficients in the auxiliary regression, except the constant, are equal to zero We note that rejection of the null may be due to genuine heteroskedasticity, but it may also be due to some other type of specification error (such as an incorrect functional form or omitted explanatory variables) 6 Autocorrelation Consider the linear regression model with AR() errors: y t = x 0 tβ + u t, u t = φu t + ε t, where φ < and ε t iid(0,σ 2 )ItcanbeshownthatE(u t )=0, E(u 2 t )= σ2 φ 2, and E(u t u t s )= φs σ 2 φ 2, s 0 8

9 Consequently, in the model y = Xβ + u, wehavee(uu 0 )=σ 2 Ω with φ φ 2 φ n Ω = φ φ φ n 2 φ 2 φ n φ n 2 φ n 3 6 OLS Estimation The OLS estimator ˆβ OLS =(X 0 X) X 0 y is consistent for β provided that µ µ plim n X0 u =0 and plim n X0 X = Q, with Q finite and positive definite Furthermore, under appropriate regularity conditions, ˆβ OLS a N β,n σ 2 Q MQ, where µ M = plim n X0 ΩX As in the case of heteroskedasticity, we may use the OLS estimator (in spite of its inefficiency) and adjust the standard errors and related test statistics so that they are valid in the presence of autocorrelation of unknown form The asymptotic covariance matrix of ˆβ OLS can be consistently estimated by (X 0 X) b Ψ(X 0 X) where bψ = nx û 2 t x t x 0 t + t= X nx j= t=j+ µ j û t û t j (x t x 0 t j + x t j x 0 + t), û t = y t x 0 tˆβ OLS and ( <n) is a so-called bandwidth (or truncation) parameter Standard errors computed from this estimator are referred to as heteroskedasticityand-autocorrelation-consistent (HAC) standard errors (or Newey West standard errors) They can be used to construct tests which are asymptotically valid in the presence of heteroskedasticity and/or autocorrelation of unspecified form 9

10 62 FGLS Estimation Under the AR() specification for the errors, E(uu 0 )=σ 2 Ω and Ω = PP 0,where p φ φ P 0 = 0 φ φ The efficient GLS estimator of β can be obtained via OLS in the transformed model y 0 = X 0 β + u 0, where y 0 = P 0 y = p φ 2 y y 2 φy, X 0 = P 0 X = p φ 2 x x 2 φx, u 0 = P 0 u y n φy n x n φx n In practice, φ (and thus Ω) are unknown FGLS estimation of β requires, therefore, a preliminary estimate of φ A natural consistent estimator of φ is à nx!, à nx! ˆφ = û t û t û 2 t, t=2 t=2 where û t = y t x 0 tˆβ OLS The resulting FGLS estimator of β is ˆβ F =(X 0 ˆP ˆP 0 X) X 0 ˆP ˆP 0 y, where ˆP 0 = q ˆφ ˆφ ˆφ ˆφ 0

11 The FGLS estimator ˆβ F is sometimes known as the Prais Winsten estimator (or the Cochrane Orcutt estimator, if the first observation (y,x ) is omitted from the calculations) It is possible to iterate such estimators to convergence Since the estimators are asymptotically efficient at every iteration, nothing is gained asymptotically by doing so 63 ML Estimation Since y,, y n are not mutually independent, the joint density f(y,y 2,,y n ) is not equal to the product of the marginal densities Q n t= f(y t) However, the likelihood function can be constructed by noticing that f(y,y 2 ) = f(y 2 y )f(y ), f(y,y 2,y 3 ) = f(y 3 y 2,y )f(y 2,y )=f(y 3 y 2,y )f(y 2 y )f(y ), f(y,y 2,,y n ) = f(y ) ny f(y t y t,,y ) t=2 From the transformation of the model obtained through premultiplication with P 0,we can see that y = x 0 β +( φ 2 ) /2 ε, y t = φy t +(x t φx t ) 0 β + ε t, t =2,,n Hence, if ε t N(0,σ 2 ),then y N(x 0 β,σ 2 /( φ 2 )), y t y t,, y N(φy t +(x t φx t ) 0 β,σ 2 ), t =2,,n Consequently, the log-likelihood function is nx log L(β,φ,σ 2 ) = logf(y )+ log f(y t y t,, y ) t=2 = log 2π +logσ 2 log( φ 2 ) ª φ2 (y 2 2σ 2 x 0 β) 2 n log 2π +logσ 2 nx (y 2 2σ 2 t φy t x 0 tβ + φx 0 t β) 2 Maximisation of log L with respect to (β,φ,σ 2 ) yields the MLE t=2

12 64 Testing for Autocorrelation 64 Durbin Watson Test The Durbin Watson test statistic is d = P n t=2 (û t û t ) P 2 n, t= û2 t where û t = y t x 0 tˆβ OLS Notethatd 2( ˆφ) The distribution of DW depends on X However, it is possible to compute upper and lower limits for the critical values of d that depend only upon n and k We will denote these upper and lower limits by d U and d L, respectively If d<d L, the null hypothesis H 0 : φ =0is rejected in favour of H : φ>0; ifd>d U, H 0 : φ =0is not rejected; if d L <d<d U, the test is inconclusive When testing against negative autocorrelation, H 0 : φ =0is rejected in favour of H : φ<0 if 4 d<d L ; H 0 : φ =0is not rejected if 4 d>d U ;ifd L < 4 d<d U, the test is inconclusive The Durbin Watson test is valid only if a constant term is included in the model and the regressors are nonstochastic 642 The Breusch Godfrey Test The Breusch Godfrey LM test can be used to test of autocorrelation of order higher than Examples of models which allow for autocorrelation of order p are: px AR(p) : u t = φ j u t j + ε t, j= MA(p) : u t = ε t + px θ j ε t j The LM test of the null hypothesis H 0 : no autocorrelation against the alternative H : u t AR(p) or H : u t MA(p) can be implemented by regressing the OLS residuals û t j= on x t, û t,,û t p to obtain the uncentred R 2 TheLMstatisticisBG = (n p)r 2,and it has a χ 2 (p) asymptotic distribution under the null (An alternative version of the test 2

13 is based on the standard F -test for the hypothesis that the coefficients on the p lagged residuals in the auxiliary regression are jointly equal to zero) The Breusch Godfrey test essentially assesses the significance of the covariance of the residuals with their lagged values, controlling for the intervening effect of the explanatory variables Importantly, it does not suffer from any of the shortcomings of the Durbin Watson test: it is valid regardless of whether X is stochastic or non-stochastic and it does not have an inconclusive region 65 Common Factor Restrictions It is not uncommon for linear regression models to suffer from dynamic misspecification The simplest example is failure to include a lagged dependent variable among the regressors In such cases, the residuals may display autocorrelation even when the errors are in fact serially uncorrelated Consider a model with AR() errors: H : y t = x 0 tβ + u t, u t = φu t + ε t, ε t iid(0,σ 2 ) It is easy to see that the model can be rewritten as H : y t = x 0 tβ + φu t + ε t = x 0 tβ + φ(y t x 0 t β)+ε t If we relax the nonlinear restriction that is implicit in this specification, we have H 2 : y t = x 0 tβ + φy t + x 0 t γ + ε t In other words, H is a special case of the unrestricted dynamic model H 2, obtained by imposing the so-called common factor (COMFAC) restriction γ = φβ To see why the restriction is called COMFAC, rewrite H as ( φl)y t =( φl)x 0 tβ + ε t, where L is the lag operator (so that Lz t = z t ) The common factor ( φl) appears on both sides of the equation 3

14 Tests for autocorrelation test wether φ =0in H Such tests are meaningful only if the COMFAC restriction is valid If the COMFAC restriction is valid, GLS in H will yield consistent and efficient estimates If, however, the COMFAC restriction is invalid, GLS in H will yield biased and inconsistent estimates The COMFAC restriction can be tested using a W, LR or LM test 7 Seemingly Unrelated Regressions 7 Model Zellner s seemingly unrelated regression equations (SURE) model consists of m linear regression equations, each of which satisfies the assumptions of the classical linear regression model: y i = X i β i + u i, i =,,m, with y i (n ), X i (n k i ), u i (n ), β i (k i ), rank(x i )=k i <nstackingthem equations, we have or, more compactly, y y 2 y m (mn ) = X X X m (mn k) β β 2 + u u 2 β m u m (k ) (mn ), y = Xβ + u, with k = P m i= k i If u it is the t th element of u i, we assume that E(u it )=0and σ ij, if t = s, E(u it u js )= 0, if t 6= s, 4

15 ie, the errors are homoskedastic and serially uncorrelated across observations but are contemporaneously correlated across equations If Σ =(σ ij ) is the m m contemporaneous covariance matrix, we have V E(uu 0 )= σ I n σ 2 I n σ m I n σ 2 I n σ 22 I n σ 2m I n σ m I n σ m2 I n σ mm I n (mn mn) = Σ I n V is positive definite whenever Σ is 72 FGLS Estimation Each equation of the model is, by itself, a classical linear regression Therefore, its parameters could be estimated consistently, if not efficiently, one equation at a time by OLS The model as a whole is a generalised linear regression model Therefore, the BLUE of β is the GLS estimator ˆβ =(X 0 V X) X 0 V y =[X 0 (Σ I n )X] X 0 (Σ I n )y, with var(ˆβ) =(X 0 V X) =[X 0 (Σ I n )X] If u N(0, Σ I n ),thenˆβ is also the MLE When Σ is unknown (which is likely to be the case in practice), we can use a FGLS estimator instead of GLS Let û i be the n vector of OLS residuals from equation i (Note that system OLS of the SURE is equivalent to OLS equation by equation) Since the elements of Σ are consistently estimated by ˆσ ij = û0 iû j n, i,j =,,m, the FGLS estimator can be obtained as ˆβ F = h i ³ X 0 (bσ I n )X X 0 Σ b I n y, 5

16 where bσ =(ˆσ ij ) Under general conditions on u and X, theglsandfglsofβ are consistent and have the same asymptotic distribution: µ n(ˆβf β) d N 0, plim(n X 0 V X) as n Theasymptoticcovariancematrixofˆβ F is consistently estimated by [X 0 (ˆΣ I n )X] Hence, the hypothesis Rβ r =0can be tested using the Wald statistic W =(Rˆβ F r) 0 {RX 0 (ˆΣ I n )XR 0 } (Rˆβ F r), which has a χ 2 (q) asymptotic distribution under the null, with q =rank(r) k 73 Comparison Between OLS and GLS There are two cases where OLS is algebraically equivalent to GLS and, hence, BLUE If σ ij =0for i 6= j, thenˆβ = ˆβ OLS To see why, ˆβ = [X 0 (Σ I n )X] X 0 (Σ I n )y σ XX σ Xy 0 0 σ 22 X2X σ 22 X2y 0 2 = 0 0 σ mm XmX 0 m σ mm Xmy 0 m σ (XX 0 ) 0 0 σ Xy 0 0 σ 22 (X2X 0 2 ) 0 σ 22 X2y 0 2 = 0 0 σ mm(xmx 0 m ) σ mm Xmy 0 m (XX 0 ) Xy 0 (X2X 0 2 ) X2y 0 2 = = ˆβ OLS (XmX 0 m ) Xmy 0 m 6

17 If X = X 2 = = X m,thenˆβ = ˆβ OLS To see why, put X = X 2 = = X m = X 0 Then, we have: ˆβ = [X 0 (Σ I n )X] X 0 (Σ I n )y = [(I m X0)(Σ 0 I n )(I m X 0 )] (I m X0)(Σ 0 I n )y = [(Σ X0)(I 0 m X 0 )] (Σ X0)y 0 = (Σ X0X 0 0 ) (Σ X0)y 0 = [Σ (X0X 0 0 ) ](Σ X0)y 0 = [I m (X0X 0 0 ) X0]y 0 = (X 0 X) X 0 y = ˆβ OLS 74 Kronecker Products Let A =(a ij ) and B =(b ij ) be m n and p q matrices, respectively The mp nq matrix A B = is the Kronecker product of A and B a B a 2 B a n B a 2 B a 22 B a 2n B a m B a m2 B a mn B (A B)(C D) =AC BD (A B) 0 = A 0 B 0 (A B) = A B,ifA and B are invertible tr (A B) = tr(a)tr(b), if A and B are square det (A B) =(deta) m (det B) n,a(m m), B(n n) 7