Generalized Leas Squares Augus 006 1 Modified Model Original assumpions: 1 Specificaion: y = Xβ + ε (1) Eε =0 3 EX 0 ε =0 4 Eεε 0 = σ I In his secion, we consider relaxing assumpion (4) Insead, assume ha Eεε 0 = Ω T T where Ω is a symmeric, posiive definie marix Why posiive definie? likelihood funcion is ½ f (y X) =(π) T/ Ω 1/ exp 1 ¾ (y Xβ)0 Ω 1 (y Xβ) The We wan (y Xβ) 0 Ω 1 (y Xβ) > 0 so ha densiy does no explode Provide oher, more inuiive reasons for Ω posiive definie: Consider a covariance marix Ω of a random vecor ε (wih Eε =0)haisnoposiivedefinie, ie, c : c 0 Ωc 0 Now consider he linear combinaion c 0 ε (noe ha c 0 ε is a scalar) Then Var(c 0 ε)=ec 0 εε 0 c = c 0 Eεε 0 c = c 0 Ω 0 c 0 Consider he special case where T =and µ σ Ω = 1 ρσ 1 σ ρσ 1 σ σ 1
Noe ha ρ = Corr (ε 1, ε ) Examples: 1 Time series wih auocorrelaion Heeroskedasiciy 3 Spacial correlaion Ω posdef σ 1 > 0, σ > 0, σ 1σ ρ σ 1σ > 0 σ 1σ 1 ρ > 0 ρ < 1 4 Random effecs models in panel daa Examples of Covariance Marices 1 Le Then y = Xβ + ε ε = ρε 1 + e e iid 0, σ e Eε =0; Eε = σ ε = E [ρε 1 + e ] = E ρ ε 1 +ρε 1 e + e = ρ σ ε + σ e σ ε = ρ σ ε + σ e σ ε = σ e 1 ρ ; Eε ε 1 = E [(ρε 1 + e ) ε 1 ] = ρσ ε = ρσ e 1 ρ ; Eε ε = E [(ρε 1 + e ) ε ] = E [ρε 1 ε + e ε ] = E [ρ (ρε + e 1 ) ε + e ε ] = ρ σ ε = ρ σ e 1 ρ ;
Eε ε k = ρ k σ ε = ρk σ e 1 ρ ; 1 ρ ρ ρ T 1 ρ 1 ρ ρ T Ω = σ e 1 ρ ρ ρ 1 ρ T 3 ρ T 1 ρ T ρ T 3 1 This srucure is called auoregressive of order 1 [AR(1)] y = Xβ + ε ε = ae + be 1 e iid 0, σ e Eε =0; Eε = E [ae + be 1 ] = E a e +abe e 1 + b e 1 = a σ e + b σ e = σ e a + b ; Eε ε 1 = E [(ae + be 1 )(ae 1 + be )] = E a e e 1 + abe e + abe 1 + b e 1 e = abσ e; Eε ε = E [(ae + be 1 )(ae + be 3 )] = 0; Eε ε k = 0 k ; a + b ab 0 0 ab a + b 0 0 Ω = σ e 0 ab a + b 0 0 0 0 a + b This srucure is called moving average of order 1 [MA(1)] 3 y = Xβ + ε ε = ρ 1 ε 1 + ρ ε + e e iid 0, σ e 3
Eε =0; Define σ k = Eε ε k k 0 Eε = σ 0 = E [ρ 1 ε 1 + ρ ε + e ] = E ρ 1ε 1 +ρ 1 ρ ε 1 ε +ρ 1 ε 1 e + ρ ε +ρ ε e + e = ρ 1σ 0 +ρ 1 ρ σ 1 + ρ σ 0 + σ e = ρ 1 + ρ σ 0 +ρ 1 ρ σ 1 + σ e = ρ 1ρ σ 1 + σ e 1 (ρ 1 + ρ ) ; () Eε ε 1 = σ 1 = E [(ρ 1 ε 1 + ρ ε + e ) ε 1 ] = ρ 1 σ 0 + ρ σ 1 = ρ 1σ 0 1 ρ (3) Equaions () and (3) are wo equaions in wo unknowns: σ 0, σ 1 These are called Yule-Walker equaions (alhouh usually wrien in a slighly differen way) and can be solved for σ 0, σ 1 Onceweknow σ 0, σ 1,we can solve ieraively for oher auocovariance erms as Eε ε k = σ k = E [(ρ 1 ε 1 + ρ ε + e ) ε k ] = ρ 1 σ k 1 + ρ σ k Thissrucureiscalledauoregressiveoforder[AR()] we mean by an AR(p) process? Whawould 4 y = Xβ + ε ε = ae + be 1 + ce e iid 0, σ e Eε = E [ae + be 1 + ce ] = a σ e + b σ e + c σ e = σ e a + b + c ; Eε ε 1 = E [(ae + be 1 + ce )(ae 1 + be + ce 3 )] = abσ e + bcσ e = σ e (ab + bc); 4
Eε ε = E [(ae + be 1 + ce )(ae + be 3 + ce 4 )] = acσ e; Eε ε k =0 k 3 This srucure is called moving average of order [MA()] we mean by an MA(q) process? Wha would 5 y = Xβ + ε px ε = ρ i ε i + i=1 e iid 0, σ e qx a i e 1 This srucure is called an auoregressive moving average of orders p, q [ARMA(p, q)] i=0 6 Ω = σ 1 0 0 0 σ 0 0 0 σ T This srucure is called heeroskedasiciy If σ = σ, hen i is called homoskedasiciy 7 8 σ 1 0 0 0 σ 0 Ω = 0 0 σ T σ Xk y i = X i β + u i + ε i ; This is a random effecs model in panel daa Provide some examples Wha do he u i errors represen? Why does his cause deviaions from Ω = σ I? 5
3 Implicaions of Generalized Error Covariance Srucure Consider he OLS esimaor, bβ = (X 0 X) 1 X 0 y = β +(X 0 X) 1 X 0 ε Noe ha Eβ b = β relies only on EX 0 ε = 0 So more general covariance srucures have no effec on he unbiasedness of OLS They also do no preven bβ being a consisen esimaor of β However, ³ D bβ = E h(x 0 X) 1 X 0 εε 0 X (X 0 X) 1i = (X 0 X) 1 X 0 Eεε 0 X (X 0 X) 1 = (X 0 X) 1 X 0 ΩX (X 0 X) 1 6= σ (X 0 X) 1 Thus, he generalized covariance srucure affecs he covariance marix of β b and hus inference as well We can correc he inference problem by adjusing formulas For example, if hen H 0 : A k (n+1) H A : Aβ 6= c, Aβ b c N (0,V); V = A (X 0 X) 1 X 0 ΩX (X 0 X) 1 A 0 ; 1 ³ Aβ k b 0 c ha (X 0 X) 1 X 0 ΩX b (X 0 X) 1 A 0i 1 ³ Aβ b c F k,t (n+1) where b Ω is a consisen esimaor of Ω 4 Generalized Leas Squares There is sill an efficiency issue: we can do beer han OLS Le H 0 H = Ω 1 wih H having full rank If Ω is posiive definie, hen here is a coninuum of such marices Using equaion (1), premuliply by H o ge Hy = HXβ + Hε, 6
and do OLS on he ransformed equaion: bβ = (HX) 0 (HX) 1 (HX) 0 (Hy) = [X 0 H 0 HX] 1 X 0 H 0 Hy = X 0 Ω 1 X 1 X 0 Ω 1 y If we know Ω (and herefore Ω 1 ) we can esimae β using β b Generalized Leas Squares (GLS) Properies of GLS: Thisiscalled bβ = X 0 Ω 1 X 1 X 0 Ω 1 y = X 0 Ω 1 X 1 X 0 Ω 1 (Xβ + ε) = X 0 Ω 1 X 1 X 0 Ω 1 Xβ + X 0 Ω 1 X 1 X 0 Ω 1 ε = β + X 0 Ω 1 X 1 X 0 Ω 1 ε ³ D bβ Noe ha EHε =0and Eβ b = β + E X 0 Ω 1 X 1 X 0 Ω 1 ε = β + X 0 Ω 1 X 1 X 0 Ω 1 Eε = β + X 0 Ω 1 X 1 X 0 Ω 1 0=β ³bβ = E β ³ bβ β 0 h X = E 0 Ω 1 X 1 X 0 Ω 1 εε 0 Ω 1 X X 0 Ω 1 X i 1 = X 0 Ω 1 X 1 X 0 Ω 1 Eεε 0 Ω 1 X X 0 Ω 1 X 1 = X 0 Ω 1 X 1 X 0 Ω 1 ΩΩ 1 X X 0 Ω 1 X 1 = X 0 Ω 1 X 1 X 0 Ω 1 X X 0 Ω 1 X 1 = X 0 Ω 1 X 1 EHεε 0 H 0 = HEεε 0 H 0 = HΩH 0 = H (H 0 H) 1 H 0 = HH 1 (H 0 ) 1 H 0 = I which saisfies he condiions for OLS Gauss-Markov Theorem: OLS is BLUE when D (ε) =σ I in general Discuss 5 Specifying Ω in Parsimoniously GLS is BLUE In general, Ω is no known So we mus esimae i, and maybe we would wan o es some characerisic of i In general, Ω has T (T +1)/ independen 7
elemens (why?), and here are only T observaions So here are oo many elemens o esimae wihou imposing more srucure on Ω Consider each of he examples above: 1 AR(1): ε = ρε 1 + e e iid 0, σ e Ω has wo independen elemens: ρ, σe MA(1): ε = ae + be 1 e iid 0, σ e Ω has hree independen elemens: a, b, σe 3 AR(): ε = ρ 1 ε 1 + ρ ε + e e iid 0, σ e Ω has hree independen elemens: ρ 1, ρ, σe An AR(p) process would have p + 1independen elemens: ρ 1, ρ,, ρ p, σe 4 MA(): ε = ae + be 1 + ce e iid 0, σ e Ω has four independen elemens: a, b, c, σe AnMA(q) process would have q + independen elemens: a 0,a 1,, a q, σe 5 ARMA(p, q): ε = px ρ i ε i + i=1 qx a i e 1 e iid 0, σ e Ω has p + q + independen elemens: {ρ i } p i=1, {a i} q i=0, e σ i=0 6 Heeroskedasiciy 1: σ 1 0 0 0 σ 0 Ω = 0 0 σ T Ω has T independen elemens 8
7 Heeroskedasiciy : Ω = σ 1 0 0 0 σ 0 ; 0 0 σ T σ X k Ω has 1 independen elemen: he proporionaliy facor 8 Random Effecs: y i = X i β + u i + ε i Ω has independen elemens: σ u, σ ε In each of hese cases, insead of rying (unsuccessfully) o esimae T (T + 1)/ independen elemens, we esimae only he number of independen parameers 6 Esimaing and Tesing for Serial Correlaion 61 Cochrane-Orcu Consider The goal is o esimae ρ 1 Do OLS: y = Xβ + ε, ε AR (1) bβ =(X 0 X) 1 X 0 y Define residuals as 3 Consruc and bε =(I P x ) y bρ = P bε bε 1 P bε bσ e = 1 T X bε 9
Noe ha P bε bε 1 plimbρ = plim P bε = plim 1 P T bε bε 1 plim 1 P T bε 1 P T bε bε 1 = plim P bε = ρσ e σ e 1 T = ρ; plimbσ e = σ e 4 Compue 1 bρ bρ bρ T 1 ³ bρ 1 bρ bρ T bω bρ, bσ e = bσ e bρ bρ 1 bρ T 3 1 bρ bρ T 1 bρ T bρ T 3 1 Noe ha plimω b ³ bρ, bσ e = Ω (why?) 5 Do GLS: bβ = ³ X 0 b Ω 1 X 1 X 0 b Ω 1 y 6 Someimes one consrucs GLS residuals, ³ bε = I X X 0 Ω b 1 1 X X 0 Ω b 1 y ³ = I PΩ,x b y and reurns o sep (3) unil convergence is reached Noe ha ³I P bω,x and PΩ,x b are boh idempoen Someimes his mehod is called feasible GLS for he AR(1) case 6 Durbin-Wason Saisic Define DW = = P (bε bε 1 ) P bε 1 P bε bε bε 1 + bε 1 P bε 1 10
Assuming ha ε AR (1), plimdw = plim = plim 1 T P ³ bε bε bε 1 + bε 1 P bε 1 P ³ plim 1 T bε bε bε 1 + bε 1 P bε 1 = σ ε ρσ ε σ =(1 ρ) ε If ρ =0,plimDW =;ifρ > 0, plimdw < ; and if ρ < 0, plimdw > The range of DW is (0, 4) If we are ineresed in esing H 0 : ρ =0vs H 0 : ρ 6= 0, we can compare he DW-saisic o However, in order o urn his ino a proper es, we need o know he criical values for he DW disribuion Mos economerics ex books have DW ables 7 Esimaing Ω in General In general, assuming we have specified Ω in erms of a small number of parameers, we can esimae hose parameers using eiher maximum likelihood esimaion (MLE) or mehod of momens (MOM) Here we discuss MOM ConsiderhecasewhereΩ has m independen parameers The basic idea is o compare m heoreical momens o he appropriae m empirical momens o ge esimaes of he m parameers Examples: 1 AR(1): See Cochrane-Orcu above Noe ha P P 1 bε bε 1 T bε bε 1 plimbρ = plim P bε = plim P bε = plim 1 P T bε bε 1 plim 1 P T bε = ρσ e σ e 1 T = ρ; MA(1): plimbσ e = σ e ε = ae + be 1 e iid 0, σ e Varε = a + b σ e; Cov (ε, ε 1 ) = abσ e 11
Noe ha we can no separaely idenify all hree parameers because a, b, σ e has he same srucure as (aσe,bσ e, 1) Thus, wih no loss in generaliy, we can se σ e = 1 and esimae only (a, b) We se 1 X bε = Varε = a + b ; T 1 X bε bε 1 = Cov (ε, ε 1 )=ab T This gives wo equaions in wo unknowns and a unique esimae of (a, b) 3 Heeroskedasiciy :σ Xk σ = σ Xk Run a regression of bε = α 0 + α 1 X k + u Given he model specificaion, α 0 =0andα 1 = σ Why is his MOM? 8 Properies of R in he Presence of Serial Correlaion In models wih significan posiive serial correlaion, R will be arificially high IfonewereocompueR for he ransformed equaion, y = X β + ε ρy 1 = X 1 β + ε 1 y ρy 1 =(X ρx 1 ) β + ε ρε 1 y ρy 1 =(X ρx 1 ) β + e, one would ge a much lower R 1