MA Advaced Ecoometrics: Properties of Least Squares Estimators Karl Whela School of Ecoomics, UCD February 5, 20 Karl Whela UCD Least Squares Estimators February 5, 20 / 5
Part I Least Squares: Some Fiite-Sample Results Karl Whela UCD Least Squares Estimators February 5, 20 2 / 5
Uivariate Regressio Model with Fixed Regressors Cosider the simple regressio model y i = βx i + ɛ i where ɛ i i =, 2,... are i.i.d. ad of mea zero ad with fiite variace. The usual textbook assumptio is that x i is a fixed determiistic regressor idepedet of all values of ɛ i. What does a fixed regressor actually mea? It meas that we are to thik of x i ot as a outcome of a radom process but merely as a fixed set of umbers. The OLS estimator is ˆβ = x iy i x 2 i = x i βx i + ɛ i x 2 i = β + x iɛ i x 2 i We ca show that this is ubiased because we ca treat the x i values as fixed costats, so xi E ˆβ = β + x i 2 E ɛ i = β 3 Karl Whela UCD Least Squares Estimators February 5, 20 3 / 5 2
Multivariate Regressio Model with Fixed Regressors This result geeralises to the multivariate model with k regressors: Y = X β + ɛ 4 where Y ad ɛ are vectors, with ɛ cotaiig i.i.d. errors with fiite variace, X is a k matrix ad β is a k vector of coefficiets. We ca show that the OLS estimators are ubiased as follows E ˆβ = E X X X Y = E X X X X β + ɛ = E β + X X X ɛ 5 6 7 = β + X X X E ɛ 8 = β 9 Karl Whela UCD Least Squares Estimators February 5, 20 4 / 5
Fixed Regressors ad Normal Errors Suppose the i.i.d. error term ɛ is ormally distributed, ɛ N 0, σ 2 I. The formula ˆβ = β + X X X ɛ 0 tells us that, whe we treat the X as a fixed set of umbers, the OLS estimator is a liear fuctio of a ormal variable. So ˆβ is also ormally distributed. We calculate the covariace of the coefficiet estimates as [ E ˆβ β ˆβ β = E X X X ɛɛ X X X ] = σ 2 [ X X X X X X ] 2 So ˆβ N β, σ 2 X X. = σ 2 X X 3 Karl Whela UCD Least Squares Estimators February 5, 20 5 / 5
Stochastic Regressors Model I reality, the data i the X matrix should usually be thought of as outcomes of radom variables, just like the Y variable. As log as the X variables are idepedet of all values i the ɛ matrix, the oe ca proceed i the same way as though the regressors are fixed. For example, oe ca demostrate ubiasedess as E ˆβ = E β + X X X ɛ 4 = β + E X X X E ɛ 5 = β 6 If the errors are ormal, the give the realised value of X oe ca say ˆβ X N β, σ 2 X X 7 This meas that, for each possible X, the distributio of β takes o a particular ormal value, so the usual t ad F statistics also work. That said, the ucoditioal distributio of β depeds upo the distributio of X ad geerally will ot be Gaussia. Karl Whela UCD Least Squares Estimators February 5, 20 6 / 5
More o the Idepedece Assumptio Oe might be tempted to thik that i the model y i = βx i + ɛ i i =, 2... 8 that oe oly eeds E x k ɛ k = 0 for OLS to be ubiased, i.e. that each error term oly eeds to be idepedet of the values of x that occur for the same observatio. However, oe eeds a stroger assumptio: x k eeds to be idepedet of all the ɛ i values for i =, 2...,. From equatio 2 we ca write E ˆβ xi = β + E x i 2 ɛ i 9 Eve if ɛ j is idepedet of x j, it may ot be idepedet of all of the x values that make up the sum x i 2. This ca iduce a correlatio betwee ad ɛ i which leads to OLS beig biased. xi x 2 i We will see that AR time series regressios are a importat example of this kid of bias. Karl Whela UCD Least Squares Estimators February 5, 20 7 / 5
Part II Asymptotic Distributios with Stochastic Regressors Karl Whela UCD Least Squares Estimators February 5, 20 8 / 5
A New Way of Lookig at OLS Estimators You kow the OLS formula i matrix form ˆβ = X X X Y. There is a useful way to restate this that allows us to make a clear coectio to the WLLN ad the CLT. Cosider the case of a regressio with 2 variables ad 3 observatios. The X matrix is thus X = x x 2 x 2 x 22 20 x 3 x 23 This meas we ca write X x x X = 2 x 3 x x 2 x x 2 x 22 x 2 x 22 2 23 x 3 x 23 x = 2 + x 2 2 + x 3 2 x x 2 + x 2 x 22 + x 3 x 23 x 2 x + x 22 x 2 + x 23 x 3 x2 2 + x 22 2 + x 23 2 22 Karl Whela UCD Least Squares Estimators February 5, 20 9 / 5
A New Way of Lookig at OLS Estimators Let x be a colum vector cotaiig the 2 datapoits o the explatory variables from the first observatio x x = 23 x 2 Multplyig this vector by its traspose, we get x x x 2 = x x 2 x 2 x x22 2 24 Goig back ad comparig this with equatio 22 from the previous slide, we get the followig ew way of describig the X X matrix X X = x i x i 25 Karl Whela UCD Least Squares Estimators February 5, 20 0 / 5
A New Formula for the OLS Estimator It follows from the previous slides that we ca re-write the matrix formula for the OLS estimator as ˆβ = X X X Y = x i x i x i y i 26 I the same way, we ca write ˆβ = β + X X X ɛ = β + x i x i x i ɛ i 27 This formula is useful because it explais how the OLS estimator depeds upo sums of radom variables. This allows us to use the Weak Law of Large Numbers ad the Cetral Limit Theorem to establish the limitig distributio of the OLS estimator. Karl Whela UCD Least Squares Estimators February 5, 20 / 5
IID Stochastic Regressors Cosider the case i which ɛ i are i.i.d. with zero mea ad the x variables are i.i.d. each period ad there exists a matrix Q xx such that x i x i p Q xx 28 This meas that x i x i p Q xx 29 We ca also say that E x i ɛ i = 0. We ca use the multivariate CLT to show that d x i ɛ i N 0, Ω 30 where Ω = E x i x i ɛ 2 i 3 Karl Whela UCD Least Squares Estimators February 5, 20 2 / 5
Limitig Distributio with IID Stochastic Regressors The OLS estimator is ˆβ = β + x i x i x i ɛ i 32 with x i x i p Q xx 33 d x i ɛ i N 0, Ω 34 Usig the Slutsky s Theorem results combiig covergece i probability with covergece i distributio, we ca say d ˆβ β N 0, Q xx ΩQxx 35 Karl Whela UCD Least Squares Estimators February 5, 20 3 / 5
Ukow Covariace Matrix Usig the termiology described i the last set of otes, we have show that ˆβ a N β, Q xx ΩQxx 36 However, we do t kow the values of the log-ru average Q xx or the covariace matrix Ω. As described i the last otes, though, we ca substitute for these with cosistet estimators. WLLN tells x i x i p Q xx 37 x i x i ɛ 2 i p Ω 38 This meas we ca substitute sample meas for the true populatio meas ad base tests o the asymptotic distributio ˆβ a N β, x i x i x i x i ɛ 2 i x i x i 39 Karl Whela UCD Least Squares Estimators February 5, 20 4 / 5
No-Idetically Distributed Stochastic Regressors I practise, it s ulikely that idepedet x i observatios will be idetically distributed. What if the x i s are idepedetly but differetly distributed? Oe ca still prove a Cetral Limit Theorem for this case. The previous CLT that we proved, for i.i.d. variables, is kow as the Lidberg-Levy CLT. Aother useful result is the Lidberg-Feller Cetral Limit Theorem: If a sequece of observatios {y, y 2,..., y } are idepedetly distributed with mea µ ad each with a differet fiite variace σi 2 the ȳ µ d N 0, σ 2. 40 where σ 2 = lim σ 2 4 σ 2 = σi 2 42 As usual, there is a atural multivariate extesio of this result i which the covariace matrix of the limitig distributio is a average of the covariace matrices of the differetly distributed x i s. Karl Whela UCD Least Squares Estimators February 5, 20 5 / 5