Simple Linear Regression: The Model task: quantifying the effect of change X in X on Y, with some constant β 1 : Y = β 1 X, linear relationship between X and Y, however, relationship subject to a random error: Y i = β 0 + β 1 X i + u i (i = 1,..., n), with dependent variable or regressand Yi, independent variable or regressor Xi, regression coefficients β0 (intercept) and β 1 (slope) disturbance or error term ui. β 0, β 1 usually estimated by ordinary least squares (OLS). Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 1
Simple Linear Regression: OLS I estimates for β 0, β 1 are obtained by finding the minimizers of min (Y i (b 0 + b 1 X i )) b 0,b 1 OLS delivers ˆβ 1 = (X i X )(Y i Y ) = s XY (X i X ) sx ˆβ 0 = Y ˆβ 1 X fitted values: Ŷi = ˆβ 0 + ˆβ 1 X i residuals: ûi = Y i Ŷ i û i = 0, Ŷ i = n Y i, û i X i = 0. Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide
Simple Linear Regression: OLS II coefficient of determination, R : gives the share of the variance of Y due to variation of X builds on the variance decomposition (Y i Y ) = } {{ } TSS (Ŷ i Y ) + } {{ } ESS û i }{{} SSR which decomposes the total sum of squares into the explained sum of squares and the sum of squared residuals. R defined as R = ESS TSS = 1 SSR TSS with 0 R 1, R = rxy = [ (X i X )(Y i Y )] (Xi X ) (Y = s XY. i Y ) sx s Y standard error of the regression: SER = sû, with s û = 1 n ûi = SSR n. Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 3
Simple Linear Regression: OLS III to study statistical properties of the OLS estimators, we need some assumptions: 1 E(u i X 1,..., X n ) 0 the regressors X i are assumed to be stochastic, regardless of the value all the X 1,..., X n take, the conditional expectation of u i given these X j (j = 1,..., n) vanishes, this implies that E(Yi X i ) = β 0 + β 1 X i, i.e. the conditional expectation of Y i is given by the regression line β 0 + β 1 X i, assumption 1 implies E(u i ) = E(E(u i X i )) = 0 and Cov(u i, X i ) = 0, in practice, therefore, assumption 1 may be inappropriate if error term and regressor are correlated, for instance because u i comprises variables that are correlated with X i or because there s mutual causation between X and Y, assumption 1 does not imply that Xi and u i are independent, it only implies that u i and X i are uncorrelated, assumption 1 does not imply that X i and u i are independent or uncorrelated, in particular it may be possible that Var(u i X 1,..., X n ) depends on X i. Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 4
Simple Linear Regression: OLS IV E(u i u j X 1,..., X n ) 0 for all i j assumption means that for i j, ui and u j are uncorrelated given X 1,..., X n this does imply that u i and u j are uncorrelated for i j: therefore, this assumption may be inappropriate if time series data are given assumption says nothing about Var(ui X 1,..., X n ); if Var(u i X 1,..., X n ) σu for all i = 1,..., n, then the errors are said to be homoskedastic, if not, they are called heteroskedastic 3 P(X 1 =... = X n ) = 0 assumption 3 ensures that β 1 is identified: if X 1 =... = X n, then the values X i are all equal to the same constant and the model would include two different indistinguishable constants Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 5
Simple Linear Regression: Properties of OLS estimators I under assumptions 1 and 3, we have: ˆβ 1 = β 1 + 1 (X i X)u i n (X i X) = β 1 + 1 n 1 n (X i X)u i s X conditionally and unconditionally unbiased for β 1 ˆβ 0 = β 0 + 1 n (X X X i)u i 1 n (X i X) = β 0 + 1 n (X X X i)u i s X conditionally and unconditionally unbiased for β 0 is both is both Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 6
Simple Linear Regression: Properties of OLS estimators II under assumptions 1,, and 3, we have: Var( ˆβ1 X 1,..., X n ) = 1 (X i X) Var(u n (sx) i X 1,..., X n ) Var( ˆβ 0 X 1,..., X n ) = 1 (X X X i) n (s X) Var(u i X 1,..., X n ) under homoskedasticity, these formulas simplify to Var( ˆβ 1 X 1,..., X n ) = 1 σu n sx ) and Var( ˆβ 0 X 1,..., X n ) = 1 n (1 + X σu. s X Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 7
Simple Linear Regression: Estimating the OLS estimators variances under assumptions 1,, and 3, Var( ˆβ 1 X 1,..., X n ) can be estimated by Var HC ( ˆβ 1 X 1,..., X n ) := 1 (X i X) n(n ) û (sx) i, Var( ˆβ0 X 1,..., X n ) can be estimated by Var HC ( ˆβ 0 X 1,..., X n ) := 1 (X X X i) n(n ) û (sx) i under homoskedasticity, σu can be estimated by sû = 1 n ûi, Var( ˆβ 1 X 1,..., X n ) can be estimated by Var( ˆβ 1 X 1,..., X n ) := 1 s û n sx Var( ˆβ 0 X 1,..., X n ) can be estimated ) by Var( ˆβ 0 X 1,..., X n ) := 1 n (1 + X sû. s X Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 8
Simple Linear Regression: distribution of the OLS estimators in case of u i i.i.d. N(0, σ u), both t 1 := t 0 := ˆβ 0 β 0 Var( ˆβ0 X 1,...,X n) ˆβ 1 β 1 Var( ˆβ 1 X 1,...,X n) and are t-distributed with n degrees of freedom this can be exploited to develop procedures for testing the hypotheses whether β0 (β 1 ) is equal to (smaller than, larger than) a given value constructing confidence intervals for β0, β 1. Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 9
Simple Linear Regression: asymptotic distribution of the OLS estimators ( ) ˆβ0 if X n and sx converge for n, then ˆβ := converges in ˆβ 1 case of ( i.i.d. ) errors u (under some technical assumptions) to β0 β :=, more precisely: the conditional distribution of β 1 n( ˆβ β) ( given X1,..., X) n converges to N (0, V ), with V := σ u µ X + σx µ X 1, µ σx µ X 1 X := lim X = lim X n n n i, and σx := lim n s X = lim 1 (X n n i X n ). V can be estimated by ( ) V := s û X X. sx X 1 this can be exploited for developing tests and confidence regions. Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 10
Simple Linear Regression: asymptotic distribution of the OLS estimators under heteroskedasticity denoting ω i := Var(u i X 1,..., X n ) and additionally assuming that 1 ω n i, 1 X n i ω i, and 1 X n i ω i converge to some constants, the conditional distribution of n( ˆβ β) given X 1,..., X n converges to N (0, V ), with V 1 1 := lim (X i X) ω i (X i X)(X X X i)ω i n (sx) n (X i X)(X X X i)ω i (X X X i) ω i which can be( estimated by ) (X i X) ûi (X i X)(X X X i)ûi 1 1 (sx) n (X i X)(X X X i)ûi (X X X i) ûi again, this can be exploited for developing tests and confidence regions. Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 11
Simple Linear Regression: Gauss-Markov Theorem if the errors are homoskedastic, then the OLS estimators are efficient among all linear (conditionally) unbiased estimators, i.e. all estimators of the form n a i Y i with a i possibly depending on X 1,..., X n, but not on Y 1,..., Y n, and with E( n a i Y i X 1,..., X n ) equal to β 1 (or β 0 ) have a variance at least as large as that of the corresponding OLS estimator if the errors are heteroskedastic, then there exist (conditionally) unbiased linear estimators for β 0, β 1 with smaller variances than those of the OLS estimators Econometric Methods & Appl. (WS 17/18) Lecture 3 Slide 1