ECON Introductory Econometrics. Lecture 6: OLS with Multiple Regressors

Transcription

1 ECON Introductory Econometrics Lecture 6: OLS with Multiple Regressors Monique de Haan (moniqued@econ.uio.no) Stock and Watson Chapter 6

2 Lecture outline 2 Violation of first Least Squares assumption Omitted variable bias violation of unbiasedness violation of consistency Multiple regression model 2 regressors k regressors Perfect multicollinearity Imperfect multicollinearity Properties OLS estimators in multiple regression model

3 Violation of first Least Squares assumption 3 Y i = β 0 + β 1 X i + u i Assumption 1: The conditional mean of u i given X i is zero E (u i X i ) = 0 The first OLS assumption states that: All other factors that affect the dependent variable Y i (contained in u i ) are unrelated to X i in the sense that, given a value of X i, the mean of these other factors equals zero. In the class size example: All the other factors affecting test scores should be unrelated to class size in the sense that, given a value of class size, the mean of these other factors equals zero.

4 Violation of first Least Squares assumption 4 Suppose that districts with small classes have few immigrants (few English learners) districts with large classes have many immigrants (many English learners) In this case class size is related to percentage of English learners Students who are still learning English likely have lower test scores Which implies that percentage of English learners is contained in u i. This implies a violation of assumption 1: E (u i ClassSize i = small) E (u i ClassSize i = large) 0

5 Omitted variable bias 5 The variable measuring the percentage of English learners in a district (el pct i ) is omitted from the simple regression model TestScore i = β 0 + β 1 ClassSize i + u i Omitting a variable from a regression analysis will lead to omitted variable bias if: 1 The omitted variable is correlated to the included regressor of interest. 2 The omitted variable is a determinant of the dependent variable.

6 Omitted variable bias 1. corr class_size el_pct (obs=420) 1. corr test_score el_pct (obs=420) 6 / / / / / / / / / / / Statistics/Data Analysis class_~e el_pct test_s~e el_pct class_size el_pct test_score el_pct Both conditions for omitted variable bias seem to be met 1 The percentage of English learners is correlated with class size 2 The percentage of English learners is correlated with test scores OLS If we omit percentage of English learners from regression, β 1 will not only estimate effect of class size on district average test scores but it will also pick up the effect of the percentage of English learners in the district on district average test scores βols 1 is biased and inconsistent.

7 Omitted variable bias: violation of unbiasedness 7 True model : Y i = β 0 + β 1 X i + β 2 W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + v i ] E [ β1 = E [ n ] i=1(x i X)(Y i Y) n i=1(x i X)(X i X) substitute for Y i, Y (true model!) [ n ] i=1(x i X)(β 0 +β 1 X i +β 2 W i +u i (β 0 +β 1 X+β 2 W +u)) = E n i=1(x i X)(X i X) rewrite (β 0 drops out) [ n ] i=1(x i X)(β 1 (X i X)+β 2(W i W)+(u i u)) = E n i=1(x i X)(X i X)

8 Omitted variable bias: violation of unbiasedness 8 ] E [ β1 = E [ n i=1(x i X)(β 1 (X i X)+β 2(W i W)+(u i u)) n i=1(x i X)(X i X) ] rewrite & use expectation rules [ β n 2 i=1(x i X)(W i W) = β 1 + E n i=1(x i X)(X i X) ] + E [ n ] i=1(x i X)(u i u) n i=1(x i X)(X i X) put β 2 in front of expectation & use algebra trick [ n ] [ i=1(x i X)(W i W) n ] i=1(x i X)u i = β 1 + β 2 E n + E i=1(x i X)(X i X) n i=1(x i X)(X i X) law of iterated expectations = β 1 + β 2 E [ n i=1(x i X)(W i W) n i=1(x i X)(X i X) ] + E [ n ] i=1(x i X)E(u i X i,w i ) n i=1(x i X)(X i X)

9 Omitted variable bias: violation of unbiasedness 9 ] E [ β1 = β 1 + β 2 E [ ni=1 ] (X i X)(W i W) ni=1 + E (X i X)(X i X) [ ni=1 ] (X i X)E(u i X i,w i ) ni=1 (X i X)(X i X) by assumption E (u i X i, W i ) = 0 [ ni=1 ] (X = β 1 + β 2 E i X)(W i W) ni=1 (X i X)(X i X) If W i is unrelated to X i (E ] E [ β1 = β 1 [ ni=1 ] ) (X i X)(W i W) ni=1 = 0 this implies that (X i X)(X i X) ] If W i is no determinant of Y i (β 2 = 0) this implies that E [ β1 = β 1 The second term is only nonzero if both conditions for omitted variable bias are met If the second term is nonzero β 1 is biased!

10 Omitted variable bias: violation of consistency 10 True model : Y i = β 0 + β 1 X i + β 2 W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + v i Plim β 1 = Plim n 1 1 ni=1 (X i X)(Y i Y) Plim n 1 1 ni=1 (X i X)(X i X) = Plim s XY Plim s 2 X = Cov(X i,y i ) Var(X i ) substitute true model for Y i = Cov(X i,β 0 +β 1 X i +β 2 W i +u i ) Var(X i ) Covariance rules Key concept 2.3 = Cov(X i,β 1 X i )+Cov(X i,β 2 W i )+Cov(X i,u i ) Var(X i )

11 Omitted variable bias: violation of consistency 11 Plim β 1 = Cov(X i,β 1 X i )+Cov(X i,β 2 W i )+Cov(X i,u i ) Var(X i ) Cov(X i, u i ) = 0 because E (u i X i, W i ) = 0 Cov(X = β i,x i ) Cov(X 1 + β i,w i ) Var(X i ) 2 Var(X i ) Cov (X i, X i ) = Var (X i ) = β 1 + β 2 Cov(X i,w i ) Var(X i )

12 Omitted variable bias: violation of consistency 12 β 1 p β 1 + β 2 Cov(X i,w i ) Var(X i ) If W i is unrelated to X i (Cov (X i, W i ) = 0) this implies that β 1 If W i is no determinant of Y i (β 2 = 0) this implies that β 1 p β 1 p β 1 If both omitted variable bias conditions are met β 1 is inconsistent!

13 Omitted variable bias: violation of consistency 13 From the omitted variable bias formula β 1 p Cov (X i, W i ) β 1 + β 2 Var (X i ) we can infer the direction of the bias of β 1 that persists in large samples Suppose W i has a positive effect on Y i, then β 2 > 0 Suppose X i and W i are positively correlated, then Cov (X i, W i ) > 0 This implies that β 1 is upward biased, it converges in probability to a larger number than the true value of β 1

14 Omitted variable bias: a simulation example 14 Lets create a data set with 100 observations W i N(0, 1) We let X i depend on W i : X i = W i + ε i ε i N (0, 1) u i N(0, 1) We define the true population model as: Y i = 1 + 2X i + W i + u i β 1 = 2 & β = 1 Tuesday January 31 11:10: Page 1 1. sum y x w set obs 100 gen w = rnormal() gen x = w + rnormal() gen y = 1 + 2*x + w + rnormal() (R) / / / / / / / / / / / / Statistics/Data Analysis Variable Obs Mean Std. Dev. Min Max y x w

15 Omitted variable bias: a simulation example 15 True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Tuesday January 31 11:22: Page 1 1. regress y x Estimated model : Y i = β 0 + β 1 X i + v i (R) / / / / / / / / / / / / Statistics/Data Analysis Source SS df MS Number of obs = 100 F( 1, 98) = Model Prob > F = Residual R-squared = Adj R-squared = Total Root MSE = y Coef. Std. Err. t P> t [95% Conf. Interval] x _cons

16 Omitted variable bias: a simulation example 16 We can Tuesday create January 999 of 31 these 11:26:40 data2017 sets with Page observations and use OLS to estimate 1. program define ols, rclass 1. drop _all 2. set obs gen w=rnormal() 4. gen x=w+rnormal() 5. gen y=1+2*x+w+rnormal() 6. regress y x 7. end Y i = β 0 + β 1 X i + v i simulate _b, reps(999) nodots : ols command: ols (R) / / / / / / / / / / / / Statistics/Data Analysis 4. sum Variable Obs Mean Std. Dev. Min Max _b_x _b_cons

17 Omitted variable bias: a simulation example n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + v i 5 OLS estimates of B 1 in 999 samples with n= OLS estimates of B 1

18 Omitted variable bias: a simulation example n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + v i 15 OLS estimates of B 1 in 999 samples with n= OLS estimates of B 1

19 Omitted variable bias: a simulation example n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + v i OLS estimates of B 1 in 999 samples with n= OLS estimates of B 1

20 . Omitted variable bias: a simulation example n=100, n=1000, n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + v i OLS estimates of B 1 in 999 samples with n=100; n=1000 and n=10000 n=100 n=1000 n= OLS estimates of B 1

21 Including the omitted variable: a simulation example 21 Natural solution to omitted variable bias is to include the variable and to estimate a multiple regression model. True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Tuesday January 31 11:47: Page 1 Estimated model : Y i = β 0 + β 1 X i + β 2 W i + v i (R) / / / / / / / / / / / / Statistics/Data Analysis 1. regress y x w Source SS df MS Number of obs = 100 F( 2, 97) = Model Prob > F = Residual R-squared = Adj R-squared = Total Root MSE = y Coef. Std. Err. t P> t [95% Conf. Interval] x w _cons

22 Including the omitted variable: a simulation example 22 We can Tuesday create January 999 of 31 these 11:49:48 data2017 sets with Page observations and use OLS to estimate Y i = β 0 + β 1 X i + β 2 W i + v i 1. program define ols, rclass 1. drop _all 2. set obs gen w=rnormal() 4. gen x=w+rnormal() 5. gen y=1+2*x+w+rnormal() 6. regress y x w 7. end simulate _b, reps(999) nodots : ols command: ols (R) / / / / / / / / / / / / Statistics/Data Analysis 4. sum Variable Obs Mean Std. Dev. Min Max _b_x _b_w _b_cons

23 Including the omitted variable: a simulation example n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + β 2 W i + v i 4 OLS estimates of B 1 in 999 samples with n= OLS estimates of B 1

24 Including the omitted variable: a simulation example n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + β 2 W i + v i 15 OLS estimates of B 1 in 999 samples with n= OLS estimates of B 1

25 Including the omitted variable: a simulation example n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + β 2 W i + v i 40 OLS estimates of B 1 in 999 samples with n= OLS estimates of B 1

26 . Including the omitted variable: a simulation example n=100, n=1000, n= True model : Y i = 1 + 2X i + W i + u i E (u i X i, W i ) = 0 Estimated model : Y i = β 0 + β 1 X i + β 2 W i + v i OLS estimates of B 1 in 999 samples with n=100; n=1000 and n=10000 n=100 n=1000 n= OLS estimates of B 1

27 Multiple regression model with 2 regressors 27 Interpretation of β 1 : Y i = β 0 + β 1 X 1i + β 2 X 2i + u i Suppose we would increase X 1 to X 1 + X 1 while keeping X 2 constant. define Y as E [Y X 1, X 2 ] = β 0 + β 1 X 1 + β 2 X 2 E [Y (X 1 + X 1 ), X 2 ] = β 0 + β 1 (X 1 + X 1 ) + β 2 X 2 Y = E [Y (X 1 + X 1 ), X 2 ] E [Y X 1, X 2 ] = β 1 X 1 this implies that β 1 is the expected change in Y due to unit change in X 1 while keeping X 2 constant!

28 Tuesday January 31 14:50: Page 1 Multiple regression model with 2 regressors Example: The effect of class size on test scores (R) / / / / / / / / / / / / Statistics/Data Analysis regress test_score class_size, robust Linear regression Number of obs = 420 F( 1, 418) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size _cons regress test_score class_size el_pct, robust Linear regression Number of obs = 420 F( 2, 417) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size el_pct _cons

29 Multiple regression model with 2 regressors Example: The effect of class size on test scores 29 test score i = class size i 0.65 el pct i The expected effect on test scores of increasing class size by 1, while keeping the percentage of English learners constant, equals -1.1 points This is about half the size of coefficient estimate when el pct i is omitted from the regression Estimated effect of class size in the simple regression model suffers from omitted variable bias Omitted variable bias formula already predicted a negative bias. β 1 p β 1 + β 2 Cov (class size i, el pct i ) Var (class size i )

30 Multiple regression model with k regressors 30 General notation for multiple regression model with k regressors: where Y i = β 0 + β 1 X 1i + β 2 X 2i β k X ki + u i Y i is the i th observation on the dependent variable X 1i,..., X ki are i th observations on the k independent variables or regressors β 0 is the intercept of the population regression line (expected value of Y when X 1i,..., X ki = 0) β 1 is the slope coefficient on X 1 ; the expected change in Y due to a unit increase in X 1i while holding X 2i,..., X ki constant. u i is the error term (all other factors, besides X 1i,..., X ki, determining Y i )

31 Multiple regression model with k regressors 31 The OLS estimators β 0, β 1,..., β k are obtained by minimizing the sum of squared prediction mistakes: n (Y i β 0 β 1 X 1i... β ) 2 k X ki i=1 Similar to the linear regression model with 1 regressor this implies taking derivatives w.r.t β 0, β 1,..., β k setting these to zero and solving for β 0, β 1,..., β k Formulas for OLS estimators in multiple regression model are best expressed using matrix algebra.

32 Multiple regression model with k regressors 32 Least squares assumption for multiple regression model: Assumption 1: The conditional distribution of u i given X 1i,..., X ki has mean zero, that is E (u i X 1i,..., X ki ) = 0 Assumption 2: (Y i, X 1i,..., X ki ) for i = 1,..., n are independently and identically distributed (i.i.d) Assumption 3: Large outliers are unlikely ( ) 0 < E X1i 4 <,..., 0 < E ( ) ( Xki 4 < & 0 < E Y 4 i ) < Assumption 4: No perfect multicollinearity

33 Perfect multicollinearity 33 Perfect multicollinearity arises when one of the regressors is a perfect linear combination of the other regressors The other regressors include the regressor on the constant term X 0i = 1 for i = 1,..., n Y i = β 0 X 0i + β 1 X 1i + β 2 X 2i β k X ki + u i If the regressors exhibit perfect multicollinearity, the OLS estimators cannot be computed Perfect multicollinearity produces division by zero in the OLS formulas Intuitively: you estimate effect of a change in one regressor on Y while holding another regressor, which is a perfect linear combination of the first regressor, constant: This does not make sense!

34 Perfect multicollinearity 34 Tuesday January 31 14:51: Page 1 (R) What happens when we include both the percentage of English learners and the share of English learners? 1. gen el_share=el_pct/ regress test_score class_size el_pct el_share, robust note: el_share omitted because of collinearity / / / / / / / / / / / / Statistics/Data Analysis Linear regression Number of obs = 420 F( 2, 417) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size el_pct el_share 0 (omitted) _cons el pct i and el share i are perfectly multicollinear

35 Perfect multicollinearity: Dummy variable trap 35 Thursday February 2 11:07: Page 1 What happens when we include both a dummy SmallClass i (=1 if class (R) size / / / / / < 20) and a dummy BigClass i (=1 if class size 20) and the constant term? 1. regress test_score SmallClass BigClass, robust note: BigClass omitted because of collinearity / / / / / / / Statistics/Data Analysis Linear regression Number of obs = 420 F(1, 418) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] SmallClass BigClass 0 (omitted) _cons BigClass i = 1 SmallClass i = X 0i SmallClass i

36 (Imperfect) multicollinearity 36 (Imperfect) multicollinearity means that two or more regressors are highly correlated, but one regressor is NOT a perfect linear function of one or more of the other regressors (imperfect) multicollinearity is not a violation of the least squares assumptions It does not impose theoretical problem for the calculation of OLS estimators If two regressors are highly correlated the the coefficient on at least one of the regressors is imprecisely estimated (high variance) With two regressors and homoskedastic errors we have that ( ) ) Var ( β1 = 1 1 σu 2 n 1 ρ 2 X 1 X 2 σx 2 1

37 Properties OLS estimators in multiple regression model 37 If the four least squares assumptions in the multiple regression model hold: The OLS estimators β 0, β 1,..., β k are unbiased ) E ( βj = β j for j = 0,..., k The OLS estimators β 0, β 1,..., β k are consistent β j p β j for j = 0,..., k The OLS estimators β 0, β 1,..., β k are normally distributed in large samples ) β j N (β j, σ 2 βj for j = 0,..., k

38 Multiple regression model: class size example Thursday February 2 14:18: Page 1 38 (R) / / / / / / / / / / / / Statistics/Data Analysis 1. regress test_score class_size, robust Linear regression Number of obs = 420 F(1, 418) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size _cons Is the average causal effect of class size on test scores equal to -2.27? Is there omitted variable bias?

39 Multiple regression model: class size example 39 Thursday February 2 14:18: Page 1 If we add the percentage of English learners as regressor in the regression model we get: 1. regress test_score class_size el_pct, robust (R) / / / / / / / / / / / / Statistics/Data Analysis Linear regression Number of obs = 420 F(2, 417) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size el_pct _cons Is the average causal effect of class size on test scores equal to -1.10? Is there omitted variable bias?

40 Multiple regression model: class size example 40 Thursday February 2 14:18: Page 1 If we add the percentage of students eligible for a free lunch as regressor (R) in the regression model we get: 1. regress test_score class_size el_pct meal_pct, robust / / / / / / / / / / / / Statistics/Data Analysis Linear regression Number of obs = 420 F(3, 416) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size el_pct meal_pct _cons Is the average causal effect of class size on test scores equal to -0.99? Is there omitted variable bias?

41 Multiple regression model: class size example 41 Thursday February 2 14:19: Page 1 If we add district average income as regressor in the regression model we get: 1. regress test_score class_size el_pct meal_pct avginc, robust (R) / / / / / / / / / / / / Statistics/Data Analysis Linear regression Number of obs = 420 F(4, 415) = Prob > F = R-squared = Root MSE = Robust test_score Coef. Std. Err. t P> t [95% Conf. Interval] class_size el_pct meal_pct avginc _cons Is the average causal effect of class size on test scores equal to -0.56? Is there omitted variable bias?

42 Multiple regression model: class size example 42 Dependent variable: district average test scores Class size *** ** *** ** (0.519) (0.433) (0.270) (0.255) Percentage of English learners *** *** *** (0.031) (0.033) (0.033) Percentage with free lunch *** *** (0.024) (0.030) Average district income 0.675*** (0.084) N