Chapter 6: Linear Regression With Multiple Regressors

Transcription

1 Chapter 6: Linear Regression With Multiple Regressors 1-1

2 Outline 1. Omitted variable bias 2. Causality and regression analysis 3. Multiple regression and OLS 4. Measures of fit 5. Sampling distribution of the OLS estimator 6-2

3 Omitted Variable Bias The error u arises because of: -Mis measurement -Randomness -Omitted variables: They influence Y but are omitted in the regression function. We turn to when the omitted variables bias the OLS estimator. 6-3

4 Omitted variable bias - definition Suppose the variable Z is omitted in the regression. By definition, there is omitted variable bias if BOTH: 1. Z is a determinant of Y (Z is part of u); 2. Z is correlated with the regressor X ( corr(z,x) 0) If either condition fails, we do not claim omitted variable bias. We will quantify the bias of the OLS estimator with a formula. 6-4

5 Role of two conditions The following illustrates why, if either condition fails, there is no bias. 1. Time of day of test affects test scores Y, but not correlated with the STR (regressor X) no bias 2. Parking space per pupil is correlated with STR (regressor X), but does not affect test scores Y no bias 6-5

6 Example: English learners & test scores In CA school districts, let Z:=Percent of English learners 1. English language ability (eg among English learners, immigrants) plausibly affects test scores (English and Math are tested): Z is a determinant of Y. 2. Communities with more English learners tend to be poorer and have higher STR: Z is correlated with X. Accordingly, there is omitted variable bias. As suggested, this implies that the OLS estimator is biased. What does common sense suggest about bias direction? What about quantifying the bias? 6-6

7 Formula for omitted variable bias Under single-regressor-ols assn s 2,3 (even if ass n 1 fails) β 1 = 1 n 1 n n i=1 n i=1 ( X i X )u i ( X i X ) 2 i p σ Xu σ X 2 σ u = = σ X Here ρ Xu :=corr(x,u) is shorthand for correlation b/w X,u. Now, ρ Xu = σ Xu σ X σ u σ u σ X ρ Xu 0 under OLS assn 1 0 under omitted variable bias 6-7

8 Reading both conditions in formula p β 1 + σ u σ X ρ Xu Indeed, if an omitted variable Z is both: 1. a determinant of Y (contained in u) 2. correlated with X, then ρ Xu 0 and the OLS estimator is biased and inconsistent (large samples do not mitigate the bias) Omitted PctEL should affect test scores negatively. In data, PctEL is positively correlated with STR. So STR should be negatively correlated with error term. By formula, OLS estimate of β is biased negatively: Overstates STR s effect. 6-8

9 Districts with fewer English Learners have higher test scores Districts with lower percent EL (PctEL) have smaller classes Among districts with comparable PctEL, the effect of class size is small (recall overall test score gap = 7.4) 6-9

10 Causality and regression analysis The test score/str/fraction English Learners example illustrates that, if an omitted variable satisfies the two conditions for omitted variable bias, then the OLS estimator in the regression omitting that variable is biased and inconsistent. So, even if n is large, will not be close to β 1. ˆβ 1 Suppose the school board decided to cut class size by 2 students per class. What would be the effect on test scores? We now have doubts about the prediction of test scores rising by an expected 4.6 points, given this bias. Already guessed that omitting PctEL causes OLS estimate to overstate STR s effect on test scores. 6-10

11 Overcoming omitted variable bias Three ways: 1. Run a randomized controlled experiment in which treatment (STR) is randomly assigned: then PctEL is still a determinant of TestScore, but PctEL is uncorrelated with STR. (Rarely feasible.) 2. Adopt the cross tabulation approach, with finer gradations of STR and PctEL within each group, all classes have the same PctEL, so we control for PctEL (But soon exhausts data; cannot also analyze other determinants: family income, parental education, ) 3. Use a regression in which the omitted variable (PctEL) is no longer omitted: Include PctEL as an additional regressor in a multiple regression! 6-11

12 The Population Multiple Regression Model Consider the case of two regressors: Y i = β 0 + β 1 X 1i + β 2 X 2i + u i, i = 1,,n Y is the dependent variable X 1, X 2 are the two independent variables (regressors) (Y i, X 1i, X 2i ) denote the i th observation on Y, X 1, and X 2. β 0 = unknown population intercept β 1 = effect on Y of a change in X 1, holding X 2 constant β 2 = effect on Y of a change in X 2, holding X 1 constant u i = the regression error (omitted factors) 6-12

13 Interpretation of coefficients Y i = β 0 + β 1 X 1i + β 2 X 2i + u i, i = 1,,n Consider changing X 1 by X 1 while holding X 2 constant: Population regression line before the change: Y = β 0 + β 1 X 1 + β 2 X 2 Population regression line, after the change: Y + Y = β 0 + β 1 (X 1 + X 1 ) + β 2 X

14 Before: Y = β 0 + β 1 (X 1 + X 1 ) + β 2 X 2 After: Y + Y = β 0 + β 1 (X 1 + X 1 ) + β 2 X 2 Difference: Y = β 1 X 1 So: Y β 1 =, holding X 2 constant X 1 β 2 = Y, holding X 1 constant X 2 β 0 = value that makes sample means fit line 6-14

15 The OLS Estimator in Multiple Regression With two regressors, by definition, the OLS estimator minimizes sum of squared residuals n i=1 min b0,b 1,b 2 [Y i (b 0 + b 1 X 1i + b 2 X 2i )] 2 The residuals are the differences between the observed Y i (data) and the predicted/fitted value given by the b s These are the OLS estimators of β 0 and β

16 Example: the California test score data Regression of TestScore against STR: TestScore = STR Now include percent English Learners in the district (PctEL): TestScore = STR 0.65PctEL Wow, effect of STR seems to have halved! This is the extent of the overstatement bias due to omitting PctEL! Advantages of OLS over tabulation: (1) data-economical, (2) quantifiable, (3) extends to multiple regressors 6-16

17 Multiple regression in STATA reg testscr str pctel, robust; Regression with robust standard errors Number of obs = 420 F( 2, 417) = Prob > F = R-squared = Root MSE = Robust testscr Coef. Std. Err. t P> t [95% Conf. Interval] str pctel _cons TestScore = STR 0.65PctEL 6-17

18 Measures of Fit of Regression Predicted value Y i : = b 0 + b 1X1i b k X ki & residual i : = i Then tautologically decompose data = prediction + residual Y = Y i + u i i u Y Y i SER = std. deviation of (with d.f. correction) RMSE = std. deviation of (without d.f. correction) R 2 R 2 R 2 = fraction of variance of Y explained by X = adjusted R 2 = R 2 with a degrees-of-freedom correction that adjusts for estimation uncertainty; < R

19 SER and RMSE The SER and the RMSE measures how much the Ys spread around the regression line: SER = n 1 uˆ n k 1 i= 1 2 i RMSE = n 1 uˆ n = i 1 2 i 6-19

20 R 2 The R 2 is the fraction of the variance predicted/explained. Exercises: * OLS goes through sample means: Also, OLS makes prediction and residual uncorrelated: cov( Y i, u i ) = 0 Sample variances of predicted Y s, actual Y s, and residuals are ESS : = ( Y ˆ i Y ) 2 TSS : = ( Y i Y ) 2 Y = Y and u = 0 SSR = ˆ 2 : u i if we ignore the common factor 1/(n-k-1). Now, expand TSS: TSS = ( Y ) ( ) 2 ( )( i + ui Y = Y i Y + ui + Y i Y ui 0) = ESS+ SSR+ factor cov( Y, i u i ) = cov= 0 ESS+ SSR Dividing this equation by TSS and rearranging, get The left is the R 2, by definition. Equation shows R 2 higher the lower the SSR (which is what the OLS minimizes). ESS TSS SSR = 1 TSS 6-28

21 R 2 & R 2 The R 2 always increases when one adds a regressor, e.g. number of bathrooms, weight of nearby ants, This because can always set coeff s on these 0, other coeffs as before, making SSR the same (and perhaps lower by minimizing) A high R2 says fit good, nothing about uncovering causality. R 2 The (the adjusted R 2 ) corrects this weakness by R 2 penalizing inclusion of another regressor. It is R 2 2 May decrease when one adds a regressor. Always, R < R, but close with large sample n. 2 n 1 SSR = 1 n k 1 TSS 6-21

22 Example: Test scores Test score example: TestScore (1) = STR, TestScore R 2 =.05, SER = 18.6 (2) = STR 0.65PctEL, R 2 R 2 =.426, =.424, SER = 14.5 Including PctEL vastly improves the fit R 2 Note: R 2 and are close because n=420 is large 6-22

23 The OLS Assumptions Y i = β 0 + β 1 X 1i + β 2 X 2i + + β k X ki + u i, i = 1,,n 1. The error term u given the X s has mean 0, E(u i X 1i = x 1,, X ki = x k ) = (X 1i,,X ki,y i ), i =1,,n, are i.i.d. 4 X 1i 4 X ki 2. Large outliers are rare: E( ),, E( ), E( ) are finite 3. There is no perfect multicollinearity NEW Y i 4 Note: (4) is true in case of sole regressor 6-23

24 Assn1: Mean of u given included X s is 0 E(u X 1 = x 1,, X k = x k ) = 0 Same interpretation as in regression with a single regressor. Again, say omitted variable bias occurs if both 1. OV influences Y (so is in u) 2. OV is correlated with an included X If possible, solution is to include OV in the regression. Another solution is to include a third variable that controls for the OV (discussed in ch7) 6-24

25 Assn2: (X 1i,,X ki,y i ), i =1,,n, are iid Automatic if data are collected by simple random sampling. Assn3: Large outliers are rare Recall, OLS is sensitive to recurring large outliers check data for outliers due to typos or coding errors 6-25

26 Assn4: No perfect multicollinearity Perfect multicollinearity: A regressor is an exact linear function of the other regressors. Example: Suppose you accidentally include STR twice: regress testscr str str, robust Regression with robust standard errors Number of obs = 420 F( 1, 418) = Prob > F = R-squared = Root MSE = Robust testscr Coef. Std. Err. t P> t [95% Conf. Interval] str str (dropped) _cons

27 . In the previous regression, β 1 is the effect on TestScore of a unit change in STR, holding STR constant (absurd) We will return to perfect (and imperfect) multicollinearity shortly, with more examples These OLS assumptions imply a sampling distribution of,,, ˆ. ˆβ 1 ˆβ 2 β k 6-27

28 Sampling Distribution of OLS Estimator Under OLS assumptions 1-4, for every regressor i=1,,k The sampling distribution of has mean β i (unbiased) ˆi β i β var( ) is inversely proportional to n. β ˆi Consistency: β β ˆi p i In large samples, approximately normally distributed: ˆ β i E( ˆ β ) i var( ˆ β ) i N(0,1) 6-28

29 Multicollinearity, Perfect and Imperfect Perfect multicollinearity: Some regressor is a linear function of the other regressors. Some more examples of perfect multicollinearity 1. The example from before: Include regressor STR twice, 2. Regress TestScore on a constant, D, and B, where: D i = 1 if STR 20, = 0 otherwise; B i = 1 if STR >20, = 0 otherwise, so B i = 1 D i 6-29

30 The dummy variable trap Suppose multiple binary/dummy variables, mutually exclusive & exhaustive: Every observation falls in exactly one dummy category (eg. Freshmen, Sophomores, Juniors, Seniors, Other). Including all dummies & intercept β 0 leads to perfect multicollinearity arises ( the dummy variable trap ). How so? Dummies, exhaustive & exclusive, add up to 1 which is the intercept s regressor To avoid the dummy variable trap: 1:Omit one group (e.g. Senior), or 2:Omit intercept How do (1) or (2) affect interpretation of the coefficients? 1: Coeffs are differences relative to omitted dummy 2: Coeffs are absolute levels for the included dummies 6-30

31 Perfect multicollinearity, ctd. Perfect multicollinearity usually reflects a mistake in the definitions of the regressors, or an oddity in the data Software will identify perfect multicollinearity and warn - by crashing or messaging error Solution: Modify list of regressors 6-31

32 Imperfect multicollinearity Imperfect and perfect multicollinearity are different. Imperfect multicollinearity: Some regressors are highly correlated Why phrase multicollinearity? If two regressors are highly correlated, their scatterplot is linear-like nearly co-linear 6-32

33 Imperfect multicollinearity, ctd. Imperfect multicollinearity implies that some coefficients will be imprecisely estimated. The idea: the coefficient on X 1 is the effect of X 1 holding X 2 constant; but if X 1 and X 2 are highly correlated, there is very little variation in X 1 once X 2 is held constant so the data don t contain much information about what happens when X 1 changes but X 2 doesn t. So the OLS estimator of the coefficient on X 1 is unreliable (high variance). Imperfect multicollinearity (correctly) results in large standard errors for one or more of the OLS coefficients. Special case clarifies (two regressors, homeskedasticity): σ 2 u var( slope Increasing as 1) = σ = 1 2 ρ β 2 X1, X 1 (as imp.col. worsens) 2 n 1 ρ X1, X σ 2 X1 6-33