6. Assessing studies based on multiple regression

Size: px

Start display at page:

Download "6. Assessing studies based on multiple regression"

Eustacia Nelson
6 years ago
Views:

1 6. Assessing studies based on multiple regression Questions of this section: What makes a study using multiple regression (un)reliable? When does multiple regression provide a useful estimate of the causal effect under consideration? Conceptual framework: Internal and external validity 139

2 Definition 6.1: (Internal and external validity) A statistical analysis is said to have internal validity if the statistical inferences about causal effects are valid for the population being studied. The analysis is said to have external validity if its inferences and conclusions can be generalized from the population and setting studied to other populations and settings. Terminology: The population studied is the population of entities (people, companies,...) from which the sample was drawn The population of interest is the population of entities to which the causal inferences from the study are to be applied By setting we mean the institutional, legal, social, and economic environment 140

3 Threats to internal validity: The estimator of a causal effect should be unbiased and consistent Hypothesis tests should have the desired significance level Confidence intervals should have the desired confidence levels Requirements for internal validity are that the OLS estimator is unbiased and consistent and that standard errors are computed in a way that makes confidence intervals have the desired confidence levels These requirements might not be met for various reasons Threats to internal validity that lead to failures of our OLS assumptions from Slide

4 Threats to external validity: Differences between the population being studied and the population of interest Example: Medical studies that use animal populations like mice (the population being studied), but aim at transfering the results to human populations (the population of interest) Even if the population studied and the population of interest are identical, the study results may not be generalized due to differences in the settings Example: The effect of an antidrinking advertising campaign on the drinking behaviour of a group of first-term students might differ at two universities if the legal penalties for drinking differ at both universities 142

5 Threats to external validity: [continued] Important questions with respect to external validity are: How to assess the external validity of a study? How to design an externally valid study? Both issues require specific knowledge of the populations and settings being studied and those of interest A rigorous treatment of both issues is beyond the scope of this lecture (cf. Shadish et al., 2002, for details) We focus on aspects of internal validity 143

6 6.1. Threats to internal validity of multiple regression analysis Objectives: Survey of five reasons why the OLS estimator of a multiple regression coefficient may be biased even in large samples (Sections ) All five sources of bias arise because the regressor is correlated with the error term in the population regression Violation of the first OLS assumption from Slide 43 What can be done to reduce this bias? 144

7 Omitted variable bias Omitted variable bias: If an omitted variable is a determinant of Y i and if it is correlated with at least one of the regressors, then the OLS estimator of at least one of the coefficients will have omitted variable bias (see Definition 3.5 on Slide 52) This bias persists even in large samples OLS estimator(s) is (are) inconsistent Mathematically, under omitted variable bias at least one of the regressors is correlated with the error term u i implying that E(u i X 1i,..., X ki ) 0 145

8 Mitigation of omitted variable bias: Inclusion of control variables in the regression equation Definition 6.2: (Control variable) A control variable is not the object of interest in a regression analysis, but is rather a regressor included to hold constant factors that, if neglected, could lead the estimated causal effect of interest to suffer from omitted variable bias. Remarks: Up to now: OLS assumptions on Slide 43 treat all regressors symmetrically Now: explicit distinction between regressors of interest and control variables 146

9 Mathematical motivation: Consider a regression with two variables X 1i (the regressor) and X 2i (the control variable) Y i = β 0 + β 1 X 1i + β 2 X 2i + u i Replace the first OLS assumption E(u i X 1i, X 2i ) = 0 by the so-called conditional-mean-independence assumption E(u i X 1i, X 2i ) = E(u i X 2i ) (6.1) β 1 has a causal interpretation, but β 2 does not (see class for details) 147

10 Intuition of (6.1): The inclusion of the control variable X 2i makes the regressor X 1i uncorrelated with u i so that the OLS estimator ˆβ 1 can estimate the causal effect on Y i of a change in X 1i By contrast, the control variable X 2i remains correlated with u i so that its coefficient β 2 is subject to omitted variable bias and does not have a causal interpretation The control variable X 2i is included because it controls for omitted factors that affect Y i and are correlated with X 1i it might (but need not) have a causal effect itself When a control variable is used, it is controlling for both, (1) its own direct causal effect (if any), and (2) for the effect of correlated omitted factors 148

11 Terminology: Complete phrasing: The coefficient β 1 on the regressor X 1i is the causal effect on Y i of a change in X 1i using the control variable X 2i both (1) to hold constant the direct effect of X 2i, and (2) to control for factors correlated with X 1i Conventional, less awkward phrasing: The coefficient β 1 on X 1i is the effect on Y i controlling for X 2i 149

12 Example: Consider the student-performance dataset Consider the regression results of TEST SCORE on STR and PCTEL (see left panel on Slide 151) Potentially omitted factor could be outside learning opportunities Factor outside learning opportunities is difficult to measure, but correlated with the students economic background Include a measure of economic background to control for omitted income-related determinants of TEST SCORE like outside learning opportunities Such a control variable is MEAL PCT measuring the percentage of students receiving a free or subsidized lunch 150

13 TEST SCORE regression results with and without the control variable MEAT PCT Dependent Variable: TEST_SCORE Method: Least Squares Date: 19/05/12 Time: 18:00 Sample: Included observations: 420 White heteroskedasticity-consistent standard errors & covariance Variable Coefficient Std. Error t-statistic Prob. C STR PCTEL R-squared Adjusted R-squared S.E. of regression Sum squared resid Log likelihood F-statistic Prob(F-statistic) tatis Mean dependent var S.D. dependent var Akaike info criterion Schwarz criterion Hannan-Quinn criter Durbin-Watson stat Dependent Variable: TEST_SCORE Method: Least Squares Date: 19/05/12 Time: 17:52 Sample: Included observations: 420 White heteroskedasticity-consistent standard errors & covariance Variable Coefficient Std. Error t-statistic Prob. C STR PCTEL MEAL_PCT R-squared Adjusted R-squared S.E. of regression Sum squared resid Log likelihood F-statistic Prob(F-statistic) tatis Mean dependent var S.D. dependent var Akaike info criterion Schwarz criterion Hannan-Quinn criter Durbin-Watson stat

14 Example: [continued] Including the control variable MEAL PCT does not substantially change the effect of STR on TEST SCORE (ˆβ 1 changes from to ) changes the size (but not the sign) of the effect of PCTEL on TEST SCORE (ˆβ 2 changes from to ) The estimated coefficient ˆβ 3 = is not reasonable, since if it were we could boost TEST SCORE by eliminating the reduced-price lunch programme Do not treat β 3 as causal 152

15 Solutions to omitted variable bias: Distinction between two situations, namely (1) one in which data on the omitted variable or on adequate control variables are available, and (2) one in which data are not available Situation #1: If data on the omitted variable is available, include it in the regression equation If you have data on adequate control variables (with the hope of achieving conditional mean independence), include these variables in the regression equation 153

16 Trade-off: Adding a variable to a regression has both costs and benefits On the one hand, omitting the variable could result in omitted variable bias On the other hand, including a variable that is not a relevant regressor (that is, when its population regression coefficient is zero) reduces the precision of the estimators of the other regression coefficients (in the form of higher variances of the OLS estimators) 154

17 Situation #2: If no data are available there are three potential ways of mitigating the omitted variable bias The use of panel data (see Stock & Watson, 2011, Section 8) The use of instrumental variables (see Section 7) The conduct of randomized controlled experiments (see Stock & Watson, 2011, Section 13) 155

18 Guidelines for deciding whether to include an additional variable: Be specific about the coefficients of interest Use a priori reasoning to identify the most important potential sources of omitted variable bias Consider a baseline specification and some questionable variables Test whether additional questionable variables have nonzero coefficients Provide full disclosure representative tabulations of your results so that others can see the effect of including questionable variables on the coefficients of interest Observe, if your results change after including a questionable control variable 156

19 Misspecification of the functional form of the regression function Definition 6.3: (Functional form misspecification) Functional form misspecification arises when the functional form of the estimated function differs from the (true) functional form of the population regression function. Two aspects of misspecification: If the (true) population regression function is nonlinear, but we estimate a linear regression equation, then the estimator of the coefficients suffer from omitted variable bias If the (true) population regression function is linear, but we estimate a nonlinear regression equation, then we estimate non-existent coefficients 157

20 Solutions to functional form misspecification: Detection of misspecification by using statistical specification tests, for example the Regression Specification Error Test (RESET) (see Econometrics I) Plot the data and the estimated regression function Correct the misspecification by trying alternative functional forms of the regression function 158

21 Measurement error and errors-in-variable bias Definition 6.4: (Errors-in-variable bias) Errors-in-variables bias in the OLS estimator arises when an independent variable is measured imprecisely. The bias depends on the nature of the measurement error and persists even if the sample size is large. Sources of measurement errors: Wrong answer of a respondent to a survey question (e.g. about her/his income) Typographical errors in data collected from computerized administrative records 159

22 Consequence: Consider a regression with a single regressor X i X i is imprecisely measured by X i The true population regression equation is Y i = β 0 + β 1 X i + u i = β 0 + β 1 X i + [ β 1 ( X i X i ) + ui ] = β 0 + β 1 X i + v i where v i = β 1 (X i X i ) + ui The population regression equation in terms of X i has an error term containing the measurement error X i X i If X i X i is correlated with X i, then the regressor X i will be correlated with v i 160

23 Consequence: [continued] Violation of OLS assumption 1 on Slide 18 ˆβ 1 will be biased and inconsistent Classical measurement error model: Suppose the measured value X i equals the unmeasured value X i plus a purely random component w i with expected value 0 and variance σ 2 w Suppose further that Corr(w i, X i ) = 0 and Corr(w i, u i ) = 0 It then follows that (see class for details) plim ˆβ 1 = σ2 X σ 2 X + σ2 w β 1 161

24 Solutions to errors-in-variables bias: Try to obtain an accurate measure of X (if possible) Use instrumental variables (see Section 7) 162

25 Missing data and sample selection We consider three cases: 1. Data are missing completely at random 2. Data are missing based on a regressor 3. Data are missing because of a selection process that is related to Y beyond depending on X (sample selection bias) 163

26 Case #1: When the data are missing completely at random (for reasons unrelated to the values of X or Y ) the effect is to reduce the sample size but not introduce bias Case #2: When the data are missing based on the value of a regressor, the effect also is to reduce the sample size but not introduce bias 164

27 Case #3: If the data are missing because of a selection process that is related to the value of the dependent variable Y beyond depending on the regressors X 1,..., X k then this selection process can introduce correlation between the error term and the regressors Bias in the OLS estimators that persists in large samples Definition 6.5: (Sample selection bias) Sample selection bias arises when a selection process influences the availability of data and that process is related to the dependent variable, beyond depending on the regressors. 165

28 Example: We consider the question as to whether stock mutual funds outperform the market To this end, many studies compare future returns on mutual funds that had high returns over the past year to future returns on other funds and on the market as a whole Some databases include historical data on funds currently available for purchase This approach implies that the most poorly performing funds are omitted from the dataset because they went out of business or were merged into other funds Only the better funds survive to be in the data set (survivorship bias) 166

29 Solutions to sample selection bias: Beyond the scope of this lecture 167

30 Simultaneous causality Definition 6.6: (Simultaneous causality bias) Simultaneous causality bias, also called simultaneous equation bias, arises in a regression of Y on X when, in addition to the causal link of interest from X to Y, there is a causal link from Y to X. Remark: The reverse causality makes X correlated with the error term Bias in the OLS estimators that persists in large samples 168

31 Example: Consider the following two regression equations that hold simultaneously: Y i = β 0 + β 1 X i + u i (6.2) X i = γ 0 + γ 1 Y i + v i (6.3) From Eq. (6.2) it follows that if u i < 0 then Y i decreases If γ 1 > 0, then a low value of Y i leads to a low value of X i in Eq. (6.3) If γ 1 > 0, then Corr(X i, u i ) > 0 in Eq. (6.2) (see class for details) 169

32 Solutions to simultaneous causality bias: Use instrumental variables (see Section 7) 170

33 Sources of inconsistency of OLS standard errors OLS standard errors: Inconsistent standard errors pose a threat to internal validity Even if the OLS estimators are consistent and the sample size is large, inconsistent standard errors will produce invalid hypothesis tests and confidence intervals Main reasons for inconsistent standard errors: Heteroskedasticity of the error terms u i Autocorrelation among the error terms u i 171

34 Remedies: Use heteroskedasticity-consistent standard errors (see Section ) Use heteroskedasticity-autocorrelation consistent (HAC) standard errors (see Section ) 172

35 6.2. Summary Five threats to internal validity: 1. Omitted variables 2. Functional form misspecification 3. Errors-in-variables 4. Sample selection 5. Simultaneous causality 173

36 Remarks: Each of these, if present, result in failure of the first OLS assumption from Slide 18: E(u i X 1i,..., X ki ) = 0 Bias in the OLS estimators that persists in large samples Incorrect calculation of standard errors poses a further threat to internal validity Applying this list of threats to a multiple regression study provides a systematic way to assess the internal validity of that study 174

2. Linear regression with multiple regressors

2. Linear regression with multiple regressors Aim of this section: Introduction of the multiple regression model OLS estimation in multiple regression Measures-of-fit in multiple regression Assumptions