Essential of Simple regression

Transcription

1 Essential of Simple regression We use simple regression when we are interested in the relationship between two variables (e.g., x is class size, and y is student s GPA). For simplicity we assume the relationship is linear: y = β 0 + β 1 x + u (1) The error term u captures other factors besides x that affect y (e.g., u includes the teacher quality). The population regression function (PRF) is E(y x) = β 0 + β 1 x, and it can be represented by a straight line: β 0 is the intercept; β 1 is the slope. Calculus indicates that, if holding u constant, β 1 = dy dx y x y β 1 x y β 1 if x = 1 () So the slope coefficient measures change in y after one-unit change in x. The slope coefficient β 1 has causal interpretation only if cov(x, u) = 0 (the regressor is exogenous). In that case the error term behaves like a constant (and we know cov(x, constant) = 0), so ceteris paribus holds. If cov(x, u) 0 the regressor is endogenous, and β 1 has no causal interpretation; it just measures the association or correlation between the two variables. β 0 and β 1 are unknown. We use sample to estimate them. There are two methods: ordinary least squares (OLS) and method of moments (MM). They two methods produce identical formula (stata command: reg y x): β 1 = S xy S x ; β 0 = y β 1x (3) So the estimated slope coefficient β 1 has the same sign as sample covariance S xy. For instance, β 1 is positive (the straight line is upward-sloping) if the two variables are positively correlated. As another example, if we double x, then the new estimated slope coefficient will be equal to old one divided by. In short we can manipulate the slope coefficient (but we cannot manipulate the t value). After obtaining β 1and β 0 we can compute the fitted value as (stata command: predict yhat) y i = β 0 + β 1x i (4) We add the index i in order to emphasize that there is a fitted value for each observation. Similarly, for each observation, we can compute the difference between the true value y i and fitted value y i, and that difference is called residual (stata command: predict uhat, r): u i = y i y i = y i β 0 β 1x i (5) Do not confuse the residual u i with the error term u i. The residual measures the prediction error. The OLS estimator n solves the problem of minimizing the sum of squared residuals (or residual sum squares, RSS). The first order i=1 u i conditions (taking partial derivatives of RSS with respect to β 1and β 0, and letting them equal zero) are This is a system of two equations. The solutions are (3). Exercise 1: how to use stata to verify (6)? i u i = 0 and i x i u i = 0 (6) 1

2 Under the assumptions of (i) E(u x) = 0 and (ii) the sample is iid sample we can show the OLS estimator is unbiased (that means, on average, β 1 is equal to the true value β 1 ) E(β 1) = β 1 (7) Under the additional assumption (iii) homoscedasticity E(u x) = σ we can show the variance of β 1 is var(β 1) = σ = σ n i=1(x i x ) (n 1)s (8) x It is easy to see the law of large number can be applied to β 1since its variance given by (8) goes to zero as the sample size n goes to infinity. Moreover, if the sample size is greater than 10, we can apply the central limit theorem to show β 1 follows normal distribution: β 1~N (β 1, σ n i=1(x i x ) ) (9) The square root of (8) is called standard error (se); the standardized β 1 is called the t value (or t statistic): t β 1 = panda face = β 1 β 1 se = β 1 β 1 σ n i=1 (x i x ) ~N(0,1) (10) In large sample, the t value follows standard normal distribution. By default, the stata command reg y x reports the t value for a special null hypothesis H 0 : β 1 = 0 (x has no effect on y or x does not matter) We reject H 0 and say x is statistically significant (and having effect on y) if the p-value is less than In large sample the 95% confidence interval for β 1 is β 1 ± 1.96(se). That interval contains the true value β 1 with 95% probability. σ is also unknown, and can be estimated using RSS: n i=1 σ = RSS n k 1 = u i n 1 1 (11) The square root of (11), or σ, is called standard error of regression (SER). Stata calls it root of mse. R measures goodness of fit, or it measures the extent to which the variation of y can be explained by x. The formula is n i=1 n i=1(y i y ) R = 1 RSS = 1 u i TSS (1) R is between 0 and 1. A high R implies that x can explain a large portion of variation of y. In this course we downplay R. Instead, we are more interested in whether β 1 provides an unbiased estimate for the causal effect of x on y. Exercise : when the variation of x rises, the t value of β 1 (rises falls). This is (good bad) news if we want to show x matters. Exercise 3: R = if we try the stata command reg y (so this regression has no independent variable)

3 An Example of Simple Regression Using House Data We are interested in the effect of aging on the house price. First we draw the scatterplot and superimpose a straight line twoway (scatter rprice age) (lfit rprice age) age rprice Fitted values Next we run the simple regression using command reg. reg rprice age Source SS df MS Number of obs = 31 F( 1, 319) = Model e e+10 Prob > F = Residual 3.134e R-squared = Adj R-squared = Total e e+09 Root MSE = 3191 rprice Coef. Std. Err. t P> t [95% Conf. Interval] age _cons The dependent variable y is. The independent variable x is. The error term u may contain. The estimated intercept coefficient β 0 is, and it can be interpreted as The age is (exogenous endogenous) because. The estimated slope coefficient β 1 is, and it can be interpreted as The t value of β 1 is. The null hypothesis is. The p-value is. The conclusion is The R is. It means that 3

4 Omitted Variable Bias and Spurious Causality Suppose the error term contains a variable w that satisfies two conditions: (A) it matters for y; and (B) it is correlated with x. This variable w is called omitted variable. Because of the omitted variable, the simple regression of y on x will yield biased estimate of the true causal effect. Suppose true model is a multiple regression in which both x and w are exogenous (i.e., cov(x, e) = 0, cov(w, e) = 0): y = β 0 + β 1 x + β w + e (13) Condition (A) implies β 0; condition (B) implies cov(x, w) 0. Now consider running a simple regression. The estimated slope coefficient is given by (3): β 1 = S xy S x cov(x,y) = cov(x,β 0+β 1 x+β w+ e) = β 1 + β cov(x,w) = β 1 + OVB (14) OVB β cov(x,w) (15) In the second step we apply the law of large number to numerator S xy and denominator S x, which converge to population covariance cov(x, y) and population variance, respectively. The last result in (14) shows β 1 in the simple regression converges to the true value β 1 plus omitted variable bias (OVB). In short, β 1 is a biased estimate for the true causal effect β 1. The sign of OVB is discussed in Table 3. (on page 90, 5 th edition). For instance, β 1 overestimates β 1 if OVB is positive. Consider the extreme case when β 1 = 0, so x has no causal effect on y at all. But, because of OVB, the simple regression may indicate a spurious causality in the sense that β 1 can be statistically significant. The reason for the biased estimate or the spurious causality is that the regressor x is endogenous in the simple regression, or cov(x, u) 0 cov(x, u) = cov(x, β w + e) = β cov(x, w) 0 (16) That is why it is always important to check whether cov(x, u) = 0. In most cases, cov(x, u) 0. So in general β 1 just measures the association between x and y. Remember: x and y can be correlated because of the omitted variable w, even when x has no causal effect on y How to fix this issue? (13) is the answer: just run a multiple regression that includes both x and w. Running a multiple regression is feasible only if you have data for w. If the omitted variable w cannot be observed (ability is an example) then β 1 becomes unidentified. In eco411, you will learn a method called instrumental variable (IV) estimator so that β 1 can be estimated (identified) even if there is no data for w. Exercise 4: What can we do to ensure cov(x, u) = 0? Exercise 5: Give me an example of omitted variable. 4

5 Consider running the following Stata codes: clear set obs 100 set seed 1345 Monte Carlo Study of Spurious Causality *w is omitted variable. By construction, x does not cause y. w does gen w = round(3*uniform()) gen x = w + invnormal(uniform()) gen y = 4*w + invnormal(uniform()) * Scatterplot just shows correlation, but not causality twoway (scatter y x)(lfit y x) * Simple regression is misleading: it indicates spurious causality reg y x * Spurious causality is gone in a multiple regression that controls for w reg y w x The advantage of using simulation or Monte Carlo study is that we know the truth. In this example, the truth is that β 1 =. That means. However, the p-value of β 1 in the simple regression is. That means the null hypothesis of H 0 : β 1 = 0 (can cannot) be rejected. In light of this, we say the simple regression indicates spurious causality. After we run the multiple regression, the p-value of β 1 in the multiple regression is. That means the null hypothesis of H 0 : β 1 = 0 (can cannot) be rejected. So the issue of spurious causality is gone in the multiple regression. Lesson: multiple regression is suitable for investigating causality; simple regression is not. Exercise 6: β = Exercise 7: cov(x, w) = Exercise 8: OVB = 5

6 Consider a multiple regression with two regressors: The first order conditions (FOC) of minimizing RSS are In short, FOC is Multiple Regression and Frisch Waugh (FW) Theorem y = β 0 + β 1 x + β w + u i u i = 0 and i x i u i = 0 and i w i u i = 0 (17) regressor residual i = 0 (18) In theory we need to solve a system of three equations in order to obtain the OLS estimators. The solution can be very complicated. However, the FW theorem indicates that there is a simple formula for the OLS estimator β 1: β 1 = r iy i (Frisch Waugh Theorem) (19) r i where r denotes the residual of regressing x on w. See Appendix 3A for proof. The FW theorem implies a two-step procedure to obtain β 1: Step 1: regress x on w and constant term, and save the residual rhat Step : regress y on rhat. In the second step it does not matter whether the constant term is included. Because r is residual, it measures the part of x that cannot be explained by w. In other words, β 1 measures the effect of x on y, after the effect of w has been netted out. The coefficient in a multiple regression measures the net or direct effect In addition we can show under homoskedasticity var(β 1) = σ r i = σ SST x (1 R x ) (0) This is equation [3.51] in theorem 3. of the textbook. SST x = (x i x ) is the total variation of x, and R x is R- squared for the step 1 regression. Multicollinearity arises when x and w are highly correlated In that case R x, var(β 1), t value is, and the null hypothesis H 0 : β 1 = 0 (can cannot) be rejected. Multiple regression can decrease the likelihood of omitted variable bias, but increase the likelihood of multicollinearity Exercise 10: what can we do to reduce var(β 1)? 6