Lecture 4: Regression Analysis

Transcription

1 Lecture 4: Regression Analysis 1

2 Regression Regression is a multivariate analysis, i.e., we are interested in relationship between several variables. For corporate audience, it is sufficient to show correlation. Causality is of no interest. Nevertheless, business people care about outliers and multicollinearity, two issues downplayed in econometrics class. 2

3 Scatter Plot A scatter plot displays the variable on the vertical axis against the variable on the horizontal axis (limitation?). Each point represents one observation. Sometimes a line fitted by OLS is attached. For instance, a downward sloping line shown in Figure 1 indicates negative correlation, though we do not know whether it is statistically significant. We don t care this correlation is due to causality, or a lurking variable. Figure 1, Scatter Plot GDP Growth Inflation 3

4 Outliers I From the scatter plot, we notice several outliers. For example, there is one observation with inflation rate less than -5 in the lower left corner. The regression using all observations is all = data.frame(infr, gro) m0 = lm(gro infr, data=all) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** infr and the regression excluding that outlier (leave-one-out regression) is Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** infr So the t value of inflation rate (infr) changes from to , a big jump. 4

5 Subsetting In stata we can use if statement to run a regression for a subset of sample. In R, we can use function subset to specify a subsample, call it suba. Then consider the function lm(y x, data=suba) The logic operators commonly used by subset are == equal to!= not equal to!x Not x x y x & y x OR y x AND y 5

6 Outliers II It turns out we can duplicate the leave-one-out regression by defining a dummy variable that equals one for that outlier, and run the regression using all observations and that dummy variable d = ifelse(infr<(-5),1,0) # -5 must be inside parentheses! lm(formula = gro infr + d, data = all) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) < 2e-16 *** infr d *** The coefficient of the dummy variable is the leave-one-out residual for that outlier observation. 6

7 Studentized Residuals I More importantly, the t value of the dummy variable, , is the studentized residual for that outlier: Studentized Residual i = êi ˆσ i (1) where ê i is the i-th residual of the regression that uses all observations; ˆσ i is the estimated standard deviation of the i-th residual. In large sample, the studentized residual follows standard normal distribution. So a studentized residual greater than 1.96 in absolute value may indicate an outlier. 7

8 Studentized Residuals II Here > 1.96, so the observation with inflation less than -5 is indeed an outlier, without which the regression result changes significantly. We can obtain all studentized residuals using the R function studres available in the MASS package, and then display all the observations with studentized residuals greater than 1.96 > library(mass) > m0 = lm(gro infr, data=all) > ehat = studres(m0) > ehat[which(abs(ehat)>1.96)] For more about outliers, see section 9.5 of Wooldridge s Introductory Econometrics textbook. 8

9 Matrix Algebra for Studentized Residual Consider the multiple regression in matrix form Y = Xβ + e The residual vector is ê = Y X ˆβ = Y X(X X) 1 X Y = (I H)Y, where H X(X X) 1 X is called hat matrix. It follows that the variance-covariance matrix of the residual vector is E(êê ) = σ 2 (I H). So the variance for the i-th residual is var(ê i ) = σ 2 (1 h ii ), where h ii, called leverage, is the i-th diagonal entry of H. Formally, the i-th studentized residual is Studentized Residual i = ê i σ 2 (1 h ii ) (2) where we estimate σ 2 by running the leave-one-out regression without the i-th observation. 9

10 Leverage The vector of fitted values is Ŷ = X ˆβ = HY. More explicitly, ŷ 1. = h y 1. ŷ n h nn y n So it is evident that h ii = ŷ i y i Therefore, the leverage measures the change of the fitted value ŷ i when y i changes by one unit. An observation with high leverage tends to pull the OLS fitted line toward it. 10

11 Summary of Outliers According to Equation (2), an outlier should satisfy both of following: (1) it has big ê i, which means unusual value of y or big discrepancy from the fitted line; (2) it has big leverage h ii, which means the value of regressor is far from the center. In short, an outlier must have an unusual X-value with an unusual Y-value given its X-value. 11

12 Least Absolute Deviations Estimation In the presence of outliers, one option is reporting the OLS regression excluding them. Alternatively, we may report the result of Least Absolute Deviations (LAD) Estimation, which minimizes the sum of the absolute values of residuals. Because the residual is not squared, the effect of outliers is diminished, meaning that LAD estimate is less sensitive to outliers than OLS estimate. In fact, LAD estimates the parameters of the conditional median of y given x. > library(l1pack) > lad(gro infr, data=all, method = "EM") Coefficients: (Intercept) infr See section 9.6 of Wooldridge s Introductory Econometrics textbook for more details. 12

13 FW Theorem Consider a multiple regression in the form of partitioned matrix Y = X 1 ˆβ1 + X 2 ˆβ2 + ê Pre-multiplying the matrix M 1 I X 1 (X 1 X 1) 1 X 1 yields ˆβ 2 = (X 2M 1 X 2 ) 1 (X 2M 1 Y ) = (ˆr ˆr) 1 (ˆr Y ) So the FW theorem states that we can obtain ˆβ 2 in two steps: first, regressing X 2 onto X 1 and keep the residual ˆr M 1 X 2, then second, regressing Y onto ˆr. It follows that var( ˆβ 2 ) = σ 2 (ˆr ˆr) 1 = σ 2 SST X2 (1 R 2 X2X1 ) where SST X2 denotes the total sum squares (TSS) for X 2, and R 2 X2X1 regressing X 2 onto X 1. is the R-squared of 13

14 Variance Inflation Factor (VIF) We obtain imprecise estimate for β 2 (with big standard error) when 1. σ 2 is big, i.e., when there are many omitted factors. 2. SST X2 is small, i.e., when there is little variation in X R 2 X2X1 is close to one, i.e., when X 2 is highly correlated with X 1, an issue called multicollinearity. In general, we can define the Variance Inflation Factor as VIF j = 1 1 R 2 j where R 2 j denotes the R-squared of regressing the j-th regressor onto all other regressors. If VIF is above 10, we conclude that multicollinearity is a problem for estimating β j. In that case, we may drop some regressors to mitigate the multicollinearity. Doing this will produce more efficient (with smaller standard error) but less unbiased estimate (since we have more omitted variables). 14

15 An Illustration I We generate two redundant variables that are highly correlated with inflation rate. As expected, we obtain big VIF after including those redundant variables r.infr1 = infr + 0.1*rnorm(n) r.infr2 = infr + 0.1*rnorm(n) m3 = lm(gro infr+r.infr1+r.infr2, data=all) library(car) vif(m3) > vif(m3) infr r.infr1 r.infr

16 An Illustration II By contrast, the VIF is small if an uncorrelated regressor is added > r.infr3 = rnorm(n) > m4 = lm(gro infr+r.infr3, data=all) > vif(m4) infr r.infr

17 Regression with Categorial Information I We can run the groupwise regression, where the group (subset) is specified by categorial information. For example sub2 = subset(all, gro>0) sub3 = subset(all, gro<=0) summary(lm(gro infr, data=sub2)) Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** infr summary(lm(gro infr, data=sub3)) Estimate Std. Error t value Pr(> t ) (Intercept) e-07 *** infr

18 Regression with Categorial Information II Of course, we get the same results by using all observations, and including a dummy and an interaction term d = ifelse(gro<=0, 1,0) i = d*infr summary(lm(gro infr+d+i, data=all)) Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** infr d <2e-16 *** i