Statistics and Quantitative Analysis U4320. Lecture 13: Explaining Variation Prof. Sharyn O Halloran

Transcription

1 Statistics and Quantitative Analysis U4320 Lecture 13: Eplaining Variation Prof. Sharyn O Halloran

2 I. Eplaining Variation: R 2 A. Breaking Down the Distances Let's go back to the basics of regression analysis. Y Money Spent 8 on Health Care 7 Y=a+b 6 Y Income How well does the predicted line eplain the variation in the independent variable money spent? X

3 I. Eplaining Variation: R 2 Y $ Spent on Health Care Total Variation Y=a+b (,y) Y-Y=deviation uneplained by regression Y-Y=deviation eplained by regression Y Y-Y=total deviation around Y Income X

4 I. Eplaining Variation: R 2 Total Deviation Y Y = ( Y$ Y ) + ( Y Y$). Total = Eplained + Uneplained Deviation Deviation Deviation The total distance from any point to Y is the sum of the distance from Y to the regression line plus the distance from the regression line to Y.

5 I. Eplaining Variation: R 2 B. Sums of Squares We can sum this equation across all the Y's and square both sides to get: 2 ( Y Y ) = ( Y Y$ ) + ( Y$ Y ) 2 2 = ( Y Y$ ) + 2 ( Y Y$ )( Y$ Y ) + ( Y $ Y ) = ( Y $ Y ) + ( Y Y$ ), 2 2 2

6 I. Eplaining Variation: R 2 1. Total Sum of Squares (SST). The term on the left-hand side of this equation is the sum of the squared distances from all points to Y. We call this the total variation in the Y's, or the Total Sum of Squares (SST). 2. Regression Sum of Squares The first term on the right hand side is the sum of the squared distances from the regression line to Y. We call it the Regression Sum of Squares, or SSR.

7 I. Eplaining Variation: R 2 3. Error Sum of Squares Finally, the last term is the sum of the squared distances from the points to the regression line. Remember, this is the quantity that least squares minimizes. We call it the Error Sum of Squares, or SSE. We can rewrite the previous equation as: SST = SSR + SSE.

8 I. Eplaining Variation: R 2 C. Definition of R2 We can use these new terms to determine how much variation is eplained by the regression line. If the points are perfectly linear, then the Error Sum of Squares is 0:

9 I. Eplaining Variation: R 2 Y Money Spent 8 on Health Care Y Income Here, SSR = SST. The variance in the Y's is completely eplained by the regression line. X

10 I. Eplaining Variation: R 2 On the other hand, if there is no relation between X and Y: Y Money Spent on Health Care Y Income Now SSR is 0 and SSE=SST. The regression line eplains none of the variance in Y. X

11 I. Eplaining Variation: R 2 3. Formula So we can construct a useful statistic. Take the ratio of the Regression Sum of Squares to the Total Sum of Squares: R 2 = SSR SST We call this statistic R 2 It represents the percent of the variation in Y eplained by the regression

12 I. Eplaining Variation: R 2 R 2 is always between 0 and 1. For a perfectly straight line it's 1, which is perfect correlation. For data with little relation, it's near 0. R 2 measures the eplanatory power of your model. The more of the variance in Y you can eplain, the more powerful your model.

13 I. Eplaining Variation: R 2 D. Eample I wanted to investigate why people have confidence in what they see on TV. 1. Dependent variable TRUSTTV = 1 if has a lot of confidence 2. Independent variables = 2 if somewhat confidence = 3 if the individual has no confidence. TUBETIME = number of Hours of TV watched a week SKOOL = years of education. LIKEJPAN = feelings towards Japan. YELOWSTN = attitudes whether the US should spend more on national parks. MYSIGN = the respondents astrological sign.

14 I. Eplaining Variation: R 2 3. Calculating R 2 a) Correlation matri TRUSTTV TUBETIME SKOOL LIKEJPAN YELOWSTN MYSIGN TRUSTTV TUBETIME SKOOL LIKEJPAN YELOWSTN MYSIGN

15 I. Eplaining Variation: R 2 b) The first model is: TRUSTTV = (TUBETIME) Analysis of Variance DF Sum of Squares Mean Square Regression Residual

16 I. Eplaining Variation: R 2 How do we calculate the Total Sum of Squares? SST = SSR + SSE SST = = Now we can calculate R 2 : R 2 SSR = SST = =. 031

17 I. Eplaining Variation: R 2 c) Each of the 4 different models has an associated R2. Eq # 2: R2 = Eq # 3: R 2 = Eq # 4 = R 2 =.04

18 I. Eplaining Variation: R 2 F. Using R 2 in Practice 1. Useful Tool 2. Measure of Uneplained Variance 3. Not a Statistical Test 4. Don't Obsess about R 2 5. You can always improve R 2 by adding variables

19 I. Eplaining Variation: R 2 G. Eample You'll notice that R 2 increase every time. No matter what variables you add you can always increase your R 2.

20 II. Adjusted R2 II. Adjusted R 2 A. Definition of Adjusted R 2 So we'd like a measure like R 2, but one that takes into account the fact that adding etra variables always increases your eplanatory power. The statistic we use for this is call the Adjusted R 2, and its formula is: 2 n 1 R = 2 1 ( 1 R ); n k n = number of observations, k = number of independent variables. So the Adjusted R 2 can actually fall if the variable you add doesn't eplain much of the variance.

21 II. Adjusted R2 B. Back to the Eample 1. Adjusted R 2 You can see that the adjusted R 2 rises from equation 1 to equation 2, and from equation 2 to equation 3. But then it falls from equation 3 to 4, when we add in the variables for national parks and the zodiac.

22 II. Adjusted R2 2. Calculating Adjusted R 2 Eample: Equation 2 Multiple R R Square Adjusted R Square Standard Error Analysis of Variance DF Sum of Squares Mean Square Regression Residual F = Signif F =.0002

23 II. Adjusted R2 Variable B SE B Beta T Sig T SKOOL TUBETIME (Constant) We calculate: 2 n 1 2 R = 1 ( 1 R ) n k = 1 ( ) =. 0314

24 II. Adjusted R2 C. Stepwise Regression One strategy for model building is to add variables only if they increase your adjusted R 2. This technique is called stepwise regression. However, I don't want to emphasize this approach to strongly. Just as people can fiate on R 2 they can fiate on adjusted R 2. ****IMPORTANT**** If you have a theory that suggests that certain variables are important for your analysis then include them whether or not they increase the adjusted R 2. Negative findings can be important!

25 III. F Tests III. F Tests A. When to use an F-Test? Say you add a number of variables into a regression model and you want to see if, as a group, they are significant in eplaining variation in your dependent variable Y. The F-test tells you whether a group of variables, or even an entire model, is jointly significant. This is in contrast to a t-test, which tells whether an individual coefficient is significantly different from zero.

26 III. F Tests B. Equations To be precise, say our original equation is: EQ 1: Y = b 0 + b 1 X 1 + b 2 X 2, and we add two more variables, so the new equation is: EQ 2: Y = b 0 + b 1 X 1 + b 2 X 2 + b 3 X 3 + b 4 X 4. We want to test the hypothesis that Η 0 : β 3 = β 4 = 0. That is, we want to test the joint hypothesis that X 3 and X 4 together are not significant factors in determining Y.

27 III. F Tests C. Using Adjusted R 2 First There's an easy way to tell if these two variables are not significant. First, run the regression without X 3 and X 4 in it, then run the regression with X 3 and X 4. Now look at the adjusted R 2 's for the two regressions. If the adjusted R 2 went down, then X 3 and X 4 are not jointly significant. So the adjusted R 2 can serve as a quick test for insignificance.

28 III. F Tests D. Calculating an F-Test If the adjusted R2 goes up, then you need to do a more complicated test, F-Test. 1. Ratio Let regression 1 be the model without X 3 and X 4, and let regression 2 include X 3 and X 4. The basic idea of the F statistic, then, is to compute the ratio: SSE SSE SSE 1 2 2

29 III. F Tests 2. Correction We have to correct for the number of independent we add. So the complete statistic is: SSE1 SSE2 F = m ; SSE2 n k m = number of restrictions; k = number of independent variables. Remember that k is the total number of independent variables, including the ones that you are testing and the constant.

30 III. F Tests 2. Correction (cont.) This equation defines an F statistic with m and n-k degrees of freedom. We write it like this: m F n k To get critical values for the F statistic, we use a set of tables, just like for the normal and t-statistics.

31 III. F Tests E. Eample 1. Adding Etra Variables: Are a group of variables jointly significant? Are the variables YELOWSTN and MYSIGN jointly significant? EQ 1: TRUSTTV = β0 + β1like JPAN + β2skool + β3tubetime. EQ 2: TRUSTTV = β0 + β1like JPAN + β2skool + β3tubetime + β4mysign + β5yelowstn

32 III. F Tests 1. Adding Etra Variables (cont.) a) State the null hypothesis H 0 : β 4 = β 5 = 0. b) Calculate the F-statistic Our formula for the F statistic is: SSE1 SSE2 F = m, SSE2 n k

33 III. F Tests What is SSE 1 --the sum of squared errors in the first regression? What is SSE 2, the sum of squared errors in the second regression? m = 2 N = 470 k = 6 The formula is: F = = 0.319

34 III. F Tests c) Reject or fail to reject the null hypothesis? The critical value at the 5 % level 2 F from the table, is F Is the F-statistic >? If yes, then we reject the null hypothesis that the variables are not significantly different from zero; otherwise we fail to reject..319 < 3.00, so we can reject the null hypothesis. B=

35 III. F Tests 2. Testing All Variables: Is the Model Significant? Equation 2 Multiple R R Square Adjusted R Square Standard Error Analysis of Variance DF Sum of Squares Mean Square Regression Residual F = Signif F =.0002

36 III. F Tests Variable B SE B Beta T Sig T SKOOL TUBETIME (Constant) a) Hypothesis: H 0 : β 1 = β 2 = 0. Again, we start with our formula: F = SSE SSE m SSE2 n k 1 2,

37 III. F Tests b) Calculate F-statistic SSE 2 = SSE 1 is the sum of squared errors when there are no eplanatory variables at all. If there are no eplanatory variables, then SSR must be 0. In this case, SSE=SST. So we can substitute SST for SSE1 in our formula. SST = SSR + SSE = = F = = This is the number reported in your printout under the F statistic.

38 III. F Tests c) Reject or fail to reject the null hypothesis? 2 The critical value at the 5% level, F from your table, is So this time we can reject the null hypothesis that β 1 = β 2 = 0.