r2, the coefficient of determination The bivariate normal assumption Diagnostic plots: Residuals and Cook's Distance R output (moved to week 3),

Transcription

1 Today's Agenda r2, the coefficient of determination The bivariate normal assumption Diagnostic plots: Residuals and Cook's Distance R output (moved to week 3), Syllabus note: We are ahead of schedule in regression, so we're taking the time to add more examples and details, like Cook's distance and residuals. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 1

2 r2, the coefficient of determination r2 is simply the Pearson correlation coefficient r, but squared. So why all the fuss about it? When x and y are correlated, we say that some of the variation in y is explained by x. The proportion explained is r2. It is called the coefficient of determination because it represents how well a value of y can be determined by x. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 2

3 Abstract case 1: If there was a perfect correlation between x and y, then the relationship between them could be described perfectly by a line. ( r = -1 or +1) Once you have the regression equation, knowing x allows you determine what y is, and without any error. In these cases, r2 is 1, meaning that 100% of the variance in y is explained by x. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 3

4 Abstract case 2: If there was NO correlation between x and y, such that r=0, then there is no linear relationship between x and y. Knowing x and using the regression equation of that (lack of) relationship would tell you literally nothing about y. In these cases, r2 is 0, so none of the variance in y is explained by x. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 4

5 Medical example: On page 4 of 8 of this paper, Pak J Physiol 2010;6(1): there are several scatterplots describing the correlation between resting heart rate (RHR) and several other possibly related variables. Consider the first scatterplot, called Figure 1A. In this figure, a regression of body-mass index (BMI, y) as a function of resting heart rate (RHR, x) is shown. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 5

6 Scatterplot of Heart Rate (x) and Body-Mass Index (y) Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 6

7 Here, the sample correlation is r = 0.305, and there is strong evidence that the population correlation is positive because p < r2 = = , so 9.3% of the variation in BMI can be explained by RHR. Also, 9.3% of the variation in RHR can be explained by BMI. Why? Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 7

8 Correlation works in both directions. In Figure 1b, the sample shows that some variation in Waistto-Hip Ratio (WHR) is explained by (and explains) RHR = or 5.3% of the variation. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 8

9 If Body-Mass Index explains 9.3% of the variation of RHR, and Waist-to-Hip Ratio explains 5.3% of the variation, could they together explain = 14.6%? Sadly, no. Since BMI and WHR are measuring very similar things, there is going to be a lot of overlap in the variation that they explain. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 9

10 But what is this 'variation'? Let's dig deeper! Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 10

11 Recall that the regression equation without the error term, α + βx, is called the least squares line. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 11

12 The 'squares' being referred to are the squared errors. Mathematically, it is the line through the data that produces the smaller sum of squared error (SSE), which is where epsilon ε is the error term that we ignored earlier: Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 12

13 The sum of squares error SSE is the amount of variation that is left unexplained by the model. We used squared errors because... - Otherwise negative and positive errors would cancel. - This way, the regression equation will favour creating many small errors instead of one big one.* - In calculus, the derivative of x2 is easy to find. * Also why Pearson correlation is sensitive to extreme values. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 13

14 The error term is in any model we use, even the null model, which is a fancy term for not regressing at all. Or In the null model, every value of y is predicted to be the average of all observed y values. So α is the sample mean of y, y-bar. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 14

15 The total squared difference from the mean of y is called the sum of squares total, or SST SST is the total square length of all the vertical red lines. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 15

16 If we fit a regression line, (most of the) errors become smaller. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 16

17 Most importantly, the squared errors get smaller. The coefficient of determination, r2, is measuring how much smaller the squared errors get. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 17

18 Here, the correlation is very strong ( r is large), and there are barely and errors at all. So SSError would be much smaller than SSTotal, and r2 is also large Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 18

19 2 The relationship between r, SSE, and SST is: SST is the total amount of variation in Y SSE is the amount of variation in Y left unexplained by X. 2 When r is zero, SSE is same as SST 2 When r is one, SSE disappears completely. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 19

20 So we now have two different interpretations of r-squared. 1. The square of the correlation efficient. 2. The proportion of Sum of Squares Total (SST) that is removed from the error term. Interpretation #1 is specific to correlation. Interpretation #2 works for simple regression, but also for AnOVa, multiple regression, general linear models! Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 20

21 R-squared is truly the go-anywhere animal. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 21

22 Bivariate Normality (and some diagnostics) Regression produces a line that minimizes sum squared errors, so a small number of extreme values (outliers) can have a strong effect on a model. Consider this Pearson r: Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 22

23 More specifically, regression is sensitive to violations of the assumption of bivariate normality. The regression model assumes: 1. The distributions of the x and y variables is normal. If you were to take a histogram of all the x values, that histogram should resemble a normal curve. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 23

24 The regression model also assumes: 2. The distribution of y, conditional on x, is normal. If you were to take a histogram of all the error terms, that histogram should ALSO resemble a normal curve. Any observations that produce errors that are too large to be in the curve are potentially influential outliers. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 24

25 In this diagram, the red line is the regression on all 54 points. The blue line is the regression without the 4 red points. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 25

26 These points are near the lower end of x, and have very large error terms associated with them, so they 'pull' the left end of the regression line down. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 26

27 Another word for these errors is residuals, literally the residue, or portion left over from the model. Here is a scatterplot of the residuals over x. A.K.A, a residual plot. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 27

28 The outliers are clearly visible from the residual plot, and from the histogram below. Their values are twice as large as any other observation. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 28

29 One way to measure how much an outlier is affecting the model is to remove that one point and see how much the model changes. We can see a big difference between the blue and red lines above, but that is a comparison by removing 4 points manually. Another, more systematic (and therefore quick, easy, and often more reliable) method is to remove one observation at a time and see how much the model changes. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 29

30 Cook's distance is a regression deletion diagnostic. It works by comparing a model with every observation to one with only the observation in question removed / deleted. The higher Cook's distance is for a value, the more that particular value is influencing the model. If there are one or two values that are having undue leverage on the model, Cook's distance will find them. This is true even if the residual plot fails to find them (which it can if the observation is 'pulling' hard enough) Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 30

31 This is Cook's Distance for all 54 data points. Note that although all 4 problem points have high Cook's distance compared to the rest, two of them are not obvious problems. Cook's distance has a hard time identifying influential observes when there are several. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 31

32 Dealing with outliers is like selecting an acceptable Type I error. There are conventions and guidelines in place, but it is a case-by-case judgement call. One question to ask is does this observation belong in my data set?, when considering things other than your model. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 32

33 If the outlier is the result of a typo, it's not the same as the rest of your sample and it should go. If other information about that observation is nonsense, such as joke answers in a survey, then that's also justification to remove that outlier observation. If it just happens to be an extreme value, but otherwise everything seems fine with it, then it is best to keep it. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 33

34 Don't rush to finish your model. Look for outliers first. Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 34

35 Next Tuesday: - Diagnostics and Regression in R. - Correlation vs Causality Read: Rubin on Causality, only Sections 1-3 for next Tuesday. Sources: xkcd.com/605 My Hobby: Extrapolating. Sand Crab Photo, by Regiane Cardillo, Brasil Pak. J. Phisol. (2010) 6:1 Mandarin Duck and Parrot on Tortoise unknown Stat 302, Winter 2016 SFU, Week 2, Hour 3, Page 35