Relationships Regression

Transcription

1 Relationships Regression BPS chapter W.H. Freeman and Company

2 Objectives (BPS chapter 5) Regression Regression lines The least-squares regression line Using technology Facts about least-squares regression Residuals Influential observations Cautions about correlation and regression Association does not imply causation

3 Correlation tells us about strength (scatter) and direction of the linear relationship between two quantitative variables. In addition, we would like to have a numerical description of how both variables vary together. For instance, is one variable increasing relative to the other one? And we would like to make predictions based on the observed association between those two variables. But which line best describes our data?

4 A regression line A regression line is a straight line that describes how a response variable y changes as an explanatory variable x changes. We often use a regression line to predict the value of y for a given value of x.

5 The regression line The least-squares regression line is the unique line such that the sum of the total vertical (y) distances is zero and sum of the squared vertical (y) distances between the data points and the line is the smallest possible. Distances between the points and line are squared so all are positive values. This is done so that distances can be properly added (Pythagoras).

6 Facts about least-squares regression 1. The distinction between explanatory and response variables is essential in regression. For instance the direction of the association determines the slope of the regression line. 2. There is a close connection between correlation and the slope of the least-squares line. 3. The least-squares regression line always passes through the point x, y 4. The correlation r describes the strength of a straight-line relationship. The square of the correlation, r 2, is the fraction of the variation in the values of y that is explained by the least-squares regression of y on x. The square of the correlation is called Coefficient of Determination.

7 Properties The least-squares regression line can be shown to have this equation: yˆ a bx, where b r s s y x or y ˆ and a y yˆ ( y rx is the predicted y value (y hat) b is the slope a is the y-intercept bx, s s y x ) r s s y x x "a" is in units of y "b" is in units of y/units of x

8 How to: First we calculate the slope of the line, b, from statistics we already know: b r s y r is the correlation s y is the standard deviation of the response variable y s x is the the standard deviation of the explanatory variable x s x Once we know b, the slope, we can calculate a, the y-intercept: a y bx where x and y are the sample means of the x and y variables This means that we don t have to calculate a lot of squared distances to find the leastsquares regression line for a data set. We can instead rely on the equation. But typically, we use a 2-var stats calculator or a stats software.

9 BEWARE!!! Not all calculators and software use the same convention: y ˆ a bx Some use instead: Make sure you know what YOUR y ˆ ax b calculator gives you for a and b before you answer homework or exam questions.

10 Software output Intercept Slope R 2 r R 2 Intercept Slope

11 The equation completely describes the regression line. To plot the regression line, you only need to plug two x values into the equation, get y, and draw the line that goes through those two points. Hint: The regression line always passes through the mean of x and y. The points you use for drawing the regression line are derived from the equation. They are NOT points from your sample data (except by pure coincidence).

12 The distinction between explanatory and response variables is crucial in regression. If you exchange y for x in calculating the regression line, you will get the wrong line. Regression examines the distance of all points from the line in the y direction only. Data from the Hubble telescope about galaxies moving away from Earth: These two lines are the two regression lines calculated either correctly (x = distance, y = velocity, solid line) or incorrectly (x = velocity, y = distance, dotted line).

13 Correlation and regression The correlation is a measure of spread (scatter) in both the x and y directions in the linear relationship. In regression we examine the variation in the response variable (y) given change in the explanatory variable (x).

14 Coefficient of determination, r 2 r 2, the coefficient of determination, is the square of the correlation coefficient. r 2 represents the fraction of the variance in y (vertical scatter from the regression line) that can be explained by changes in x. b r s y s x

15 r = 1 r 2 = 1 Changes in x explain 100% of the variations in y. y can be entirely predicted for any given value of x. r = 0.87 r 2 = 0.76 r = 0 r 2 = 0 Changes in x explain 0% of the variations in y. The value(s) y takes is (are) entirely independent of what value x takes. Here the change in x only explains 76% of the change in y. The rest of the change in y (the vertical scatter, shown as red arrows) must be explained by something other than x.

16 r =0.7 r 2 =0.49 There is quite some variation in BAC for the same number of beers drunk. A person s blood volume is a factor in the equation that was overlooked here. We changed the number of beers to the number of beers/weight of a person in pounds. r =0.9 r 2 =0.81 In the first plot, number of beers only explains 49% of the variation in blood alcohol content. But number of beers/weight explains 81% of the variation in blood alcohol content. Additional factors contribute to variations in BAC among individuals (like maybe some genetic ability to process alcohol).

17 Grade performance If class attendance explains 16% of the variation in grades, what is the correlation between percent of classes attended and grade? 1. We need to make an assumption: Attendance and grades are positively correlated. We also assume that the association between attendance and grades is linear. So r will be positive too. 2. r 2 = 0.16, so r = = A weak correlation.

18 Residuals The distances from each point to the least-squares regression line give us potentially useful information about the contribution of individual data points to the overall pattern of scatter. These distances are called residuals. Points above the line have a positive residual (under estimation). Points below the line have a negative residual (over estimation). ^ Predicted y Observed y dist. The sum of these residuals is always 0. ( y yˆ) residual

19 Residual plots Residuals are the distances between y-observed and y-predicted. We plot them in a residual plot. If residuals are scattered randomly around 0, chances are your data fit a linear model, were normally distributed, and you didn t have outliers.

20 The x-axis in a residual plot is the same as on the scatterplot. The line on both plots is the regression line. Only the y-axis is different.

21 Residuals are randomly scattered good! A curved pattern means the relationship you are looking at is not linear. A change in variability across plot is a warning sign. You need to find out why it is and remember that predictions made in areas of larger variability will not be as good.

22 Outliers and influential points Outlier: An observation that lies outside the overall pattern of observations. Influential individual : An observation that markedly changes the regression if removed. This is often an outlier on the x-axis. Child 19 = outlier in y direction Child 19 is an outlier of the relationship. Child 18 = outlier in x direction Child 18 is only an outlier in the x direction and thus might be an influential point.

23 Outlier in y-direction All data Without child 18 Without child 19 Are these points influential? Influential

24 Correlation/regression using averages Many regression or correlation studies use average data. While this is appropriate, you should know that correlations based on averages are usually quite higher than when made on the raw data. The correlation is a measure of spread (scatter) in a linear relationship. Using averages greatly reduces the scatter. Therefore, r and r 2 are typically greatly increased when averages are used.

25 Boys Boys Each dot represents an average. The variation among boys per age class is not shown. These histograms illustrate that each mean represents a distribution of boys of a particular age. Should parents be worried if their son does not match the point for his age? If the raw values were used in the correlation instead of the mean, there would be a lot of spread in the y-direction,and thus, the correlation would be smaller.

26 That s why typically growth charts show a range of values (here from 5th to 95th percentiles). This is a more comprehensive way of displaying the same information.

27 Making predictions: Interpolation The equation of the least-squares regression allows you to predict y for any x within the range studied. This is called interpolating. ˆ y x Nobody in the study drank 6.5 beers, but by finding the value of ŷ from the regression line for x = 6.5, we would expect a blood alcohol content of mg/ml. yˆ yˆ * mg / ml

28 (in 1000 s) Year Powerboats Dead Manatees ˆ y 0.125x 41.4 There is a positive linear relationship between the number of powerboats registered and the number of manatee deaths. The least-squares regression line has for equation: ˆ y 0.125x 41.4 Thus, if we were to limit the number of powerboat registrations to 500,000, what could we expect for the number of manatee deaths? yˆ 0.125(500) 41.4 yˆ Roughly 22 manatees.

29 Caution with regression Do not use a regression on inappropriate data. Pattern in the residuals Presence of large outliers Use residual plots for help. Clumped data falsely appearing linear Recognize when the correlation/regression is performed on averages. A relationship, however strong, does not itself imply causation. Beware of lurking variables. Avoid extrapolating (going beyond interpolation).

30 Lurking variables A lurking variable is a variable not included in the study design that does have an effect on the variables studied. Lurking variables can falsely suggest a relationship. What is the lurking variable in these examples? How could you answer if you didn t know anything about the topic? Strong positive association between the number firefighters at a fire site and the amount of damage a fire does Negative association between moderate amounts of wine drinking and death rates from heart disease in developed nations

31 Extrapolation Extrapolation is the use of a regression line for predictions outside the range of x values used to obtain the line. This can be misleading, as seen here.

32 Bacterial growth rate changes over time in closed cultures: If you only observed bacterial growth in test tubes during a small subset of the time shown here, you could get almost any regression line imaginable. Extrapolation = big mistake

33 The y-intercept Sometimes the y-intercept is not biologically possible. Here we have negative blood alcohol content, which makes no sense But the negative value is y-intercept shows negative blood alcohol appropriate for the equation of the regression line. There is a lot of scatter in the data and the line is just an estimate.

34 Always plot your data! The correlations all give r 0.816, and the regression lines are all approximately ŷ = x. For all four sets, we would predict ŷ= 8 when x = 10.

35 Always plot your data! However, making the scatterplots shows us that the correlation/ regression analysis is not appropriate for all data sets. Moderate linear Obvious nonlinear One point deviates from Just one very influential association; relationship; the (highly linear) point and a series of regression OK. regression pattern of the other other points all with the inappropriate. points; it requires same x value; a examination before a redesign is due here regression can be done.

36 reading index Association and causation Association, however strong, does NOT imply causation. Only careful experimentation can show causation. Strong positive linear relationship Children reading skills w ith shoe size Shoe size Not all examples are so obvious

37 There is quite some variation in BAC for the same number of beers drunk. A person s blood volume is a factor in the equation that we have overlooked. Now we change the number of beers to the number of beers/weight of a person in pounds. Note how much smaller the variation is. An individual s weight was indeed influencing our response variable blood alcohol content.

38 Vocabulary: lurking vs. confounding LURKING VARIABLE A lurking variable is a variable that is not among the explanatory or response variables in a study and yet may influence the interpretation of relationships among those variables. CONFOUNDING Two variables are confounded when their effects on a response variable cannot be distinguished from each other. The confounded variables may be either explanatory variables or lurking variables. But you often see them used interchangeably

39 Association and causation It appears that lung cancer is associated with smoking. How do we know that both of these variables are not being affected by an unobserved third (lurking) variable? For instance, what if there is a genetic predisposition that causes people to both get lung cancer and become addicted to smoking, but the smoking itself doesn t CAUSE lung cancer? We can evaluate the association using the following criteria: 1) The association is strong. 2) The association is consistent. 3) Higher doses are associated with stronger responses. 4) The alleged cause precedes the effect. 5) The alleged cause is plausible.

40 WINS Linear Regression y LinReg (a+bx) Y1 b a ERA x 40

41 The Least-Square Regression Finds the best fit line by minimizing the areas formed by the difference: Observed Value - Predicted Value 41

42 WINS The Least-Square Regression y Finds the best fit line by minimizing the areas formed by the difference: Observed Value - Predicted Value ERA x 42

43 The Least-Square Regression Statisticians use a slightly different version of slopeintercept form. ˆ a y bx y a bx b s r s y x a y bx 43

44 WINS Predicting Model On the calculator LSRL onto Scatterplot Use: LinReg(a+bx) Y 1 OR Use: Y 1 = RegEQ NOTE: Y 1 VARS:Y-Vars:Fcn:Y 1 RegEQ VARS:Statistics:Eq:RegEQ Then use TRACE, TABLE or Y 1 to find: y values that correspond to particular x values. y ERA x 44

45 Fact about least-square regression Make sure you know which is the explanatory variable (x) & which is the response variable (y). Switching them gives a different regression line. 45

46 Least-Squares Regression Facts: Regression line always goes through the point: ( x, y The correlation coefficient (r) explains the strength of the linear relationship. The square of the correlation (r-squared) indicates the variation in y that is explained by x. % of the variation of is explained by. ) 46

47 r 2 coefficient of explanation In the regression of ERA vs. WINS: r 2 =.4512 LinReg (a+bx) Y1 Interpretation: 45% of the variation in WINS can be explained by ERA b a 47

48 Outliers vs. Influential Data An OUTLIER is an observation outside the overall pattern An INFLUENTIAL POINT has a large effect on the regression line. Removing it significantly changes both the regression line & calculations. 48

49 Outliers vs. Influential Data 49

50 RESIDUAL Note: the observed value almost never match the predicted values exactly Predicted Value ( ŷ): 5.3 ERA Wins RESIDUAL = difference b/t observed & predicted values: RESIDUAL = y yˆ y yˆ Observed Value Predicted Value Residual: y yˆ = Observed Value (y): 5.3 ERA, 43 Wins 50

51 Residuals When observed value is below predicted value: residual is negative Predicted Value ( ŷ): 5.3 ERA Wins When observed value is above predicted value: residual is positive Residual: y yˆ = Observed Value (y): 5.3 ERA, 43 Wins 51

52 Residual Plots Does a LINEAR MODEL fit the original data well? Make a RESIDUAL PLOT to see if the LINEAR MODEL describes our orig data well. 52

53 Residual Plots This residual plot shows... Scattered residuals & NO OBVIOUS PATTERN. Interpretation Orig scatterplot (y vs x) appears to be a LINEAR RELATIONSHIP. y Residuals yˆ x 53

54 Residual Plot has a PATTERN Orig Scatterplot is NOT LINEAR y yˆ x 54

55 Residual Plot has NO PATTERN y yˆ Orig Scatterplot is LINEAR x 55