Unit 10: Simple Linear Regression and Correlation

Transcription

1 Unit 10: Simple Linear Regression and Correlation Statistics 571: Statistical Methods Ramón V. León 6/28/2004 Unit 10 - Stat Ramón V. León 1

2 Introductory Remarks Regression analysis is a method for studying the relationship between two or more numerical variables In regression analysis one of the variables is regarded as a response, outcome or dependent variable The other variables are regarded as predictor, explanatory or independent variables An empirical model approximately captures the main features of the relationship between the response variable and the key predictor variables Sometimes it is not clear which of two variable should be the response. In this case correlation analysis is used. In this unit we consider only two variables 6/28/2004 Unit 10 - Stat Ramón V. León 2

3 A Probabilistic Model for Simple Linear Regression 2 Note that the Yi are independent N( µ i = β0 + β1xi, σ ) r.v.'s Constant variability about the regression line Notice that the predictor variable is regarded as nonrandom because it is assumed to be set by the investigator 6/28/2004 Unit 10 - Stat Ramón V. León 3

4 A Probabilistic Model for Simple Linear Regression Let x1, x2,..., xn be specific settings of the predictor variable. Let y1, y2,..., yn be the corresponding values of the response variable. Assume that yi is the observed value of a random variable (r.v.) Yi, which depends on xi according to the following model: Y = β + β x + ε ( i = 1, 2,..., n). i 0 1 i i 2 Here εi is the random error with E( εi) = 0 and Var( εi) = σ Thus EY ( i) = µ i = β0 + β1xi (true regression line). We assume that εi are independent, identically distributed (i.i.d.) r.v.'s We also usually assume that the are normally distributed. ε i 6/28/2004 Unit 10 - Stat Ramón V. León 4

5 Reflections on the Method of Tire Data Collection: Experimental Design Mileage Depth Mileage in 1000 miles Depth in mils Was one tire or nine used in the experiment? (Read the book!) What effect would this answer have on the experimental error? on our ability to generalize to all tires of this brand? Was the data collected on one or several cars? Does it matter? Was the data collected in random order or in the order given? Does it matter? Is there a possible confounding problem? 6/28/2004 Unit 10 - Stat Ramón V. León 5

6 Example 10.1: Tread Wear vs. Mileage Using the Fit Y by X platform in the Analyze menu 6/28/2004 Unit 10 - Stat Ramón V. León 6

7 Scatter Plot: Tread Wear vs. Mileage 6/28/2004 Unit 10 - Stat Ramón V. León 7

8 Least Squares Line Fit y = ˆ β + ˆ 0 1 β x 6/28/2004 Unit 10 - Stat Ramón V. León 8

9 Illustration: Least Squares Line Fit The least squares line fit is the line that minimizes the sum of the squares of the lengths of the vertical segments 6/28/2004 Unit 10 - Stat Ramón V. León 9

10 i= 1 Mathematics of Least Squares Find the line, i.e., values of β and β that minimizes the sum of the squared deviations: How? [ ( β β )] Solve for values of β and β for which Q β n Q= y + x i Q = 0 and = 0 β i 6/28/2004 Unit 10 - Stat Ramón V. León 10

11 Predict Mean Grove Depth for a Given Mileage For x = 25 (25,000 miles) the predicted y: yˆ = ˆ β + ˆ β (25) = = mils. 0 1 Add a row to the table and leave the y value blank. 6/28/2004 Unit 10 - Stat Ramón V. León 11

12 Goodness of Fit of the LS Line Fitted values of y : yˆ = ˆ β + ˆ β x, i = 1,2,..., n. i i 0 1 i ( ˆ β ˆ ) 0 β1 Residuals: e = y yˆ = y + x = segment length i i i i i 6/28/2004 Unit 10 - Stat Ramón V. León 12

13 Sums of Squares Error Sum of Squares (SSE) = Total Sum of Squares (SST) = n n 2 ei = yi yi i= 1 i= 1 i= 1 Regression Sum of Squares (SSR) = n ( y y) i n i= 1 ( ˆ ) 2 ( yˆ y) n 2 n 2 n 2 ( y y) = ( yˆ y) + ( y yˆ ) i i i i i= 1 i= 1 i= 1 SST = SSR + SSE i 2 2 These definitions apply even if the model is more complex than linear in x, for example, if it is quadratic in x SST is interpreted as the total variation in y JMP calls SSR the Model Sums of Square 6/28/2004 Unit 10 - Stat Ramón V. León 13

14 y i yi y y i yˆi y ˆ i y y Illustration of Sums of Squares 6/28/2004 Unit 10 - Stat Ramón V. León 14

15 Coefficient of Determination SST = SSR + SSE 2 SSR SSE r = = 1 = proportion of the variation in SST SST that is accounted for by regression on x y SSR SST = 50, , /28/2004 Unit 10 - Stat Ramón V. León 15

16 s Estimation of σ 2 n n n 2 e i yi y i yi 0 1xi 2 i= 1 i= 1 i= 1 ( ) 2 ˆ ( ˆ β + ˆ β ) 2 = = = n 2 n 2 n 2 s 2 s This estimate has n-2 degrees of freedom because two unknown parameters are estimated. 6/28/2004 Unit 10 - Stat Ramón V. León 16

17 Statistical Inference for β 0 and β 1 2. Select ˆ β0 β ˆ 0 β1 β1 N(0,1) and N(0,1) SD( ˆ β ) SD( ˆ β ) 0 1 ˆ β β ˆ β β t and SE( ˆ β ) SE( ˆ β ) n 2 n (1- α)% CI : ˆ β ± t SE( ˆ β ) = 7.28 ± t = 1 n 2, α 2 1 7,0.025 [ ] 7.28 ± = 8.733, ˆ β ˆ 0 ± tn 2, α 2SE( β0) = ± t7, t 1. Right click on the Parameter Estimate area 6/28/2004 Unit 10 - Stat Ramón V. León 17

18 Test of Hypothesis for β 0 and β 1 H : β = 0 vs. H : β Reject H at α level if t = t ˆ β 0 ˆ β = > t SE( ˆ β ) SE( ˆ β ) n 2, α 2 Tire example: ˆ β 7.28 SE( ˆ β ) = = = t7,.025 = , Strong evidence that mileage affects tread depth. (Two-sided p-value) 6/28/2004 Unit 10 - Stat Ramón V. León 18

19 The Mean Square (MS) is the Sums of Square divided by its degrees of freedom, e.g. MSE = SSE/df = /7 = H Analysis of Variance : β = 0 vs. H : β MSR F = MSE = F = = ( ) = t 2 2 F and t relationship holds only when the F numerator has one degree of freedom 6/28/2004 Unit 10 - Stat Ramón V. León 19

20 Example: Prediction of a Future y or Its Mean For a fixed value of x*, are we trying to predict -the average value of y? -the value of a future observation of y? Do I want to predict the average selling price of all 4,000 square feet houses in my neighborhood. Or do I want to predict the particular future selling price of my 4,000 square feet house? Which prediction is subject to the most error? 6/28/2004 Unit 10 - Stat Ramón V. León 20

21 Prediction of a Future y or Its Mean: Prediction Interval or Confidence Interval For a fixed value of x*, are we trying to predict -the average value of y? -the value of a future observation of y? 6/28/2004 Unit 10 - Stat Ramón V. León 21

22 JMP: Prediction of the Mean of y Grove Depth (in mils) Mileage (in 1000 Miles 6/28/2004 Unit 10 - Stat Ramón V. León 22

23 JMP: Prediction of a Future Value of y Grove Depth (in mils) Mileage (in 1000 Miles 6/28/2004 Unit 10 - Stat Ramón V. León 23

24 Formulas for Confidence and Prediction Intervals xx n i= 1 ( ) 2 S = x x i Prediction Interval of Chapter 7: 6/28/2004 Unit 10 - Stat Ramón V. León 24

25 Confidence and Prediction Intervals with JMP Using the Fit Model Platform not the Fit Y by X platform 6/28/2004 Unit 10 - Stat Ramón V. León 25

26 CI for Mean for µ* 6/28/2004 Unit 10 - Stat Ramón V. León 26

27 Prediction Interval for Y* 6/28/2004 Unit 10 - Stat Ramón V. León 27

28 Prediction for the Mean of Y or a Future Observation of Y Point estimate prediction is the same in both cases: But the error bands are different Narrower for the mean of Y: [158.73, ] Wider for a future value of Y: [129.44, ] 6/28/2004 Unit 10 - Stat Ramón V. León 28

29 Calibration (Inverse Regression) 6/28/2004 Unit 10 - Stat Ramón V. León 29

30 Inverse Regression in JMP 5: Confidence Interval In Fit Model Platform unchecked 6/28/2004 Unit 10 - Stat Ramón V. León 30

31 Inverse Regression: Two Types of Confidence Intervals Calibration is usually used in measurement. Given that you got a certain measurement what is the true value of the measured quantity? Which inverse prediction interval makes more sense? 6/28/2004 Unit 10 - Stat Ramón V. León 31

32 Measurement Application Suppose you get a measurement of 6.50 on an object. What is a calibration interval for its true value? Data was supposedly obtained by measuring standards of known true measurement with a calibrated instrument How the data was really generated 6/28/2004 Unit 10 - Stat Ramón V. León 32

33 Residuals The residuals are the differences between the observed value of y and the predicted value of y yˆ = ˆ β + ˆ β (32) = = mils 0 1 6/28/2004 Unit 10 - Stat Ramón V. León 33

34 Regression Diagnostics The residuals e = y yˆ can be viewed as the "left over" i i i after the model is fitted. If the model is correct, then the e i can be viewed as the "estimates" of the random errors ε ' s. As such, the residuals are vital for checking the model assumptions. i Residual plots If the model is correct the residuals should have no structure, that is, they should look random on their plots. 6/28/2004 Unit 10 - Stat Ramón V. León 34

35 Mathematics of Residuals If the assumed regression model is correct, then the normally distributed with Ee ( ) = 0 and ( xi x) 2 Var( ei ) = σ 1 σ if n is large. n Sxx The e ' s are not independent (even though the ε 's are i n i i i i=1 i=1 i n i e ' s are independent) because they satisfy the following contraints: e = 0, xe = 0. However, the dependence introduced by these constraints is neglible in n is modestly large, say greater than 20 or so. i xx n i= 1 ( ) 2 S = x x i 6/28/2004 Unit 10 - Stat Ramón V. León 35

36 Basic Principle Underlying Residual Plots If the assumed model is correct then the residuals should be randomly scattered around 0 and should show no obvious systematic pattern. 6/28/2004 Unit 10 - Stat Ramón V. León 36

37 In Fit Y by X platform: In Fit Model platform: Red triangle menu next to Linear Fit. Residuals in JMP 6/28/2004 Unit 10 - Stat Ramón V. León 37

38 Checking for Linearity Should EY ( ) = β + β x+ β x be fitted rather than EY ( ) = β + β x? Grove Depth (in mils) Residuals Grove Depth (in mils) Mileage (in 1000 Miles Mileage (in 1000 Miles 6/28/2004 Unit 10 - Stat Ramón V. León 38

39 Predicted by Residual Plot Residuals Grove Depth (in mils) Predicted Grove Depth (in mils) Mileage (in 1000 Miles 6/28/2004 Unit 10 - Stat Ramón V. León 39 Residuals Grove Depth (in mils) What are the advantages of this plot?

40 Tread Wear Quadratic Model 6/28/2004 Unit 10 - Stat Ramón V. León 40

41 Residuals for Quadratic Model Residuals have a random pattern The Fit Y by X platform was used to get these plots Left plot can also be obtained using the Plot Residuals option 6/28/2004 Unit 10 - Stat Ramón V. León 41

42 Tread Wear Quadratic Model R 2 was for model linear in x 6/28/2004 Unit 10 - Stat Ramón V. León 42

43 Checking for Constant Variance: Plot of Predicted vs. Residual If assumption is incorrect, often Var( Y ) is some function of EY ( ) = µ 6/28/2004 Unit 10 - Stat Ramón V. León 43

44 Other Model Diagnostics Based on Residuals Checking for normality of errors: Do a normal plot of the residuals Warning: Don t plot the response y on a normal plot This plot has no meaning when one has regressors Don t transform the data on the basis of this plot Many students make this mistake in their project. Don t be one of them. 6/28/2004 Unit 10 - Stat Ramón V. León 44

45 Other Model Diagnostics Based on Residuals Fit Model Platform: Checking for independence of errors: Plot the residual in time order. The order of data collection should be always recorded. Use Durbin-Watson statistic to test for autocorrelation. 6/28/2004 Unit 10 - Stat Ramón V. León 45

46 Other Model Diagnostics Based on Residuals Checking for outliers: See if any standardized (Studentized) residual exceeds 2 (standard deviations) in absolute value. * ei ei ei ei = =, i= 1, 2,..., n. SE( e ) 2 i 1 ( xi x) s s 1 n S xx Also do a box plot of residuals to check for outliers 6/28/2004 Unit 10 - Stat Ramón V. León 46

47 Standardized (Studentized) Residuals in JMP 4 Using the Fit Model JMP platform with linear model Should we omit the first observation and refit the model? Possible outlier The model fitted here is the one linear in x, not the quadratic model 6/28/2004 Unit 10 - Stat Ramón V. León 47

48 Checking for Influential Observations An observation can be influential because it has an extreme x-value, an extreme y-value, or both Note that the influential observation above are not outliers since their residuals are quite small (even zero) 6/28/2004 Unit 10 - Stat Ramón V. León 48

49 Checking for Influential Observations n yˆ = h y where the h are some functions of the x's i ij j ij j= 1 H = h ij is called the hat matrix We can think of the h as the leverage exerted by y on the fitted value yˆ. i ij If yˆ i is largely determined by yi with very small contribution from the other y 's, then we say that the ith observations is influential (also called high leverage). j j 6/28/2004 Unit 10 - Stat Ramón V. León 49

50 e * i Checking for Influential Observations Since h detemines how much ˆ ii yi depends on yi it is used as a measure of the leverage of the i-thobservation. n Now h = k+ 1 or the average of h is ( k+ 1) / n, i= 1 ii where k is the number of predictor variables variables. Rule of thumb: Regard any hii > 2( k+ 1) / n as high leverage. In simple regression k = 1 and so h > 4 / n is regarded as high leverage. ei ei = = SE( e ) s 1 h i ii ii 6/28/2004 Unit 10 - Stat Ramón V. León 50 ii Standardized (Studentized) residuals will be large relative to the residual for high leverage observations.

51 Graph of Anscombe Data Observation in row 8 6/28/2004 Unit 10 - Stat Ramón V. León 51

52 Checking for Influential Observations Fit Model Platform: Anscombe Data Illustration: > 4 / n= 4 /11 = 0.36 Notice: Observation No. 8 is not an outlier since e8 = y ˆ 8 y8 = 0 6/28/2004 Unit 10 - Stat Ramón V. León 52 h ii Thus observation 8 is influential

53 How to Deal with Outliers and Influential Observations They can give a misleading picture of the relationship between y and x Are they erroneous observations? If yes, they should be discarded. If the observation are valid then they should be included in the analysis Least Squares is very sensitive to these observations. Do analysis with and without outlier or influential observation. Or use robust method: Minimize the sum of the absolute deviations n i= 1 y i ( β + β x ) 0 1 These estimates are analogous to the sample median. LS estimates are analogous to the sample mean. 6/28/2004 Unit 10 - Stat Ramón V. León 53 i

54 Data Transformations If the functional relationship between x and y is known it may be possible to find a linearizing transformation analytically: β For y = αx take log on both sides: log y = logα + βlog x β x For y = αe take log on both sides: log y = logα + βx If we are fitting an empirical model, then a linearizing transformation can be found by trial and error using the scatter plot as a guide. 6/28/2004 Unit 10 - Stat Ramón V. León 54

55 Scatter Plots and Linearizing Transformations 6/28/2004 Unit 10 - Stat Ramón V. León 55

56 Tire Tread Wear vs. Mileage: Exponential Model y = αe β x log y = logα β x 6/28/2004 Unit 10 - Stat Ramón V. León 56

57 Tire Tread Wear vs. Mileage: Exponential Model WEAR = e e = e MILES.0298MILES 6/28/2004 Unit 10 - Stat Ramón V. León 57

58 Tire Tread Wear vs. Mileage: Exponential Model For linear model: R 2 =.953 Residual plot is still curved mainly because observation 1 is an outlier 6/28/2004 Unit 10 - Stat Ramón V. León 58

59 Weights and Gas Mileages of Model Year Cars 6/28/2004 Unit 10 - Stat Ramón V. León 59

60 Weights and Gas Mileages of Model Year Cars Using Fit Y by X platform 100 Use transformation y. The new variable has units y of gallons per 100 miles. This transformation has the advantage of being physically interpretable. A spline fits the data locally. Our prior curve fits have been global. 6/28/2004 Unit 10 - Stat Ramón V. León 60

61 Weights and Gas Mileages of Model Year Cars 6/28/2004 Unit 10 - Stat Ramón V. León 61

62 Variance Stabilizing Transformation α Suppose σ µ We want to find a transformation of y that yields a constant variance. Suppose the transformation is a power of the original data, say y* = Y y λ For example: Take the square root of Poisson count data since for data so distributed the mean and the variance are the same 6/28/2004 Unit 10 - Stat Ramón V. León 62

63 Box-Cox Transformation: Empirical Determination of Best Transformation Fit Model Platform: y λ λ 1 ( y 1 ) y 0 λ λ λ = yln y λ = 0 ( ) where y = n n yi = i= 1 geometric mean 6/28/2004 Unit 10 - Stat Ramón V. León 63

64 Transformation in Tire Data A better fit than than the polynomial and with one less parameter (3 versus 4) Transform Mileage (in 1000 Miles 6/28/2004 Unit 10 - Stat Ramón V. León 64

65 Weights and Gas Mileages of Model Year Cars: Box-Cox Transformation Reciprocal transformation among suggested transformations 6/28/2004 Unit 10 - Stat Ramón V. León 65

66 Correlation Analysis When it is not clear which is the predictor variable and which is the response variable When both variables are random Try the bivariate normal distribution as a probability model for the joint distribution of two r.v. s Parameters: µ X, µ Y, σ X, σy and Cov (X, Y) ρ = Corr(X, Y) = Var(X)Var(Y) The correlation ρ is a measure of association between the two random variables X and Y. Zero correlation corresponds to no association. Correlation of 1 or -1 represents perfect association. 6/28/2004 Unit 10 - Stat Ramón V. León 66

67 Bivariate Normal Density Function 6/28/2004 Unit 10 - Stat Ramón V. León 67

68 Statistical Inference on the Correlation Coefficient R H = n ( X X)( Y Y) i i i= 1 n n 2 2 ( X i X) ( Yi Y) i= 1 i= 1 : ρ = 0 vs H : ρ Test statistic Approximate test statistic available to Test nonzero null hypotheses Obtain confidence intervals for ρ T = R n-2 1 R 6/28/2004 Unit 10 - Stat Ramón V. León 68 2 t n 2 Equivalent to testing H : β = 0 vs. H : β See textbook

69 Exercise /28/2004 Unit 10 - Stat Ramón V. León 69

70 Exercise /28/2004 Unit 10 - Stat Ramón V. León 70

71 Exercise By default, a 95% bivariate normal density ellipse is imposed on each scatterplot. If the variables are bivariate normally distributed, this ellipse encloses approximately 95% of the points. The correlation of the variables is seen by the collapsing of the ellipse along the diagonal axis. If the ellipse is fairly round and is not diagonally oriented, the variables are uncorrelated 6/28/2004 Unit 10 - Stat Ramón V. León 71