28. SIMPLE LINEAR REGRESSION III

Transcription

1 28. SIMPLE LINEAR REGRESSION III Fitted Values and Residuals To each observed x i, there corresponds a y-value on the fitted line, y = βˆ + βˆ x. The are called fitted values. ŷ i They are the values of y which would be predicted by the estimated linear regression model, at the observed values of x. Eg: For the TV example, the fitted People Meter readings are where x is the Nielsen Index. ˆi 0 1 i y ˆ = x,

2 Regression Analysis The regression equation is People Meter = Nielsen Rating Predictor Coef StDev T P Constant Nielsen S = R-Sq = 94.9% R-Sq(adj) = 94.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total

3 Nielsen Index People Meter The Cosby Show Family Ties Cheers Moonlighting Night Court Growing Pains Who's The Boss Family Ties (Special) Murder She Wrote Minutes The Family Ties Special had a Nielsen Index of Therefore, the corresponding fitted value is yˆ = ( ) + (1.0261)(17.5) = Note that this "prediction" is larger than the actual People Meter reading for this show, y = 13.3.

4 In general, the observed data values y i will be different from the fitted values ŷ i. The differences between the data and fitted values are called the residuals, e = y y. i Eg: In the TV example, the residual corresponding to the Family Ties Special is = We can think of the residuals e i as estimators of the underlying errors ε i. A given residual will be positive if the data point lies above the fitted line, and negative if the point is below the fitted line. Because of the way the least squares estimators are determined, the residuals will always sum to zero. i ˆi

5 The residuals are very important, since they tell us how well the least squares line fits the data. To assess the quality of the fit, we can plot the residuals against the x i 's, or against the fitted values. If the model is adequate, then these plots should show no systematic patterns or structure. If they do show structure, then the model may need to be modified.

6 Eg: For the TV example, the plot of residuals versus the Nielsen Index shows little, if any, systematic structure. Instead, the residuals seem randomly scattered around zero, without any severe outliers. (Even the most extreme residual, 1.94 for the Family Ties Special, does not seem completely out of line with the other residuals.) Further, the variability seems not to change radically with x. So far, then, our data has not strongly contradicted any of the assumptions underlying the linear regression model. Residual plots should be a routine part of any regression analysis. They often reveal unexpected features in the data, such as nonlinear relationships, non-constant variance, and outliers, which might never have been discovered otherwise.

7 Regression Plot Y = X R-Sq = 94.9 % 25 People Meter Nielsen Ratings

8 Residuals Versus the Fitted Values (response is People Meter) 1 Standardized Residual Fitted Value

9 A numerical measure of the closeness of the least squares line to the data is provided by the sum of squared residuals, i= 1 If all the data points lie exactly on a line, then SSE will be zero. (Why?) SSE stands for "Sum of Squares for Error". = n But SSE is not equal to the sum of squares of the actual errors, n 2 ε. (Why?) i= 1 i SSE Nevertheless, we can use SSE to estimate the variance σ 2 of the actual errors ε. e 2 i = n i= 1 ( y i yˆ i ) 2

10 An unbiased estimator of σ 2 is given by 2 SSE s = = Resid. Mean Square. (1) n 2 Since we are estimating two parameters, we have n 2 "degrees of freedom". We can think of s 2 as an "average squared prediction error", since SSE is the sum of the squared deviations between the actual and predicted values ( y and yˆ). You might be tempted to use s 2 or SSE to compare different models fitted to a given set of y's. This is actually a very bad way to pick a model! You need to first adjust for the fact that different models may have different numbers of explanatory variables. (We will describe how to do this later if time permits).

11 Eg: For the TV data, s 2 = Thus, we can estimate the standard deviation of the People Meter rating for a given Nielsen Index value to be s = 1.07 Rating Points. This tells us about the natural variability which would be expected in the People Meter ratings, even if the Nielsen Index were held fixed. Regression Analysis The regression equation is People Meter = Nielsen Rating Predictor Coef StDev T P Constant Nielsen S = R-Sq = 94.9% R-Sq(adj) = 94.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total

12 Don't confuse s 2 with the sample variance of a batch of numbers, 1 n 2 ( y i y). (2) n 1i= 1 We also used the name s 2 for (2), but it's not the same as (1). There is a connection, though, since we can think of (2) as an "average squared prediction error" based on a simpler predictor of y: Just use y to predict all the y's, and forget about the x's.

13 Testing Whether β 1 is Zero: Is the Straight-Line Model Useful for Predicting y from x? A basic principle of model building is that we should always use the simplest model which adequately describes the data. Recall the Forecasting Lab, and the Taste Test. Why do we think that the Earth goes around the Sun? What is the best predictor for the number of homeruns that Sammy Sosa will hit next year?

14 Accordingly, unless there is strong evidence of a linear relationship between x and y, it is better to adhere to the null hypothesis of no linear relationship, H 0 : β 1 = 0. This produces a much simpler model, y i = β 0 + ε i, in other words, the y's are a batch of data with a normal distribution (mean β 0, variance σ 2 ). In fact, this is the same model used earlier in the course to make inferences about the population mean µ based on the sample mean and standard deviation. So if β 1 = 0, the best predictor of a future y value is just the average of the observed y values. It is only if β 1 0 that the values of the "explanatory variable" x can improve our ability to predict a future y. If β 1 = 0, the use of x will usually degrade the quality of the forecast!

15 Of course, we can use a scatterplot of y versus x as well as a residual plot to help us to decide whether a linear relationship exists, but it would be nice to have a more objective tool. Why can't we just check whether βˆ 1 = 0? Because even if β 1 = 0, the estimator βˆ is a random variable, and 1 will not in general be exactly zero. So βˆ has a sampling distribution. This distribution is normal, 1 with mean β 1. The standard error of βˆ can be estimated from the 1 data. Sincich calls this. s βˆ1 Minitab calls it "StDev", but it's really an estimated standard error!

16 For the Stock Market data, s βˆ1 is Regression Analysis The regression equation is TODAY = YESTERDAY 8430 cases used 2 cases contain missing values Predictor Coef StDev T P Constant YESTERDA S = R-Sq = 1.5% R-Sq(adj) = 1.5% Analysis of Variance Source DF SS MS F P Regression Residual Error Total

17 Linear Regression of Today vs Yesterday Y = 2.81E X R-Sq = 1.5 % TODAY YESTERDAY

18 Now, we can test H 0 : β 1 = 0 versus H 1 : β 1 0 by converting into its own t-score, βˆ 0 t =. 1 s βˆ 1 βˆ1 The result is given by Minitab as "T". For the Stock Market data, t = = If β 1 = 0, then t will have a t distribution with n 2 degrees of freedom. The p-value corresponding to a 2-tailed test is given by Minitab as "P".

19 For the Stock Market data, we get p = Conclusion: Strong evidence of a linear relationship between today's return and yesterday's return! Eg: For the TV example, the t-value for β 1 is 12.20, yielding p = (to four significant figures). There is very strong evidence of a linear relationship between People Meter readings and Nielsen Ratings. This does not mean that the relationship is purely a linear one (there might be nonlinear factors at work here as well). Recall, though, that the residual plot (shown earlier) did not show any patterns, such as a parabola, or a wedge. (A parabola would indicate the presence of a nonlinear association, while a wedge would indicate that the variance of the error term depends on x). At this point, then, we can be reasonably comfortable with using the simple linear regression model for the TV data.

20 Regression Analysis The regression equation is People Meter = Nielsen Rating Predictor Coef StDev T P Constant Nielsen S = R-Sq = 94.9% R-Sq(adj) = 94.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total

21 Making Inferences About the Slope β 1 For any value of the true slope β 1, the quantity βˆ1 β 1 s has a t-distribution with n 2 degrees of freedom. This allows us to construct a confidence interval or a test for β 1. A (1 α)100% confidence interval for β 1 is given by βˆ ± t, where t α/2 is based on n 2 degrees of freedom. βˆ1 1 α 2 sβˆ 1

22 For the TV example, a 95% confidence interval for β 1 is given by where from Table 6, t = (df=8). From the Minitab output, we obtain the interval ± (2.306) (0.0841), which reduces to (0.832, 1.22). Note that this interval contains 1.0, so it is plausible that β 1 = 1. (What would be the practical importance of this?) βˆ 1 ± t0.025 sβˆ 1 To test H 0 : β 1 = 1 versus H 1 : β 1 1 at level α = 0.05, we can use the test statistic βˆ t = = = 0.31, s βˆ1 which does not come close to the critical value of 2.306, so we do not reject H 0.,

23 Clearly, the p-value is very large, but we cannot compute it without additional tables. (Actually, there is a way to get tail areas for a t-distribution in Minitab, but we won't bother here). In any case, there is little or no statistical evidence to suggest that β 1 is different from 1. In this example, it was more reasonable to test if β 1 = 1 than to test if β 1 = 0. Keep in mind, though, that Minitab bases its "T" and "P" on a null hypothesis that the true parameter values are zero. If you are interested in a different null, you must compute the test statistic manually, as above.

24 One more note: We can get confidence intervals and hypothesis test for the intercept β 0 in a similar way. Use the standard errors for βˆ, as given by Minitab. 0

25 Eg: Monthly salary for the first post-mba job vs. heights of 1986 male MBA graduates of the University of Pittsburgh (from the Wall Street Journal). Data File: MBA.MTP Salary vs. Height Salary Height

26 Regression Analysis The regression equation is Salary = Height Predictor Coef StDev T P Constant Height S = R-Sq = 71.4% R-Sq(adj) = 70.3% Analysis of Variance Source DF SS MS F P Regression Residual Error Total

27 Residuals Versus the Fitted Values (response is Salary) Residual Fitted Value

28 Eg: Batting Averages of 1992 Major League Baseball Players compared to their Salaries. Data File: MLB92.MTP 1992 Batting Averages vs. Salaries BA Salary(1000s)

29 Regression Analysis The regression equation is BA = Salary(1000s) Predictor Coef StDev T P Constant Salary( S = R-Sq = 7.6% R-Sq(adj) = 7.4% Analysis of Variance Source DF SS MS F P Regression Residual Error Total

30 Residuals Versus the Fitted Values (response is BA) Residual Fitted Value