Lecture 18: Simple Linear Regression

Transcription

1 Lecture 18: Simple Linear Regression BIOS 553 Department of Biostatistics University of Michigan Fall 2004

2 The Correlation Coefficient: r The correlation coefficient (r) is a number that measures the strength and direction of a linear relationship between two quantitative variables The correlation coefficient is defined as r = 1 n 1 Properties of r ( ) ( ) x i x yi ȳ s x 1 r measures the strength of linear relationships 2 r is between -1 and 1 s y = 1 (zx ) (z y ) n 1 3 r is sensitive to outliers (influential observations) 4 r does not depend on units

3 Algebraically The model for simple linear regression is y i = β 0 + β 1 x i + ɛ i ɛ i N(0, σ) That is, the y value (y i ) for a given x value (x i ) is determined by the equation of a line (ŷ i = β 0 + β 1 x i ), and an error term (+ɛ i ). One way of thinking about the error term is that the line is predicting for each value of x the mean value of y. There is still variation among different individuals with the same value for x. The error term measures how far y i is from the the predicted mean according to the line. These errors are assumed to be normally distributed with a standard deviation σ that does not depend on x (or on anything else).

4 Regression Model Assumptions To summarize, the assumptions for the linear regression model are 1 The means µ x fall along a line. 2 The distribution of y values for all individuals with the same x value is normal. 3 The variation in the y values is the same for each value of x

5 Some Consequences of the Model Since the model is based on normal distributions and we don t know σ: Regression is going to be sensitive to outliers. Outliers with especially large or small values of the independent variable are especially influential. We can do inference based on the t-distributions We can check if the model is reasonable by looking at our residuals. residual = observed expected residual = vertical distance from observed (x i, y i ) to the regression line.

6 Example Suppose we want to predict SAT scores from ACT scores. We sample scores from a number of student who have taken both tests. Here is a scatter plot of their test scores: SAT score ACT score

7 Parameters and Statistics The simple regression model has 3 parameters: β 0 (intercept), β 1 (slope), and σ (standard deviation). Estimates based on data: b 0, b 1 and s. b 0 and b 1 come from the regression line: b 1 = slope = r sy s x b 0 = intercept. We can solve for b 0 using the fact that the point ( x, ȳ) is on the regression line. (residual) 2 s =, n 2 where residual = y i ŷ i = observed predicted In practice, the values of b 0, b 1, s and r (and others) are calculated by software.

8 Distributions We can do inference for β 0, β 1, etc. using the t distributions, we just need to know the corresponding SE and degrees of freedom. parameter SE df 1 β 0 SE b0 = s n + x 2 (xi x) 2 n 2 s β 1 SE b1 = n 2 (xi x) 2 1 ˆµ (prediction of mean) SEˆµ = s n + ŷ (individual prediction) SEŷ = s x x (xi x) 2 n n + x x (xi x) 2 n 2 We won t ever compute these SE s by hand, but notice that they are made up of pieces that look familiar (square roots, n in the denominator,

9 Locating Things in Computer Output SE b0 and SE b1 are easy to identify in the computer output: R Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) *** ACT e-15 *** The t value and P-value for the two-sided hypothesis tests: Inference for β 0. (H 0 : β 0 = 0) This is usually not interesting. Inference for β 1. (H 0 : β 1 = 0) This is much more interesting for two reasons. the slope is often a very interesting parameter to know this is a measure of how useful the model is for making predictions.

10 Confidence Intervals for the Parameters Let s compute a confidence interval for β 1. Minitab Predictor Coef SE Coef T P Constant ACT S = R-Sq = 66.7% R-Sq(adj) = 66.1% 95% CI for β 1 : 31 ± t SE = 31 ± (2.0017)(2.895) = 31 ±

11 Confidence Intervals and Prediction Intervals Recall that our goal was to make predictions of y from x. As you would probably expect, such predictions will also be described using confidence intervals. Actually there are two kinds of predictions: 1 Confidence intervals for the mean response (for all individuals with a certain level of the explanatory variable) 2 Prediction intervals are confidence intervals for a future observation.

12 Confidence and Prediction Intervals predicted SAT ACT score parameter SE df 1 ˆµ (prediction of mean) SEˆµ = s n + ŷ (individual prediction) SEŷ = s x x (xi x) 2 n n + x x (xi x) 2 n 2

13 Looking at Residuals: Histograms Histograms can be used to check overall normality. We are looking for a roughly bell-shaped histogram. Histogram of residuals (ACT vs SAT) Frequency residual It can be hard to judge normality by eye, so there is a better plot for this.

14 Looking at Residuals: Normal Quantile Plots A normal quantile plot is a better way to check for normality. Normal quantile plots are linear for normal distributions. Standardized residuals Normal Q Q plot Theoretical Quantiles lm(formula = SAT ~ ACT, data = sat)

15 How Normal Quantile Plots Work Scatter plot two values for each observation/subject: 1 the residual (or standardized residual) 2 the theoretical quantile use the rank of the residual to determine its percentile theoretical quantile = value in standard normal distribution with this same percentile example the 5th lowest of 60 residuals has percentile 5/60 = P(Z < 1.383) = , so the theoretical quantile for the 5th of 60 values is example (improved) the 5th lowest of 60 residuals has percentile 4.5/60 = P(Z < ) = 0.075, so the theoretical quantile for the 5th of 60 values is

16 Looking at Residuals: Residual Plots We can also look at scatter plots to see if the standard deviation appears to remain constant throughout. residuals vs x, or residuals vs. order, or residuals vs. fit (fit = value predicted by regression line ) If the model is a good match, we should NOT see any clear pattern in these plots. A pattern would indicate something other than randomness is influencing the residuals. Residuals vs Fitted Residuals Fitted values lm(formula = SAT ~ ACT, data = sat)