Linear Regression is a very popular method in science and engineering. It lets you establish relationships between two or more numerical variables.

Transcription

1 Lab 13. Linear Regression Note: the things you will read or type on the computer are in the Typewriter Font. All the files mentioned can be found at Linear Regression is a very popular method in science and engineering. It lets you establish relationships between two or more numerical variables. 1 Simple Linear Regression The goal of regression is to establish a relationship between predictor (or independent variable) X and response (or dependent variable) Y. Simple Linear Regression tries to determine the best fit for the equation Y i = β 0 + β 1 X i + ε i, i = 1, 2,..., n in the sense of minimizing the sum of squared residuals 1 e i = Y i ( ˆβ 0 + ˆβ 1 X i ) = Observations - Fits As an example, consider predicting the house Price Y (in $1,000) as a function of its Area X (in square feet). The data are in the file house.csv. We will use the function lm as follows: House = read.csv("house.csv") lm1 = lm(price ~ Area, data = House) summary(lm1) plot(house$area, House$Price) abline(lm1) # put the line on the plot The summary of the fit is given by R in the following format Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) Area e-15 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 61 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 61 DF, p-value: 6.57e-15 1 Thus, this analysis is usually called Least Squares regression. 1

2 We see that ˆβ 0 = 3.72 and ˆβ 1 = are the estimates of the intercept and the slope, respectively. For example, the house is supposed to be worth about $55 more for each extra square foot of area. Some other important information about our model is given in the output: The Residual standard error (which will be called S in class) estimates the standard deviation of residuals, that is, the typical difference between actual price and the predicted price. So, the predicted price will typically be off by about $21,000. The models with smaller S are better fit. Coefficient of determination R 2 is called here Multiple R-squared and equals Higher values of R 2 indicate better model fit. For simple linear regression, R 2 also equals the square of the correlation coefficient between X and Y. F-statistic and the corresponding p-value tell us whether the entire model is statistically significant. For simple linear regression, this is the same as testing H 0 : β 1 = 0. Small p-value means that the slope is statistically different from 0. Here, p-value of 6.57e-15 is very small! t value and Pr(> t ) show the result of t-test for H 0 : β 1 = 0 and its p-value. For simple linear regression, this is the same as the F-test discussed previously. In fact, F-statistic = = T 2 = Also, you see the standard error of the slope estimate (Std.Error) reported. This quantity tells you how uncertain the slope estimate of is. For example, 95% C.I. for the slope will be Estimate ± t α/2 Std.Error. Here, the value of t α/2 will be discussed in class, but it s close to 2. Problem 1 We will analyze the Global Warming data found at GlobAnom.csv (these are annual anomalies expressed in degrees Celsius, relative to the mean). See also globe/land_ocean/ytd (a) Run linear regression and produce the plot and the summary table. Is the slope term significantly positive? By how much, in your estimation, do the global temperatures have risen per decade, on average? (b) Calculate an approximate 95% CI for the slope. (c) Do you notice any deviations from a linear pattern in your data? 2

3 (d) Give an estimate of the residual standard deviation, S. (e) Let s try and predict the average global temperature anomaly for First, do it by hand using the regression equation obtained in (a). Then, obtain a 95% prediction interval as follows: newdata = data.frame(year = 2020) predict(lm1, newdata, interval = "prediction") In light of (c), do you expect your prediction to be biased? Is it, in your opinion, too high or too low? 1.1 Transformations One possible way to handle non-linear relationships is to use a transformation. For example, to implement the model Y = a 0 e a1x we might log both sides 2 and obtain log(y ) = log(a 0 ) + a 1 X. This means you should use the linear regression of log(y ) on X. To implement a power model Y = a 0 X a 2 you might want to log-transform both X and Y. Problem 2 Open the file mammals.csv. Analyze how the average life span of a mammal species depends on its average weight. Predict the average life span of humans (average weight = 150 lbs). 2 Multiple Regression For multiple regression, we want to use more than one predictor. A typical multiple regression equation is Y i = β 0 + β 1 X (1) i + β 2 X (2) i β p X (p) i + ε i For our first dataset, we may hope to predict the house price better if we use more information. In particular, our predictors are going to be X 1 = Area (sq. ft.), X 2 = number of rooms, X 3 = number of bedrooms, X 4 = number of baths and X 5 = age. lm2 = lm(price ~., data = House) summary(lm2) # the dot means "use *all* the variables" 2 Always a natural log! 3

4 and here are the results Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) Area e-05 *** Rooms * Bedrooms age ** bath Signif. codes: 0 *** ** 0.01 * Residual standard error: on 57 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: 31.1 on 5 and 57 DF, p-value: 4.027e-15 The overall regression model has improved somewhat (for example, S = is lower now, and R 2 = is higher). However, some terms, like Bedrooms and bath are not significant, judging by their p-values in the last column. R 2 -adjusted is better than R 2 because it subtracts a penalty for adding more predictors, i.e. it s biased towards using simpler models. Thus, higher R 2 - adjusted values typically indicate a better model. We cannot make a plot of all predictors simultaneously vs. Y. Another plot, called Residuals vs Fits plot is used for diagnostics. plot(lm2$fit, lm2$res) A good Residuals vs Fits plot should lack any apparent structure. Here, it has a funnel shape widening from left to right. This might indicate that a log transformation of Y is needed. Also, you can see unusual observations or outliers. You can probably spot two outliers here: one is on top and another in the lower-right of the plot. If we now use the log-transformed Y and also eliminate the non-significant predictors with the following lm3 = lm(log(price) ~ Area + Rooms + age, data = House) we can see that R 2 -adjusted has improved. 4

5 Let s revisit the example of mercury in fish. We ve been trying to link the mercury content in fish to some environmental variables. The data are in the file MercBass.csv. Problem 3 (a) Run the multiple regression of Avg.Merc variable vs. the rest of the variables. Which variables are deemed to be significant according to the t-test output? (b) Look at the plot of residuals vs fits. If has a typical funnel shape indicating that a log transformation of Y might be useful. (c) Re-run the analysis with log(y ) as the response variable. Which variables are deemed significant now? Compare the R 2 -adjusted value with the one from part (a). (d) Predict the average mercury content in a lake with Alkalinity = 30 ph = 6.5 Calcium = 15 Chlorophyll = 9 5