22s:152 Applied Linear Regression. Chapter 5: Ordinary Least Squares Regression. Part 2: Multiple Linear Regression Introduction

Transcription

1 22s:152 Applied Linear Regression Chapter 5: Ordinary Least Squares Regression Part 2: Multiple Linear Regression Introduction Basic idea: we have more than one covariate or predictor for modeling a dependent variable Examples: predict salary from years of experience AND years in school predict force at which a metal alloy rod bends based on iron content AND temperature of rod 1

2 Multiple Linear Regression Example: Predicting Highway MPG in cars based on car specifications Data set: 93 observations, 26 variables > names(car.data) [1] "Manuf" "Model" "Type" [4] "MinPrice" "MidRangePrice" "MaxPrice" [7] "CityMPG" "HwyMPG" "AirBags" [10] "DriveTrain" "Cyl" "EngSize" [13] "HPW" "RPM" "Rev" [16] "ManTran" "GasTank" "PassCap" [19] "Length" "WheelBase" "Width" [22] "U-turn" "RearSeat" "Luggage" [25] "Weight" "Domest" Consumer Reports: The 1993 Cars - Annual Auto Issue (April 1993), Yonkers, NY: Consumers Union When predicting Highway MPG, there are perhaps many informative predictors... 2

3 ... we ll start by looking at two possible predictors, such as the vehicle Length and Width. > plot(car.data[,c(8,19,21)]) HwyMPG Length Width Length and Width are both correlated with MPG (making them potentially good predictors), and it looks like these potential predictors of Length and Width are also correlated with each other. 3

4 Multiple Linear Regression A multiple linear regression model with two independent variables: Y i = β 0 + β 1 x 1i + β 2 x 2i + ɛ i Y i is the response or dependent variable for observation i x 1i is the observed predictor, explanatory variable, independent variable, covariate for variable 1 observation i x 2i is the observed predictor, explanatory variable, independent variable, covariate for variable 2 observation i ɛ i is the error term with ɛ i iid N(0, σ 2 ) So, E[Y i x 1i, x 2i ] = β 0 + β 1 x 1i + β 2 x 2i 4

5 In this model, we assume Y and X 1 are linearly related, and Y and X 2 are linearly related Notation in the book is for the fitted model: Ŷ = A + B 1 x 1 + B 2 x 2 with A as the estimated intercept with B 1 as the estimated regression coefficient (slope) for variable 1 with B 2 as the estimated regression coefficient (slope) for variable 2 5

6 Multiple Linear Regression Example: Return to MPG example After predicting MPG from Width, what % of the variability in MPG is left to be explained? > attach(car.data) > lm.out=lm(hwympg ~ Width) > summary(lm.out)$r.squared [1] Does including Length as a predictor improve the overall % explained? > lm.out.2=lm(hwympg ~ Width + Length) > summary(lm.out.2)$r.squared [1] It explains only a little bit more than Width alone. 6

7 From the scatterplot, it looked like Length and MPG (covariate and response) were negatively correlated. But, Length was also correlated with Width (the other predictor variable in the model). > cor(hwympg,length) [1] > cor(width,length) [1] So, there is a lot of redundant information in Length and Width when trying to predict MPG. We gained little by including Length after Width was already in the model. A simple linear regression model including only MPG and Length provides an R 2 of 0.30 (a fair predictor), but this is not as high as when Width as the only predictor. > lm.out.length=lm(hwympg ~ Length) > summary(lm.out.length)$r.squared [1]

8 Multiple Linear Regression For other variables or data sets, it may be the case where we may gain a lot from the inclusion of more predictors. - Example: Price of Antique Clocks at Auction The data gives the selling price at auction of 32 antique grandfather clocks. Also recorded is the age of the clock and the number of people who made a bid. VARIABLES Age - Age of the clock (years) Bidders - Number of individuals participating in the bidding Price - Selling price (pounds Stirling) We re interested in modeling the Price (dependent variable) based on Age and Bidders. 8

9 Get the correlation of the variables, and look at the scatter plots of the bivariate relationships: > clock.data=read.delim("antique_clocks.txt", sep="\t",header=true) > head(clock.data) Age Bidders Price > cor(clock.data) Age Bidders Price Age Bidders Price > plot(clock.data,pch=16) 9

10 Age Bidders Price Price is linearly related to Age and Bidder Age and Bidder have little correlation. The # of bidders on young clocks is about the same as the # of bidders on old clocks. 10

11 Let s fit a SLR with Age as predictor: > lm.out=lm(price ~ Age, data=clock.data) > summary(lm.out).. Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) Age e-06 *** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: 273 on 30 degrees of freedom Multiple R-Squared: ,Adjusted R-squared: F-statistic: on 1 and 30 DF, p-value: 2.096e % of the total variability in Price is explained by the Age of the clock. 11

12 Let s fit a multiple linear regression with Age and Bidders as predictors: > lm.out.2=lm(price ~ Age + Bidders, data=clock.data) > summary(lm.out.2)... Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-08 *** Age e-14 *** Bidders e-11 *** --- Multiple R-Squared: ,Adjusted R-squared: % of the total variability in Price is explained by the Age of the clock and the number of Bidders. It was very useful to include both Age and number of Bidders in modeling the Price. 12

13 What was left unexplained in Price after Age was included in the model? How much of this unexplained variability can Bidders explain if it is also included in the model? We can get a feel for this by using a Partial Regression Plot (a.k.a. added-variable plot). Partial Regression Plot A plot that shows a certain type of residual on the Y -axis AND a certain type of residual on the X-axis. In the two predictor model, we have Y, x 1 and x 2. The partial regression plot for x 2 gives us information on including x 2 in the model AFTER x 1 has already been included, and is created based on the residuals from these two R models: 1) lm(y x 1 ) 2) lm(x 2 x 1 ) 13

14 The partial regression plot (or AV-plot) is VERY informative in the multiple linear regression framework. Example: Antique clocks Let x 1 Age and x 2 Bidders. 1) lm(price Age) 2) lm(bidders Age) Plot the residuals from the regression of Price on Age (this represents the variability in Price still left unexplained after accounting for Age) against the residuals from the regression of Bidders on Age (this represents the nonredundant part of Bidders and Age, unique information in Bidders). ## SLR Y on x1: > lm.out=lm(price ~ Age, data=clock.data) ## Regressing Bidders on Age (the two predictors): > lm.bid.age=lm(bidders ~ Age, data=clock.data) ## Plot the residuals against each other: > plot(lm.bid.age$residuals,lm.out$residuals,pch=16) 14

15 Partial Regression Plot Vertical axis: The residuals from the regression of Price on Age are on the Y axis. Horizontal axis: The residuals from the regression of Bidders on Age are on the X axis. lm.out$residuals lm.bid.age$residuals If I see a linear relationship here, it s worth it to include Bidders as a second predictor. 15

16 How much of the variability left unexplained by Age can be explained by adding Bidders to the model? Fit a simple linear regression to the plot on the previous page. > lm.part.reg=lm(lm.out$residuals ~ lm.bid.age$residuals) > summary(lm.part.reg). Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) 1.347e e e-16 1 lm.bid.age$residuals 8.582e e e Multiple R-Squared: The earlier SLR of Price on Age explained 53.32% of the variability in Price. By adding Bidders to the model, we explain 77.01% of the variability that was left unexplained (this is the NOT the R 2 for the full model with both predictors). 16

17 Total variability explained by the full model with both Age and Bidders is: *( )= or 89.27% as a percentage. From the multiple linear regression model... > lm.out.2=lm(price ~ Age + Bidders, data=clock.data) > summary(lm.out.2)... Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-08 *** Age e-14 *** Bidders e-11 *** --- Multiple R-Squared: ,Adjusted R-squared: The R 2 for the full model is

18 The fitted line in the Partial Regression Plot also gives us the regression coefficient for Bidders in the multiple linear regression model: 18

19 From the Partial Regression Plot fit: > lm.part.reg=lm(lm.out$residuals ~ lm.bid.age$residuals) > summary(lm.part.reg). Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) 1.347e e e-16 1 lm.bid.age$residuals 8.582e e e-11 From the multiple linear regression fit: > lm.out.2=lm(price ~ Age + Bidders, data=clock.data) > summary(lm.out.2). Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-08 *** Age e-14 *** Bidders e-11 *** 19

20 This is why the regression coefficient β k in multiple regression is often referred to as a partial regression coefficient. A Partial Regression Plot is also called an Added-Variable Plot Multiple linear regression brings predictors together to predict a response When predictors are correlated (or share information about the response), things can get a little complicated The Partial Regression Plot helps us understand what s left to be explained by a new added predictor We ask, given all other variables in the model, is this still a useful predictor? 20

21 Z Interpretation of regression coefficients The interpretation of a regression coefficient in a multiple linear regression, is stated in terms of holding all other covariates constant. For a model with two covariates X 1 and X 2, the mean response surface is a plane. y x1 In SLR, the mean is a line. 21

22 Example: Return to clock example > lm.out.2=lm(price ~ Age + Bidders, data=clock.data) > summary(lm.out.2). Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-08 *** Age e-14 *** Bidders e-11 *** For clocks with a given number of bidders, a 1 year increase in Age is associated with an increase in pounds in the mean Price. For clocks of a given age, a 1 bidder increase in the number of bidders on the clock is associated with an increase in pounds in the mean Price. 22

23 Order of predictors in model The order doesn t matter because we estimate the regression coefficient after including all other variables The summary table would have looked the same if we had fit: > lm.out.2=lm(price ~ Bidders + Age,data=clock.data) 23

24 Multiple Linear Regression Lease squares estimates We use the same criteria for multiple regression as we did for simple linear regression for choosing the best estimates of the coefficients β 0, β 1, and β 2. We minimize the sum of the squared residuals ni=1 (Y i Ŷi) 2 In other words we find b 0, b 1, and b 2 such that ni=1 (Y i (b 0 + b 1 x 1i + b 2 x 2i )) 2 is minimized. With 2 covariates, the residuals are the distance from the observed data value to the fitted plane. 24

25 Multiple Linear Regression Lease squares estimates With k covariates, we just extend this to minimize ni=1 (Y i (b 0 + b 1 x 1i b k x ki )) 2 To get the parameter estimates, we take the derivative with respect to each regression coefficient, set the equations equal to 0, and solve. 25

26 Multiple Linear Regression Estimate for σ 2 σ 2 is estimated as before with the residuals, but we now divide the RSS by (n-(k+1))=(n-k-1) since there are k+1 parameters estimated. ˆσ 2 = S 2 E = n k 1 RSS ni=1 = (Y i Ŷi) 2 n k 1 SSE = RSS E[ RSS n k 1 ] = σ2 S E is the standard error for the regression ˆσ = S E = S 2 E Another term often used, is MSE where MSE = SSE n k 1 = RSS n k 1 26