Unit 11: Multiple Linear Regression

Transcription

1 Unit 11: Multiple Linear Regression Statistics 571: Statistical Methods Ramón V. León 7/13/2004 Unit 11 - Stat Ramón V. León 1 Main Application of Multiple Regression Isolating the effect of a variable by adjusting for the effect of other variables Regressing health index versus wealth gives a negative slope for wealth effect The higher the wealth the lower the health What one does when one fits a simple linear regression model Regressing health index versus wealth while adjusting for age gives a positive slope for wealth The higher the wealth the higher the health when one adjusts for the effect of the confounder age Multiple linear regression models allow us to determine the effect of regressors while adjusting For confounding For effect modification 7/13/2004 Unit 11 - Stat Ramón V. León 2

2 Why Do Regression Coefficients Have the Wrong Sign? yˆ = x yˆ = x x /13/2004 Unit 11 - Stat Ramón V. León 3 Multiple Linear Regression y = β + β x + β x + + β x + ε Model: k k Model is called linear because it is linear in the β s not necessarily in the x s. So y = + x + x + + x + 2 k β0 β1 β2... βk ε is considered a linear model and so is y = β + β x + β x + β x x + β x + β x + ε /13/2004 Unit 11 - Stat Ramón V. León 4

3 A Probabilistic Model for Multiple Linear Regression Yi = β0 + β1xi 1+ β2xi βkxik + εi, i= 1,2,..., n where εi is a random error with E( εi) = 0, and β0, β1,..., βk are unknown parameters. Usually we assume that the εi are 2 independent N(0, σ ) r.v.'s. 2 This implies that Yi's are independent N( µ i, σ ) r.v.'s with µ = EY ( ) = β + β x + β x β x i i 0 1 i1 2 i2 k ik 7/13/2004 Unit 11 - Stat Ramón V. León 5 Tire Wear Example xi1 x i 2 k=2 i y n=9 i = 1,2,..., n= 9 7/13/2004 Unit 11 - Stat Ramón V. León 6

4 Minimize: Least Squares Fit n i 0 1 i1 2 i2 k ik i= 1 ( ) 2 β β β... β Q= y + x + x + + x By solving the following equations: Q Q Q = 0, = 0,..., = 0 β β β 0 1 Solution are Least Square regression estimates: ˆ β, ˆ β,..., ˆ β 0 1 k k 7/13/2004 Unit 11 - Stat Ramón V. León 7 Sums of Squares yˆ = ˆ β + ˆ β x ˆ β x i= 1,2,..., n i 0 0 i1 k ik n n 2 2 e ( ˆ i = yi yi) i= 1 i= 1 n 2 ( yi y) i= 1 n 2 ( yˆ i y) i= 1 n ( ) 2 n ( ) 2 n y ˆ ( ˆ ) 2 i y = yi y + yi yi i= 1 i= 1 i= 1 Error Sum of Squares (SSE) = Total Sum of Squares (SST) = Regression Sum of Squares (SSR) = SST = SSR + SSE As also obtained for simple linear regression 7/13/2004 Unit 11 - Stat Ramón V. León 8

5 Coefficient of Multiple Determination r 2 SSR SSE = = 1 SST SST Percentage of the total variation in Y that is explained by the model r = r 2, is called the multiple correlation coefficient. Remark: R 2 square always increases with the number of variables included in the model, but this does not mean that more variables always leads to a better model. 7/13/2004 Unit 11 - Stat Ramón V. León 9 The Tire Wear Example: Quadratic Fit Using Fit Y by X platform: 7/13/2004 Unit 11 - Stat Ramón V. León 10

6 Tire Wear Example: JMP Output We used the Fit Y by X platform of JMP Reduces multicollinearty and differences in order of magnitudes among different powers of x Notice that the polynomial is centered at the average of Mileage = 16 7/13/2004 Unit 11 - Stat Ramón V. León 11 Tire Wear Example: JMP In this method of fitting, using the Fit Model platform, JMP does not know that Mileage^2 is the square of Mileage. The Mileage^2 column was created using JMP s calculator 7/13/2004 Unit 11 - Stat Ramón V. León 12

7 Tire Wear Example: JMP Output Polynomial is not centered here. Why? 7/13/2004 Unit 11 - Stat Ramón V. León 13 JMP Fit Model Platform Square term is added by highlighting Mileage in both Select Columns and Construct Model Effects and then clicking the Cross button Polynomials are centered by default 7/13/2004 Unit 11 - Stat Ramón V. León 14

8 Fit Model Platform Output Now polynomial is centered again 7/13/2004 Unit 11 - Stat Ramón V. León 15 Statistical Inference on Individual β s ˆ β j β j Tn ( k+ 1) ( j = 0,1,..., k) SE( ˆ β ) j 100(1-α)% CI for β : ˆ β ± t j n ( k+ 1), α 2 j j SE( ˆ β ) H : β = 0 versus H : β 0 oj j 1 j j ˆ β j Reject if t = > t SE ( ˆ β j ) j n ( k+ 1), α 2 Recall that the CI are obtained by right clicking on the parameter estimate area and from the popup menu selecting Columns and then Upper 95% and Lower 95% 7/13/2004 Unit 11 - Stat Ramón V. León 16

9 H Statistical Inference on Simultaneous β s : β = β =... = β = 0 vs. H : At least one β k 1 j MSR Reject if F = > fkn k MSE, ( + 1), α ANOVA table for quadratic fit of tire wear data. 7/13/2004 Unit 11 - Stat Ramón V. León 17 Compares the full model: Other Test of Hypothesis H0 : β = k m 1... = + β = k 0 Y = β + β x β x + ε ( i= 1,2,..., n) i 0 1 i1 k ik i to the partial model: Y = β + β x β x + ε ( i = 1, 2,..., n) i 0 1 i1 k m i, k m i Does adding additional predictor variables to the model lead to a significant reduction in the Error Sums of Squares (SSE)? 7/13/2004 Unit 11 - Stat Ramón V. León 18

10 Exercise 11.3: Type I and III Sums of Squares 7/13/2004 Unit 11 - Stat Ramón V. León 19 Type III Sums of Squares Does adding a variable to the model last lead to a significant decrease in the error sum of squares? Adding weight to the model - once brain size and height are in the model - does not help predict IQ Part of JMP s standard output for the Fit Model platform 7/13/2004 Unit 11 - Stat Ramón V. León 20

11 Type I or Sequential Sum of Squares Decrease in the error sums of squares when variables are added one at the time. Here order matters. Conclusion: Weight does not provide additional information about IQ once brain size and height are in the model 7/13/2004 Unit 11 - Stat Ramón V. León 21 Sequential SS Add to the Model SS (SSR) SSR = SSR = SST = SSR = Type III SS do not add up to the Model SS (SSR) 7/13/2004 Unit 11 - Stat Ramón V. León 22

12 Type I or Sequential Sum of Squares Looks a how much of SST is moved from the SSE into the SSR as you add additional regressors SST remains constant no matter how many regressors are in the model Recall that SST = SSR + SSE 7/13/2004 Unit 11 - Stat Ramón V. León 23 Variable Selection Methods In practice, a common problem is that there is a large set of candidate predictor variables. One wants to choose a small subset so that the resulting regression model is simple and has good predictive ability Stepwise regression Best subsets regression Optimality criteria Take Stat 572: Applied Linear Models to learn more Reading assignment: Section 11.8 A Strategy for Building a Multiple Regression Model. This strategy will help you with your project. 7/13/2004 Unit 11 - Stat Ramón V. León 24

13 Stepwise Regression in JMP 7/13/2004 Unit 11 - Stat Ramón V. León 25 Stepwise Regression in JMP Stepwise regression drops Weight from the model Sequential Sums of Square 7/13/2004 Unit 11 - Stat Ramón V. León 26

14 Reduced Model Compared with Full Model Brain size, height, and weight in the models Only brain size and height in the model Adjusted R 2 improves when Weight is dropped from the model since it has a penalty for number of variables in the model R 2 always increases with number of variables added to the model, but in this case the increase is negligible. This confirms that weight does not add much additional information to the model once brain weight and height are in the model. 7/13/2004 Unit 11 - Stat Ramón V. León 27 PRESS Statistic 7/13/2004 Unit 11 - Stat Ramón V. León 28

15 PRESS in JMP 7/13/2004 Unit 11 - Stat Ramón V. León 29 Cross-Validation Divide the data into training and test sets using random mechanism. Fit the models with training set data. Predict the test set data. Choose model that predicts best. 7/13/2004 Unit 11 - Stat Ramón V. León 30

16 Analysis Methods that Are Similar to Those in Simple Linear Regression Confidence intervals for the average response of Y at a particular value of the regressors Prediction intervals for a future value of Y for a particular value of the regressors Plot of residuals against individual predictor variables included or not in the model against predicted value on normal plot in time order if data is time ordered (Durbin-Watson test) Standardized residuals ( > 2 indicates an outlier) Influential observations if 2( k + 1) hii > n 7/13/2004 Unit 11 - Stat Ramón V. León 31 Transformations For the predictor variables as well as the response to make the model linear A model that can be transformed so that it becomes linear in in its unknown parameters is called intrinsically linear otherwise it is called a nonlinear model Unless a common transformation of y linearizes its relationship with all x s, effort should be focused on transforming the x s, leaving y untransformed, at least initially. It is usually assumed that the random error enters additively into the final form of the model 7/13/2004 Unit 11 - Stat Ramón V. León 32

17 Illustration of Multicollinearity: Cement Data Note that x1, x2, x3, and x4 add up to approximately 100% for all observations. 7/13/2004 Unit 11 - Stat Ramón V. León 33 Correlation Matrix: Cement Data Note that the x 1, x 2, x 3, and x 4 add up to approximately 100% for all observations. This approximate linear relationship among the x s results in multicollinearity. Multicollinearity means that the predictor variable are linearly related. Correlations X1 x2 x3 X4 X1 x2 x3 X /13/2004 Unit 11 - Stat Ramón V. León 34

18 Scatterplot Matrix Scatter Plot Matrix of Cement Data X1 x2 x X /13/2004 Unit 11 - Stat Ramón V. León 35 Multicolliniarity 7/13/2004 Unit 11 - Stat Ramón V. León 36

19 Multicollinearity Multicollinearity can cause serious numerical and statistical difficulties in fitting the regression model unless extra predictor variables are deleted. Example:If income, expenditure, and savings are used as predictor variables one should be deleted since saving = income expenditure Multicollinearity leads to these problems: -The estimates ˆ β j are subject to numerical errors and are unreliables. This is reflected in large changes in their magnitudes with small changes in data. -Most of the estimated coefficients have large standard errors and as a result are statistically nonsignificant even if the overall F-statistic is significant. 7/13/2004 Unit 11 - Stat Ramón V. León 37 Regression Coefficients in the Presence of Multicollinearty: Cement Data Parameter Estimates Term Intercept X1 x2 x3 X4 Source Model Error C. Total Estimate Analysis of Variance DF Std Error Sum of Squares t Ratio Mean Square Prob> t F Ratio Prob > F <.0001 Notice that all the regression coefficients are nonsignificant, although the overall F = is highly significant 7/13/2004 Unit 11 - Stat Ramón V. León 38

20 Measures of Multicollinearity: Correlations and Variance Inflation Factor (VIF) The simplest measure is the correlations matrix between pairs of predictor variables. A more direct measure is the VIF. VIFj = Diagonal elements of inverse of correlation matrix 1 We can show that VIFj =, i = 1, 2,..., k 1 r 2 where rj is the coefficient of multiple determination when regressing xj on the remaining k-1 predictor variables. Generally, VIF j values greater than 10, corresponding to 2 r >.9, are regarded as unacceptable. j 2 j 7/13/2004 Unit 11 - Stat Ramón V. León 39 VIFs for Cement Data Parameter Estimates Term Intercept X1 x2 x3 X4 Estimate Std Error t Ratio Prob> t VIF Notice that all the VIFs are greater than 10, a strong indication of multicollinearity 7/13/2004 Unit 11 - Stat Ramón V. León 40

21 VIF in JMP: How? Right click on the Parameter Estimates area. Select VIF from the columns submenu that pops up: 7/13/2004 Unit 11 - Stat Ramón V. León 41 Polynomial Regression Remarks Problems Power of x are highly correlated: collinearity Powers of x can have large differences in order of magnitude Solutions Center the x around the mean: x x Standardizing the x is even better: x x s x 7/13/2004 Unit 11 - Stat Ramón V. León 42

22 Dummy Predictor Variables Many applications involve categorical predictor variables, e.g. gender, season, prognosis (poor, average, good) For ordinal variables may use scores, e.g. use 1, 2, and 3 for the prognosis categories For nominal variables with c categories use c-1 indicator variables x 1,x 2,,x c-1 called dummy variables. Example: 1 if Winter x1 = 0 if another season Model: 1 if Spring x2 = Y = β0 + β1x1 + β2x2 + β3x3 + ε 0 if another season Fall is taken as reference point: 1 if Summer x3 = 0 if another season β 0 7/13/2004 Unit 11 - Stat Ramón V. León 43 Why 1-c Dummy Variables? 1 if Winter x1 = 0 if another season If 1 if Spring x2 = 0 if another season 1 if Summer x3 = 0 if another season Then x x x x = 1 Multicollinearity x 4 1 if Fall = 0 if another season 7/13/2004 Unit 11 - Stat Ramón V. León 44

23 JMP s Dummy Variables 1 if Winter 1 if Spring 1 if Summer z1 = -1 if Fall z2 = -1 if Fall z3 = -1 if Fall 0 if another season 0 if another season 0 if another season Y = β + β ( z ) + β ( z ) + β ( z ) + ε Four season average is taken as reference point:β 0 + Y Y Y Y = β + β ( 1) + β ( 1) + β ( 1) + ε Fall = β + β (1) + ε Winter 0 1 = β + β (1) + ε Spring 0 2 = β + β (1) + ε Summer 0 3 = 4β + ε 0 ε Y = β /13/2004 Unit 11 - Stat Ramón V. León 45 Quarterly Sales of Soda Cans (in Millions of Cans) The x s are the textbook (SAS) dummy variables. Thez s are JMP s dummy variables. 7/13/2004 Unit 11 - Stat Ramón V. León 46

24 New platform Runs Chart of Soda Can Sales 1 if Winter x1 = 0 if another season 1 if Spring x2 = 0 if another season 1 if Summer x3 = 0 if another season Try Model : Y = β + β x + β x Q β x + β QUARTER + ε Linear time trend and cyclical seasonal trend 7/13/2004 Unit 11 - Stat Ramón V. León 47 JMP Analysis: Textbook (SAS) Dummy Variables After adjusting for the linear time trend, the Spring and Summer sales differ significantly from the Fall sales, but those of Winter do not. Notice that these variables are declared continuous in JMP Winter Spring Summer 7/13/2004 Unit 11 - Stat Ramón V. León 48

25 JMP Analysis: JMP Dummy Variables After adjusting for the linear time trend Winter, Spring, and Summer (and Fall) differ significantly from the average of the four quarters Winter Spring Summer 7/13/2004 Unit 11 - Stat Ramón V. León 49 JMP Analysis: The Easy Way Nominal To make Fall the -1 z variable. 7/13/2004 Unit 11 - Stat Ramón V. León 50

26 How to Tell Season Effect is Important* Effect Tests Source Quarter Season Np arm 1 3 DF 1 3 Sum of Squares F Ratio Prob > F <.0001 <.0001 There statistical evidence of a seasonal effect *Part the output of the command in the previous visual 7/13/2004 Unit 11 - Stat Ramón V. León 51 7/13/2004 Unit 11 - Stat Ramón V. León 52

27 Confidence and Prediction Intervals Notice that one should not use the Mean Confidence Interval Predicted sales for seventeenth quarter (next Winter) in last line 7/13/2004 Unit 11 - Stat Ramón V. León 53 Time Series Reference Forecasting Principles and Applications by Stephen A. DeLurgio. Inwin McGraw- Hill 7/13/2004 Unit 11 - Stat Ramón V. León 54

28 No interaction term in the model One Dummy and One Continuous Predictor Variables E( Weight) = β0 + β1sex + β2height where SEX = 0 for females and 1 for males. Equivalent formulation: β0 + β2height for females E( Weight) = ( β0 + β1) + β2height for males Parallel lines Lines have the same slope Height is an effect confounder for the gender effect. 7/13/2004 Unit 11 - Stat Ramón V. León 55 No interaction term in the model One Dummy and One Continuous Predictor Variables E( Weight) = β0 + β1sex + β2height where SEX = 0 for females and 1 for males. Equivalent formulation: β0 + β2height for females E( Weight) = ( β0 + β1) + β2height for males Lines have the same slope Height is an effect confounder for the gender effect. 7/13/2004 Unit 11 Female - Stat Ramón Male V. León 56 F

29 Interaction E( Weight) = β0 + β1sex + β2height + β3sex * HEIGHT where SEX = 0 for females and 1 for males. Interaction Equivalent formulation: Term β0 + β2height for females E( Weight) = ( β0 + β1) + ( β2 + β3) Height for males Lines have different slopes Height is an effect modifier for the gender effect Interactions are also important to consider among continuous and among categorical variables. This is a key step in model building. 7/13/2004 Unit 11 - Stat Ramón V. León 57 Considering Interactions 7/13/2004 Unit 11 - Stat Ramón V. León 58

30 7/13/2004 Unit 11 - Stat Ramón V. León 59 7/13/2004 Unit 11 - Stat Ramón V. León 60

31 The interaction of Brain Size and Height is not significant 7/13/2004 Unit 11 - Stat Ramón V. León 61 Fallacy of Doing Simple Linear Regression When Multiple Linear Regression is Called For Regressing health index versus wealth gives a negative slope for wealth effect The higher the wealth the lower the health This is what one does when one fits a simple linear regression model Regressing health index versus wealth while adjusting for age gives a positive slope for wealth The higher the wealth the higher the health This is what one does when one fits a multiple linear regression model of the health index versus wealth and age 7/13/2004 Unit 11 - Stat Ramón V. León 62

32 Logistic Regression and Logit Models Logistic Regression used when the response variable is binary (or more generally categorical), e.g., a patient survives or dies. Model: PY ( = 1 x1, x2,..., x) k ln = β0 + β1x1+ β2x βkxk PY ( = 0 x1, x2,..., xk ) Equivalently, β0+ β1x1+ β2x βkxk e PY ( = 1 x1, x2,..., xk ) = β0+ β1x1+ β2x βkxk 1 + e 7/13/2004 Unit 11 - Stat Ramón V. León 63 Age and Coronary Heart Disease: Status of 100 Subjects Age CHD Age CHD Age CHD Age CHD Age CHD /13/2004 Unit 11 - Stat Ramón V. León 64

33 Coronary Heart Disease by Age 7/13/2004 Unit 11 - Stat Ramón V. León 65 Logistic Regression in JMP Notice that CHD is declared nominal 7/13/2004 Unit 11 - Stat Ramón V. León 66

34 JMP Output The probability of coronary heart disease increases with age. CHD AGE Ask JMP with the question mark tool for the complete interpretation of this plot 7/13/2004 Unit 11 - Stat Ramón V. León 67 JMP Output SSR SSE SST Whole Model Test Model Difference Full Reduced -LogLikelihood RSquare (U) Observations (or Sum Wgts) Converged by Gradient Parameter Estimates Term Intercept AGE For log odds of 0/1 Estimate DF Std Error ChiSquare ChiSquare Prob>ChiSq <.0001 Counterpart to the Analysis of Variance table in multiple linear regression with similar interpretation. Prob>ChiSq <.0001 < /13/2004 Unit 11 - Stat Ramón V. León 68

35 Logistic Regression Using JMP s Fit Model Platform: How to do Multiple Logistic Regression 7/13/2004 Unit 11 - Stat Ramón V. León 69 Whole Model Test Model Difference Full Reduced -LogLikelihood RSquare (U) Observations (or Sum Wgts) Converged by Gradient Lack Of Fit Source Lack Of Fit Saturated Fitted Intercept AGE DF Parameter Estimates Term Estimate DF LogLikelihood ChiSquare Prob>ChiSq Std Error ChiSquare Prob>ChiSq <.0001 ChiSquare For log odds of 0/1 Effect Wald Tests Source Nparm DF Wald ChiSquare Prob>ChiSq AGE Effect Likelihood Ratio Tests Source Nparm DF L-R ChiSquare Prob>ChiSq AGE Prob>ChiSq Lower 95% <.0001 < Upper 95% JMP Output This test is also available in multiple linear regression when there are repeat observations at certain values of the regressors X 7/13/2004 Unit 11 - Stat Ramón V. León 70 Y Interpreted as the Type III sums of squares in multiple linear regression

36 Do You Need to Know More? 572 Applied Regression Analysis (3) Simple linear regression. Matrix approach to multiple linear regression. Partial and sequential sums of squares, interaction and confounding, use of dummy variables, model selection. Leverage, influence and collinearity. Autocorrelated errors. Generalized linear models, maximum likelihood estimation, logistic regression, analysis of deviance. Nonlinear models, inference, ill-conditioning. Robust regression, M-estimators, iteratively reweighted least squares. Nonparametric regression, kernel, splines, testing lack of fit. Prereq: 571 and matrix algebra. F,Sp Recommended References: Introduction to Linear Regression Analysis, Third Edition by Douglas C. Montgomery, Elizabeth A. Peck, and G. Geoffrey Vining. A very well-written introduction to regression including some modern methods. Applied Logistic Regression, Second Edition by David W. Hosmer and Stanley Lemeshow. Wiley Interscience. A great introduction to logistic regression for those already familiar with the elements of multiple regression analysis. 7/13/2004 Unit 11 - Stat Ramón V. León 71