Lecture 4 Multiple linear regression

Transcription

1 Lecture 4 Multiple linear regression BIOST 515 January 15, 2004

2 Outline 1 Motivation for the multiple regression model Multiple regression in matrix notation Least squares estimation of model parameters Maximum likelihood estimation of model parameters Hypothesis testing

3 Multiple linear regression 2 We are now considering the model y i = β 0 + β 1 x i1 + + β p x ip + ɛ i, (1) i = 1,..., n, E[ɛ i ] = 0, var(ɛ i ) = σ 2 and cov(ɛ i, ɛ j ) = 0. We will also require that p < n. Additional assumption: The predictors (x 1,..., x p ) are fixed and measured without error

4 Why multiple linear regression? 3 Previously we ve examined the case with one predictor and one outcome (simple linear regression). There are a variety of reasons we may want to include additional predictors in the model. Scientific question Adjustment for confounding Gain precision

5 Scientific Question 4 May dictate inclusion of particular predictors Predictors of interest The scientific factor under investigation can be modeled by multiple predictors (eg - dummy variables, polynomials, etc.) Effect modifiers The scientific question may relate to detection of effect modification Confounders The scientific question may have been stated in terms of adjusting for known (or suspected) confounders

6 Confounding 5 Sometimes the scientific question of greatest interest is confounded by associations in the data. From KKMN, pg. 187: In general, confounding exists if meaningfully different interpretations of the relationship of interest result when an extraneous variable is ignored or included in the analysis.

7 Precision 6 Adjusting for an additional covariate changes the standard error of the slope estimate Standard error is decreased by having smaller within group variance Standard error is increased by having correlations between the predictor of interest and other covariates in the model

8 General comments on multiple regression 7 Can be difficult to choose the best model, since many reasonable candidates may exist More difficult to visualize the fitted model More difficult to interpret the fitted model

9 The model in matrix notation 8 y = Xβ + ɛ y = y 1 y 2. y n, X = 1 x 11 x 12 x 1p 1. x 21. x 22. x 2p. 1 x n1 x n2 x np, β = β 0 β 1. β p, ɛ = ɛ 1 ɛ 2. ɛ n

10 Least Squares Estimates 9 S(β) = (y Xβ) (y Xβ) Least squares estimates are obtained by solving S(β) β = 2X y + 2X Xβ = 0 for β. This gives us the least squares normal equations: X X ˆβ = X y. To get the least squares estimator of β multiply each side by (X X) 1 which gives ˆβ = (X X) 1 X y

11 provided (X X) 1 exists (which it will if the regressors are linearly independent). 10 The vector of fitted y-values, ŷ, corresponding to the observed y-values y is ŷ = X ˆβ = X(X X) 1 X y = Hy. The n n matrix H is often referred to as the hat matrix. The residuals, can also be rewritten as e = ŷ y, e = y X ˆβ = y Hy = (I H)y.

12 Properties of least squares estimates 11 ˆβ is an unbiased estimator of β E( ˆβ) = E[(X X) 1 X y] = (X X) 1 X Xβ = β The variance of ˆβ is expressed by the covariance matrix cov( ˆβ) = E{[ ˆβ E( ˆβ)][ ˆβ E( ˆβ)] } = σ 2 (X X) 1 If we let C = (X X) 1, the variance of ˆβ j = σ 2 C jj and the covariance between ˆβ i and ˆβ j is σ 2 C ij.

13 Estimation of σ 2 12 As in simple linear regression, we develop an estimator of σ 2 from SSE (residual sum of squares). SSE = (y X ˆβ) (y X ˆβ) ˆσ 2 = MSE = SSE n p 1 As in simple linear regression ˆσ 2 is an unbiased estimator of σ 2.

14 Example 13 Cardiovascular Health Study (CHS) A population-based, longitudinal study of coronary heart disease in people over 65. Primary aim: To identify risk factors related to the onset and course of coronary heart disease and stroke. Secondary aim: Describe prevalence and distributions of risk factors.

15 Scientific question 14 How is a person s weight related to blood pressure? We might expect people who are more overweight to have higher blood pressure. However, a simple linear regression model with weight as a predictor might be misleading. Why? We will examine a subset of 500 of these subjects to see how height and weight are related to blood pressure.

16 15 Scatterplot matrix pairs(cbind(chs$diabp,chs$height,chs$weight), labels=c("blood Pressure", "Height", "Weight")) Blood Pressure Height Weight

17 16 3d scatterplot library(scatterplot) scatterplot3d(chs$height,chs$weight,chs$diabp, xlab="height", ylab="weight", zlab="blood pressure") height weight Blood pressure

18 Regression models we may be interested in: 17 E[BP i ] = β 0 + β 1 height i E[BP i ] = β 0 + β 1 weight i E[BP i ] = β 0 + β 1 height i + β 2 weight i We ve fit models similar to the first 2, but not the 3rd. Note: 2 observations will be omitted from the analysis because the subjects weights are missing.

19 18 >lmht <- lm(diabp~height,data=chs) >summary(lmht) BP i = β 0 + β 1 height i + ɛ i Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-09 *** HEIGHT * --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 496 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 1 and 496 DF, p-value:

20 19 lmwt <- lm(diabp~weight,data=chs) summary(lmwt) BP i = β 0 + β 1 weight i + ɛ i Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) < 2e-16 *** WEIGHT ** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 496 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 1 and 496 DF, p-value:

21 20 BP i = β 0 + β 1 height i + β 2 weight i + ɛ i lmhtwt <- lm(diabp~height+weight,data=chs) summary(lmhtwt) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-10 *** HEIGHT WEIGHT * --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 495 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 2 and 495 DF, p-value:

22 Dummy (indicator) variables 21 So far we have only dealt with predictors that are continuous. However, we will often have variables that can take on only a small number of levels. Examples: Smoking status (current/former/never) Race Sex (Male/Female) Treatment group in a clinical trial

23 To model the association between these predictors and a response, we assign some sort of numerical scale to them. For example, we could define sex as (0/1), (-1,1), (1,2). Then if 22 y i = β 0 + β 1 sex i + ɛ i and sex = 1 if female and 0 if male, E(y i ) = β 0, if male and E(y i ) = β 0 + β 1, if female. What does β 1 represent?

24 How is being a current smoker related to blood pressure? 23 >lmsmk <- lm(chs$diabp~smoker) >summary(lmsmk) BP i = β 0 + β 1 smoker i + ɛ i Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** smoker Residual standard error: on 496 degrees of freedom Multiple R-Squared: , Adjusted R-squared: F-statistic: on 1 and 496 DF, p-value:

25 We might also want to know if current smoking status is an effect modifier for weight? In other words, is the relationship between blood pressure and weight different for people who smoke than for those who don t smoke? 24 BP i = β 0 +β 1 weight i +β 2 smoker i +β 3 weight i smoker i +ɛ i How do we interpret β 0, β 1, β 2, and β 3?

26 > summary(lm(chs$diabp~smoker*chs$weight)) 25 Call: lm(formula = chs$diabp ~ smoke * chs$weight) Residuals: Min 1Q Median 3Q Max e e e e e+01 Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) < 2e-16 *** smoker chs$weight ** smoker:chs$weight Residual standard error: on 494 degrees of freedom Multiple R-Squared: , Adjusted R-squared:

27 weight blood pressure Smokers Non smokers

28 We know if a subject is a current/former/never smoker. How should we model this relationship with blood pressure? 27 Option 1: smoke i = 1, never smoked 2, f ormer smoker 3, current smoker BP i = β 0 + β 1 smoke i + ɛ, how do we interpret β 1? ˆβ 1 = 1.08 and ŝe( ˆβ 1 ) =

29 Option 2: Create two dummy variables 28 smoke 1i = { 1, never smoked 0, otherwise and smoke 2i = { 1, former smoker 0, otherwise. BP i = β 0 + β 1 smoke 1i + β 2 smoke 2i + ɛ i How do we interpret β 0, β 1 and β 2?

30 > lmsmk2 <- lm(chs$diabp~smoke1+smoke2) > summary(lmsmk2) 29 Call: lm(formula = chs$diabp ~ smoke1 + smoke2) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** smoke1true smoke2true Signif. codes: 0 *** ** 0.01 * Residual standard error: on 495 degrees of freedom

31 Summary 30 So far, we ve discussed The motivation for multiple linear regression models Matrix notation for multiple linear regression Least squares estimates for multiple linear regression Recoding factors into dummy variables Intrepretation of linear models

32 Next, we ll discuss hypothesis testing in multiple linear regression 31 Overall tests : does the entire set of independent predictors contribute significanly to the prediction of y? Test for addition of a single variable: does the addition of one variable significantly improve the prediction of y over other independent predictors already present in the model? Test for addition of a group of variables: does the addtion of some group of variables improve the prediction of y over other independent predictors alreay present in the model?