1 Multiple Regression

Transcription

1 1 Multiple Regression In this section, we extend the linear model to the case of several quantitative explanatory variables. There are many issues involved in this problem and this section serves only as an introduction. We start with an example. Example 1.1. The dataset fat in the faraway package contains several body measurements of 252 adult males. Included in this dataset are two measures of the percentage of body fat, the Brozek and Siri indices. Each of these indices computes the percentage of body fat from the density (in gm/cm 3 ) which in turn is approximated by an underwater weighing technique. This is a time-consuming procedure and it might be useful to be able to estimate the percentage of body fat from easily obtainable measurements. For example, it might be nice to have a relationship of the following form: density = f(x 1,..., x k ) for k easily measured variables x 1,..., x k. We will first investigate the problem of approximating body fat by a function of only weight and abdomen circumference. The data on the first two individuals is given for illustration. > fat[1:2,] brozek siri density age weight height adipos free neck chest abdom hip thigh knee ankle biceps forearm wrist The notation gets a bit messy. We will continue to use y for the response variable and we will use x 1,..., x k for the k explanatory variables. We will again assume that there are n individuals and use the subscript i to range over individuals. Therefore, the i th data point is (x 1i, x 2i,..., x ki, y i ). The standard linear model now becomes the following. The standard linear model The standard linear model is given by the equation y = β 0 + β 1 x β k x k + ε (1) where 1. ε has mean 0 2. ε has standard deviation σ 3. and ε has a normal distribution. We again assume that the n data points are the result of independent ɛ 1,..., ɛ n. To find good estimates of β 0,..., β k we proceed exactly as in the case of one predictor and find the least squares estimates. Specifically, let b i be an estimate of β i and define ŷ i = b o + b 1 x ii + b 2 x 2i + + b k x ki. We choose these estimates so that we minimize SSE where SSE = n (y i ŷ i ) 2. i=1 It is routine to find the values of the b i s that minimize SSE. R computes them with dispatch. Suppose that we use weight and abdomen circumference to try to predict the Brozek measure of body fat.

2 > l=lm(brozek~weight+abdom,data=fat) > l lm(formula = brozek ~ weight + abdom, data = fat) (Intercept) weight abdom In the case of multiple predictors, we need to be very careful in how we interpret the various coefficients of the model. For example b 1 = 0.14 in this model seems to indicate that body fat is decreasing as a function of weight. This is counter to our intuition and our experience which says that the heaviest men tend to have more body fat than average. On the other hand, the coefficient b 2 = seems to be consistent with the relationship between stomach girth and body fat that we know. The key here is that the coefficient b 1 measures the effect of weight on body fat for a fixed abdomen circumference. This makes more sense. Among individuals with a fixed abdomen circumference, the heavier individuals tend to be taller and so have perhaps less body fat. Even this interpretation needs to be expressed carefully however. It is misleading to say that body fat decreases as weight increases with abdomen circumference held fixed since increasing weight tends to increase abdomen circumference. We will come back to this relationship in a moment but first we investigate the problem of inference in this linear model. The short story of inference is that all of the results for the one predictor case have the obvious extensions to more than one variable. To estimate σ, we again use the residual standard error, s e, except that we define it by SSE s e = n (k + 1). The denominator in s e is simply n p where p is the number of estimated coefficients in the model. Using the estimate s e of σ, we can again produce an estimate se(b j ) of the standard deviation of b j and produce confidence intervals for β j. For the body fat data we have > summary(l) lm(formula = brozek ~ weight + abdom, data = fat) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) < 2e-16 *** weight e-11 *** abdom < 2e-16 *** Residual standard error: on 249 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 2 and 249 DF, p-value: < 2.2e-16 > confint(l) 2.5 % 97.5 % (Intercept) weight abdom From the output we observe the following. Our estimate for σ is the residual standard error, We note that 249 degrees of freedom are used which is since there are three parameters. We can compute the confidence interval for β 1 from the summary table (b 1 = 0.14 and se(b 1 ) = 0.019) using the t distribution with 249 degrees of freedom or from the R function confint.

3 We can compute confidence intervals for the expected value of body fat and prediction intervals for an individual observation as well. Investigating what happens for a male weighing 180 pounds with an abdomen measure of 82 cm gives the following prediction and confidence intervals: > d=data.frame(weight=180, abdom=82) > predict(l,d,interval='confidence') fit lwr upr > predict(l,d,interval='prediction') fit lwr upr The average body fat of such individuals is likely to be between 7.9% and 10.4%. An individual male not part of the dataset is likely to have body fat between 0.91% and 17.4%. We now return to the issue of interpreting the coefficients in the linear model. In the case of the body fat example, let s fit a model with weight as the only predictor. > lm(brozek~weight,data=fat) lm(formula = brozek ~ weight, data = fat) (Intercept) weight Notice that the sign of the relationship between weight and body fat has changed! Using weight alone, we predict an increase of 0.16 in percentage of body fat for each pound increase in weight. What has happened? Let s first restate the two fitted linear relationships: brozek = weight abdom (2) brozek = weight (3) In order to understand the relationships above, it is important to understand that there is a relationship between weight and the abdomen measurement. One more regression is useful. > lm(abdom~weight,data=fat) lm(formula = abdom ~ weight, data = fat) (Intercept) weight Now suppose that we change weight by 10 pounds. The last analysis says that we would predict that the abdomen measure increases by 3.3 cm. Using (2) we see that a increase in 10 pounds of weight and an increase of 3.3 cm in abdomen circumference causes an increase of 10 ( 0.14) (3.3) = 1.6% in Brozek index. But this is precisely what an increase in 10 pounds of weight should produce according (3). The fact that our predictors are linearly related in the set of data (and so presumably in the population that we are modeling) is known as multicollinearity. The presence of multicollinearity makes it difficult to give simple interpretations of the coefficients in a multiple regression. Interaction terms Consider our linear relationship, brozek = weight abdom. This model implies that for any fixed value of abdom, the slope of the line relating brozek to weight is always An alternative (and more complicated) model would be that the slope of this line also changes as the value of abdom changes. One strategy

4 for incorporating such behavior into our model is to add an additional term, an interaction term. The equation for the linear model with an interaction term in the case that there are only two predictor variables is Y = β 0 + β 1 x 1 + β 2 x 2 + β 1,2 x 1 x 2 + ɛ. While this is not the only way that two variables could interact, it seems to be the simplest possible way. R allows us to add an interaction term using a colon. > lm(brozek~weight+abdom+weight:abdom,data=fat) lm(formula = brozek ~ weight + abdom + weight:abdom, data = fat) (Intercept) weight abdom weight:abdom While the coefficient for the interaction term ( ) seems small, one should realize that the values of the product of these two variables are large so that this term contributes significantly to the sum. On the other hand, in the presence of this interaction term, the contribution of the term for weight is now very small. With all the possible variables that we might include in our model and with all the possible interaction terms, it is important to have some tools for evaluating different choices. We take up this issue in the next section. 2 Evaluating Models In the previous section, we considered several different linear models for predicting the Brozek body fat index from easily determined physical measurements. Other models could be considered by using other physical measurements that were available in the dataset. How should we evaluate one of these models and how should we choose among them? One of the principle tools used to evaluate such models is known as the analysis of variance. Given a linear model (any model, really), we choose the parameters to minimize SSE. Recall SSE = n (y i ŷ i ) 2. i=1 Therefore it seems reasonable to suppose that a model with smaller SSE is better than one with large SSE. Such a model seems to explain or account for more of the variation in the y i. Consider the two models for body fat, one using only abdomen circumference and the other only weight.

5 > la=lm(brozek~abdom,data=fat) > anova(la) Analysis of Variance Table Response: brozek Df Sum Sq Mean Sq F value Pr(>F) abdom < 2.2e-16 *** Residuals > lw=lm(brozek~weight,data=fat) > anova(lw) Analysis of Variance Table Response: brozek Df Sum Sq Mean Sq F value Pr(>F) weight < 2.2e-16 *** Residuals Among other things, the function anova() tells us that SSE = 5, 095 for the linear model using abdomen circumference and SSE = 9, 410 for the model using only weight. While this comparison seems clearly to indicate that abdomen circumference predicts Brozek index better on average than does weight, using SSE as an absolute measure of goodness of fit is has two shortcomings. First, the units of SSE are in terms of the squares of y units which means that SSE will tend to be large or small according as the observations are large or small. Second, we will obviously reduce SSE by including more variables in the model so that comparing SSE does not give us a good way of comparing, say, the model with abdomen circumference and weight to the model with abdomen circumference alone. We address the first issue first. We would like to transform SSE into a dimension free measurement. The key to doing this is to compare SSE to the maximum possible SSE. To do this, define SSTotal = n (y i ȳ) 2, i=1 The quantity SSTotal is just the quantity SSE for the model with only a constant term. The quantity SSTotal can be computed from the output of the function anova() by summing the column labeled Sum Sq. For the body fat data, that number is SSTotal = 1, We first note that 0 SSE SSTotal. This is because choosing b 0 = ȳ and b 1 = 0 would already achieve SSE = SSTotal but SSE is the minimum among all choices of b 0, b 1. Using this fact, we have a first measure of the fit of a linear model. Define R 2 = 1 SSE SSTotal. We have that 0 R 2 1 and R 2 is close to 1 if linear part of the model fits the data well. The number R 2 is sometimes called the coefficient of determination of the model and is often read as a percentage. In the model for Brozek index which uses only abdomen circumference, we can compute R 2 from the statistics in the analysis of variance table or else we can read it from the summary of the regression where it is labeled Multiple R-Squared. We read the result below as abdomen circumference explains 66.2% of the variation in Brozek index and weight explains 37.6% of the variation in the Brozek index.

6 > summary(la) lm(formula = brozek ~ abdom, data = fat) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** abdom <2e-16 *** Residual standard error: on 250 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 250 DF, p-value: < 2.2e-16 > summary(lw) lm(formula = brozek ~ weight, data = fat) Residuals: Min 1Q Median 3Q Max Estimate Std. Error t value Pr(> t ) (Intercept) e-05 *** weight < 2e-16 *** Residual standard error: on 250 degrees of freedom Multiple R-squared: 0.376, Adjusted R-squared: F-statistic: on 1 and 250 DF, p-value: < 2.2e-16 The number R 2 values for two different models with the same number of parameters gives us a reasonable way to compare their usefulness. However R 2 is a misleading tool for comparing models with differing numbers of parameters. After all, if we allow ourselves n different parameters (i.e., we have n different explanatory variables), we will be able to fit the data exactly and so achieve R 2 = 100%. There are a many ways to compare two such models the one that we explore here compares two nested models based on their respective SSE. Suppose that we have two different models, one of which is nested in the other. For example, the models model1 : model2 : brozek = β 0 + β 1 weight brozek = β 0 + β 1 weight + β 2 abdom are nested in this way. We compute a statstic, called the F statistic, that compares the SSE of the two models. In the following notation, suppose we have two models, model1 with p 1 parameters and sum of squared residuals equal to SSE 1 and similarly for model2. Here model2 contains all the parameters of model1 so that p 1 < p 2. Define F = (SSE 1 SSE 2 )/(p 2 p 1 ) SSE 1 /(n p 1 )

7 If model2 is substantially better than model1, then the F statistic is large but if not then the F statistic is small. We now take as our null hypothesis that all the extra parameters in model2 are 0. The anova() function of R computes the F statstic and the p-value for this null hypothesis: > anova(lw,l) Analysis of Variance Table Model 1: brozek ~ weight Model 2: brozek ~ weight + abdom Res.Df RSS Df Sum of Sq F Pr(>F) < 2.2e-16 *** This result (given the extremely low p value) should be read as including abdom in the model explains significantly more variation that the model that has weight alone. In particular, this p-value tests the null hypothesis H 0 : β 2 = 0.