Multiple regression. Partial regression coefficients

Transcription

1 Multiple regression We now generalise the results of simple linear regression to the case where there is one response variable Y and two predictor variables, X and Z. Data consist of n triplets of values (X, Z, Y ) (X n, Z n, Y n ). We want to predict the value of Y associated with particular combination of values of X and Z, or to describe the relationship between Y, X and Z, or to estimate the effect of changes in X and Z on Y. Once again there are two situations in which this type of problem arises. a) Predictors X, Z and response Y are all random. b) Predictors X and Z are fixed, e.g. by experimental design. In either case, there is a prediction equation Y = b 0 + b X + b 2 Z + e () The prediction error e is assumed N(0, σ 2 ). Geometrically, we can imagine a 3-d picture with perpendicular horizontal axes X and Z and a vertical axis Y. The equation Y = b 0 + b X + b 2 Z represents a plane surface whose position is determined by b 0 and its orientation by b and b 2. Partial regression coefficients The coefficients b and b 2 in equation () are partial regression coefficients. For example, b represents the effect on Y of changing X while holding Z constant. The parameters b and b 2 can be estimated by least squares, i.e. chosen so that the sum of squares n (Y i= i b 0 b X i b 2 Z i ) 2 is minimized. The least squares estimates b and b 2 are solutions of the two equations [ S xx S xz S zx S zz ][ b b 2 ] = [ S xy S zy ] For the case where all variables X, Z, and Y are random, the corrected sums of squares and products would be replaced by variances and covariances. Solving the equations gives b = (S xy S xz S yz /S zz )/(S xx S 2 xz /S zz) The formula for b 2 is obtained by switching x and z in the formula for b. It sometimes helps to give regression coefficients more informative labels. For example, if we denote the regression coefficient of Y on X (ignoring Z) by b yx and the partial regression coefficient of Y on X (accounting for Z) by b yx,z then b yx = b yx,z + b yz,x b zx

2 If X changes by a small amount δx, there is a small concomitant change in Z of amount δz = b zx δx, which is invisible when we regress Y on X (ignoring Z). The total change in Y is δy = b yx,z δx + b yz,x δz and b yx = δy/δx. Residuals and fitted values The fitted value for the ith observation is Y i = b 0 + b X i + b 2 Z i, and the equation which we derived for simple linear regression still holds: n i= (Y i Y ) 2 = n i= ( Y i Y ) 2 + n i= (Y i Y i ) 2 The regression sum of squares (first term on r.h.s.) simplifies to b S xy + b 2 S zy, with 2 d.f. The total sum of squares (on l.h.s.) has n d.f., as before. The residual sum of squares (calculated as the difference between total and regression sums of squares) has n 3 d.f. The ANOVA table now has the form Source of variation DF Regression 2 Residual n 3 Total n Mean squares for regression and residual are calculated in the usual way, and F is the ratio of these two mean squares. The F statistic is used to test H 0 : b = b 2 = 0, i.e. that E(Y ) = b 0 (a constant). Extra sums of squares The regression sum of squares with 2 d.f. is usually split into two components: the SSQ associated with fitting X alone ( d.f.), and the extra SSQ obtained when Z is added to the equation (this also has d.f.). Alternatively, it is the SSQ associated with fitting Z alone, plus the extra SSQ obtained by adding X to the equation. Note that, for example, the SSQ associated with fitting X alone is not generally the same as the extra SSQ obtained by fitting X after Z. However, whichever way we chose to do the split, the two components add up to the regression SSQ with 2 d.f. Hypothesis testing As well as testing the hypothesis that b = b 2 = 0, we can test each partial regression coefficient separately for significance. The test of H 0 : b = 0 is based on the extra sum of squares obtained by fitting X after Z: if H 0 is true, (extra sum of squares)/s 2 F with and n 3 d.f. 2

3 Two examples ) In animal breeding, Y might be the breeding value of an animal for a particular trait, and X and Z values of the trait measured on two of its relatives. For example, if Y is the milk yield breeding value of a heifer, and X and Z milk yields of its mother and paternal half-sister, we might predict Y from X and Z. Covariance matrix for X, Z and Y is V P 0 2 V A 0 V P 2 V A 4 V A 4 V A V A Prediction equation is Y = h 2 ( 2 X + 4 Z), Y = b X + b 2 Z, where b V P = 2 V A, b 2 V P = 4 V A, so where h 2 = V A /V P is the heritability of the trait. 2) The difficulty of a hill race is measured by (i) the total distance covered, and (ii) the total climb required. The file hills.csv gives the distance (miles), climb (000 feet), and record time (minutes) for 35 Scottish hill races. Multiple regression can be used to find a relationship between record time and the two measures of difficulty. Fitting distance (time = distance) produces a reasonable fit, but there are large residuals for two difficult races, Bens of Jura (+75.7) and Lairig Ghru ( 39.4). With both distance and climb fitted, (time = distance +.75 climb), there is a significant improvement in overall fit. For example, the residuals for both Bens of Jura (+27.9) and Lairig Ghru (+3.3) are considerably reduced. Using lm( ) to fit this model to the hill race data gives the following estimates for the partial regression coefficients: Estimate Std. Error t value Distance Climb and the ANOVA table is: Df Sum Sq Mean Sq F value Distance Climb Residuals The t statistic for the Climb partial regression coefficient is the square root of the F statistic for the effect of fitting Climb after Distance. These two tests are equivalent. 3

4 Miscellaneous The results given above describe how to predict Y using two r.v.s X and Z. These results generalise in a straightforward way to the problem of predicting Y from any number of predictors X, Z, W, etc. Comments already made about residuals, outliers, cause and effect, etc, for simple linear regression remain relevant for multiple regression. It often happens that one of the predictor variables (X, say) is of primary interest, and the other (Z) is included as a potential confounder. In other words, Z is included not because we are interested in its effect, but to ensure that the effect of X is adjusted for the effect of Z. For example, in a study to determine whether drinking coffee might have a beneficial effect on health, we might also include annual earnings. Unless we do this, an association between good health and high earnings might appear misleadingly as an association with level of coffee consumption (if, for example, high earners consume large amounts of coffee and low earners do not, because they cannot afford to). If there is a strong correlation between X and Z, the partial regression coefficients will have large standard errors. In extreme cases, when the correlation is close to ± ( collinearity ), the fitting procedure breaks down and one or other variable must be dropped from the regression equation. Multiple regression in R The lm( ) function which we have already used for simple linear regression also deals with multiple regression. Diagnostic plots (residuals against fitted values, etc), and analysis of variance tables are produced as for simple linear regression. R code for the hill-race example is given below. If fit is a fitted lm object, summary(fit) produces estimates and standard errors for partial regression coefficients. These are adjusted for all other effects in the model, and do not depend on the order of terms in the model equation. anova(fit) produces sequential sums of squares, which depend on the order of terms. In the example below, the first row of the anova table will have a sum of squares ( d.f.) for the effect of distance (unadjusted). The second row will have a sum of squares ( d.f.) for the effect of climb, adjusted for the effect of distance. If the model equation had been given as time ~ climb + dist, a different ANOVA would be produced with sums of squares for climb (unadjusted) and distance (adjusted for climb). The sum of the two sums of squares (unadjusted and adjusted) would be the same in both cases. hills <- read.table("hills.txt") fit <- lm(time ~ dist + climb, data = hills) summary(fit) anova(fit) plot(fit) 4

5 Factors and dummy variables The following is a typical example of an analysis of variance model. We will consider such models next week. Here we show that the anova model can be considered as a special case of multiple regression. A flock of sheep consists of three breeds: Scottish Blackface, Welsh Mountain and the Blackface Welsh cross. A blood sample is taken from a random sample of each breed, and the Cu content measured. Do the breeds differ in Cu concentrations? Blackface Welsh Cross Assume the data are normally distributed with constant variance and a mean value that depends on breed. This can be treated as a multiple regression E(Y ) = b 0 + b X + b 2 Z, where the values assigned to X and Z depend on breed as follows: Breed X Z b 0 + b X + b 2 Z Blackface 0 0 b 0 Welsh 0 b 0 + b Cross 0 b 0 + b 2 The parameters b 0, b, b 2 represent the mean value for the Blackface breed, the difference between Welsh and Blackface, and the difference between Cross and Blackface. The multiple regression F test (with 2 d.f. in the numerator) tests H 0 : b 2 = b 3 = 0 (no difference among breeds). This is easily generalized to compare any number of groups. Dummy variables usually arise through use of factors in model formulas. For example, in the R code given below, the formula ~ breed is equivalent to ~ X + Z, where X and Z are the dummy variables described above. However, in this case, the usual splitting of the regression sums of squares does not take place. breed <- factor(rep(c("blackface", "Welsh", "Cross"), c(5,5,5))) Cu <- c(6.5,7.9,7.4,6.8,8.,0.4,9.8,., 0.6,9.2,6.9,9.2,8.4,7.6,9.7) fit <- lm(cu ~ breed) anova(fit) 5

6 Example model formulas Suppose x and z are numeric, and that A is a factor. Some possibilities for the right-hand side of an lm( ) formula are: Formula x x + z poly(x,2) A A + x A * x Interpretation simple linear regression multiple regression polynomial regression one-way analysis of variance parallel lines (analysis of covariance) separate lines (intercept and slope) for each level of A For example, if factor A has two levels, the model formula for parallel lines gives E(Y ) as b 0 + b 2 X for the first level of A and b 0 + b + b 2 X for the second level. The common slope of the two lines is b 2, the intercept for the first level of A is b 0, and b is the difference between the intercepts (the constant vertical separation of the two lines). This model ( analysis of covariance ) is often used when primary interest is in the factor A, and x is a potential confounder. 6