STAT 571A Advanced Statistical Regression Analysis. Chapter 8 NOTES Quantitative and Qualitative Predictors for MLR

Transcription

1 STAT 571A Advanced Statistical Regression Analysis Chapter 8 NOTES Quantitative and Qualitative Predictors for MLR 2015 University of Arizona Statistics GIDP. All rights reserved, except where previous rights exist. No part of this material may be reproduced, stored in a retrieval system, or transmitted in any form or by any means electronic, online, mechanical, photoreproduction, recording, or scanning without the prior written consent of the course instructor.

2 8.1: Polynomial Regression Mentioned in passing in 6.1, we now study polynomial regression in more detail. This is technically a special form of MLR, since it has more than one β k parameter. Simplest case: 2nd-order/single predictor model: Y i = β 0 + β 1 (X i X) + β 2 (X i X) 2 + ε i (i = 1,...,n) with ε i ~ i.i.d.n(0,σ 2 ).

3 Polynomial Regression (cont d) For simplicity, write x i = (X i X): Y i = β 0 + β 1 x i + β 2 x i 2 + ε i (Why? Centering usually reduces multicollinearity with 2nd-order, and higher, predictors. Just do it.) This quadratic regression can be a useful approximation to data that deviate from strict linearity. See Fig. 8.1

4 Quadratic Regression

5 Cubic Regression (Fig Examples of 3rd-order, cubic regression polynomials.)

6 2nd-Order Response Surface (Fig Examples of 2nd-order response surface, as in 6.1.)

7 Testing Polynomial Models Sequential testing: for testing purposes, we start with the highest-order term and work down the order ( up the ladder ). Suppose E{Y i } = β 0 + β 1 x i + β 11 x i2 + β 111 x i 3 First test H o :β 111 =0 via partial F-test and SSR(x 3 x 1,x 2 ). If signif., STOP and conclude cubic polynomial is significant. If H o :β 111 =0 is NOT signif., drop β 111 and go up ladder to test H o: β 11 =0 via SSR(x 2 x 1 ).

8 Polynomial Regression (cont d) E{Y i } = β 0 + β 1 x i + β 11 x i2 + β 111 x i 3 (cont d) If H o :β 11 =0 is signif., STOP and conclude quadratic polynomial is significant. If H o :β 11 =0 is NOT signif., drop β 11 and go up ladder to test H o: β 1 =0 via SSR(x 1 ). If H o :β 1 =0 is signif., STOP and conclude simple linear model is significant. Etc. Once sequential testing is complete, we usually go back and fit the final model in terms of the orig. X k s to get cleaner b k s and std. errors.

9 Example: Power Cell Data (CH08TA01) Power Cell Data example: Y = {# cycles} and we have 2 predictors (X 1 = charge rate & X2 = temp.); see Table 8.1. Consider a 2nd-order response surface MLR: > Y = c(150, 86, 49,..., 279, 235, 224) > X1 = c(0.6, 1.0, 1.4,..., 0.6, 1.0, 1.4) > X2 = c( rep(10,3), rep(20,5), rep(30,3) ) > x1 = (X1 - mean(x1))/0.4 > x2 = (X2 - mean(x2))/min(x2) > > x1sq = x1*x1 > x2sq = x2*x2 > x1x2 = x1*x2

10 Power Cell Data (CH08TA01) (cont d) Selection of X 1 and X 2 was controlled. note the zero/near-zero correlations among the (transformed) x-variables: > cor( cbind(x1, x2, x1sq, x2sq, x1x2) ) x1 x2 x1sq x2sq x1x2 x1 1.00e e e e e+00 x2 0.00e e e e e-17 x1sq -4.04e e e e e+00 x2sq -1.99e e e e e+00 x1x2 0.00e e e e e+00

11 Power Cell Data (CH08TA01) (cont d) Compare this to (non-trivial) correlations among orig. X k s, etc : > cor( cbind(x1, X2, X1sq, X2sq, X1X2) ) X1 X2 X1sq X2sq X1X2 X1 1.00e e X2 0.00e X1sq 9.91e X2sq -4.16e e X1X2 6.05e e

12 Power Cell Data (CH08TA01) (cont d) Full 2nd-order model fit with transfrm d x s: > summary( lm(y ~ x1 + x2 + x1sq + x2sq + x1x2) ) Call: lm(formula = Y ~ x1 + x2 + x1sq + x2sq + x1x2) Coefficients: Estimate Std.Error t value Pr(> t ) (Intercept) x x x1sq x2sq x1x

13 Power Cell Data (CH08TA01) (cont d) Residual analysis shows no serious issues: > plot( resid(ch08ta01.lm) ~ fitted(ch08ta01.lm) ) > abline( h=0 ) > qqnorm( resid(ch08ta01.lm) )

14 Power Cell Data (CH08TA01) (cont d) Lack of Fit test. Only joint replication is at x 1 =x 2 =0, so need to set up the factor term carefully in R: > LOFfactor = factor( c(seq(-4,-1), rep(0,3), seq(1,4)) ) > anova( CH08TA01.lm, lm(y ~ LOFfactor) ) Analysis of Variance Table Model 1: Y ~ x1 + x2 + x1sq + x2sq + x1x2 Model 2: Y ~ LOFfactor Res.Df RSS Df Sum of Sq F Pr(>F) LOF stat. is F* = 1.82 (P = 0.374). No signif. lack of fit.

15 Power Cell Data (CH08TA01) (cont d) Partial F-test of 2nd-order terms (H o :β 11 = β 22 = β 12 = 0): > anova( lm(y ~ x1+x2), CH08TA01.lm ) Analysis of Variance Table Model 1: Y ~ x1 + x2 Model 2: Y ~ x1 + x2 + x1sq + x2sq + x1x2 Res.Df RSS Df Sum of Sq F Pr(>F) Partial 3 df F-statistic is F* = 0.78 (P = 0.553). No signif. deviation from 0 seen in 2nd-order terms.

16 Power Cell Data (CH08TA01) (cont d) Fit reduced 1st-order model: > summary( lm(y ~ x1+x2) ) Call: lm(formula = Y ~ x1 + x2) Coefficients: Estimate Std.Error t value Pr(> t ) (Intercept) e-08 x x Multiple R-squared: Adjusted R-squared: F-stat.: on 2 and 8 DF, p-val.:

17 Power Cell Data (CH08TA01) (cont d) Bonferroni-adjusted simultaneous conf. intervals on 1st-order β-parameters (using original X-variables): > g = length( coef(lm(y ~ X1+X2)) ) - 1 > confint( lm(y ~ X1+X2), level = 1-(.10/g) ) X X (cf. Textbook p. 305)

18 Interaction Terms Interaction cross-product terms can be included in any MLR to allow for interactions between the X k -variables. E.g., E{Y} = β 0 + β 1 X 1 + β 2 X 2 + β 3 X 1 X 2 The cross-product creates a departure from additivity in the mean response. If β 3 = 0, the mean response is strictly additive in X 1 and X 2.

19 Interaction Terms (cont d) Notice that the usual interpretation for the β k parameters is muddied here. What does it mean to increase X 1 by +1 unit while holding X 1 X 2 fixed?!? Alt. interpretation: cross-product terms allow for synergistic or antagonistic interactions between the X k -variables. It s a special kind of departure from additivity: synergy occurs for β 3 > 0, antagonism for β 3 < 0.

20 Figure 8.8 Graphics for (a) additive, (b) synergistic, or (c) antagonistic response surfaces.

21 Interaction Caveats Need to be careful with interactions. If they exist and they are ignored, very poor inferences on E{Y} will result. On the other hand, adding a kitchen sink of all possible interactions can overwhelm the MLR. With 3 X k -variables there are 3 possible pairwise interactions (not incl. the tri-way!) With 8 X k -variables there are 28 possible pairwise interactions (not incl. multi-ways!) Things get unwieldy fast...

22 Body Fat Data (CH07TA01) (cont d) To the 3 original X k -variables now include all pairwise interactions. Center each X-variable (about its mean) first to assuage problems with multicollinearity: x ik = X ik X k (k = 1,2,3) MLR now has six predictor terms and p=7 β-parameters: E{Y} = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 3 + β 4 x 1 x 2 + β 5 x 1 x 3 + β 6 x 2 x 3

23 Body Fat Data (CH07TA01) (cont d) R can fit interaction terms using a special * operator: e.g., x1*x2 fits x1 and x2 and x1:x2 all with just 1 term. For the Body Fat data, construct centered x-variables as x1 = X1 mean(x1), etc. Then call > anova( lm(y ~ x1 + x2 + x3), lm(y ~ x1*x2 + x1*x3 + x2*x3) ) Output follows

24 Body Fat Data (CH07TA01) (cont d) Output from partial F-test of all pairwise interactions: Analysis of Variance Table Model 1: Y ~ x1 + x2 + x3 Model 2: Y ~ x1 * x2 + x1 * x3 + x2 * x3 Res.Df RSS Df Sum of Sq F Pr(>F) Partial 3 df F-statistic is F* = 0.53 (P = 0.670). No signif. pairwise interactions are seen.

25 Body Fat Data (CH07TA01) (cont d) Can also include tri-way interactions: > anova( lm(y ~ x1+x2+x3), lm(y ~ x1*x2*x3) ) Analysis of Variance Table Model 1: Y ~ x1 + x2 + x3 Model 2: Y ~ x1 * x2 * x3 Res.Df RSS Df Sum of Sq F Pr(>F) d.f. partial F-statistic is F* = 0.45 (P = 0.771). no signif. pairwise or tri-way interactions.

26 Qualitative Predictors We ve seen cases where the X-variable was either 0 or 1 (called a binary indicator). If this indicated a qualitative state (say, 1 = or 0 = ) then the numbering is arbitrary. The predictor is actually qualitative, not quantitative. (Still 0 vs. 1 is usually as good a pseudoquantification as any.) Question: What happens when binary indicators are combined with true quantitative predictors?

27 Example: Insur. Innov n Data Suppose we study Y = Insurance method adoption time (mos.) in insurance companies, with X 1 = size of firm (quantitative) X 2 = type of firm: 1 = stock, 0 otherwise X 3 = type of firm: 1 = mutual, 0 otherwise Design matrix is (n = 4): X 0 X 1 X 2 X 3 X = 1 X X X X

28 Design Matrix Problem But wait, there s a problem with this design matrix X. Notice that X 0 = X 2 + X 3 so the predictors are not linearly independent: rank(x X) = 3 < 4 = p. (See p. 314.) The MLR will fail! Solution is (usually) to eliminate X 3 and model E{Y} = β 0 + β 1 X 1 + β 2 X 2. Model interpretation here is actually sorta intriguing

29 Two Straight Lines For E{Y} = β 0 + β 1 X 1 + β 2 X 2, with X 1 quantitative and X 2 a 0-1 indicator, consider: When X 2 = 0 (mutual firm), E{Y} = β 0 + β 1 X 1, an SLR on X 1 with slope β 1 and Y-intercept β 0. When X 2 = 1 (stock firm), E{Y} = (β 0 + β 2 ) + β 1 X 1, an SLR on X 1 with same slope β 1 but new Y-intercept (β 0 + β 2 ). So we have two parallel straight lines each with same σ 2 one for stock firms (X 2 = 1) and one for mutual firms (X 2 = 0).

30 ANCOVA Graphic

31 Tests in Equal-Slopes ANCOVA For this equal-slopes ANCOVA model, some obvious hypotheses are (first) H o : β 2 = 0 (i.e., no diff. between type lines are same) (next) H o : β 1 = 0 (i.e., no effect of size lines are flat) Data are in Table 8.2; n = 20. R code/analysis follows

32 Insur. Innov n Data (CH08TA02) Y = Insurance method adoption time X 1 =size of firm X 2 = type of firm (mutual vs. stock) > Y = c(17, 26,..., 30, 14) > X1 = c(151, 92,..., 124, 246) > X2 = c( rep(0,10), rep(1,10) ) Scatterplot using > plot( Y ~ X1, pch=1+(18*x2) ) (next slide ) shows two separate scatterlines, one for each type of firm.

33 Plot shows dual linear relationship, indexed by type of firm Insur. Innov n Data (CH08TA02) Scatterplot

34 Insur. Innov n (CH08TA02) (cont d) Equal-slopes ANCOVA in R: > CH08TA02.lm = lm( Y ~ X1 + X2 ) > summary( CH08TA02.lm ) Call: lm(formula = Y ~ X1 + X2) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-13 X e-09 X e-05 Multiple R-squared: , Adjusted R-squ.: F-statistic: 72.5 on 2 and 17 DF, p-value: 4.77e-09

35 Insur. Innov n (CH08TA02) (cont d) ANOVA table (with sequential SSRs): > CH08TA02.lm = lm( Y ~ X1 + X2 ) > anova( CH08TA02.lm ) Analysis of Variance Table Response: Y Df Sum Sq Mean Sq F value Pr(>F) X e-09 X e-05 Resid Partial F* = for X 2 (P = ), so the two types are signif. different (cf. Table 8.3).

36 Insur. Innov n (CH08TA02) (cont d) Pointwise conf. intervals from ANCOVA: > CH08TA02.lm = lm( Y ~ X1 + X2 ) > confint( CH08TA02.lm ) 2.5 % 97.5 % (Intercept) X X So, e.g., if interest is in effect of type of firm (X 2 ), we see stock firms take between 4.98 β months longer to adopt the innovation.

37 Insur. Innov n Data (CH08TA02) (cont d) Scatterplot with separate, equal-slope lines overlaid (cf. Fig. 8.12)

38 If more than 2 levels are represented by the qualitative factor, just include more (parallel) lines: one line for each level of the factor. See, e.g., Fig Multiple-Level ANCOVA

39 Unequal-Slopes ANCOVA How to incorporate differential slopes in a (two-factor/two-level) ANCOVA? Easy! Just add an X 1 X 2 interaction term: E{Y} = β 0 + β 1 X 1 + β 2 X 2 + β 3 X 1 X 2. When X 2 = 0, E{Y} = β 0 + β 1 X 1, an SLR on X 1 with slope β 1 and Y-intercept β 0. When X 2 = 1, E{Y} = (β 0 + β 2 ) + (β 1 + β 3 )X 1, an SLR on X 1 with new slope (β 1 + β 3 ) and new Y-intercept (β 0 + β 2 ).

40 Unequal-Slopes ANCOVA For instance, with the Insur. Innov n Data (CH08TA02), Fig conceptualizes the unequal-slopes model

41 Insur. Innov n (CH08TA02) (cont d) Unequal-slopes ANCOVA in R, via interaction term and * operator: > summary( lm(y ~ X1*X2) ) Call: lm(formula = Y ~ X1 * X2) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-10 X e-07 X X1:X Multiple R-squared: , Adjusted R-squ.: F-statistic: on 3 and 16 DF, p-val.: 4.68e-08

42 Insur. Innov n (CH08TA02) (cont d) ANOVA table (with sequential SSRs) for unequalslopes ANCOVA model: > anova( lm(y ~ X1*X2) ) Analysis of Variance Table Response: Y Df Sum Sq Mean Sq F value Pr(>F) X e-09 X e-05 X1:X Resid X1*X2 interaction P-value >.05, so no signif. departure from equal slopes is indicated (cf. Table 8.4)

43 8.6: Multi-Factor ANCOVA The ANCOVA model can be extended to more than one quantitative X-variable. The concepts are essentially unchanged, just in a higher-dimensional space: the hyperplanes are all parallel and the qualitative predictor changes locations of the hyper-intercepts. Sounds trickier, but not really that different and not much harder to program.

44 Only Qualitative Predictors What if all the predictor variables are qualitative (0-1) indicators? In effect, the model structure is more circumspect, since we are now just comparing the mean responses across the levels of each qualitative factor. This is known as ANOVA modeling, and is studied in STAT 571B.

45 8.7: Comparing Multiple Regression Curves Let s do a fully coordinated example of how to compare two regression functions. Example: Production Line Data (CH08TA05) with Y = Soap Production Scrap X 1 = Product n Line Speed X 2 = Line Indicator (Line 1 vs. Line 2) Start with a scatterplot

46 Plot shows dual linear relationship, indexed by prod n line Sec. 8.7: Product n Line Data (CH08TA05) Scatterplot

47 Product n Line Data (CH08TA05) (cont d) Unequal-slopes ANCOVA in R, via interaction term and * operator: > summary( lm(y ~ X1*X2) ) Call: lm(formula = Y ~ X1 * X2) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) X e-13 X X1:X Multiple R-squ.: , Adjusted R-squ.: F-statistic: on 3 and 23 DF, p-val.: 1.34e-14

48 Product n Line Data (CH08TA05) (cont d) Per-line residual plots (cf. Fig. 8.17):

49 Product n Line Data (CH08TA05) (cont d) Brown-Forsythe test for equal σ 2 between the two product n lines shows insignif. P = 0.53: > library( lawstat ) > BF.htest = levene.test( resid( CH08TA05.lm ), group=x2, location= median ) modified robust Brown-Forsythe Levene-type test based on the absolute deviations from the median data: ei Test Statistic = , p-value = > sqrt( BF.htest$statistic ) Test Statistic #BF t*-stat. (cf. p.333)

50 Product n Line Data (CH08TA05) (cont d) ANOVA table (with sequential SSRs) for unequalslopes model: > anova( lm(y ~ X1*X2) ) Analysis of Variance Table Response: Y Df Sum Sq Mean Sq F value Pr(>F) X e-15 X e-06 X1:X Residuals (cf. Table 8.6)

51 Product n Line Data (CH08TA05) (cont d) ANOVA partial F-test for identity of lines (H o :β 2 =β 3 =0): > anova( lm(y ~ X1), lm(y ~ X1*X2) ) Analysis of Variance Table Model 1: Y ~ X1 Model 2: Y ~ X1 * X2 Res.Df RSS Df Sum of Sq F Pr(>F) e-06 2 d.f. partial F* = (P = ), so two lines are significantly different (somehow).

52 Product n Line Data (CH08TA05) (cont d) ANOVA partial F-test for identity of slopes (H o :β 3 = 0): > anova( lm(y ~ X1+X2), lm(y ~ X1*X2) ) Analysis of Variance Table Model 1: Y ~ X1 + X2 Model 2: Y ~ X1 * X2 Res.Df RSS Df Sum of Sq F Pr(>F) d.f. partial F* = 1.88 (P = ), so two lines have insignificantly different slopes. (NB: Should adjust the 2 inferences on β 2 and β 3 for multiplicity.)