6. Multiple Linear Regression

Transcription

1 6. Multiple Linear Regression SLR: 1 predictor X, MLR: more than 1 predictor Example data set: Y i = #points scored by UF football team in game i X i1 = #games won by opponent in their last 10 games X i2 = #healthy starters for UF (out of 22) in game i i points X i1 X i

2 Simplest Multiple Linear Regression (MLR) Model: Y i = β 0 + β 1 X i1 + β 2 X i2 + ɛ i, i = 1, 2,..., n ɛ i iid N(0, σ 2 ) β 0, β 1, β 2, and σ 2 are unknown parameters X ij s are known constants. SLR: E(Y ) = β 0 + β 1 X β 1 is the change in E(Y ) corresponding to a unit increase in X. MLR: E(Y ) = β 0 + β 1 X 1 + β 2 X 2 When we have more than 1 predictor, we have to worry about how they affect each other. 2

3 Suppose we fix X i1 = 5 (games won by ith opponent): Suppose we fix X i1 = 7: E(Y i ) = β 0 + β 1 (5) + β 2 X i2 = (β 0 + β 1 (5)) + β 2 X i2 E(Y i ) = β 0 + β 1 (7) + β 2 X i2 = (β 0 + β 1 (7)) + β 2 X i2 We ve got SLR models with different intercepts but equal slopes. Plot of E(Y ) vs X 2 for fixed values of X 1 3

4 E(points) β 0 + β 1 (5) 1 1 β 2 β 2 opponent won 5/10 (β 0 + β 1 (5)) + β 2 X i2 opponent won 7/10 (β 0 + β 1 (7)) + β 2 X i2 By this model, we assumed that, for any fixed value of X i1 (opponent wins), the change in E(Y ) corresponding to the addition of 1 healthy starter is β 2 for all games. Is this reasonable? β 0 + β 1 (7) #healthy starters 4

5 Suppose AU is winless in their last 10 games. Our model says that if we add 1 healthy starter, we expect that UF scores β 2 more points. Suppose BU won their last 10 games. Again, if we add 1 healthy starter, we expect to score β 2 more points. Starters probably won t play against AU, so we expect to gain nothing if a starter becomes healthy. Maybe the plot should look like: 5

6 opponent won 5/10 E(points) 1 <β 2 β 2 opponent won 7/10 Smaller slope since starters are less important against bad teams. Q: How can we change our model to allow for this? 1 A: Add an interaction term #healthy starters 6

7 E(Y i ) = β 0 + β 1 X i1 + β 2 X i2 + β 3 X i1 X i2 This function is not a simple plane any more! When X i1 = 5: When X i1 = 7: E(Y i ) = (β 0 + β 1 (5)) + (β 2 + β 3 (5))X i2 E(Y i ) = (β 0 + β 1 (7)) + (β 2 + β 3 (7))X i2 7

8 opponent won 5/10 E(points) β 0 + β 1 (5) 1 1 β 2 + β 3 (5) opponent won 7/10 β 2 + β 3 (7) Now the gain in expected points corresponding to the addition of 1 healthy starter depends on X i1 as it should. β 1 < 0, β 2 > 0, β 3 > 0 β 0 + β 1 (7) #healthy starters 8

9 General Linear Regression Model Data (X i1, X i2,..., X i,p 1, Y i ), i = 1, 2,..., n Model Equation and Assumptions Y i = β 0 + β 1 X i1 + β 2 X i2 + + β p 1 X i,p 1 + ɛ i ɛ i iid N(0, σ 2 ) β 0, β 1, β 2,..., β p 1 and σ 2 are unknown param s X ij s are known constants. 9

10 Two cases: 1. p 1 different predictors 2. some of the predictors are functions of the others (a) polynomial regression Y i = β 0 + β 1 X i + β 2 Xi 2 + ɛ i Let Z i1 = X i and Z i2 = X 2 i then Y i = β 0 + β 1 Z i1 + β 2 Z i2 + ɛ i (b) interaction effects Y i = β 0 + β 1 X i1 + β 2 X i2 + β 3 X i1 X i2 + ɛ i Let X i3 = X i1 X i2 and we re back to the general linear regression model 10

11 (c) both of (a) and (b) Y i = β 0 + β 1 X i1 + β 2 X i2 + β 3 X 2 i1 + β 4 X 2 i2 + β 5 X i1 X i2 + ɛ i With Z i1 = X i1, Z i2 = X i2, Z i3 = X 2 i1, Z i4 = X 2 i2, Z i5 = X i1 X i2 this transforms to the general linear regression model Y i = β 0 + β 1 Z i1 + β 2 Z i2 + β 3 Z i3 + β 4 Z i4 + β 5 Z i5 + ɛ i 11

12 General Linear Model in Matrix Terms Y n 1 = Y 1 Y 2. Y n β p 1 = X n p = β 0 β 1. β p 1 1 X 11 X X 1,p 1 1 X 21 X X 2,p X n1 X n2... X n,p 1 ɛ n 1 = ɛ 1 ɛ 2. ɛ n 12

13 Model: Y = Xβ + ɛ Assumptions: ɛ N(0, σ 2 I) β and σ 2 are unknown parameters X is a (n p) matrix of fixed known constants 13

14 Least Squares Estimates: b p 1 = b 0 b 1. b p 1 = (X X) 1 X Y Fitted Values: Ŷ n 1 = Ŷ 1 Ŷ 2. Ŷ n = b 0 + b 1 X b p 1 X 1,p 1 b 0 + b 1 X b p 1 X 2,p 1. b 0 + b 1 X n b p 1 X n,p 1 = Xb 14

15 Residuals: e n 1 = Y Ŷ = Y Xb = Y X(X X) 1 X Y = (I H)Y with the (n n) hat matrix H = X(X X) 1 X 15

16 ANalysis Of VAriance Formulas are exactly the same. Remember SSTO = SSR + SSE n n n (Y i Ȳ )2 = (Ŷi Ȳ )2 + (Y i Ŷi) 2 i=1 i=1 but their degrees of freedom (df) change: SSTO still has n 1 df SSR now has p 1 because of the p param s in Ŷi SSE therefore has n p df i=1 16

17 ANOVA Table for MLR: Source variat. Sum of Squares (SS) df mean SS Regr. SSR = i (Ŷi Ȳ )2 p 1 SSR p 1 Error SSE = i (Y i Ŷi) 2 n p SSE n p Total SSTO = i (Y i Ȳ )2 n 1 17

18 Overall F-Test for Regression Relation H 0 : β 1 = β 2 = = β p 1 = 0 H A : not all β j (j = 1,..., p 1) equal zero. H 0 states that all predictors X 1,..., X p 1 are useless (no relation between Y and the set of X variables), whereas H A says that at least one is useful. Test Statistic F = MSR MSE Rejection Rule: reject H 0, if F > F (1 α; p 1, n p) Note: when p 1 = 1, this is the F-test for H 0 : β 1 = 0 in the SLR. 18

19 Coefficient of Multiple Determination: it s the same as in SLR s, R 2 = SSR SSTO = 1 SSE SSTO It measures the relative reduction in the total variation (SSTO) due to the MLR. 19

20 Inferences about Regression Parameters Since with C p n = (X X) 1 X we can write b = (X X) 1 X Y = c c 1n. c p1... c pn Y 1. Y n Thus, every element of b is a linear combination of the Y s and is therefore a normal r.v. Again E(b) = (X X) 1 X E(Y) = β Thus b is an unbiased estimator for β. Moreover Var(b) = σ 2 (X X) 1 20

21 This means that for any k = 0, 1,..., p 1 we have b k N ( β k, σ 2 [ (X X) 1] ) k+1,k+1 where [ ] jj is the jth diagonal element of the matrix. 21

22 Thus σ 2 b k β k ] [(X X) 1 and because the MSE now has df = n p k+1,k+1 N(0, 1) MSE b k β k ] [(X X) 1 k+1,k+1 t(n p) Using this we can construct tests and CI s for each individual β k Test Statistic: t = MSE b k [(X X) 1 ] Rejection Rule: reject H 0 if t > t(1 α/2; n p) 22 k+1,k+1

23 (1 α)100% CI for the parameter β k b k ± t(1 α/2; n p) MSE [(X X) 1 ] k+1,k+1 (1 α)100% CI for the mean of Y at X h = (1 X h1 X h2... X h,p 1 ) Say we want a CI for the mean #points scored by UF when the opponent win 90% (X h1 = 9) and there are 20 healthy starters (X h2 = 20). So X h = (1 9 20) The point estimate of E(Y h ) = X h β is Ê(Y h ) = Ŷh = X hb Because this equals X h (X X) 1 X hy, it is a linear combination of normals and is thus normal with E(Ê(Y h)) = X h E(b) = X hβ 23

24 (unbiased) and Var(Ê(Y h)) = X h Var(b) X h = σ 2 X h(x X) 1 X h Thus and Ê(Y h ) X h β σ2 X h (X X) 1 X h N(0, 1) Ê(Y h ) X h β MSE X h (X X) 1 X h t(n p) The CI for X hβ is constructed in the usual manner. (1 α)100% Prediction Interval for a New Observation at X h = (1 X h1 X h2... X h,p 1 ) 24

25 Call the new observation Y h(new) and use Y h(new) Ê(Y h(new)) } MSE {1 + X h (X X) 1 X h t(n p) with Ê(Y h(new) ) = X hb 25

26 House Price Example using R > houses <- read.table("houses.dat", col.names = + c("price", "area", "bed", "bath", "new")) > attach(houses) > plot(area, price); plot(bed, price) 26

27 price price area bed 27

28 > model <- lm(price ~ area + bed) > model Coefficients: (Intercept) area bed > model.i <- lm(price ~ area + bed + area*bed) > summary(model.i, corr=t) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) area ** bed area:bed Residual standard error: on 89 df Multiple R-Squared: 0.814, Adjusted R-squared: F-statistic: on 3 and 89 df, p-value: 0 28

29 price price area area 29

30 > anova(model.i) Analysis of Variance Table Response: price Df Sum Sq Mean Sq F value Pr(>F) area < 2e-16 *** bed area:bed Residuals Sig.codes: 0 *** ** 0.01 *