Lecture 20: Multiple linear regression

Transcription

1 Lecture 20: Multiple linear regression Statistics 101 Mine Çetinkaya-Rundel April 5, 2012

2 Announcements Announcements Project proposals due Sunday midnight: Respsonse variable: numeric Explanatory variables: at least one numeric and one at least categorical variable Total of at least 3 variables, more the merrier You might want to make a few quick plots of your dataset to see if the relationship between your response and explanatory variables are linear (or could be made liner using transformations). If the relationship is not linear, you may want to consider another dataset otherwise your findings will be inconclusive. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

3 Recap Review question One waiter recorded information about each tip he received over a period of a few months. Below is the regression model for predicting tip amount from total bill amount (both in $). Which of the below is correct? (Intercept) total bill (a) For each $1 increase in total bill amount, we would expect tip to increase on average by 92 cents. (b) The regression model is tip = total bill. (c) For a bill amount of 0 dollars we would expect the tip amount to be 11 cents on average. (d) The explanatory variable is tips and the response variable is total bill amount. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

4 1 Multiple regression Categorical variables with two levels Many variables in a model Adjusted R 2 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, 2012

5 Multiple regression Simple linear regression: Bivariate - two variables: y and x Multiple linear regression: Multiple variables: y and x 1, x 2, Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

6 Categorical variables with two levels GPA vs. Greek Relationship between Greek organization or an SLG and GPA based on class survey: 4.0 gpa no yes greek Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

7 Categorical variables with two levels GPA vs. Greek - linear model gpa_greek = lm(gpa greek, data = survey) summary(gpa_greek) Call: lm(formula = gpa greek, data = survey) Residuals: Min 1Q Median 3Q Max Coefficients: (Intercept) < 2e-16 *** greekyes ** --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 203 degrees of freedom (13 observations deleted due to missingness) Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 203 DF, p-value: Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

8 Categorical variables with two levels GPA vs. Greek - linear model (cont.) (Intercept) greek:yes The variable greek has two levels: yes and no no is the reference level, hence not shown on the output Linear model: ĝpa = greek : yes Intercept: The estimated mean GPA of students who do not belong to a Greek organization is This is the value we get if we plug in 0 for the explanatory variable Slope: The estimated mean GPA of students who belong to a Greek organization is 0.11 higher than those who do not Then, the estimated mean GPA of students who do belong to a Greek organization is = 3.64 This is the value we get if we plug in 1 for the explanatory variable Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

9 Categorical variables with two levels GPA vs. Greek - inference (Intercept) greek:yes Hypotheses: H 0 : β 1 = 0 H A : β 1 0 p value = 0.01 The data provide convincing evidence that the true slope parameter is different than 0, hence there appears to be statistically significant relationship between belonging to a Greek organization or an SLG and GPA Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

10 Categorical variables with two levels Another approach? Clicker question What other approach could we use to evaluate the relationship between belonging to a Greek organization or an SLG and GPA? Remember: 205 students reported their GPA, 87 belong to a Greek organization or an SLG and 118 do not. (a) chi-squared test of independence (b) chi-squared test of goodness-of-fit (c) test for comparing means of dependent groups (d) test for comparing means of independent groups (e) test for comparing proportions of independent groups Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

11 Categorical variables with two levels Another example... Relationship between how politically active students are (on a scale of 1 to 5) and whether or not they are pro-life or pro-choice: 5 politic_active pro choice pro life life_choice Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

12 Categorical variables with two levels Identifying the reference level Clicker question Based on the regression output given below, what is the reference level of the variable life choice? (Intercept) life choice:pro-life (a) pro-choice (b) pro-life (c) neither (d) cannot tell from the information given Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

13 Categorical variables with two levels Identifying the reference level Clicker question Based on the regression output, which of the below is correct? (Intercept) life choice:pro-life The estimated mean political activity level of students who are (a) pro-choice is 0.06 higher than those who are pro-life (b) pro-choice is 0.06 lower than those who are pro-life (c) pro-life is 0.06 lower than those who are pro-choice (d) pro-life is 0.06 higher than those who are pro-choice Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

14 Categorical variables with two levels Identifying the reference level Clicker question Does whether or not a student is pro-life or pro-choice have a statistically significant relationship with their political activity level? (a) no (b) yes (Intercept) life choice:pro-life (c) cannot tell from the information given Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

15 Many variables in a model Weights of books weight (g) volume (cm 3 ) cover hc hc hc hc hc hc hc pb pb pb pb pb pb pb pb l w h Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

16 Many variables in a model Weights of books (cont.) Clicker question The scatterplot shows the relationship between weights and volumes of books as well as the regression output. Which of the below is correct? weight (g) weight = volume R 2 = 80% volume (cm 3 ) (a) Weights of 80% of the books can be predicted accurately using this model. (b) Books that are 10 cm 3 over average are expected to weigh 7 g over average. (c) The correlation between weight and volume is R = = (d) The model underestimates the weight of the book with the highest volume. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

17 Many variables in a model If you d like to replicate the analysis in R... install.packages("daag") library(daag) data(allbacks) From: Maindonald, J.H. and Braun, W.J. (2nd ed., 2007) Data Analysis and Graphics Using R Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

18 Many variables in a model Modeling weights of books using volume somewhat abbreviated output... m1 = lm(weight volume, data = allbacks) summary(m1) Coefficients: (Intercept) volume e-06 Residual standard error: on 13 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 13 DF, p-value: 6.262e-06 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

19 Many variables in a model Weights of hard cover and paperback books Can you identify a trend in the relationship between volume and weight of hardcover and paperback books? 1000 hardcover paperback weight (g) volume (cm 3 ) Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

20 Many variables in a model Modeling weights of books using volume and cover type m2 = lm(weight volume + cover, data = allbacks) summary(m2) Coefficients: (Intercept) ** volume e-08 *** cover:pb *** Residual standard error: 78.2 on 12 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 2 and 12 DF, p-value: 1.455e-07 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

21 Many variables in a model Determining the reference level Clicker question Based on the regression output below, which level of cover is the reference level? Note that pb: paperback. (Intercept) volume cover:pb (a) paperback (b) hardcover Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

22 Many variables in a model Determining the reference level Clicker question Which of the below correctly describes the roles of variables in this regression model? (Intercept) volume cover:pb (a) response: weight, explanatory: volume, paperback cover (b) response: weight, explanatory: volume, hardcover cover (c) response: volume, explanatory: weight, cover type (d) response: weight, explanatory: volume, cover type Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

23 Many variables in a model Linear model (Intercept) volume cover:pb weight = volume cover : pb 1 For hardcover books: plug in 0 for cover weight = volume = volume 2 For paperback books: plug in 1 for cover weight = volume = volume Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

24 Many variables in a model Visualising the linear model 1000 hardcover paperback weight (g) volume (cm 3 ) Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

25 Many variables in a model Interpretation of the regression coefficients (Intercept) volume cover:pb Slope of volume: All else held constant, for each 1 cm 3 increase in volume we would expect weight to increase on average by 0.72 grams. Slope of cover: All else held constant, the model predicts that paperback books weigh 184 grams lower than hardcover books. Intercept: Hardcover books with no volume are expected on average to weigh 198 grams. Obviously, the intercept does not make sense in context. It only serves to adjust the height of the line. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

26 Many variables in a model Prediction Clicker question Which of the following is the correct calculation for the predicted weight of a paperback book that is 600 cm 3? (Intercept) volume cover:pb (a) * * 1 (b) * * 1 (c) * * 0 (d) * * 600 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

27 Many variables in a model Another example: Modeling kid s test scores Predicting cognitive test scores of three- and four-year-old children using characteristics of their mothers. Data are from a survey of adult American women and their children - a subsample from the National Longitudinal Survey of Youth. kid score mom hs mom iq mom work mom age 1 65 yes yes yes yes no no yes yes 25 Gelman, Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models. (2007) Cambridge University Press. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

28 Many variables in a model Interpreting the slope What is the correct interpretation of the slope for mom s IQ? (Intercept) mom hs:yes mom iq mom work:yes mom age Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

29 Many variables in a model Interpreting the slope What is the correct interpretation of the intercept? (Intercept) mom hs:yes mom iq mom work:yes mom age Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

30 Many variables in a model Interpreting the slope Clicker question What is the correct interpretation of the slope for mom work? (Intercept) mom hs:yes mom iq mom work:yes mom age All else being equal, kids whose moms worked during the first three year s of the kid s life (a) are estimated to score 2.54 points lower (b) are estimated to score 2.54 points higher than those whose moms did not work. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

31 Adjusted R 2 Revisit: Modeling poverty poverty metro_res white hs_grad female_house Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

32 Adjusted R 2 Predicting poverty using % female householder pov_slr = lm(poverty female_house, data = poverty) (Intercept) female house % in poverty R = 0.53 R 2 = = % female householder Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

33 Adjusted R 2 Another look at R 2 R 2 can be calculated in three ways: 1 square the correlation coefficient of x and y (how we have been calculating it): > cor(poverty$poverty, poverty$female_house)ˆ2 [1] square the correlation coefficient of y and ŷ: > cor(poverty$poverty, pov_slr$fitted.values)ˆ2 [1] based on definition: R 2 = explained variability in y total variability in y Using ANOVA we can calculate the explained variability and total variability in y. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

34 Adjusted R 2 Sum of squares anova(pov_slr) Df Sum Sq Mean Sq F value Pr(>F) female house Residuals Total Sum of squares of y: SS Total = Sum of squares of residuals: SS Error = (y ȳ) 2 = total variability e 2 i = unexplained variability Sum of squares of x: SS Model = SS Total SS Error explained variability = = R 2 = explained variability total variability = = 0.28 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

35 Adjusted R 2 Why bother? Why bother with another approach for calculating R 2 when we had a perfectly good way to calculate it as the correlation coefficient squared? Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

36 Adjusted R 2 Predicting poverty using % female hh + % white Linear model: (Intercept) female house white ANOVA: Df Sum Sq Mean Sq F value Pr(>F) female house white Residuals Total R 2 = explained variability total variability = = 0.29 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

37 Adjusted R 2 Does adding the variable white to the model add valuable information that wasn t provided by female house? poverty metro_res white hs_grad female_house Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

38 Adjusted R 2 Collinearity between explanatory variables poverty vs. % female head of household (Intercept) female house poverty vs. % female head of household and % female hh (Intercept) female house white Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

39 Adjusted R 2 Collinearity between explanatory variables poverty vs. % female head of household (Intercept) female house poverty vs. % female head of household and % female hh (Intercept) female house white Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

40 Adjusted R 2 Collinearity between explanatory variables (cont.) Two predictor variables are said to be collinear when they are correlated, and this collinearity complicates model estimation. Remember: Predictors are also called explanatory or independent variables, so they should be independent of each other. We don t like adding predictors that are associated with each other to the mode, because often times the addition of such variable brings nothing to the table. Instead, we prefer the simplest best model, i.e. parsimonious model. While it s impossible to avoid collinearity from arising in observational data, experiments are usually designed to control for correlated predictors. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

41 R 2 vs. adjusted R 2 Multiple regression Adjusted R 2 R 2 Adjusted R 2 Model 1 (SLR) Model 2 (MLR) When any variable is added to the model R 2 increases. But if the added variable doesn t really provide any new information, or is completely unrelated, adjusted R 2 does not increase. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

42 Adjusted R 2 Multiple regression Adjusted R 2 Adjusted R 2 ( R 2 adj = 1 SSError n 1 ) SS Total n p 1 where n is the number of cases and p is the number of predictors (explanatory variables) in the model. Because p is never negative, R 2 adj will always be smaller than R2. R 2 adj applies a penalty for the number of predictors included in the model. Therefore, we choose models with higher R 2 adj over others. Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40

43 Calculate adjusted R 2 Multiple regression Adjusted R 2 ANOVA: Df Sum Sq Mean Sq F value Pr(>F) female house white Residuals Total ( R 2 SSError adj = 1 n 1 ) SS Total n p 1 ( = ) ( = ) 48 = = 0.26 Statistics 101 (Mine Çetinkaya-Rundel) L20: Multiple linear regression April 5, / 40