Regression Pitfalls. Pitfall Noun: A hidden or unsuspected danger or difficulty. A covered pit used as a trap.

Transcription

1 Regression Pitfalls Pitfall Noun: A hidden or unsuspected danger or difficulty. A covered pit used as a trap. Multiple regression is a widely used and powerful tool. It is also one of the most abused statistical techniques. 1 / 17 Some Regression Pitfalls Introduction

2 Observational versus Experimental Data Recall: In some investigations, the independent variables x 1, x 2,..., x k can be controlled; that is, held at desired values. The resulting data are called experimental. In other cases, the independent variables cannot be controlled, and their values are simply observed. The resulting data are called observational. 2 / 17 Some Regression Pitfalls Observational vs Experimental Data

3 Observational example Cocaine Use During Pregnancy Linked To Development Problems Two groups of new mothers, 218 used cocaine during pregnancy, 197 did not. IQ tests of infants at age 2 showed lower scores for children of users. Correlation does not imply causation. 3 / 17 Some Regression Pitfalls Observational vs Experimental Data

4 The study does not show that cocaine use causes development problems. It does show association, which might be used in prediction. For instance, it could help identify children at high risk of having development problems. 4 / 17 Some Regression Pitfalls Observational vs Experimental Data

5 Experimental example Animal-assisted therapy. 76 heart patients randomly assigned to three therapies: T: visit from a volunteer and a trained dog; V: visit from a volunteer only; C: no visit. Response y is decrease in anxiety. 5 / 17 Some Regression Pitfalls Observational vs Experimental Data

6 Result: ȳ T = 10.5, ȳ V = 3.9, ȳ C = 1.4. Model: E(Y ) = β 0 + β 1 x 1 + β 2 x 2, where x 1 is the indicator variable for group T and x 2 is the indicator variable for group V. The model-utility F -test shows significant differences among groups. Because of random assignment, the differences can be assumed to be caused by the treatments. 6 / 17 Some Regression Pitfalls Observational vs Experimental Data

7 Parameter Estimability ST 430/514 Recall The normal equations X Xˆβ = X y that define least squares parameter estimates always have a solution. But if X X is singular, they have many solutions. An individual parameter that is not uniquely estimated is called nonestimable. 7 / 17 Some Regression Pitfalls Parameter Estimability

8 Example The animal-assisted therapy data. Suppose we tried to fit the model E(Y ) = β 0 + β 1 x 1 + β 2 x 2 + β 3 x 3, where x 3 is the third indicator variable, for group C. One solution is ˆβ 0 = 0, ˆβ 1 = ȳ T = 10.5, ˆβ 2 = ȳ V = 3.9, ˆβ 3 = ȳ C = 1.4. The more usual solution is ˆβ 0 = ȳ C = 1.4, ˆβ 1 = ȳ T ȳ C = 9.1, ˆβ 2 = ȳ V ȳ C = 2.5, ˆβ 3 = 0. 8 / 17 Some Regression Pitfalls Parameter Estimability

9 All estimates change, so no parameter is estimable. The conventional solution is to leave out one variable, or equivalently to constrain one parameter to be zero. Another possibility is to constrain β 1 + β 2 + β 3 = 0, which is appealing in its symmetry, but rarely used in practice. In more complex cases, estimability may be harder to understand. 9 / 17 Some Regression Pitfalls Parameter Estimability

10 Multicollinearity Two independent variables are orthogonal if their sample correlation coefficient is zero. If all pairs of independent variables are orthogonal, X X is diagonal, and the normal equations are trivial to solve. In a controlled experiment, the variables are often orthogonal by design. If some pairs are far from orthogonal, the equations may be nearly singular. 10 / 17 Some Regression Pitfalls Multicollinearity

11 If X X is nearly singular, its inverse (X X) 1 exists but will have large entries. So the least squares estimates ˆβ = (X X) 1 X y are very sensitive to small changes in y. That makes their standard errors large. 11 / 17 Some Regression Pitfalls Multicollinearity

12 Example Carbon monoxide from cigarettes ST 430/514 cigar <- read.table("text/exercises&examples/ftccigar.txt", header = TRUE) pairs(cigar) cor(cigar) summary(lm(co ~ TAR, cigar)) summary(lm(co ~ TAR + NICOTINE + WEIGHT, cigar)) The standard error of ˆβ TAR increases nearly five-fold when NICOTINE is added to the model. Note the negative coefficients for NICOTINE and WEIGHT. But both are positively correlated with CO. 12 / 17 Some Regression Pitfalls Multicollinearity

13 Multicollinearity is sometimes measured using the Variance Inflation Factor (VIF). For variable x i, the VIF is VIF i = 1 1 R 2 i 1, where Ri 2 is the coefficient of determination in the regression of x i on the other independent variables {x j, j i}. VIF i is related to the increase in the standard error of ˆβ i when the other variables are included. VIF i = 1 if x i is orthogonal to the other independent variables. 13 / 17 Some Regression Pitfalls Multicollinearity

14 Extrapolation A regression model is an approximation to the complexities of the real world. It may fit the sample data well. If it fits well, it will usually give a reliable prediction for a new context that is similar to those in the sample data. With several variables, deciding when the new context is too different for reliable prediction may be difficult, especially in the presence of multicollinearity. 14 / 17 Some Regression Pitfalls Extrapolation

15 Transformation In many problems, one or more of the variables (dependent and independent) may be measured and recorded in a form that is not the best from a modeling perspective. Linear transformations are usually pointless, as a linear model is essentially unchanged by it. Among nonlinear transformations, logarithms are most widely useful, followed by powers of the variables. 15 / 17 Some Regression Pitfalls Variable Transformation

16 The primary goal of transformation is to find a good approximation to the way E(Y ) depends on x. Another goal is to make the variance of the random error reasonably constant. ɛ = Y E(Y ) Finally, if a transformation makes ɛ approximately normally distributed, that is worth achieving. 16 / 17 Some Regression Pitfalls Variable Transformation

17 Example 7.8 Impact of price of coffee on demand: coffee <- read.table("text/exercises&examples/coffee.txt", header = TRUE) with(coffee, plot(price, DEMAND)) Example 7.8 models Y (DEMAND) against p 1, where p = PRICE. We could also consider log(y ) and log(p), as well as other powers of p. 17 / 17 Some Regression Pitfalls Variable Transformation