Advanced Regression Topics: Violation of Assumptions

Transcription

1 Advanced Regression Topics: Violation of Assumptions Lecture 7 February 15, 2005 Applied Regression Analysis Lecture #7-2/15/2005 Slide 1 of 36

2 Today s Lecture Today s Lecture rapping Up Revisiting residuals. Outliers aside, what other things are important to look for: Nonconstant variance... Lecture #7-2/15/2005 Slide 2 of 36

3 Snow Geese Snow geese Regression Analysis Assumptions Residual Plot rapping Up From eisberg (1985, p. 102): Aerial survey methods are regularly used to estimated the number of snow geese in their summer range areas west of Hudson Bay in Canada. To obtain estimates, small aircraft fly over the range and, when a flock of geese is spotted, an experienced person estimates the number of geese in the flock. To investigate the reliability of this method of counting, an experiment was conducted in which an airplane carrying two observers flew over 45 flocks, and each observer made an independent estimate of the number of birds in each flock. Also, a photograph of the flock was taken so that an exact count of the number of birds in the flock could be made (data by Cook and Jacobsen, 1978). Lecture #7-2/15/2005 Slide 3 of 36

4 Snow Geese Lecture #7-2/15/2005 Slide 4 of 36

5 Hudson Bay Lecture #7-2/15/2005 Slide 5 of 36

6 Regression Analysis Snow geese Regression Analysis Assumptions Residual Plot Using the first observer in the plane, we consider the relationship between this person s count and that from the photograph: rapping Up photo count observer 1 count Lecture #7-2/15/2005 Slide 6 of 36

7 Regression Analysis Snow geese Regression Analysis Assumptions Residual Plot One way of analyzing these data is to fit a regression that attempts to predict the count in the photo from the count by the observer. Using SPSS, this regression was estimated, giving the following statistics: rapping Up Coefficient Estimate SE t p-value a - intercept b - slope < Statistic Estimate SS reg 254, SS res 84, R Lecture #7-2/15/2005 Slide 7 of 36

8 Assumptions Snow geese Regression Analysis Assumptions Residual Plot But, remember, before we can interpret these results, we must first check our assumptions. Assumptions of regression analyses revolve around the residuals, e = Y Y = Y (a + bx). rapping Up In particular, we specified that all residuals were: Independent (or non-correlated). Identically distributed. Distribution was normal, with: A zero mean. A constant variance. e N(0, σ 2 e) Lecture #7-2/15/2005 Slide 8 of 36

9 Residual Plot Snow geese Regression Analysis Assumptions Residual Plot From a previous lecture, recall that an easy way to check assumptions was to look at a plot of the standardized residuals against the unstandardized predicted values: rapping Up Standardized Residual Unstandardized Predicted Value Do you notice any problems from this plot? Lecture #7-2/15/2005 Slide 9 of 36

10 Detection Example Test Remedies Alternate Estimation Transformations One of the primary assumptions in a linear regression is that var(e i ) = σ 2 e for all i = 1,...,N observations. Standardized Residual rapping Up Unstandardized Predicted Value In our example, this assumption is clearly violated. Lecture #7-2/15/2005 Slide 10 of 36

11 Detection Detection Example Test Remedies Alternate Estimation Transformations rapping Up Detecting nonconstant variance is often accomplished by examining the residual plot. However, using visual inspection can lead to some problems: Subjective interpretation, relying on experience. How much is too much? Nonconstant variance is really a matter of degree. For this reason, one can construct a statistical hypothesis test for the constancy of variance. Lecture #7-2/15/2005 Slide 11 of 36

12 Detection Test Detection Example Test Remedies Alternate Estimation Transformations rapping Up Note that var(e i ) is caused by: The response, y. The predictors X. Some other quantity not involved in the regression, such as: Observations over time. Observations related by space (spatial orientation). Any (or all) of these features can be put into a large matrix, Z, so each observation i has a row vector z i. Lecture #7-2/15/2005 Slide 12 of 36

13 Detection Test Detection Example Test Remedies Alternate Estimation Transformations rapping Up Given our suspected cause of nonconstancy of variance for each observation z i, we can assume: var(e i ) = σ 2 e[exp(λ z i )] This is a very technical way of making variance a function of other variables. This form specifies the following constraints on our variance: 1. var(e i ) > 0 for all observations z i. 2. The variance depends on z i and λ, but only because in the linear function λ z i. 3. var(e i ) is monotonic (either increasing or decreasing) across each component of z i. 4. If λ = 0, then var(e i ) = σ 2 e for all i. Lecture #7-2/15/2005 Slide 13 of 36

14 Detection Test: Steps Detection Example Test Remedies Alternate Estimation Transformations rapping Up STEP 0: Determination of what is causing nonconstant variance. Let s go back to our geese data example, and assume the nonconstancy in variance was due to the predictor. Human s have a more difficult time detecting the number of geese consistently as the number observed gets very large. Because we feel that the nonconstancy in variability is caused by our predictor variables, we will construct Z from X. Note that X has a column vector of ones for the intercept. Lecture #7-2/15/2005 Slide 14 of 36

15 Detection Test: Steps Detection Example Test Remedies Alternate Estimation Transformations 1. Estimate regression line for original model (Y = a + bx), and save the unstandardized residuals for each observation: ê i = Y i (â + ˆbX i ). 2. For each observation, compute scaled squared residuals, u i : rapping Up u i = ê2 i σ 2 e, where σ 2 e is the ML estimate of σ 2 e (differing from MS error because of denominator of N rather than N k 1): σ 2 e = N i=1 ê2 i N Lecture #7-2/15/2005 Slide 15 of 36

16 Detection Test: Steps Detection Example Test Remedies Alternate Estimation Transformations 3. Compute the regression of u i onto z i. Obtain from this regression the SS reg. Obtain the df reg, where this is the number of predictors in Z (not including the intercept). rapping Up 4. Compute the Score statistic (using SS reg from step 3): S = SSreg 2 5. Test the hypothesis that λ = 0 by obtaining a p-value for S, which is distributed χ 2 (df reg ) where df reg is from step 3. Lecture #7-2/15/2005 Slide 16 of 36

17 From Our Example Detection Example Test Remedies Alternate Estimation Transformations rapping Up 1. Original regression estimates found from SPSS: Analyze...Regression...Linear. Save unstandardized residuals from Save button menu. 2. For each observation, compute scaled squared residuals, u i = ê2 i σ : e 2 Compute σ 2 e = N i=1 ê2 i N. In SPSS: Transform...Compute, and make a new variable that is the squared value of the unstandardized residual. Then find average of the new variable from Analyze...Descriptive Statistics...Descriptives. Alternative: take SS res from step 1 output and divide by N. σ 2 e = 1, Compute u i in SPSS by going to Transform...Compute. Lecture #7-2/15/2005 Slide 17 of 36

18 From Our Example Detection Example Test Remedies Alternate Estimation Transformations rapping Up 3. Compute the regression of u i onto z i. In SPSS: Analyze...Regression...Linear. 4. Compute the Score statistic S = SSreg 2, using SS reg from step 3. Get this from SPSS output. S = /2 = This has df reg = Get p-value for S. In Excel type =chidist(81.41,1). p < Based on these results, we reject the null hypothesis of constant variance, and conclude that our example violates the constant variance assumption. Lecture #7-2/15/2005 Slide 18 of 36

19 ...Now hat? Detection Example Test Remedies Alternate Estimation Transformations rapping Up e found in our example that we have statistical evidence for nonconstant variance. The biggest result of nonconstant variance is that our regression line does not accurately represent all cases in our sample. Also a problem is that the hypothesis tests we use are based on the assumption that e i N(0, σ 2 e). hen nonconstant variance is found, two options are possible: 1. Estimate the regression using alternate methods. eighted least squares. Median regression. 2. Transform either the response or the predictors. Lecture #7-2/15/2005 Slide 19 of 36

20 Remedy #1: Alternate Estimation Algorithms Detection Example Test Remedies Alternate Estimation Transformations rapping Up Much like the mean is extremely sensitive to highly skewed (outlying) observations, least squares estimates have what is called a high breakdown point. Instead of finding regression parameters that minimize: N (Y Y ) 2, i=1 alternative optimization criteria exist. Lecture #7-2/15/2005 Slide 20 of 36

21 Alternate Estimation Algorithms Detection Example Test Remedies Alternate Estimation Transformations Two possible alternatives: eighted Least Squares (LS): N w i (Y Y ) 2 i=1 Can be performed in SPSS. rapping Up Minimum absolute deviation: Much more technical. N Y Y i=1 Simplex optimization method involves linear programming. Lecture #7-2/15/2005 Slide 21 of 36

22 Remedy #2: Variance Stabilizing Transformations Detection Example Test Remedies Alternate Estimation Transformations The second (and perhaps most commonly used) remedy for nonconstant variance is to transform the response variable Y. (eisberg, 1985; p. 134) Transformation Situation Reason Y var(ei ) E(Y i ) Poisson counts rapping Up Y + Y + 1 Poisson, small Y values ln(y ) var(e i ) [E(Y i )] 2 Broad range of Y ln(y + 1) Some Y are zero 1 Y var(e i ) [E(Y i )] 4 Y bunched near zero 1 (Y +1) Some Y are zero sin 1 ( Y ) var(e i ) E(Y i )(1 E(Y i )) Binomial proportions Lecture #7-2/15/2005 Slide 22 of 36

23 Nonlinear Relationship Between Predictor(s) and Y Detection Remedy Situations occur where a non-linear relationship between predictors and Y is present. For example, imagine that the true relationship between Y and X is something like: rapping Up Y = ax b To use linear regression techniques we are well aware of, this function must be transformed (both Y and X): ln(y ) = ln(a) + b ln(x) Lecture #7-2/15/2005 Slide 23 of 36

24 Nonlinear Relationship Between Predictor(s) and Y Depending on the situation, not all functions can be made linear: Detection Remedy rapping Up Y = a 1 e b 1X 1 + a 2 e b 2X 2 Furthermore, depending on the error dependency (multiplicative or additive), transformations will not lead to errors with the distributional assumptions of linear regression. Linear regression can only go so far, so if data have a functional relationship other methods may be better suited. Lecture #7-2/15/2005 Slide 24 of 36

25 Standardized Residual Detection of Often, nonlinearity is detected visually, through use of the residual plots. Detection Remedy rapping Up Unstandardized Predicted Value Lecture #7-2/15/2005 Slide 25 of 36

26 Possible Remedy for Detection Remedy As discussed, a possible remedy for nonlinearity is to use a transformation of both Y and X. Situations occur where a non-linear relationship between predictors and Y is present. rapping Up (eisberg, 1985; p. 142) Y Transformation X Transformation Form ln(y ) ln(x) Y = ax b 1 1 Xb Xb k k ln(y ) X Y = ae b 1 X 1 +b 2 X b k X k Y ln(x) Y = a + b 1 ln(x 1 ) + b 2 ln(x 2 ) ln(b k )X k 1 1 Y = 1 Y X a+(b 1 /X 1 )+(b 2 /X 2 )+...+(b k /X k ) 1 X Y = 1 Y a+(b 1 X 1 )+(b 2 X 2 )+...+(b k X k ) Y 1 Y = a + (b 1 X 1 + b 1 X b 1 X2 k Xk Lecture #7-2/15/2005 Slide 26 of 36

27 Parameterizing Transformations Transformation Parameter 1 Transformation Parameter 2 Instead of choosing some type of transformation function seemingly arbitrarily, statistical techniques have been developed to transform both Y and the set of all predictors X based on known functions. These techniques bear mention because from time to time you will encounter estimates based on these. rapping Up Furthermore, a clear functional relationship between Y and X may not be known, either from substantive theory or empirical results. In these situations, linear methods are often easier to rely upon because of parsimony (real or perceived). Lecture #7-2/15/2005 Slide 27 of 36

28 Transformation of Y : Optimization Consider the family of regression models (power models): y λ = Xb + e. Transformation Parameter 1 Transformation Parameter 2 rapping Up Finding λ gives an idea of the power relationship between the response and the predictor variables. Such transformations are often referred to as Box-Cox transformations, and involve iterative techniques to find the most likely value of λ. Another transformation method is called Atkinson s score method. Lecture #7-2/15/2005 Slide 28 of 36

29 Transformation of X: Optimization Similar methods have been developed for transforming X. Transforming X is inherently more difficulty. Transformation Parameter 1 Transformation Parameter 2 rapping Up Often times absurd results can be the outcome. Many times non-significant relationships obscure any needed transformations. Lecture #7-2/15/2005 Slide 29 of 36

30 The last assumption to be checked for is the normality of the residuals. Detecting nonnormality can be tricky, and is often based upon sample size. Detection 1: Plots Detection 2: Tests Special Cases rapping Up Statistically speaking, there is not a hypothesis test that can conclude that a variable is normally distributed. Violations of this assumption lead to inaccurate p-values in hypothesis tests (only). The p-values, however, are fairly robust to violations of this assumption. Lecture #7-2/15/2005 Slide 30 of 36

31 Detection of : Probability Plots Detection 1: Plots Detection 2: Tests Special Cases rapping Up The easiest way to detect nonnormal errors is to use a Q-Q plot. A Q-Q plot is a plot of an ordered variable against what its expected value should be for a variable from a normal distribution with size N. In SPSS: Graphs...Q-Q (check Standardize values and be sure Test distribution is Normal). The plot of the data should fall on the line produced on the plot. If not, the data are not from a normal distribution. Lecture #7-2/15/2005 Slide 31 of 36

32 Detection of : Probability Plots Detection 1: Plots Detection 2: Tests Special Cases rapping Up Lecture #7-2/15/2005 Slide 32 of 36

33 Detection of : Hypothesis Tests Detection 1: Plots Detection 2: Tests Special Cases rapping Up Additionally, statistical hypothesis tests have been developed to test the null hypothesis that the data (in this case the residuals) can from a normal distribution. Shapiro-ilk test. Kolmogorov-Smirnov test. In SPSS: get the unstandardized residuals and go to Analyze...Descriptive Statistics...Explore. Put the residuals in the Dependent List box. Click on Plots and check Normality plots with tests". If p-value is less than some α, then reject the null hypothesis...residuals are not from a normal distribution. Little validity under small samples. Lecture #7-2/15/2005 Slide 33 of 36

34 Test for Correlated Errors Finally, there is a hypothesis test for seriation effects in the predictor variables. The Durbin-atson statistic tests for correlation between adjacent observations. Detection 1: Plots Detection 2: Tests Special Cases rapping Up Really, this test is only valid if observations were made on equal time intervals. In SPSS: Analyze...Regresssion...Linear. Click on the Statistics box. Under Residuals check Durbin-atson. Lecture #7-2/15/2005 Slide 34 of 36

35 Here it is: Your Moment of Zen rapping Up Moment of Zen Next Class Regression diagnostics are important parts of an analysis that must not be overlooked. Often times, inferences can be wrong if assumptions of the regression have not been met. Transformations can take forever, and may not get you closer to a good result with respect to assumptions. Listen to your data, they are trying to tell you something. Lecture #7-2/15/2005 Slide 35 of 36

36 Next Time Bringing it all together: Case studies in regression. rapping Up Moment of Zen Next Class Lecture #7-2/15/2005 Slide 36 of 36