Robust Regression Diagnostics. Regression Analysis

Size: px

Start display at page:

Download "Robust Regression Diagnostics. Regression Analysis"

Roderick Thornton
5 years ago
Views:

Robust Regression Diagnostics 1.1 A Graduate Course Presented at the Faculty of Economics and Political Sciences, Cairo University Professor Ali S.

1 Robust Regression Diagnostics 1.1 A Graduate Course Presented at the Faculty of Economics and Political Sciences, Cairo University Professor Ali S. Hadi The American University in Cairo and Cornell University ahadi@aucegypt.edu ali-hadi@cornell.edu Copyright 017 by Ali S. Hadi Regression Analysis 1. Input Computer Output Data Model Fitting Method Assumptions Estimated Parameters Test Statistics Graphs Tables We like to know how sensitive the output is to small perturbation in the input.

2 Motivating Example 1 New York Rivers Data: 1.3 In a 1976 study on land use and water quality in New York rivers, the total nitrogen content was used as a measure of water quality in the 0 New York State river basins. New York Rivers Olean. Cassadaga 3. Oatka 4. Neversink 5. Hackensack 6. Wappinger 7. Fishkill 8. Honeoye 9. Susquehanna 10.Chenango 11. Tioughnioga 1.West Canada 13. East Canada 14.Saranac 15. Ausable 16. Black 17. Schoharie 18. Raquette 19. Oswegatchie 0.Cohocton See map of NY State

3 Variables Used Active Agriculture (X 1 ): percentage of land area currently in agricultural use Forest (X ): percentage of land area in forest Residential (X 3 ): percentage of residential land area Commercial/Industrial (X 4 ): percentage of land area used in either commercial or manufacturing Total Nitrogen (Y): mean concentration (mg/liter) based on samples taken at regular intervals during the spring, summer, and fall months 1.5 River X 1 X X 3 X 4 Y

4 Regression Summary Observation Deleted T-value None t t t t t t j ˆ j ; s. e.( ˆ ) j j 0,1,,3,4. Motivating Example 1.8 Homicides Data: This data set is a result of a study investigating the role of firearms in accounting for the rising homicide rate in Detroit. The data is for the years

5 Variables Used FTP: # of full-time police per 100,000 population UEMP: % of the population unemployed MAN: # of manufacturing workers (in thousands) LIC: # of handgun licenses issued per 100,000 population CLEAR: Percent of homicides cleared by arrest WM: # of white males in the population GOV: # of government workers (in thousands) HOM: # of homicides per 100,000 population 1.9 Estimated Coefficient (T-value) 1.10 Coef. Model 1 Model Model 3 Const (.4) (11.1) (-1.5) MAN (-4.5) (-5.4) (-.8) WM (-.7) (-15.9) GOV (0.7) (1.0)

6 Model Selection Criteria 1.11 Minimum Residual Mean Square (RMS): ˆ SSE, n p where SSE = ( y i yˆ i n i 1 is the residual sum of squares, n is the number of observations, p is the number of regression coefficients. ) Model Selection Criteria 1.1 Maximum R-Square: where SST squares. R SSE 1, SST n ( y i y i 1 is the total sum of Note: Not good for comparing models with different number of predictors. )

7 Model Selection Criteria 1.13 Maximum Adjusted R-Square: SSE /( n p) R a 1, SST /( n 1) where SST squares. n ( y i y i 1 is the total sum of Note: The sum of squares are adjusted for their degrees of freedom. It imposes a penalty for including insignificant variables. ) Model Selection Criteria 1.14 Mallows C-p: For a model with p predictors, C p Y T (I P)Y ( p n), ˆ where ˆ is a good estimate of (usually obtained from the full model). Note: The above are standard well-known criteria, used to judge the adequacy of fit and to guide variable selection procedures.

8 Variable Selection Methods Backward Elimination: Start with the full model, then delete the least significant variable (the one with the smallest T-value or largest p- value) Repeat until all regression coefficients in the model are significant. Variable Selection Methods 1.16 Forward Selection: Start with the empty model, then add the most significant variable (the one with the largest T-value or smallest p- value). Repeat until all candidate variables to enter the model have insignificant regression coefficients.

Variable Selection Methods 1.17 Stepwise Method: A combination of the Backward and Forward methods. Other Methods: See any textbook on regression analysis.

9 Variable Selection Methods 1.17 Stepwise Method: A combination of the Backward and Forward methods. Other Methods: See any textbook on regression analysis. Let us apply some of these methods to the Homicides Data. Backward Elimination Method 1.18 Variable RMS Adjusted Removed ˆ R a None GOV MAN WM Accordingly, GOV is the least important variable.

10 Forward Selection Method 1.19 Variable RMS Adjusted Added ˆ GOV MAN WM Accordingly, GOV is the most important variable. R a Reasons for Inconsistency 1.0 GOV MAN GOV WM

11 Summary 1.1 Conclusions drawn from fitted models that are highly sensitive to a particular data point, a particular variable, or a particular assumption should be treated cautiously. Course Outline Motivating Examples. Selected References 3. Review of Least Squares (LS) Regression Analysis 4. The Iterative Nature of Regression Analysis 5. The Projection Matrix and its Properties

12 Course Outline Sensitivity of the LS fit with Respect to: Variables (column sensitivity) Observations (row sensitivity) Errors of Measurements Probability Law of Errors Course Outline Robust Regression and Outlier Detection: The Brute Force Method The LMS Method The LAV Method The BACON Approach The RIRLS Method

13 Selected References: Selected Books 1.5 Birkes, D. and Dodge, Y. (1993), Alternative Methods of Regression, New York: Wiley. Chatterjee, S. and Hadi, A.S. (1988), Sensitivity Analysis in Linear Regression, New York: Wiley. Chatterjee, S. and Hadi (006), Regression Analysis By Examples, Fifth Edition, New York: Wiley. Rousseeuw, P. J. and Leroy, A. (1987), Robust Regression and Outlier Detection, New York: Wiley. Selected References: Selected Articles 1.6 Gould, W. and Hadi, A. S. (1993), Identifying Multivariate Outliers, Stata Technical Bulletin, 11, 5. Hadi, A. S. (199), Identifying Multiple Outliers in Multivariate Data, Journal of the Royal Statistical Society, (B), 54, No. 3, Hadi, A. S. (199), A New Measure of Overall Potential Influence in Linear Regression, Computational Statistics and Data Analysis, 14, 1 7.

14 Selected References: Selected Articles 1.7 Hadi, A. S. (1994), A Modification of a Method for the Detection of Outliers in Multivariate Samples, Journal of the Royal Statistical Society, Series (B), 56, Hadi, A. S. and Simonoff, J. S. (1993), Procedures for the Identification of Multiple Outliers in Linear Models, Journal of the American Statistical Association, 88, Selected References: Articles Hadi, A. S. and Simonoff, J. S. (1994), Improving the Estimation and Outlier Identification Properties of the Least Median of Squares and Minimum Volume Ellipsoid Estimators, Parisankhyan Sammikkha, 1, Hadi, A. S. and Simonoff, J. S. (1997), A More Robust Outlier Identifier for Regression Data, Bulletin of the International Statistical Institute, Munier, S. (1999), Multiple Outlier Detection in Logistic Regression, Student, 3,

15 Course Outline Motivating Examples. Selected References 3. Review of Least Squares (LS) Regression Analysis 4. The Iterative Nature of Regression Analysis 5. The Projection Matrix and its Properties

Regression Analysis By Example

Regression Analysis By Example Third Edition SAMPRIT CHATTERJEE New York University ALI S. HADI Cornell University BERTRAM PRICE Price Associates, Inc. A Wiley-Interscience Publication JOHN WILEY & SONS,