Sensitivity Analysis in Linear Regression

Transcription

1 Sensitivity Analysis in Linear Regression

2 Sensitivity Analysis in Linear Regression SAMPRIT CHAlTERJEE New York University New York, New York ALI S. HAD1 Cornell University Ithaca, New York WILEY JOHN WLEY & SONS New York Chichester Brisbane Toronto. Singapore

3 A NOTE TO THE READER This book has been electronically reproduced firom digital information stored at John Wiley & Sons, Inc. We are pleased that the use of this new technology will enable us to keep works of enduring scholarly value in print as long as there is a reasonable demand for them. The content of this book is identical to previous printings. Copyright by John Wiley & Sons, Inc. All rights reserved. Published simultaneously in Canada Reproduction or translation of any part of this work beyond that permitted by Section 107 or 108 of the 1976 United States Copyright Act without the permission of the copyright owner is unlawful. Requests for permission or further information should be addressed to the Permissions Department, John Wiiey & Sons, Inc. Library of Congtess Cataloging in Publication Data: Chatterjee, Samprit Sensitivity analysis in linear regression!samprit Chatterjee, Ali S. Hadi. (Wiley series in probability and mathematical statistics, Applied probability and statistics.) p. cm. Bibliography: p. Includes index. ISBN Regression analysis. 2. Perturbation (Mathematics) 3. Mathematical optimization. I. Hadi, Ali S. 11. Title. QA278.2.CS '36-4~ CIP ISBN

4 Dedicated to the people of South Africa and their struggle for equality and human dignity.

5 Preface The past twenty years have Seen a great surge of activity in the general area of model fitting. The linear regression model fitted by least squares is undoubtedly the most widely used statistical procedure. In this book we concentrate on one important aspect of the fitting of linear regression models by least squares. We examine the factors that determine the fit and study the sensitivity of the fit to these factors. Several elements determine a fitted regression equation: the variables, the observations, and the model assumptions. We study the effect of each of these factors on the fitted model in turn. The regression coefficient for a particular variable will change if a variable not currently included in the model is brought into the model. We examine methods for estimating the change and assessing its relative impaance. Each observation in the data set plays a role in the fit. We study extensively the role of a single observation and multiple observations on the whole fitting procedure. Methods for the study of the joint effect of a variable and an observation are also presented, Many variables included in a regression study are measured with error. but the standard least squares estimates do not take this into account. The effects of mea- surement errors on the estimated regression coefficients are assessed. Assessment of the effects of measurement emrs is of great importance (for example in epidemiological studies) where regression coefficients are used to apportion effects due to different variables. The implicit assumption in least squares fitting is that the random disturbances present in the model have a Gaussian distribution. The generalized linear models proposed by Nelder and Weddehurn (1972) can be used to examine the sensitivity of the fitted model to the probability laws of the "errors" in the model. The object of this analysis is to assess qualitatively and quantitatively (numerically) the robustness of the regression fit This book does not aim at theory but brings together, Erom a practical point of view, scattered results in regression analysis, We rely heavily on examples to illus- vii

6 Viii PREFACE trate the theory. Besides numerical measures, we focus on diagnostic plots to assess sensitivity; rapid advances in statistical graphics make it imperative to incorporate diagnostic plots in any newly proposed methodology. This book is divided into nine chapters and an appendix. The chapters are more or less self-contained. Chapter 1 gives a summary of the standard least squares regression results, reviews the assumptions on which these results are based, and introduces the notations that we follow in the text. Chapter 2 discusses the properties of the prediction brojection) matrix which plays a pivotal role in regression. Chapter 3 discusses the role of variables in a regression equation. Chapters 4 and 5 examine the impact of individual and multiple observations on the fit. The nature of an observation (outlier, leverage point, influential point) is discussed in considerable detail. Chapter 6 assesses the joint impact of a variable and an observation. Chapter 7 examines the impact of measurement errors on the regression coefficients (the classical "error-in-variables" problem) from a numerical analyst's point of view. Chapter 8 presents a methodology for examining the effect of error laws of the "random disturbances" on the estimated regression parameters. Chapter 9 outlines some of the computational methods for efficiently executing the procedures described in the previous chapters. Since some readers might not be familiar with the concept of matrix norms, the main properties of norms are presented in the Appendix. The Appendix also contains proofs of some of the results in Chapters 4 and 5. The methods we discuss attempt to provide the data analyst with a clear and complete picture of how and why the data at hand affect the results of a multiple regression analysis. The material should prove useful to anyone who is involved in analyzing data. For an effective use of the book, some matrix algebra and familiarity with the basic concepts of regression analysis is needed. This book could serve as a text for a second course in regression analysis or as a supplement to the basic text in the first course. The book brings together material that is often scattered in the litera- ture and should, therefore, be a valuable addition to the basic material found in most regression texts. Certain issues associated with least squares regression are covered briefly and others are not covered at all; because we feel there exist excellent texts that deal with these issues. A detailed discussion of multicollinearity can be found in Chatteqiee and Price (1977) and Belsley, Kuh, and Welsch (1980); transformations of response

7 PREFACE ix and/or explanatory variables are covered in detail in Atkinson (1985) and Carroll and Ruppen (1988); the problems of heteroscedasticity and autocorrelation are addressed in Chatterjee and Price (1977) and Judge et al. (1985); and robust regression can be found in Huber (1981) and Rousseeuw and Leroy (1987). Interactive, menu-driven, and user-friendly computer programs implementing the statistical procedures and graphical displays presented in this book are available from Ali S. Hadi, 358 Ives Hall, Cornell University, Ithaca, NY These programs are written in APL; but user s knowledge of APL is not necessary. Two versions of these programs are available; one is tailored for the Macintosh and the other for the IBM PC. Some of the material in this book has been used in courses we have taught at Cornell University and New York University. We would like to thank our many students whose comments have improved the clarity of exposition and eliminated many errors. In writing this book we have been helped by comments and encouragement from our many friends and colleagues. We would like particularly to mention Isadore Blumen, Sangit Chatterjee, Mary Dowling, Andrew Forbes, Glen Heller, Peter Lenk, Robert Ling, Philip McCarthy, Douglas Momce, Cris Negm, Daryl Pregibon, Mary Rouse, David Ruppert, Steven Schwager, Karen Shane, Gary Simon, Jeffrey Simonoff, Leonard Stefanski, Paul Switzer, Chi-Ling Tsai, Paul Velleman, Martin Wells, and Roy Welsch. For help in preparing the manuscript for publication we would like to thank Janet Brown, Helene Croft, and Evelyn Maybe. We would also like to thank Robert Cooke and Ted Sobel of Cooke Publications for providing us with a pre-release version of their software MathWriterm that we have used in writing the mathematical expressions in this book. We would like to thank Bea Shube of John Wiley & Sons for her patience, understanding, and encouragement. Eagle Island, Maine Ithaca, New York October, 1987 SAMPRlT CHArnRJEE ALI s. HAD1

8 Contents PREFACE... CONTENTS... vii xi 1. INTRODUCTION Innoduction Notations Standard Estimation Results in Least Squares Assumptions Iterative Regression Process Organization of the Book PREDICTION MATRIX Introduction Roles of P and (I -P) in Linear Regression Properties of the Prediction Matrix General Properties Omitting (Adding) Variables Omitting (Adding) an Observation Conditions for Large Values of pii Omitting Multiple Rows of X Eigenvalues of P and (I - P) Distribution of pii Examples ROLE OF VARIABLES IN A REGRESSION EQUATION Introduction Effects of Underfitting xi

9 xii CONTENTS 3.3. Effects of Overfining Interpreting Successive Fining Computing Implications for Successive Fitting Introduction of One Additional Regressor Comparing Models: Comparison Criteria Diagnostic Plots for the Effects of Variables Added Variable (Partial Regression) Plots Residual Versus Predictor Plots Component-Plus-Residual (Partial Residual) Plots Augmented Partial Residual Plots Effects of an Additional Regressor EFFECTS OF AN OBSERVATION ON A REGRESSION EQUATION Introduction Omission Approach Measures Based on Residuals Testing for a Single Outlier Graphical Methods Outliers, High.leverage. and Influential Points Measures Based on Remoteness of Points in X-Y Space Diagonal Elements of P Mahalanobis Distance Weighted Squared Standardized Distance Diagonal Elements of Pz Influence Curve Definition of the Influence Curve Influence Curves for 6 and Approximating the Influence Curve Measures Based on the Influence Curve Cook's Distance Welsch-Kuh's Distance Welsch's Distance Modified Cooks Distance Measures Based on the Volume of Confidence Ellipsoids Andrews-Pregibon Statistic VarianceRatio Cook-Weisberg Statistic Measures Based on the Likelihood Function

10 CONTENTS xi Measures Based on a Subset of the Regression Coefficients Influence on a Single Regression Coefficient Ilnfluence on Linear Functions of fi Measures based on the Eigensmcture of X Condition Number and Collinearity Indices Collinearity-Influential Points Effects of an Observation on the Condition Number Diagnosing Collinearhy-Influential Observations Differentiation Approach Summary and Concluding Remarks ASSESSING THE EFFECTS OF MULTIPLE OBSERVATIONS Introduction Measures Based on Residuals Measures Based on the Influence Curve Sample lnfluence Curve Empirical Influence Curve Generalized Cook's Distance Generalized Welsch's Distance Measures Based on Volume of Confidence Ellipsoids Generalized Andrews-Pregibon Statistic Generalized Variance Ratio Measures Based on the Likelihood Function Measures Based on a Subset of the Regression Coefficients Identifying Collinearity-Influential Points Identifying Influential Observations by Clustering Example: Demographic Data JOlNT IMPACT OF A VARIABLE AND AN OBSERVATION 6.1. Introduction Notations Impact on the Leverage Values Impact on Residual Sum of Squares Impact on the Fitted Values Partial F-Te~ts Summary of Diagnostic Measures Examples Concluding Remarks

11 xiv CONTENTS 7. ASSESSING THE EFFECTS OF ERRORS OF MEASUREMENT Introduction Errors in the Response Variable Emrs in X: Asymptotic Approach Emrs in X: Perturbation Approach Errors in X: Simulation Approach STUDY OF MODEL SENSITIVITY BY THE GENERALIZED LINEAR MODEL APPROACH Introduction Generalized Linear Models (GLM) Exponential Family Link Function Parameter Estimation for GLM Judging the Goodness of Fit for GLM Example COMPUTATIONAL CONSIDERATIONS Introduction Triangular Decomposition Definitions Algorithm for Computing L and D Q-R Decomposition Properties of L and D Efficient Computing of Regression Diagnostics APPENDIX A. 1. Summary of Vector and Matrix Norms A.2. Another Proof of Theorem A.3. Proof of (4.60a) and (5.31a) REFERENCES INDEX

12 Sensitivity Analysis in Linear Regression