MULTIVARIATE TECHNIQUES, ROBUSTNESS

Transcription

1 MULTIVARIATE TECHNIQUES, ROBUSTNESS Mia Hubert Associate Professor, Department of Mathematics and L-STAT Katholieke Universiteit Leuven, Belgium Peter J. Rousseeuw 1 Senior Researcher, Renaissance Technologies, USA peter@rousseeuw.net The usual multivariate analysis techniques include location and scatter estimation, principal component analysis, factor analysis, discriminant analysis, canonical correlation analysis, multiple regression and cluster analysis. These methods all try to describe and discover structure in the data, and thus rely on the correlation structure between the variables. Classical procedures typically assume normality (i.e. gaussianity) and consequently use the sample mean and sample covariance matrix to estimate the true underlying model parameters. Below are three examples of multivariate settings used to analyze a data set with n objects and p variables, forming an n p data matrix X = (x 1,...,x n ) with x i = (x i1,..., x ip ) the ith observation. 1. Hotelling s T 2 -statistic for inference about the center of the (normal) underlying distribution is based on the sample mean x = n i=1 x i and the sample covariance matrix S x = 1 n n 1 i=1 (x i x)(x i x). 2. Classical principal component analysis (PCA) uses the eigenvectors and eigenvalues of S x to construct a smaller set of uncorrelated variables. 3. In the multiple regression setting, also a response variable y = (y 1,...,y n ) is measured. The goal of linear regression is to estimate the parameter θ = (β 0, β) = (β 0, β 1,...,β p ) relating the response variable and the predictor variables in the model y i = β 0 + β 1 x i1 + + β p x ip + ε i. 1 Dr Peter Rousseeuw was Professor and Head (since 1992) of the Division of Applied Mathematics, Universiteit Antwerpen, Belgium. Currently he is a Senior Researcher at Renaissance Technologies in New York. He has (co-)authored over 160 papers, two edited volumes and three books, including Robust Regression and Outlier Detection (with A.M. Leroy, Wiley-Interscience, 1987). In 2003 ISI-Thompson included him in their list of Highly Cited Mathematicians. His paper Least Median of Squares Regression (1984), Journal of the American Statistical Association, 79, ) which proposed new robust methods for regression and covariance, has been reprinted in Breakthroughs in Statistics III (the 3-volume collection consists of the 60 most influential publications in statistics from 1850 to 1990), Kotz and Johnson 1997, Springer-Verlag, New York. He is an Elected Member, International Statistical Institute (1991) and an Elected Fellow of Institute of Mathematical Statistics (elected 1993) and American Statistical Association (elected 1994). He was Associate Editor, Journal of the American Statistical Association ( ), and Computational Statistics and Data Analysis ( ). He has supervised 20 PhD students. 1

2 The least squares slope estimator can be written as ˆβ LS = S 1 x s xy with s xy = 1 n n 1 i=1 (y i ȳ)(x i x) the cross-covariance vector. The intercept is given by ˆβ 0 = ȳ ˆβ LS x. These classical estimators often possess optimal properties under the gaussian model assumptions, but they can be strongly affected by even a few outliers. Outliers are data points that deviate from the pattern suggested by the majority of the data. Outliers are more likely to occur in datasets with many observations and/or variables, and often they do not show up by simple visual inspection. When the data contain nasty outliers, typically two things happen: the multivariate estimates differ substantially from the right answer, defined here as the estimates we would have obtained without the outliers; the resulting fitted model does not allow to detect the outliers by means of their residuals, Mahalanobis distances, or the widely used leave-one-out diagnostics. The first consequence is fairly well-known (although the size of the effect is often underestimated). Unfortunately the second consequence is less well-known, and when stated many people find it hard to believe or paradoxical. Common intuition says that outliers must stick out from the classical fitted model, and indeed some of them do so. But the most harmful types of outliers, especially if there are several of them, may affect the estimated model so much in their direction that they are now well-fitted by it. Once this effect is understood, one sees that the following two problems are essentially equivalent: Robust estimation: find a robust fit, which is similar to the fit we would have found without the outliers; Outlier detection: find all the outliers that matter. Indeed, a solution to the first problem allows us, as a byproduct, to identify the outliers by their deviation from the robust fit. Conversely, a solution to the second problem would allow us to remove or downweight the outliers followed by a classical fit, which yields a robust estimate. It turns out that the more fruitful approach is to solve the first problem and to use its result to answer the second. This is because from a combinatorial viewpoint it is more feasible to search for sufficiently many good data points than to find all the bad data points. Many robust multivariate estimators have been constructed by replacing the empirical mean and covariance matrix with a robust alternative. Currently the most popular estimator for this purpose is the Minimum Covariance Determinant (MCD) estimator [27]. The MCD method looks for the h observations (out of n) whose classical covariance matrix has the lowest possible determinant. The raw 2

3 MCD estimate of location is then the average of these h points, whereas the raw MCD estimate of scatter is a multiple of their covariance matrix. Based on these raw estimates one typically carries out a reweighting step, yielding the reweighted MCD estimates [30]. The MCD location and scatter estimates are affine equivariant, which means that they behave properly under affine transformations of the data. Computation of the MCD is non-trivial, but can be performed efficiently by means of the FAST- MCD algorithm [30] which is available in standard SAS, S-Plus, and R. A useful measure of robustness is the finite-sample breakdown value [11, 12]. The breakdown value is the smallest amount of contamination that can have an arbitrarily large effect on the estimator. The MCD estimates of multivariate location and scatter have breakdown value (n h)/n. The MCD has its highest possible breakdown value of 50% when h = [(n + p + 1)/2]. Note that no affine equivariant estimator can have a breakdown value above 50%. Another measure of robustness is the influence function [12], which measures the effect on an estimator of adding a small mass of data in a specific place. The MCD has a bounded influence function, which means that a small contamination at any position can only have a small effect on the estimator [7]. In regression, a popular estimator with high breakdown value is the Least Trimmed Squares (LTS) estimator [27, 31]. The LTS is the fit that minimizes the sum of the h smallest squared residuals (out of n). Other frequently used robust estimators include S-estimators [28] and MM-estimators [35], which can achieve a higher finite-sample efficiency than the LTS. Robust multivariate estimators have been used to robustify the Hotelling T 2 -statistic [34], PCA [8, 32], multiple regression with one or several response variables [29, 1], discriminant analysis [14, 20, 5], factor analysis [26], canonical correlation [6], and cluster analysis [13]. Another important group of robust multivariate methods are based on projection pursuit (PP) techniques. They are especially useful when the dimension p of the data is larger than the sample size n, in which case the MCD is no longer welldefined. Robust PP methods project the data on many univariate directions and apply robust estimators of location and scale (such as the median and the median absolute deviation) to each projection. Examples include the Stahel-Donoho estimator of location and scatter [24] and generalizations [36], robust PCA [23, 9, 18, 2], discriminant analysis [25], canonical correlation [3], and outlier detection in skewed data [4, 19]. The hybrid ROBPCA method [17, 10] combines PP techniques with the MCD and has led to the construction of robust principal component regression [22], partial least squares [21], and classification for high-dimensional data [33]. A more extensive description of robust multivariate methods and their applications can be found in [16, 15]. 3

4 References [1] J. Agulló, C. Croux, and S. Van Aelst, The multivariate least trimmed squares estimator, Journal of Multivariate Analysis 99 (2008), [2] G. Boente, A.M. Pires, and I. Rodrigues, General projection-pursuit estimates for the common principal components model: Influence functions and Monte Carlo study, Journal of Multivariate Analysis 97 (2006), [3] J.A. Branco, C. Croux, P. Filzmoser, and M.R. Oliviera, Robust canonical correlations: A comparative study, Computational Statistics 20 (2005), [4] G. Brys, M. Hubert, and P.J. Rousseeuw, A robustification of Independent Component Analysis, Journal of Chemometrics 19 (2005), [5] C. Croux and C. Dehon, Robust linear discriminant analysis using S-estimators, The Canadian Journal of Statistics 29 (2001), [6], Analyse canonique basée sur des estimateurs robustes de la matrice de covariance, La Revue de Statistique Appliquée 2 (2002), [7] C. Croux and G. Haesbroeck, Influence function and efficiency of the Minimum Covariance Determinant scatter matrix estimator, Journal of Multivariate Analysis 71 (1999), [8], Principal components analysis based on robust estimators of the covariance or correlation matrix: influence functions and efficiencies, Biometrika 87 (2000), [9] C. Croux and A. Ruiz-Gazen, High breakdown estimators for principal components: the projection-pursuit approach revisited, Journal of Multivariate Analysis 95 (2005), [10] M. Debruyne and M. Hubert, The influence function of the Stahel-Donoho covariance estimator of smallest outlyingness, Statistics and Probability Letters 79 (2009), [11] D.L. Donoho and P.J. Huber, The notion of breakdown point, A Festschrift for Erich Lehmann (Belmont) (P. Bickel, K. Doksum, and J.L. Hodges, eds.), Wadsworth, 1983, pp [12] F.R. Hampel, E.M. Ronchetti, P.J. Rousseeuw, and W.A. Stahel, Robust statistics: The approach based on influence functions, Wiley-Interscience, New York, [13] J. Hardin and D.M. Rocke, Outlier detection in the multiple cluster setting using the minimum covariance determinant estimator, Computational Statistics and Data Analysis 44 (2004), [14] D.M. Hawkins and G.J. McLachlan, High-breakdown linear discriminant analysis, Journal of the American Statistical Association 92 (1997), [15] M. Hubert and M. Debruyne, Minimum Covariance Determinant, Wiley Interdisciplinary Reviews: Computational Statistics 2 (2010), [16] M. Hubert, P.J. Rousseeuw, and S. Van Aelst, High breakdown robust multivariate methods, Statistical Science 23 (2008), [17] M. Hubert, P.J. Rousseeuw, and K. Vanden Branden, ROBPCA: a new approach to robust principal components analysis, Technometrics 47 (2005),

5 [18] M. Hubert, P.J. Rousseeuw, and S. Verboven, A fast robust method for principal components with applications to chemometrics, Chemometrics and Intelligent Laboratory Systems 60 (2002), [19] M. Hubert and S. Van der Veeken, Outlier detection for skewed data, Journal of Chemometrics 22 (2008), [20] M. Hubert and K. Van Driessen, Fast and robust discriminant analysis, Computational Statistics and Data Analysis 45 (2004), [21] M. Hubert and K. Vanden Branden, Robust methods for Partial Least Squares Regression, Journal of Chemometrics 17 (2003), [22] M. Hubert and S. Verboven, A robust PCR method for high-dimensional regressors, Journal of Chemometrics 17 (2003), [23] G. Li and Z. Chen, Projection-pursuit approach to robust dispersion matrices and principal components: primary theory and Monte Carlo, Journal of the American Statistical Association 80 (1985), [24] R.A. Maronna and V.J. Yohai, The behavior of the Stahel-Donoho robust multivariate estimator, Journal of the American Statistical Association 90 (1995), [25] A.M. Pires, Robust discriminant analysis and the projection pursuit approach: Practical aspects, in: Developments in Robust Statistics (R. Dutter, P. Filzmoser, U. Gather, and P.J. Rousseeuw, eds.), Physika Verlag, Heidelberg, 2003, [26] G. Pison, P.J. Rousseeuw, P. Filzmoser, and C. Croux, Robust factor analysis, Journal of Multivariate Analysis 84 (2003), [27] P.J. Rousseeuw, Least median of squares regression, Journal of the American Statistical Association 79 (1984), [28] P.J. Rousseeuw and A.M. Leroy, Robust regression and outlier detection, Wiley- Interscience, New York, [29] P.J. Rousseeuw, S. Van Aelst, K. Van Driessen, and J. Agulló, Robust multivariate regression, Technometrics 46 (2004), [30] P.J. Rousseeuw and K. Van Driessen, A fast algorithm for the Minimum Covariance Determinant estimator, Technometrics 41 (1999), [31], Computing LTS regression for large data sets, Data Mining and Knowledge Discovery 12 (2006), [32] M. Salibian-Barrera, S. Van Aelst, and G. Willems, PCA based on multivariate MMestimators with fast and robust bootstrap, Journal of the American Statistical Association 101 (2006), [33] K. Vanden Branden and M. Hubert, Robust classification in high dimensions based on the SIMCA method, Chemometrics and Intelligent Laboratory Systems 79 (2005), [34] G. Willems, G. Pison, P.J. Rousseeuw, and S. Van Aelst, A robust Hotelling test, Metrika 55 (2002), [35] V.J. Yohai, High breakdown point and high efficiency robust estimates for regression, The Annals of Statistics 15 (1987), [36] Y. Zuo, H. Cui, and X. He, On the Stahel-Donoho estimator and depth-weighted means of multivariate data, The Annals of Statistics 32 (2004),