High Breakdown Point Estimation in Regression

Transcription

1 WDS'08 Proceedings of Contributed Papers, Part I, 94 99, ISBN MATFYZPRESS High Breakdown Point Estimation in Regression T. Jurczyk Charles University, Faculty of Mathematics and Physics, Prague, Czech Republic. Abstract. In robust regression theory the estimators, which can resist contamination of nearly fifty percent of the data, due to the fact that they are highly important in practice, were intensively studied. In this paper we describe three methods with high breakdown point: the least trimmed squares (LTS), the least median of squares (LMS) and their generalization the least weighted squares (LWS) estimator 1. Instead of showing how powerful these procedures can be, we are especially interested in potential problems and situations when the methods can behave rather strange. These are illustrated by various data examples. We want to show that the high breakdown point regression should be performed with caution. Introduction During few decades before 1984 there was a great effort in searching for some multivariate regression estimators which will have breakdown point of nearly 50% (the exact definition of the breakdown point is mentioned below). It was due to the belief that such estimators can give a hint which of the estimates are near to the true model (see Rousseeuw and Leroy [1987]). The first proposal of such method was based on an idea by Hampel [1975] and presented by Rousseeuw [1984]. This method is called the least median of squares. In the same paper (Rousseeuw [1984]) another important estimator the least trimmed squares was also introduced. Before we give the exact definitions, let us set up some notations. We consider the linear regression model p Y i = X iβ 0 e i = X ij βj 0 e i, i = 1, 2,..., n. j=1 For any β R p r i (β) = Y i X i β denotes the i-th residual and r2 (j)(β) the j-th order statistic among the squared residuals. Definition 1 Let n/2 h n, then ˆβ (LMS,n,h) = argmin r(h) 2 (β) and ˆβ(LT S,n,h) = argmin β R p β R p h r(i) 2 (β) are called the least median of squares (LMS) and the least trimmed squares (LTS) estimator, respectively. We are going to define also estimator proposed by Víšek [2001]. This estimator is also based upon ordered squared residuals but they are weighted in addition. Definition 2 Let for any n N i=1 1 = w 1 w 2... w n 0 be some weights. Then ˆβ (LW S,n,w) = argmin β R p is called the least weighted squares (LWS) estimator. n w i r(i) 2 (β) Notice please that the single weight isn t related directly to some observation (don t confuse it with another regression method the weighted least squares). The estimator itself assigns the weights implicitly to the observations (so as in LTS case). From both definitions we can clearly see that the LTS are special case of the LWS with w i = I{i h} (I denotes indicator). The ordinary least squares (OLS) are the special case of the LWS as well. (If we wanted, we could define the LWS even more generally without the restriction on monotonicity of the weights, then also the LMS would be the special case of the LWS. But we use only the LWS in the form of definition 2 in what follows). i=1 1 LWS estimator has breakdown point in dependance on the choice of its weights. 94

2 Properties of the methods Let us summarize some important properties of just defined estimators. But at first we need recall some definitions. Definition 3 An estimator T is called regression equivariant if T ({(X i, Y i X i v); i = 1,..., n}) = T ({(X i, Y i ); i = 1,..., n}) v, scale equivariant if T ({(X i, cy i ); i = 1,..., n}) = ct ({(X i, Y i ); i = 1,..., n}) and we say that T is affine equivariant if T ({(X i A, Y i ); i = 1,..., n}) = A 1 T ({(X i, Y i ); i = 1,..., n}), where v is any column vector, c is any constant and A is any nonsingular square matrix. Definition 4 Let Z be any sample of n data points (X 1, Y 1 ),..., (X n, Y n ), T (Z) is estimate of β 0. Let Zm be sample, where any m of the original data points are replaced by arbitrary values (could be also infinite). Then the breakdown point of the estimator T at Z is ɛ (T, Z) = min { m n : sup Z m T (Z ) T (Z) is infinite }. Remark 1: Notice that this definition counts also with replacing of X ij, which is in general more serious problem than coping with outlier in Y -values. Remark 2: An estimator has high breakdown point if it can resist contamination of nearly 50% of the data (i.e. ɛ (T, Z) tends to 0.5 as n goes to infinity). Unfortunately, we don t have enough space to show correctness of all following LTS and LMS properties but they can be found in Rousseeuw and Leroy [1987]. Some important properties of the LMS and LTS method: They are regression, scale and affine equivariant (in terms of Definition 3). If we put h in Definition 1 equal to n 2 p1 2 ( denotes integer part), then LMS and LTS will attain the breakdown point ɛ max = 1 n ( n p 2 1) (any regression equivariant estimator has the breakdown point equal or less than ɛ max). Both estimators also have exact fit property if strictly more than 1 2 (n p 1) observations satisfy model exactly and are in general position (when h equals to above defined value). If strictly more than 1 2 (n p 1) observations satisfy model Y i = X i β 0 exactly and are in general position, then estimates of both methods are equal to β 0 (when h equals to above defined value). Remark 3: This last relation is called the exact fit property. In words: If the majority of the data follows a linear relationship exactly and regression method yields this equation, then this regression technique is said to possess the exact fit property. Concerning LWS we look at the LWS estimator satisfying the requirement: w 1,..., w h are strictly positive and others are 0. Then also this estimator has all above mentioned properties like LMS and LTS have. Throughout the paper this special LWS will be called the LWS with high breakdown point. Remark 4: Computing the LMS estimate in simple regression (regression with one explanatory variable and intercept) is equivalent to finding the narrowest stripe covering h observations (see Rousseeuw and Leroy [1987]). Negative consequences of high breakdown point In following part of the paper we will talk about some problems which we can come across using high breakdown point estimators. These are illustrated especially through data examples. Before we show this data examples we should say some words about used software. All calculations were performed by the R software. Used LMS method is implemented in the library MASS (function lqs). For LWS calculations we had to program own LWS procedure according to the article by Víšek [2008], because the LWS in the R language were still missing 2. Then the program was checked out (as its special case of LTS) on various data examples, of which we know the exact solution for LTS. The algorithm gives quite satisfactory results. The source code of the LWS procedure is available upon request on the address jurczyk@karlin.mff.cuni.cz. The LTS estimates were computed by our LWS procedure. 2 Our LWS procedure works for nonincreasing weights and for data points in general position. 95

3 a) Instability of the estimators The whole discussion about instability of high breakdown point estimators started with the paper by Hettmansperger and Sheather [1991]. They introduced the Engine Knock Data with 16 observations and 4 explanatory variables. They typed inadvertently one wrong value (among 80 values) 15.1 (we shall call this data damaged data) instead of the original value 14.1 (correct data). After they had computed estimates for the correct and damaged data, they discovered that the estimates for the LMS estimator were considerably different. It was very surprising because of the common belief that LMS is highly resistant method. We show values of the estimates for all three introduced methods for both damaged and correct data sets in Table 1 (exact choices of LWS weights are in following remark). We can see that such an unstable behaviour is common to other methods, too. Unstable in the sence that a small change of the data can cause big change of the estimates. Remark 5 (choice of LWS weights): One of the ways how to choose LWS weights is to choose some nonincreasing function f : [0, 1] [0, 1] with f(0) = 1 and then assign w i = f((i 1)/n). LWS 1, LWS 2 and LWS 3 in our example correspond to f 1 (x) = 1 if x x 0.3 if x (0.35, 0.65) 0 if x 0.65 f 2 (x) = 1 if x x 0.1 if x (0.55, 0.65) 0 if x 0.65 f 3 (x) = 1 2 arctan(50x 30) 2 arctan( 30), f 1 and f 2 is defined to make h = 11 strictly positive weights (notice please that h is tied with n and p), f 3 provides the possible smooth low breakdown point alternative to the LTS method and should be important when we need all positive weights, i.e. when we don t want to discard any observation from the data. Table 1. Engine Knock Data: n = 16, p = 5, h = = 11 Estimates for correct data Estimates for damaged data OLS LMS LTS LWS 1 LWS 2 LWS 3 OLS LMS LTS LWS 1 LWS 2 LWS 3 Intercept X X X X Like Hettmansperger and Sheather [1991] we construct also an exact fit example for simple regression (Figure 1). It is clear that we can construct data where arbitrary small change of one datum can cause large change of estimate (if estimator has the exact fit property). The increasing number of observations also doesn t solve the problem. The theory of robustness is based on the presumtion that small change of underlying model cause small change of robust estimate. But if we think about it for a while, we will realize that the situation here is different, because small change of the data changes the whole true model. The true model in robust regression means the model which covers the majority of the data. Hence, when we look once more at the exact fit example, we realize that this unstable behaviour is not suprising at all. And in this case the strange behaviour in Figure 1 is exactly what we expect from good robust procedure. The only problem here is to realize that a small change of the data can cause the large change of underlying model. b) Diversity of the estimates When we use for example LTS and LMS on some data, according to various experiments (e.g. Víšek [1996]), usually (rlms 2 ) (h) from the LMS fit is smaller than (rlt 2 S ) (h) from the LTS fit and conversely h i=1 (r2 LT S ) (i) is smaller than h i=1 (r2 LMS ) (i). Therefore the estimates for LTS and LMS are usually somehow different. An artificial data example (in Figure 2) shows that such a diversity may be pretty large. For this data, the LTS and LMS estimates are nearly orthogonal to each other. If we want, we will be able to construct examples where LWS and LTS give different estimates as well (like in Figure 4). Construction of such artificial data is possible due to the fact that investigated estimators use for estimating β 0 only n(1 ɛ ) 1 chosen observations. Each estimator may find important different observations (like in our example), thus the estimates may be also different. The increasing number of observations again doesn t help. 96

4 Figure 1. The exact fit example: A change in position of one datum (signed as a circle) causes the large change of the estimate. This is because of the exact fit property of the estimator. The regression line will follow the majority of the data. LMS LTS Figure 2. LTS estimate perpendicular to LMS estimate: n = 11, h = 6, the stripe covering 6 observations in squares is narrower than the stripe covering 6 observations in circles and at the same time the sum of the squared residuals of encircled observations is less than it would be for the LTS estimate using 6 observations in squares. c) Subsample stability Now, we are interested in how the estimate from the whole data set will change if we use for calculation only a subsample of the data. If the estimator is stable with respect to subsamples, then this change of the estimates shouldn t be large. And it is quite natural to require such subsample stability for a good estimator. Let us explain it a bit. If some method recognize the true model for the whole data, then it should recognize (approximately) the same model for subsample. The requirement of subsample stability is also connected with consistency, because consistency may be viewed as a stability for large sample sizes. Hereafter, we will study only the difference of the estimates for the whole data and for the data without one obsevation. Denote ˆβ (M,n 1,l,h) (resp. ˆβ(LW S,n 1,l,w) ) the estimate by method M (resp. LWS) for the data without l-th observation, h has the same meaning as it had in the definitions of the methods. Some results of subsample stability for the LTS method are available in Víšek [2006]. They derived the asymptotic representation of the n( ˆβ (LT S,n,h) ˆβ (LT S,n 1,l,h) ) and concluded that the LTS estimator has in some way similar behaviour as M-estimators with discontinuous ψ-function, which means that it can be sometimes sensitive even to deletion of one observation. One of their proposals, which may solve this problem, is to use LWS estimator with smoothly redescending weights. We were searching for an example of such improvement by using LWS estimator and we found it in already mentioned Engine Knock Data. We used the same three sets of weights for the LWS as in Remark 5 (for computing the LWS estimate for subsample with n 1 observations the same weights w as for ˆβ (LW S,n,w) were used but without n-th coordinate. We prefer this choice because of the compatibility with the results shown by Víšek [2006]). The results for Engine Knock Data are displayed in Figure 3. Due to the limited space there are shown only results for the first explanatory variable, but other variables 97

5 behave in the same way. We can see (left picture in Figure 3) that at least for one subsample the value of β 0 2 estimate for LMS and LTS is larger than 4, although estimates for the full data is around 0. We conclude that in this example the LWS estimators really have better behaviour and are much more stable with respect to subsamples than the LTS estimator (and also than the LMS estimator). In order to understand the principle why the LWS are better, we constructed also an artificial data with 9 observations (upper part of Figure 4), where the LTS estimator is unstable while the LWS estimator is stable. It seems that we finally find some improvement of LMS and LTS methods through the LWS with high breakdown point. But unfortunately it isn t so simple, because on the contrary it can be also constructed an alternative example where the LWS are unstable and the LTS aren t (bottom pictures in Figure 4). And also from other numerical examples (including simulated as well as real data sets) with different sample sizes and different number of explanatory variables, which we tried, we can t say that LWS are better than LTS or LMS in terms of subsample stability. Simply there exist cases of data sets of both kinds (like in the artificial data) where LWS can solve LTS instability and vice versa ols lms lts lws1 lws2 lws Figure 3. Engine Knock data subsample sensitivity: Pictures for the first explanatory variable (M,n 1,l,h) (second coordinate of β). The left picture is a boxplot of { ˆβ 2, l = 1,..., 16}, each box belongs to one method M, red circles are estimates from the whole data for corresponding methods. The second (LT S,n,h) (LT S,n 1,l,h) (LW S1,n,w) picture is comparison of LTS and LWS 1, 16 points here are { ˆβ 2 ˆβ 2 ˆβ 2 (LW S1,n 1,l,w[1:(n 1)]) ˆβ 2, l = 1..., 16} in dependence on l, positive values correspond to better behaviour of LWS 1. Conclusion The paper shows that some caution is inevitable. We have done already some conclusions about presented issues in previous text, we saw that above discussed problems and maybe also others (e.g. discussion about arbitrary low efficiency at central model with respect to ordinary least squares in finite samples by Stefanski [1991]) may occur for some data and are inherent with high breakdown point methods. When one works with such procedure he/she should take it into consideration and don t blindly believe in stability under any circumstances. Further, when we meet with the data where different high breakdown methods have rather different estimates, then we should reconsider what is the real structure of data in question. Although proposed high breakdown point LWS methods share some negative consequeces of other high breakdown point methods, we believe that the possibility of continuous weighting and also weighting with small instead of zero weights could be benefitial. Acknowledgments. This work was supported by grants GAČR 202/06/0408 and 201/05/H007. The author also thanks to Prof. Jan Ámos Víšek for the professional help. 98

6 LTS weights: (1,1,1,1,1,0,0,0,0) LWS weights: (1,1,1,1,0.5,0,0,0,0) LTS weights: (1,1,1,1,1,0,0,0,0) LWS weights: (1,1,1,1,0.5,0,0,0,0) Figure 4. Subsample instability: upper data: x 1 = ( 1, 0.7, 0.7, 1, 0.2, 0.1, 0, 0.1, 0.2), y = (0, 0, 0, 0, 0.7, 0.2, 0, 0.4, 0.6) ; bottom data x 1 = ( 2.1, 1.9, 0.1, 0.1, 1.9, 2.1, 0.5, 1, 0.5), y = ( 1.9, 2.1, 0.1, 0.1, 2.1, 1.9, 0.5, 1, 1.15). In each picture there are black line as the whole data estimate and 9 lines for the estimates based upon the data without one observation (LTS - left pictures, LWS - right pictures). The weights are in titles of pictures, for smaller data is utilized first 8 coordinates from the weights. References Hampel, F. R., Beyond location parameters: Robust concepts and methods, Bulletin of the International Statistical Institute, Rousseeuw, P. J., Least median of squares regression, Journal of The American Statistical Association, 79, , Rousseeuw, P. J., and Leroy, A. M., Robust Regression and Outlier Detection, John Wiley & Sons, Hettmansperger, T. P., and Sheather, S.J., L. A., A Cautionary Note on the Method of Least Median Squares, The American Statistician, 46, 79-83, Stefanski, L. A., A note on high-breakdown estimators, Statistics & Probability Letters, 11, , Víšek, J. Á., On High Breakdown Point Estimation, Computational Statistics, 11, , Víšek, J. Á., Regression with high breakdown point, Robust 2000 (eds. Jaromír Antoch & Gejza Dohnal, published by Union of Czech Mathematicians and Physicists), Prague: matfyzpress, , Víšek, J. Á., The Least Trimmed Squares. Sensitivity Study., Proceedings of the Prague Stochastics 2006, eds. Marie Hušková & Martin Janžura, Prague: MatFyz Press, , Víšek, J. Á., Consistency of the Instrumental Weighted Variables, To appear in Annals of the Institute of Statistical Mathematics,