Breakdown points of Cauchy regression-scale estimators

Transcription

1 Breadown points of Cauchy regression-scale estimators Ivan Mizera University of Alberta 1 and Christine H. Müller Carl von Ossietzy University of Oldenburg Abstract. The lower bounds for the explosion and implosion breadown points of the simultaneous Cauchy M-estimator Cauchy MLE of the regression and scale parameters are derived. For appropriate tuning constants, the breadown point attains the maximum possible value. Key Words. Breadown point, explosion, implosion, Cauchy maximum lielihood, M- estimator, simultaneous estimation of regression and scale parameters. 1 Corresponding author. Department of Mathematical and Statistical Sciences, University of Alberta, Edmonton, Alberta, T6G G1, Canada. mizera@stat.ualberta.ca. This wor was supported in part by the National Sciences and Engineering Research Council of Canada. Department of Mathematics, Carl von Ossietzy University of Oldenburg, Postfach 503, D-6111 Oldenburg, Germany. mueller@math.uni-oldenburg.de. 1

2 1. Introduction Consider a general linear model y = Xβ + ɛ where y = y 1, y,..., y N IR N is a vector of observations, β IR p is an unnown parameter vector, ɛ = ɛ 1, ɛ,..., ɛ N IR N a vector of errors, and X = x 1, x,..., x N IR N p represents the design, the nown matrix of regressors design points. A well-nown robustness measure for studying the behavior of an estimator in the presence of outliers is the finite sample breadown point, the minimum proportion of outliers which may cause an arbitrary high bias of the estimator; see Huber 1981, Hampel et al. 1986, Rousseeuw and Leroy The definition that prevails in the literature reflect situations when regressors are prone to outliers for instance, random. Employing this definition, Maronna et al determined that M-estimators have the smallest possible breadown point 1/N, a poor behavior exhibited even by estimators generally considered robust for instance, by the l 1 estimator. However, as indicated by He et al and Ellis and Morgenthaler 199, this behavior considerably changes under a definition of breadown point that considers regressors error-free, non-stochastic. Such an approach suites particularly models with qualitative factors lie ANOVA, or with regressors fully under experimental control. Let N X := max β 0 #{n : x n β = 0}, the symbol # hereafter standing for the number of elements of a set. The maximal possible breadown point within regression equivariant estimators is N N N X + 1 see Müller 1995, Mizera and Müller 1999 pointed out that this upper bound is attained by M-estimators whose score functions has variation exponent zero, the class containing maximum lielihood estimators MLE based on t-distribution with γ degrees of freedom, in particular MLE based on Cauchy γ = 1 distribution hereafter Cauchy estimator. The results of Mizera and Müller 1999 were proved under the simplifying assumption that the scale is nown and equal to one. In applications, however, redescending estimators are very sensitive to the scale factor; if it is too large, their behavior may be fairly non-robust. Therefore, it is of considerable interest whether the same breadown bounds apply also for the more general setting with unnown scale. In this note, we answer this ;

3 question affirmatively for the Cauchy estimator. In the context of breadown point considering outliers in regressors, He et al. 000 found that the Cauchy estimator has the optimal performance among the MLE based on t-distribution. In Section, we define the Cauchy simultaneous estimator of regression and scale; a choice of a tuning constant for the scale leads to several possible versions. In Section 3, we show that the breadown point of all versions is reasonable, and that there is a version attaining the upper bound 1.1. We derive the explosion and implosion breadown point of scale estimators separately; the overall breadown point is taen to be their minimum. Finally, a short conclusion is given in Section 4.. Cauchy regression-scale estimators The maximization of the lielihood for regression and scale parameters for the errors with the Cauchy distribution leads to the minimization of N x n β lβ,, y, X := log N log with respect to β, Θ := IR p 0,. Let ϕu := log1 + u, and for K > 0, let N x n β D K β,, y, X := ϕ + K log. If K = N, then D K β,, y, X is equal to lβ,, y, X. If K N, we call the lielihood function tuned; its minimization leads to the Cauchy simultaneous regression and scale estimators 3.1 β, arg min{d K β,, y, X : β, Θ}. where Θ = IR p 0,. A solution of.1 in Θ satisfies the system of equations N y n x ψ β n. x n = 0, N y n x n χ β.3 = K, where ψu := u and χu := u. Since χ u 1+u 1+u = 0 for all > 0 if u = 0, and χ u 0 if and u 0, we have that N x n β.4 χ 0 if.

4 4 On the other hand, χ u 1 if 0 and u 0; since χ u is decreasing in for u 0, we have N x n β.5 χ #{n : y n x n β} if 0. By.4 and.5, there is no solution of.1 in Θ, if Let Note that N X = max β 0 N β, 0, X. N β, y, X := #{n : y n = x n β} N K. N y, X := max N β, y, X. β Theorem 1. Suppose that N X < N K. i If N y, X < N K, then there is a compact Θ 0 Θ and β 0, 0 Θ 0 such that D K β 0, 0, y, X < D K β,, y, X for all β, Θ \ Θ 0. ii If there exists β 0 such that N β 0, y, X N K, then for all 0,. lim D Kβ 0,, y, X < D K β 0,, y, X 0 Proof. i Tae β 0 = 0 and 0 = 1. Assume that for every IN there exists β, with D K β 0, 0, y, X D K β,, y, X such that β, β 0, 0 > or 1 >. There are three possible cases: a β IN is bounded and IN contains a subsequence convergent to zero thus, we may w.l.o.g. assume that 0. b IN is bounded and β IN contains a subsequence tending to infinity thus, we may w.l.o.g. assume that β. c A subsequence of IN converges to ; w.l.o.g. we may assume that. We will show that D K β,, y, X is unbounded in all three cases a contradiction proving i. Case a. We may assume w.l.o.g. that β β. Then N x n β log K log

5 = = x n β log K log n:y n x n β 1 log + y n x n β + K log n:y n x n β log + y n x n β #{n : y n x n β} log + K log n:y n x n β 5 because n:y n x n β log + y n x n β > if 0 and β β, log if 0, and #{n : y n x n β} + K = K N + #{n : y n = x n β} K N + N y, X < 0. Case b. W.l.o.g. we may assume β / β β. Then / β 0, y n / β 0, x n β / β x n β. Since is bounded and we obtain K #{n : x n β 0} K N + N X < 0, K #{n : x n β 0} log >. This implies N x n β log K log = =. n:x n β 0 n:x n β 0 log n:x n β 0 x n β log K log log β β + β + β x n β x n β β β + K log β + #{n : x n β 0} log β + K #{n : x n β 0} log

6 6 Case c. If, then N x n β log K log K log. ii For every 0,, N x n β 0 D K β 0,, y, X = log K log 1 = log + y n x n β 0 + K log n: y n x n β 0 = log + y n x n β 0 + K #{n : y n x n β 0 } log. n: y n x n β 0 The assumption N β 0, y, X N K yields K #{n : y n x n β 0 } = K N + N β 0, y, X 0. If K N + N β 0, y, X > 0 that is, K is odd, then lim 0 D K β 0,, y, X =. If K N + N β 0, y, X = 0, then D K β 0,, y, X = n: y n x n β 0 log + y n x n β 0 and ii holds with a finite limit. In view of Theorem 1, we may formulate the following definition. Let Θ = IR p [0,. Definition 1. A Cauchy regression-scale estimate with tuning constant K is, for given y and X, any point β 0, 0 in Θ such that β 0, 0 arg min β, Θ D Kβ,, y, X if N β, y, X < N K for all β IRp ; = β 0, 0, if N β 0, y, X N K. For Cauchy regression-scale estimators, we use the notation Cy, X = βy, X, y, X, suppressing the dependence on K. 3. Breadown points There are two types of breadown of a regression-scale estimator caused by outliers see Hampel et al. 1986, Rousseeuw and Croux 199, 1993: explosion, when the estimator for the regression parameters or the scale estimator goes to infinity, or implosion, which means that the scale estimator goes to zero. Assuming that the regressors are error-free

7 and non-stochastic, we allow only for outliers in the observations y n, in accord with He et al. 1990, Ellis and Morgenthaler 199, and Müller 1995, Definition Breadown point. Let C = β, be an estimator for β,. The explosion breadown point ɛ C, y, X of C is defined by } ɛ C, y, X := {M 1 N min : sup Cy, X = y By,M the implosion breadown point ɛ 0 C, y, X of C is defined by ɛ 0 C, y, X := 1 } {M N min : inf y, X = 0, y By,M where By, M := { y IR N : #{n : y n y n } M }. The breadown point ɛ C, y, X of C is defined to be ɛ C, y, X = min{ɛ C, y, X, ɛ 0 C, y, X}. If the estimator is not unique, then we adopt the least favorable values those maximizing Cy, X and minimizing y, X. Proposition Explosion without implosion. Suppose there is a sequence y By, M such that the Cauchy estimates Cy, X = βy, X, y, X satisfy Cy, X and 1 y,x = O1. Then M min{ K, N K N X}. Proof. Assume y n = for n = 1,..., N M for all IN, and let β := βy, X, := y, X. Then one of the following cases taes place: a and β = O1. b and / β. c, / β = O1, and β / = O1. d and β /. e β, = O1, and 1/ = O1. Cases a and b. If, then N y, X < N K for almost all IN so that β, is a solution of.1, lying in Θ. Since it satisfies.3, x K = n β N y + χ n x n β 1 + x n β n=+1 ; 7

8 8 In Case a we have y n x n β + y n x n β = y n x n β + y n x n β + M. 1 y nx n β because y n x n β is bounded for all n = 1,..., N M. It follows that M K. In Case b we may assume w.l.o.g. that β / β β 0. Since y n / β 0, we have y n x n β y + y n x n β = n β x β n β + β y n β x n 0, β β yielding M K in Case b too. Case c: Again, is away from 0, so that β, are solutions of.1, lying in Θ. Since ϕ is symmetric and subadditive, there exists L > 0 such that ϕs ϕt L ϕt s and ϕs L ϕt s ϕt for all s, t IR; see Mizera and Müller This property implies that 0 D K β,, y, X D K 0, 1, y, X = D K β,, y, X D K 0,, y, X + D K 0,, y, X D K 0, 1, y, X x n β N y = ϕ + ϕ n x n β + K log + + ϕ ϕ ϕy n + x ϕ n β ϕ n=+1 N n=+1 N n=+1 N n=+1 y ϕ n K log y ϕ n + K log ϕy n ϕ ϕy n N n=+1 x ϕ n β N M L M L

9 Setting A := + N n=+1 y ϕ n x ϕ n β N n=+1 N n=+1 ϕy n + K log. x ϕ n β N L ϕ ϕy n, 9 we have N 0 A + n=+1 N = A + n=+1 y ϕ n N ϕ + K log n=+1 y log 1 + n N log 1 + n=+1 + M log + K M log N 1 + y n = A + log K M log = A + n=+1 N n=+1 log + + K M log 1 +. Using the fact that 1 for large, we obtain dividing by log gives A + K M log ; 0 1 log A + K M. We may assume w.l.o.g. that β / β β 0 and / β c. Then together with y n / 0 we obtain that A = x β n N β ϕ x β n β ϕ N L ϕ ϕy n β β n=+1 is bounded. Hence A /log converges to 0 so that the limit of 3.1 gives M K/. Cases d and e. Since is greater than 0, we can use.3. Assuming w.l.o.g. that

10 10 β / β β 0, we obtain with y n / β 0 K = y n x n β + y n x n β + β + y n β x n β β y n β x n N n=+1 y χ n x n β N M N X. β β This implies M N K N X. Proposition 3 Explosion and implosion. Suppose there is a sequence y By, M such that the corresponding Cauchy estimates Cy, X = βy, X, y, X satisfy βy, X and y, X 0. Then M N K N X. Proof. Again, assume y n = y n for n = 1,..., N M for all IN, and let β := βy, X, := y, X. There are two possibilities. We have > 0 for almost all IN; then the assertion follows as for the Cases d and e in the proof of Proposition. Or, we have = 0 for almost IN. In the latter case, β satisfies #{n : y n = x n β } = N β, y, X N K. Since β the equality y n = y n = x n β can be satisfied for n {1,..., N M} only if y n = 0 = x n β, that means it is satisfied for at most N X elements of {1,..., N M}. Hence M + N X N β, y, X N K, so that M N K N X follows. Theorem 4 Explosion breadown point. The explosion breadown point ɛ C, y, X of the Cauchy regression-scale estimator satisfies ɛ C, y, X 1 N min { K, N K N X }. The lower bound is 1 N X, N if K = N; 1 N X, N if K = N N X.

11 Proof. Let y IN any sequence in By, M with Cy, X = βy, X, y, X. Then we have explosion with or without implosion so that the assertion follows from Proposition and 3. The Cauchy regression-scale estimator is regression equivariant: βy + Xβ, X = βy, X + β for all y, X and β. Hence, the upper bound 1.1 for regression equivariant estimators provides ɛ C, y, X 1 NN X+1, where equality holds in the case N K = N N X. Therefore, the Cauchy regression-scale estimator with tuning constant K = N N X has the highest explosion breadown point which is possible within regression equivariant estimators. However, the untuned Cauchy estimator K = N exhibits similar features if N X is small. A similar result holds for the implosion breadown point. Since the Cauchy scale estimator is allowed to be equal to 0, we investigate the implosion only for data points y, X with y, X > 0, when N y, X < N K. Proposition 5 Implosion without explosion. Let y, X > 0. If there is a sequence y By, M such that Cy, X = βy, X, y, X satisfy βy, X = O1 and y, X 0, then M N K N y, X. Proof. There are two possibilities: a For almost all IN we have y, X = 0: there exists β with N β, y, X N K. This implies M N β, y, X N y, X N K N y, X. b There is a subsequence with := y, X > 0. We may w.l.o.g. assume that β := βy, X β 0. According to.3, we obtain K = y n x n β + y n x n β + y n x n β + y n x n β,...,: y n x n β 0 N n=+1 y n x n β + y n x n β #{n = 1,..., N M : y n x n β 0 } y n x n β + y n x n β = N M #{n = 1,..., N M : y n = x n β 0 } N M N β 0, y, X N M N y, X, 11 so that M N K N y, X.

12 1 Theorem 6 Implosion breadown point. If y, X > 0, then the implosion point ɛ 0 C, y, X of the Cauchy regression-scale estimator satisfies ɛ 0 C, y, X 1 { N min N K N y, X, N K } N X. Proof. The assertion follows from Proposition 3 and 5, for any sequence y IN sequence in By, M with y, X 0. any Usually with probability 1 for data sampled from continuous populations we have that N y, X N X. Theorems 4 and 6 then imply that the breadown point of the Cauchy regression-scale estimator bounded from below by { 1 K N min, N K } N X, and attains the maximal value when tuning constant K = N N X. 4. Conclusion Our results imply, in particular, that the breadown point of the simultaneous location and scale Cauchy estimator is 1/. Copas 1975 showed that in this case the solution of the estimating equations is unique, except for the very special case when half of the observations is concentrated at one point and the another half at another point. Although Gabrielsen 198 showed that this property in general does not hold in regression-scale setting, Lange et al indicated that it is true for reasonable well-behaved datasets. Nevertheless, theoretical results that good breadown behavior of redescending M-estimators requires their appropriate definition: while breadown point of arbitrary roots may be poor, the breadown point of the regression-scale Cauchy estimator defined via global minimization is high when tuned optimally, maximal. References Copas, J.B On the unimodality of the lielihood for the Cauchy distribution. Biometria 6, Ellis, S. P. and Morgenthaler, S Leverage and breadown in L 1 regression. J. Amer. Statist. Assoc. 87, Gabrielsen, G On the unimodality of the lielihood for the Cauchy distribution: Some comments. Biometria 69, Hampel, F. R., Ronchetti, E. M., Rousseeuw, P. J. and Stahel, W. A Robust Statistics The Approach Based on Influence Functions. Wiley, New Yor.

13 He, X., Jurečová, J., Koener, R. and Portnoy, S Tail behavior of regression estimators and their breadown points. Econometrica 58, He, X., Simpson, D.G. and Wang, G Breadown points of t-type regression estimators. Biometria 87, Huber, P. J Robust Statistics. Wiley, New Yor. Huber, P. J Finite sample breadown of M- and P-estimators. Ann. Statist. 1, Lange, K.L., Little, R.J.A. and Taylor, J.M.G Robust statistical modeling using the t distribution. J. Amer. Statist. Assoc. 84, Maronna, R. A., Bustos, O. H. and Yohai, V. J Bias- and efficiency-robustness of general M-estimators for regression with random carriers. In: Smoothing Techniques for Curve Estimation, eds. T. Gasser and M. Rosenblatt, Lecture Notes in Mathematics 757, Springer, Berlin, Mizera, I. and Müller, Ch. H Breadown points and variation exponents of robust M-estimators in linear models. Ann. Statist. 7, Müller, Ch. H Breadown points for designed experiments. J. Statist. Plann. Inference. 45, Müller, Ch. H Robust Planning and Analysis of Experiments. Lecture Notes in Statistics 14, Springer, New Yor. Rousseeuw, P.J. and Croux, C Explicit scale estimators with high breadown point. In L 1 -Statistical Analysis and Related Methods, ed. Y. Dodge, North-Holland, Amsterdam, Rousseeuw, P.J. and Croux, C Alternatives to the median absolute deviation. J. Amer. Statist. Assoc. 88, Rousseeuw, P. J. and Leroy, A. M Robust Regression and Outlier Detection. Wiley, New Yor. 13