A NEW ROBUST AND EFFICIENT ESTIMATOR FOR ILL-CONDITIONED LINEAR INVERSE PROBLEMS WITH OUTLIERS

Transcription

1 A NEW ROBUST AND EFFICIENT ESTIMATOR FOR ILL-CONDITIONED LINEAR INVERSE PROBLEMS WITH OUTLIERS Marta Martinez-Caara 1, Michael Mua 2, Abdelhak M. Zoubir 2, Martin Vetterli 1 1 School of Coputer and Counication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland, arta.artinez-caara@epfl.ch artin.vetterli@epfl.ch 2 Signal Processing Group, Technische Universität Darstadt, Merckstr. 25, DE Darstadt, Gerany, ua@spg.tu-darstadt.de zoubir@spg.tu-darstadt.de ABSTRACT Solving a linear inverse proble ay include difficulties such as the presence of outliers and a iing atri with a large condition nuber. In such cases a regularized robust estiator is needed. We propose a new τ-type regularized robust estiator that is siultaneously highly robust against outliers, highly efficient in the presence of purely Gaussian noise, and also stable when the iing atri has a large condition nuber. We also propose an algorith to copute the estiates, based on a regularized iterative reweighted least squares algorith. A basic and a fast version of the algorith are given. Finally, we test the perforance of the proposed approach using nuerical eperients and copare it with other estiators. Our estiator provides superior robustness, even up to 40% of outliers, while at the sae tie perforing quite close to the optial aiu likelihood estiator in the outlier-free case. 1. INTRODUCTION Let a linear inverse proble be given by y = A + n. (1) The unknown source signal R n 1 is to be estiated. The easureent vector y R 1 and the iing atri A R n are known. Finally, the noise n R n 1 is in general unknown. This type of proble appears frequently in signal processing [1], as well as in physics, geosciences [2, 3], finance, and engineering in general. Traditionally, n is assued to have independent and identically distributed (iid) Gaussian rando entries. Under this assuption, the optial estiator is the least squares (LS) estiator [4]: = arg in y A 2 2. (2) However, in any odern engineering probles this idealistic noise odel does not hold. The distribution of n is far fro Gaussian when it contains outliers, which causes the distribution to be heavy This work was supported by a SNF Grant: Non-linear Sapling Methods FNS , by the project HANDiCAMS which acknowledges the financial support of the Future and Eerging Technologies (FET) prograe within the Seventh Fraework Prograe for Research of the European Coission, under FET-Open grant nuber: and by an ERC Advanced Grant Support for Frontier Research SPARSAM Nr: tailed [5]. In such scenarios, the LS-estiator s perforance drastically degrades. As a solution, any statistically robust estiators have been proposed in the literature. The first systeatic approach to statistically robust estiation is M-estiation [6], which generalizes aiu likelihood (ML) estiation. However M-estiators are non-robust, i.e., have a breakdown point (BP) equal to zero, when leverage points are present (errors in the entries of A). The BP is the proportion of outliers in the data that an estiator can handle before giving an uninforative (e.g. arbitrarily large) result. A second fundaental class of robust estiators are the ones that are based on the iniization of a robust residual scale. This class includes estiators such as the least-tried-squares (LTS) [7], the least-edian-of-squares (LMS) [7], and the S-estiators [8]. The latter iniizes the M-estiate of scale of the residuals. S- estiators can be highly robust, i.e., have a high BP. However, in this case, they do not have high efficiency, i.e., the variance of the S-estiator, when it is tuned for high robustness, is uch larger than for the ML-estiator. This led to the developent of a third class of robust estiators, which are siultaneously robust and efficient. The τ-estiator [9] belongs to this class. Another coon difficulty is A having a large condition nuber. In such cases, the LS solution in (2) does not depend continuously on the proble data y and A, i.e., it is unstable. Such inverse probles are said to be ill-posed. To stabilize, it is possible to introduce additional a priori inforation in the proble. This is referred to as regularization. Tikhonov regularization is one of the oldest and ost well-known techniques for dealing with ill-posed inverse probles [10]: = arg in y A λ 2 2. (3) Moreover, soe inverse probles suffer fro both outliers and a atri A with a large condition nuber [11]. In these cases, a regularized robust estiator is needed. Recently, there has been increased interest in regularized estiators in the robust statistics counity. In particular, regularized M-estiators [12], S-estiators [13], and MM-estiators [14] were proposed. Of these, only the MM-estiator is siultaneously efficient and robust against leverage points [5]. We propose a regularized estiator of the τ-type, which is siultaneously highly robust, efficient, and stable when A has a large condition nuber. An advantage of our proposed estiator over the /15/$ IEEE 3422 ICASSP 2015

2 regularized MM-estiator is that it provides a high BP and a highly efficient estiate of the scale of the errors, in addition to the regularized estiate. We also derive an algorith to copute the estiates, which forulates the τ-iniization proble as a regularized iterative reweighted least squares (IRLS) proble of a nonconve function. A fast version of the algorith is provided, which is highly practical. Siulation studies are also conducted to copare our estiator to the regularized LS-estiator and M-estiator in ters of solving an outlier-containated inverse proble, where the atri A has a high condition nuber. A brief discussion about how the regularization affects the robustness of the estiators is provided by running the sae eperient, but using a atri A with a low condition nuber. This paper is organized as follows. Section 2 introduces the proposed regularized τ-estiator. Section 3 contains the derivation of an algorith to solve the τ-iniization proble as a regularized IRLS proble of a non-conve function in a coputationally feasible way. Section 4, provides a Monte-Carlo siulation study that copares the proposed approach to the regularized LS-estiator and M-estiator in ters of the nor of the error. Section 5 concludes the paper and provides a brief outlook on future research. 2. THE REGULARIZED τ-estimator The τ-scale of the residuals σ τ (r()) [9] is defined by στ 2 (r()) = σm 2 (r()) 1 ( ) ri() ρ 2. (4) σ M (r()) Here, σ M (r()) is the robust M-scale estiate, given by 1 ( ) ri() ρ 1 = b, (5) σ M (r()) with b = E F [ρ 1(r())], E F [ ] denoting the epectation w.r.t. the standard Gaussian distribution F, and r i = y i A, for i {1 }. The choice of ρ 2, controls the efficiency of the τ-estiator, e.g., full efficiency for the Gaussian distribution can be obtained by setting ρ 2(r) = r 2, which corresponds to the ML-estiator under the Gaussian assuption. The BP of the τ-estiator, on the other hand, is controlled by ρ 1 and is the sae as that of an S-estiator [8] that uses ρ 1. In this way, one can obtain BP=0.5 siultaneously with high efficiency [9]. In this paper, we introduce the regularized τ-estiator of regression as the iniizer of the su of the τ-estiate of scale of the residuals and the Tikhonov regularization: =arg in τ 2 R(r()), (6) where τ 2 R(r()) = σ 2 τ (r()) + λ 2 2 and λ 0 is the regularization paraeter. 3. COMPUTING THE ESTIMATES A ajor challenge of the regularized τ-estiator is its coputation. We note that (6) is a non-conve function, so finding the eact global iniu ay be coputationally prohibitive. Thus, to iniize (6), we propose a odification of the heuristic algorith given in [15]. The ain idea behind this heuristic algorith is intuitively suarized as follows: find several local inia, and select the best one aong the as the final solution. There is, of course, no guarantee that this solution is the global iniu of (6) Finding local inia The paraeter values that locally iniize τ 2 R(r()) are the ones with a derivative w.r.t. being equal to zero. yields Defining r() := r() σ M (r()) (7) ψ j() := ρj(), (8) τr(r()) 2 σm (r()) 1 = 2σ M (r()) ρ 2( r i())+ [ ] + σm 2 (r()) 1 Aiσ M (r()) r i() σ M (r) ψ 2( r i()) σ 2 M (r()) + λ 2 i = 0. (9) To find σ M (r), we take the derivative of (5) w.r.t. : σ M (r()) = σ M (r()) ψ1( ri())ai ψ1( ri())ri() (10) Replacing (10) in (9) we obtain ( W ()ψ 1( r i()) ψ 2( r i()) where W () := ) σ M (r())a i + + λ 2 i = 0, (11) 2ρ2( ri()) + ψ2( ri()) ri() ψ1( ri()) ri(). (12) We can see that (11) is also f() where f() is the regularized weighted least squares estiator with weights n f() = w i()(y i A i) 2 + λ 2 i (13) w i() = ψτ () 2 r i(), (14) where ( ψ τ () = 1 ) W ()ψ 1( r i()) ψ 2( r i()). (15) 3423

3 So f() is just another way of writing (6). Hence, f() can be iniized iteratively as a regularized IRLS [16] proble with weights given by (14). The regularized IRLS algorith for our proble is detailed in Algorith 1. As f() is a non-conve function, the result of this iniisation depends on the initial condition [0] given to the IRLS. Algorith 1 Regularized IRLS INPUT: y R, A R n, λ, ɛ, [0], K require λ 0, ɛ 0, K 0 for k = 0 to K 1 do r[k] y A[k] Copute σ M [k] Copute W[k] [k + 1] (A W [k]w[k]a + λ 2 ) 1 A W [k]w[k]y if [k + 1] [k] < ɛ then break end if return [k + 1] Fro 11, we deduce that the regularized tau-estiator is asyptotically equivalent to a regularized M-estiator with data-adaptive psi-function. Then the basic algorith is shown below as Algorith 2. Algorith 2 Basic algorith INPUT: y R, A R n, λ, ɛ, K, Q require λ 0, ɛ 0, K 0, Q 0 for q = 0 to Q 1 do q[0] rando initial condition q Algorith 1 (y, A, λ, ɛ, q[0], K) return arg in τr(r( 2 q)) q Naturally, the value of K ust be large enough to ensure sufficient convergence of the solutions, typically on the order of hundreds of iterations. Hence, Algorith 2 ay require large run ties, especially when high nubers of candidate solutions, Q, are considered Fast algorith To reduce the coputational cost of the basic version of the algorith, we adopt the idea proposed in [15]. It is a two-step algorith, as shown below. Algorith 3 Fast algorith INPUT: y R, A R n, λ, ɛ, K, Q,M require λ 0, ɛ 0, K 0, Q 0, M 0 for q = 0 to Q 1 do q[0] rando initial condition q Algorith 1 (y, A, λ, ɛ, q[0], K) Z set of M best solutions q for z Z do z Algorith 1 (y, A, λ, ɛ, z, ) return arg in τr(r( 2 z)) z In the first step, a very sall K is chosen, in the order of 3 to 5. Hence, this step, though identical to the basic algorith, takes uch less tie. After the first step, instead of choosing just one best solution, as in Algorith 2, the M best solutions are kept as input to the second step. Note that in the second step, all the M solutions are forced to run to coplete convergence by setting K. Fro these M converged solutions, the best solution is selected as the final output. 4. NUMERICAL EXPERIMENTS In this section, we copare the perforance of the regularized τ- estiator with that of the regularized LS-estiator and the regularized M-estiator. The siulation setup is as follows: we generate a atri A R with rando iid Gaussian entries and a condition nuber of We use the sae piece-wise constant for all the eperients. Additive outliers and Gaussian noise are included in generating y: y = A + n G + n o, (16) where n G R has rando iid Gaussian entries. On the other hand, n o R is a sparse vector. The few non-zero randoly selected entries have a rando unifor value with zero ean and variance equal to 10 ties that of A. These entries represent the outliers. With y and A as input, is estiated with the three different estiators. To show the efficiency and robustness of the estiators in ters of the nor of the error e = ˆ, several eperients with different percentages of outliers (0%, 10%, 20%, 30%, 40%) are carried out. For the loss function ρ(t) of the M-estiator, we choose the Huber function [17], while for the τ-estiator we choose the so-called optial weight function with clipping paraeters c 1 = and c 2 = They are plotted in Figure 1. These loss functions produce a τ-estiator with BP = 0.5 and efficiency of 95% [15]. In each realization, the optial λ was found eperientally and used in the eperient. Figure 2 shows the average of the e evaluated over 100 Monte-Carlo runs. We can observe in Figure 2 that the e of the regularized LSestiator and regularized M-estiator strongly increase already for 10% of outliers. The regularized τ-estiator, on the other hand, provides superior robustness, even up to 40% of outliers. The LSestiator, which coincides with the ML-estiator, provides the best 3424

4 1 = = Fig. 1. Optial weight [15] loss functions ρ 1(t) (solid line) and ρ 2(t) (dashed line) used in the regularized-τ estiator. Thus, the aiu e = = 3.6 for the particular source we chose. This intuitively represents the BP of the regularized estiator. In the regularized case, the LS saturates to this aiu e, while the M-estiator reains under this bound. Its perforance is uch better than in the non-regularized case. The τ-estiator perfors well up to 40% of outliers in the regularized case, but in the non-regularized case it is not robust for 40% of outliers. Fro these results, we can conclude that the regularization iproves the robustness of the estiation. It includes in the proble ore a priori inforation about the solution, which perits the estiator to copute a solution closer to the real one, and to discard unlikely solutions which cause a breakdown LS M tau 4 3 Reg. LS Reg. M Reg. tau % outliers % outliers Fig. 2. Average of the produced by the regularized LSestiator, M-estiator, and τ-estiator evaluated over 100 Monte- Carlo runs. perforance in the outlier-free case. The τ-estiator has a high relative efficiency, producing estiates which are quite close to the LS-estiates. In order to deonstrate how the regularization affects the robustness of the estiators, we run the sae eperient, but using a atri A with a low condition nuber of the order of 10. Then, is estiated using the non-regularized LS-estiator, M-estiator, and τ-estiator, respectively. Results are shown in Figure 3. First, we note that the errors in the non-regularized case are, in general, one order of agnitude larger than in the regularized case. This is due to the regularization liiting the aiu error: we chose the optiu regularization paraeter. The aiu error is thus obtained when the paraeter is large enough to ake = 0. Fig. 3. Average of the produced by the non-regularized LSestiator, M-estiator, and τ-estiator evaluated over 200 Monte- Carlo runs. 5. CONCLUSIONS In this paper, we proposed a new regularized robust estiator that is siultaneously highly robust against outliers, highly efficient in the presence of purely Gaussian noise, and also stable when the iing atri has a large condition nuber. A large part of this paper was dedicated to deriving a fast way of coputing this estiator based on a regularized iterative reweighted least squares algorith. We showed that our estiator provides superior robustness, copared to eisting regularized estiators, and approiately aintains a perforance quite close to the optial aiu likelihood estiator in the outlier-free case, even up to 40% of outliers. The proposed estiator sees well suited to solve inverse transport odelling of atospheric eissions, where the iing atri ay also contain outliers. This real data application will be part of our future work. Furtherore, fundaental robustness concepts, such as the breakdown point, the aiu bias curve and the influence function are influenced by the regularization. We showed via nuerical eperients that a breakdown occurs when the nor of the errors converges to a fied value that depends on the regularization. A full statistical robustness analysis of the proposed estiator will provide new insights on the interrelation of robustness and regularization. 3425

5 6. REFERENCES [1] A. Ribes and F. Schitt, Linear inverse probles in iaging, IEEE Signal Process. Mag., pp , July [2] R. H. Stolt and A. B. Weglein, Seisic iaging and inversion, Cabridge, [3] M. Martinez-Caara, I. Dokanic, J. Ranieri, R. Scheibler, M. Vetterli, and A. Stohl, The Fukushia inverse proble, in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, May 2013, pp [4] P. J. Rousseeuw and A. M. Leroy, Robust regression and outlier detection, John Wiley & Sons, Ltd, [5] A. M. Zoubir, V. Koivunen, Y. Chakhchoukh, and M. Mua, Robust estiation in signal processing: a tutorial-style treatent of fundaental concepts, IEEE Signal Process. Mag., vol. 29, no. 4, pp , July [6] P. J. Huber, Robust estiation of a location paraeter, Ann. Math. Statist., vol. 35, no. 1, pp , [7] P. J. Rousseeuw, Least edian of squares regression, J. A. Statist. Assoc., vol. 79, pp , [8] P. J. Rousseeuw and V. J. Yohai, Robust regression by eans of S-estiators, Lect. Notes Stat., vol. 26, pp , [9] V. J. Yohai and R.H. Zaar, High breakdown-point estiates of regression by eans of the iniization of an efficient scale, J. Aer. Statist. Assoc., vol. 83, no. 402, pp , [10] A.N. Tikhonov and V.Y. Arsenin, Solution of ill-posed probles, Winston-Willey, New York, [11] M. Martinez-Caara, B. Béjar Haro, A. Stohl, and M. Vetterli, A robust ethod for inverse transport odelling of atospheric eissions using blind outlier detection, Geosci. Model Dev. Discuss., vol. 7, no. 3, pp , [12] M. J. Silavapulle, Robust ridge regrssion based on an M- estiator, Aust. J. Stat., vol. 33, no. 3, pp , [13] K. Thararatna, G. Claeskens, C. Crou, and M. Salibian- Barrera, S-estiation for penalised regression splines, J. Coput. Graph. Stat., vol. 19, pp , [14] R. A. Maronna, Robust ridge regression for high-diensional data, Technoetrics, vol. 53, no. 1, pp , [15] M. Salibian-Barrera, G. Willes, and R.H. Zaar, The fasttau estiator for regression, J. Coput. Graph. Stat., vol. 17, pp , [16] R. Wolke and H. Schwetlick, Iteratively reweighted least squares: Algoriths, convergence analysis, and nuerical coparisons, SIAM J. Sci. and Stat. Coput., vol. 9, no. 5, pp , [17] P. J. Huber and E. M. Rochetti, Robust Statistics, vol. 2, John Willey & Sons,