Robust Linear Regression: A Review and Comparison

Transcription

1 Roust Liner Regression: A Review nd Comprison rxiv: v1 [stt.me] 24 Apr 2014 Chun Yu 1, Weixin Yo 1, nd Xue Bi 1 1 Deprtment of Sttistis, Knss Stte University, Mnhttn, Knss, USA Astrt Ordinry lest-squres (OLS) estimtors for liner model re very sensitive to unusul vlues in the design spe or outliers mong y vlues. Even one single typil vlue my hve lrge effet on the prmeter estimtes. This rtile ims to review nd desrie some ville nd populr roust tehniques, inluding some reent developed ones, nd ompre them in terms of rekdown point nd effiieny. In ddition, we lso use simultion study nd rel dt pplition to ompre the performne of existing roust methods under different senrios. Key words: Brekdown point; Roust estimte; Liner Regression. 1 Introdution Liner regression hs een one of the most importnt sttistil dt nlysis tools. Given the independent nd identilly distriuted (iid) oservtions (x i, y i ), i = 1,..., n, 1

2 in order to understnd how the response y i s re relted to the ovrites x i s, we trditionlly ssume the following liner regression model y i = x T i β + ε i, (1.1) where β is n unknown p 1 vetor, nd the ε i s re i.i.d. nd independent of x i with E(ε i x i ) = 0. The most ommonly used estimte for β is the ordinry lest squre (OLS) estimte whih minimizes the sum of squred residuls n (y i x T i β) 2. (1.2) However, it is well known tht the OLS estimte is extremely sensitive to the outliers. A single outlier n hve lrge effet on the OLS estimte. In this pper, we review nd desrie some ville roust methods. In ddition, simultion study nd rel dt pplition re used to ompre different existing roust methods. The effiieny nd rekdown point (Donoho nd Huer 1983) re two trditionlly used importnt riteri to ompre different roust methods. The effiieny is used to mesure the reltive effiieny of the roust estimte ompred to the OLS estimte when the error distriution is extly norml nd there re no outliers. Brekdown point is to mesure the proportion of outliers n estimte n tolerte efore it goes to infinity. In this pper, finite smple rekdown point (Donoho nd Huer 1983) is used nd defined s follows: Let z i = (x i, y i ). Given ny smple z = (z i,..., z n ), denote T (z) the estimte of the prmeter β. Let z e the orrupted smple where ny m of the originl points of z re repled y ritrry d dt. Then the finite smple rekdown point δ is defined s { } m δ (z, T ) = min 1 m n n : sup T (z ) T (z) =, (1.3) z 2

3 where is Euliden norm. Mny roust methods hve een proposed to hieve high rekdown point or high effiieny or oth. M-estimtes (Huer, 1981) re solutions of the norml eqution with pproprite weight funtions. They re resistnt to unusul y oservtions, ut sensitive to high leverge points on x. Hene the rekdown point of n M-estimte is 1/n. R-estimtes (Jekel 1972) whih minimize the sum of sores of the rnked residuls hve reltively high effiieny ut their rekdown points re s low s those of OLS estimtes. Lest Medin of Squres (LMS) estimtes (Siegel 1982) whih minimize the medin of squred residuls, Lest Trimmed Squres (LTS) estimtes (Rousseeuw 1983) whih minimize the trimmed sum of squred residuls, nd S-estimtes (Rousseeuw nd Yohi 1984) whih minimize the vrine of the residuls ll hve high rekdown point ut with low effiieny. Generlized S-estimtes (GS-estimtes) (Croux et l. 1994) mintin high rekdown point s S-estimtes nd hve slightly higher effiieny. MM-estimtes proposed y Yohi (1987) n simultneously ttin high rekdown point nd effiienies. Mllows Generlized M-estimtes (Mllows 1975) nd Shweppe Generlized M-estimtes (Hndshin et l. 1975) downweight the high leverge points on x ut nnot distinguish good nd d leverge points, thus resulting in loss of effiienies. In ddition, these two estimtors hve low rekdown points when p, the numer of explntory vriles, is lrge. Shweppe one-step (S1S) Generlized M-estimtes (Cokley nd Hettmnsperger 1993) overome the prolems of Shweppe Generlized M-estimtes nd re lulted in one step. They oth hve high rekdown points nd high effiienies. Reently, Gervini nd Yohi (2002) proposed new lss of high rekdown point nd high effiieny roust estimte lled roust nd effiient weighted lest squres estimtor (REWLSE). Lee et l. (2011) nd She nd Owen (2011) proposed new lss of roust methods sed on the regulriztion of sespeifi prmeters for eh response. They further proved tht the M-estimtor with 3

4 Huer s ψ funtion is speil se of their proposed estimtor. The rest of the pper is orgnized s follows. In Setion 2, we review nd desrie some of the ville roust methods. In Setion 3, simultion study nd rel dt pplition re used to ompre different roust methods. Some disussions re given in Setion 4. 2 Roust Regression Methods 2.1 M-Estimtes By repling the lest squres riterion (1.2) with roust riterion, M-estimte (Huer, 1964) of β is ˆβ = rg min β n ( ) yi x T i β ρ, (2.1) ˆσ where ρ( ) is roust loss funtion nd ˆσ is n error sle estimte. The derivtive of ρ, denoted y ψ( ) = ρ ( ), is lled the influene funtion. In prtiulr, if ρ(t) = 1 2 t2, then the solution is the OLS estimte. The OLS estimte is very sensitive to outliers. Rousseeuw nd Yohi (1984) indited tht OLS estimtes hve rekdown point (BP) of BP = 1/n, whih tends to zero when the smple size n is getting lrge. Therefore, one single unusul oservtion n hve lrge impt on the OLS estimte. One of the ommonly used roust loss funtions is Huer s ψ funtion (Huer 1981), where ψ (t) = ρ (t) = mx{, min(, t)}. Huer (1981) reommends using = in prtie. This hoie produes reltive effiieny of pproximtely 95% when the error density is norml. Another possiility for ψ( ) is Tukey s isqure funtion ψ (t) = t{1 (t/) 2 } 2 +. The use of = produes 95% effiieny. If ρ(t) = t, then lest solute devition (LAD, lso lled medin regression) estimtes re hieved y 4

5 minimizing the sum of the solute vlues of the residuls ˆβ = rg min β n yi x T i β. (2.2) The LAD is lso lled L 1 estimte due to the L 1 norm used. Although LAD is more resistent thn OLS to unusul y vlues, it is sensitive to high leverge outliers, nd thus hs rekdown point of BP = 1/n 0 (Rousseeuw nd Yohi 1984). Moreover, LAD estimtes hve low effiieny of 0.64 when the errors re normlly distriuted. Similr to LAD estimtes, the generl monotone M-estimtes, i.e., M-estimtes with monotone ψ funtions, hve BP = 1/n 0 due to lk of immunity to high leverge outliers (Mronn, Mrtin, nd Yohi 2006). 2.2 LMS Estimtes The LMS estimtes (Siegel 1982) re found y minimizing the medin of the squred residuls ˆβ = rg min β Med{( y i x T i β ) 2 }. (2.3) One good property of the LMS estimte is tht it possesses high rekdown point of ner 0.5. However, the LMS estimte hs t est n effiieny of 0.37 when the ssumption of norml errors is met (see Rousseeuw nd Croux 1993). Moreover, LMS estimtes do not hve well-defined influene funtion euse of its onvergene rte of n 1 3 (Rousseeuw 1982). Despite these limittions, the LMS estimte n e used s the initil estimte for some other high rekdown point nd high effiieny roust methods. 5

6 2.3 LTS Estimtes The LTS estimte (Rousseeuw 1983) is defined s ˆβ = rg min β q r (i) (β) 2, (2.4) where r (i) (β) = y (i) x T (i) β, r (1) (β) 2 r (q) (β) 2 re ordered squred residuls, q = [n (1 α) + 1], nd α is the proportion of trimming. Using q = ( n 2 ) +1 ensures tht the estimtor hs rekdown point of BP = 0.5, nd the onvergene rte of n 1 2 (Rousseeuw 1983). Although highly resistent to outliers, LTS suffers dly in terms of very low effiieny, whih is out 0.08, reltive to OLS estimtes (Stromerg, et l. 2000). The reson tht LTS estimtes ll ttentions to us is tht it is trditionlly used s the initil estimte for some other high rekdown point nd high effiieny roust methods. 2.4 S-Estimtes S-estimtes (Rousseeuw nd Yohi 1984) re defined y ˆβ = rg min β ˆσ (r 1 (β),, r n (β)), (2.5) where r i (β) = y i x T i β nd ˆσ (r 1 (β),, r n (β)) is the sle M-estimte whih is defined s the solution of 1 n n ( ) ri (β) ρ = δ, (2.6) ˆσ for ny given β, where δ is tken to e E Φ [ρ (r)]. For the iweight sle, S-estimtes n ttin high rekdown point of BP = 0.5 nd hs n symptoti effiieny of 0.29 under the ssumption of normlly distriuted errors (Mronn, Mrtin, nd Yhi 2006). 6

7 2.5 Generlized S-Estimtes (GS-Estimtes) Croux et l. (1994) proposed generlized S-estimtes in n ttempt to improve the low effiieny of S-estimtors. Generlized S-estimtes re defined s ˆβ = rg min β S n(β), (2.7) where S n (β) is defined s S n (β) = sup { S > 0; ( ) 1 n ( ri r j ρ 2 S i<j ) k n,p }, (2.8) where r i = y i x T i β, p is the numer of regression prmeters, nd k n,p is onstnt whih might depend on n nd p. Prtiulrly, if ρ(x) = I( x 1) nd k n,p = (( n ) ( 2 hp ) ) ( / n ) 2 with hp = n+p+1, generlized S-estimtor yields speil se, the 2 lest qurtile differene (LQD) estimtor, whih is defined s ˆβ = rg min β Q n(r 1,..., r n ), (2.9) where Q n = { r i r j ; i < j} ( hp 2 ) (2.10) is the ( h p 2 ) th order sttisti mong the ( n 2) elements of the set { ri r j ; i < j}. Generlized S-estimtes hve rekdown point s high s S-estimtes ut with higher effiieny. 2.6 MM-Estimtes First proposed y Yohi (1987), MM-estimtes hve eome inresingly populr nd re one of the most ommonly employed roust regression tehniques. The MM-estimtes 7

8 n e found y three-stge proedure. In the first stge, ompute n initil onsistent estimte ˆβ 0 with high rekdown point ut possily low norml effiieny. In the seond stge, ompute roust M-estimte of sle ˆσ of the residuls sed on the initil estimte. In the third stge, find n M-estimte ˆβ strting t ˆβ 0. In prtie, LMS or S-estimte with Huer or isqure funtions is typilly used s the initil estimte ˆβ 0. Let ρ 0 (r) = ρ 1 (r/k 0 ), ρ(r) = ρ 1 (r/k 1 ), nd ssume tht eh of the ρ-funtions is ounded. The sle estimte ˆσ stisfies 1 n n ρ 0 r i (ˆβ) ˆσ = 0.5. (2.11) If the ρ-funtion is iweight, then k 0 = 1.56 ensures tht the estimtor hs the symptoti BP = 0.5. Note tht n M-estimte minimizes L(β) = (ˆβ) n ρ r i. (2.12) ˆσ Let ρ stisfy ρ ρ 0. Yohi (1987) showed tht if ˆβ stisfies L(ˆβ) (ˆβ 0 ), then ˆβ s BP is not less thn tht of ˆβ 0. Furthermore, the rekdown point of the MM-estimte depends only on k 0 nd the symptoti vrine of the MM-estimte depends only on k 1. We n hoose k 1 in order to ttin the desired norml effiieny without ffeting its rekdown point. In order to let ρ ρ 0, we must hve k 1 k 0 ; the lrger the k 1 is, the higher effiieny the MM-estimte n ttin t the norml distriution. Mronn, Mrtin, nd Yhi (2006) provides the vlues of k 1 with the orresponding effiienies of the iweight ρ-funtion. Plese see the following tle for more detil. Effiieny k However, Yohi (1987) indites tht MM-estimtes with lrger vlues of k 1 re more 8

9 sensitive to outliers thn the estimtes orresponding to smller vlues of k 1. In prtie, n MM-estimte with isqure funtion nd effiieny 0.85 (k 1 = 3.44) strting from isqure S-estimte is reommended. 2.7 Generlized M-Estimtes (GM-Estimtes) Mllows GM-estimte In order to mke M-estimte resistent to high leverge outliers, Mllows (1975) proposed Mllows GM-estimte tht is defined y ) n r i (ˆβ w i ψ ˆσ x i = 0, (2.13) where ψ(e) = ρ (e) nd w i = 1 h i with h i eing the leverge of the ith oservtion. The weight w i ensures tht the oservtion with high leverge reeives less weight thn oservtion with smll leverge. However, even good leverge points tht fll in line with the pttern in the ulk of the dt re down-weighted, resulting in loss of effieny Shweppe GM-estimte Shweppe GM-estimte (Hndshin et l. 1975) is defined y the solution of ) n r i (ˆβ w i ψ w iˆσ x i = 0, (2.14) whih djusts the leverge weights ording to the size of the residul r i. Crroll nd Welsh (1988) proved tht the Shweppe estimtor is not onsistent when the errors re symmetri. Furthermore, the rekdown points for oth Mllows nd Shweppe GMestimtes re no more thn 1/(p + 1), where p is the numer of unknown prmeters. 9

10 2.7.3 S1S GM-estimte Cokley nd Hettmnsperger (1993) proposed Shweppe one-step (S1S) estimte, whih extends from the originl Shweppe estimtor. S1S estimtor is defined s ˆβ = ˆβ n 0 + ψ ) (ˆβ0 r i ˆσw i 1 x i x i ) n ˆσw i ψ r i (ˆβ0 x i, (2.15) ˆσw i where the weight w i is defined in the sme wy s Shweppe s GM-estimte. The method for S1S estimte is different from the Mllows nd Shweppe GMestimtes in tht one the initil estimtes of the residuls nd the sle of the residuls re given, finl M-estimtes re lulted in one step rther thn itertively. Cokley nd Hettmnsperger (1993) reommended to use Rousseeuw s LTS for the initil estimtes of the residuls nd LMS for the initil estimtes of the sle nd proved tht the S1S estimte gives rekdown point of BP = 0.5 nd results in 0.95 effiieny ompred to the OLS estimte under the Guss-Mrkov ssumption. 2.8 R-Estimtes The R-estimte (Jekel 1972) minimizes the sum of some sores of the rnked residuls n n (R i ) r i = min, (2.16) where R i represents the rnk of the ith residul r i, nd n ( ) is monotone sore funtion tht stisfies n n (i) = 0. (2.17) R-estimtes re sle equivlent whih is n dvntge ompred to M-estimtes. However, the optiml hoie of the sore funtion is unler. In ddition, most of R-estimtes 10

11 hve rekdown point of BP = 1/n 0. The ounded influene R-estimtor proposed y Nrnjo nd Hettmnsperger (1994) hs firly high effiieny when the errors hve norml distriution. However, it is proved tht their rekdown point is no more thn REWLSE Gervini nd Yohi (2002) proposed new lss of roust regression method lled roust nd effiient weighted lest squres estimtor (REWLSE). REWLSE is muh more ttrtive thn mny other roust estimtors due to its simultneously ttining mximum rekdown point nd full effiieny under norml errors. This new estimtor is type of weighted lest squres estimtor with the weights dptively lulted from n initil roust estimtor. Consider pir of initil roust estimtes of regression prmeters nd sle, ˆβ 0 nd ˆσ respetively, the stndrdized residuls re defined s r i = y i x T ˆβ i 0. ˆσ A lrge vlue of r i would suggest tht (x i, y i ) is n outlier. Define mesure of proportion of outliers in the smple { d n = mx F + ( r i>i (i) ) 0 } + (i 1), (2.18) n where { } + denotes positive prt, F + denotes the distriution of X when X F, r (1)... r (n) re the order sttistis of the stndrdized solute residuls, nd { } i 0 = mx i : r (i) < η, where η is some lrge quntile of F +. Typilly η = 2.5 nd the df of norml distriution is hosen for F. Thus those nd n oservtions with lrgest stndrdized solute residuls re eliminted (here is the lrgest integer less 11

12 thn or equl to ). The dptive ut-off vlue is t n = r (in) with i n = n nd n. With this dptive ut-off vlue, the dptive weights proposed y Gervini nd Yohi (2002) re 1 if r i < t n w i = 0 if r i t n. (2.19) Then, the REWLSE is ˆβ = (X T W X) 1 X T W y, (2.20) where W = dig(w 1,, w n ), X = (x 1,..., x n ) T, nd y = (y 1,, y n ). If the initil regression nd sle estimtes with BP = 0.5 re hosen, the rekdown point of the REWLSE is lso 0.5. Furthermore, when the errors re normlly distriuted, the REWLSE is symptotilly equivlent to the OLS estimtes nd hene symptotilly effiient Roust regression sed on regulriztion of se-speifi prmeters She nd Owen (2011) nd Lee et l. (2011) proposed new lss of roust regression methods using the se-speifi inditors in men shift model with regulriztion method. A men shift model for the liner regression is y = Xβ + γ + ε, ε N(0, σ 2 I) where y = (y 1,, y n ) T, X = (x 1,..., x n ) T, nd the men shift prmeter γ i is nonzero when the ith oservtion is n outlier nd zero, otherwise. Due to the sprsity of γ i s, She nd Owen (2011) nd Lee et l. (2011) proposed to 12

13 estimte β nd γ y minimizing the penlized lest squres using L 1 penlty: L(β, γ) = 1 n 2 {y (Xβ + γ)}t {y (Xβ + γ)} + λ γ i, (2.21) where λ re fixed regulriztion prmeters for γ. Given the estimte ˆγ, ˆβ is the OLS estimte with y repled y y γ. For fixed ˆβ, the minimizer of (2.21) is ˆγ i = sgn(r i )( γ i λ) +, tht is, 0 if r i λ; ˆγ i = y i x T ˆβ i if r i > λ. Therefore, the solution of (2.21) n e found y itertively updting the ove two steps. She nd Owen (2011) nd Lee et l. (2011) proved tht the ove estimte is in ft equivlent to the M-estimte if Huer s ψ funtion is used. However, their proposed roust estimtes re sed on different perspetive nd n e extended to mny other likelihood sed models. Note, however, the monotone M-estimte is not resistent to the high leverge outliers. In order to overome this prolem, She nd Owen (2011) further proposed to reple the L 1 penlty in (2.21) y generl penlty. The ojetive funtion is then defined y L p (β, γ) = 1 2 {y (Xβ + γ)}t {y (Xβ + γ)} + n p λ ( γ i ), (2.22) where p λ ( ) is ny penlty funtion whih depends on the regulriztion prmeter λ. We n find ˆγ y defining thresholding funtion Θ(γ; λ) (She nd Owen 2009). She nd Owen (2009, 2011) proved tht for speifi thresholding funtion, we n lwys find the orresponding penlty funtion. For exmple, the soft, hrd, nd smoothly lipped solute devition (SCAD; Fn nd Li, 2001) thresholding solutions of γ orrespond to 13

14 L 1, Hrd, nd SCAD penlty funtions, respetively. Minimizing the eqution (2.22) yields sprse ˆγ for outlier detetion nd roust estimte of β. She nd Owen (2011) showed tht the proposed estimtes of (2.22) with hrd or SCAD penlties re equivlent to the M-estimtes with ertin redesending ψ funtions nd thus will e resistent to high leverge outliers if high rekdown point roust estimtes re used s the initil vlues. 3 Exmples In this setion, we ompre different roust methods nd report the men squred errors (MSE) of the prmeter estimtes for eh estimtion method. We ompre the OLS estimte with seven other ommonly used roust regression estimtes: the M estimte using Huer s ψ funtion (M H ), the M estimte using Tukey s isqure funtion (M T ), the S estimte, the LTS estimte, the LMS estimte, the MM estimte (using isqure weights nd k 1 = 4.68), nd the REWLSE. Note tht we didn t inlude the se-speifi regulriztion methods proposed y She nd Owen (2011) nd Lee et l. (2011) sine they re essentilly equivlent to M-estimtors (She nd Owen (2011) did show tht their new methods hve etter performne in deteting outliers in their simultion study). Exmple 1. We generte n smples {(x 1, y 1 ),..., (x n, y n )} from the model Y = X + ε, where X N(0, 1). In order to ompre the performne of different methods, we onsider the following six ses for the error density of ε: Cse I: ε N(0, 1)- stndrd norml distriution. Cse II: ε t 3 - t-distriution with degrees of freedom 3. 14

15 Cse III: ε t 1 - t-distriution with degrees of freedom 1 (Cuhy distriution). Cse IV: ε 0.95N(0, 1) N(0, 10 2 ) - ontminted norml mixture. Cse V: ε N (0,1) with 10% identil outliers in y diretion (where we let the first 10% of y s equl to 30). Cse VI: ε N (0,1) with 10% identil high leverge outliers (where we let the first 10% of x s equl to 10 nd their orresponding y s equl to 50). Tles 1 nd 2 report the men squred errors (MSE) of the prmeter estimtes for eh estimtion method with smple size n = 20 nd 100, respetively. The numer of replites is 200. From the tles, we n see tht MM nd REWLSE hve the overll est performne throughout most ses nd they re onsistent for different smple sizes. For Cse I, LSE hs the smllest MSE whih is resonle sine under norml errors LSE is the est estimte; M H, M T, MM, nd REWLSE hve similr MSE to LSE, due to their high effiieny property; LMS, LTS, nd S hve reltive lrger MSE due to their low effiieny. For Cse II, M H, M T, MM, nd REWLSE work etter thn other estimtes. For Cse III, LSE hs muh lrger MSE thn other roust estimtors; M H, M T, MM, nd REWLSE hve similr MSE to S. For Cse IV, M, MM, nd REWLSE hve smller MSE thn others. From Cse V, we n see tht when the dt ontin outliers in the y-diretion, LSE is muh worse thn ny other roust estimtes; MM, REWLSE, nd M T re etter thn other roust estimtors. Finlly for Cse VI, sine there re high leverge outliers, similr to LSE, oth M T nd M H perform poorly; MM nd REWLSE work etter thn other roust estimtes. In order to etter ompre the performne of different methods, Figure 1 shows the plot of their MSE versus eh se for the slope (left side) nd interept (right side) prmeters for model 1 when smple size n = 100. Sine the lines for LTS nd LMS re ove the other lines, S, MM, nd REWLSE of the interept nd slopes outperform 15

16 LTS nd LMS estimtes throughout ll six ses. In ddition, the S estimte hs similr performne to MM nd REWLSE when the error density of ε is Cuhy distriution. However, MM nd REWLSE perform etter thn S-estimtes in other five ses. Furthermore, the lines for MM nd REWLSE lmost overlp for ll six ses. It shows tht MM nd REWLSE re the overll est pprohes in roust regression. Exmple 2. Y = X 1 + X 2 + X 3 + ε, where X i N(0, 1), i = 1, 2, 3 nd X i s re independent. We onsider the following six ses for the error density of ε: Cse I: ε N(0, 1)- stndrd norml distriution. Cse II: ε t 3 - t-distriution with degrees of freedom 3. Cse III: ε t 1 - t-distriution with degrees of freedom 1 (Cuhy distriution). Cse IV: ε 0.95N(0, 1) N(0, 10 2 ) - ontminted norml mixture. Cse V: ε N(0, 1) with 10% identil outliers in y diretion (where we let the first 10% of y s equl to 30). Cse VI: ε N(0, 1) with 10% identil high leverge outliers (where we let the first 10% of x s equl to 10 nd their orresponding y s equl to 50). Tles 3 nd 4 show the men squred errors (MSE) of the prmeter estimtes of eh estimtion method for smple size n = 20 nd n = 100, respetively. Figure 2 shows the plot of their MSE versus eh se for three slopes nd the interept prmeters with smple size n = 100. The results in Exmple 2 tell similr stories to Exmple 1. In summry, MM nd REWLSE hve the overll est performne; LSE only works well when there re no outliers sine it is very sensitive to outliers; M-estimtes (M H nd 16

17 M T ) work well if the outliers re in y diretion ut re lso sensitive to the high leverge outliers. Exmple 3: Next, we use the fmous dt set found in Freedmn et l. (1991) to ompre LSE with MM nd REWLSE. The dt set re shown in Tle 5 whih ontins per pit onsumption of igrettes in vrious ountries in 1930 nd the deth rtes (numer of deths per million people) from lung ner for Here, we re interested in how the deth rtes per million people from lung ner (dependent vrile y) dependent on the onsumption of igrettes per pit (the independent vrile x). Figure 3 is stter plot of the dt. From the plot, we n see tht USA (x = 1300, y = 200) is n outlier with high leverge. We ompre different regression prmeters estimtes y LSE, MM, nd REWLSE. Figure 3 shows the fitted lines y these three estimtes. The LSE line does not fit the ulk of the dt, eing ompromise etween USA oservtion nd the rest of the dt, while the fitted lines for the other two estimtes lmost overlp nd give etter representtion of the mjority of the dt. Tle 6 lso gives the estimted regression prmeters of these three methods for oth the omplete dt nd the dt without the outlier USA. For LSE, the interept estimte hnges from (omplete dt set) to 9.14 (without outlier) nd the slope estimte hnges from 0.23 (omplete dt set) to 0.37 (without outlier). Thus, it is ler tht the outlier USA strongly influenes LSE. For MM-estimte, fter deleting the outlier, the interept estimte hnges slightly ut slope estimte remins lmost the sme. For REWLSE, oth interept nd slope estimtes remin unhnged fter deleting the outlier. In ddition, note tht REWLSE for the whole dt gives lmost the sme result s LSE without the outlier. 17

18 4 Disussion In this rtile, we desrie nd ompre different ville roust methods. Tle 7 summrizes the roustness ttriutes nd symptoti effiieny of most of the estimtors we hve disussed. Bsed on Tle 7, it n e seen tht MM-estimtes nd REWLSE hve oth high rekdown point nd high effiieny. Our simultion study lso demonstrted tht MM-estimtes nd REWLSE hve overll est performne mong ll ompred roust methods. In terms of rekdown point nd effiieny, GM-estimtes (Mllows, Shweppe), Bounded R-estimtes, M-estimtes, nd LAD estimtes re less ttrtive due to their low rekdown points. Although LMS, LTS, S-estimtes, nd GS-estimtes re strongly resistent to outliers, their effiienies re low. However, these high rekdown point roust estimtes suh s S-estimtes nd LTS re trditionlly used s the initil estimtes for some other high rekdown point nd high effiieny roust estimtes. Referenes Crroll, R. J. nd Welsh, A. H. (1988), A Note on Asymmetry nd Roustness in Liner Regression. Journl of Amerin Sttisitl Assoition, 4, Cokley, C. W. nd Hettmnsperger, T. P. (1993), A Bounded Influene, High Brekdown, Effiient Regression Estimtor. Journl of Amerin Sttistil Assoition, 88, Croux, C., Rousseeuw, P. J., nd Hössjer O. (1994), Generlized S-estimtors. Journl of Amerin Sttistil Assoition, 89, Donoho, D. L. nd Huer, P. J. (1983), The Nottion of Brek-down Point, in A Festshrift for E. L. Lehmnn, Wdsworth 18

19 Freedmn, W. L., Wilson, C. D., nd Mdore, B. F. (1991), New Cepheid Distnes to Nery Glxies Bsed on BVRI CCD Photometry. Astrophysil Journl, 372, Fn, J. nd Li, R. (2001). Vrile seletion vi nononve penlized likelihood nd its orle properties. Journl of the Amerin Sttistil Assoition, 96, Gervini, D. nd Yohi, V. J. (2002), A Clss of Roust nd Fully Effiient Regression Estimtors. The Annls of Sttistis, 30, Hndshin, E., Kohls, J., Fiehter, A., nd Shweppe, F. (1975), Bd Dt Anlysis for Power System Stte Estimtion. IEEE Trnstions on Power Apprtus nd Systems, 2, Huer, P.J. (1981), Roust Sttistis. New York: John Wiley nd Sons. Jkel, L.A. (1972), Estimting Regression Coeffiients y Minimizing the Dispersion of the Residuls. Annls of Mthemtil Sttistis, 5, Lee, Y., MEhern, S. N., nd Jung, Y. (2011), Regulriztion of Cse-Speifi Prmeters for Roustness nd Effiieny. Sumitted to the Sttistil Siene. Mllows, C.L. (1975), On Some Topis in Roustness. unpulished memorndum, Bell Tel. Lortories, Murry Hill. Mronn, R. A., Mrtin, R. D. nd Yohi, V. J. (2006), Roust Sttistis. John Wiley. Nrnjo, J.D., Hettmnsperger, T. P. (1994), Bounded Influene Rnk Regression. Journl of the Royl Sttistil Soiety B, 56, Rousseeuw, P.J.(1982), Lest Medin of Squres regression. Reserh Report No. 178, Centre for Sttistis nd Opertions reserh, VUB Brussels. 19

20 Rousseeuw, P.J.(1983), Multivrite Estimtion with High Brekdown Point. Reserh Report No. 192, Center for Sttistis nd Opertions reserh, VUB Brussels. Rousseeuw, P.J. nd Croux, C.(1993), Alterntives to the Medin Asolute Devition. Journl of Amerin Sttistil Assoition, 94, Rousseeuw, P.J. nd Yohi, V. J. (1984). Roust Regression y Mens of S-estimtors. Roust nd Nonliner Time series, J. Frnke, W. Härdle nd R. D. Mrtin (eds.),letures Notes in Sttistis 26, , New York: Springer. She, Y. (2009). Thresholding-Bsed Itertive Seletion Proedures for Model Seletion nd Shrinkge. Eletroni Journl of Sttistis, 3, 384C415. She, Y. nd Owen, A. B. (2011), Outlier Detetion Using Nononvex Penlized regression. Journl of Amerin Sttistil Assoition, 106, Siegel, A.F. (1982), Roust Regression Using Repeted Medins. Biometrik, 69, Stromerg, A. J., Hwkins, D. M., nd Hössjer, O. (2000), The Lest Trimmed Differenes Regression Estimtor nd Alterntives. Journl of Amerin Sttistil Assoition, 95, Yohi, V. J. (1987), High Brekdown-point nd High Effiieny Roust Estimtes for Regression. The Annls of Sttistis, 15,

21 Tle 1: MSE of Point Estimtes for Exmple 1 with n = 20 TRUE OLS M H M T LMS LTS S MM REWLSE Cse I: ε N(0, 1) β 0 : β 1 : Cse II: ε t 3 β 0 : β 1 : Cse III: ε t 1 β 0 : β 1 : Cse IV: ε 0.95N(0, 1) N(0, 10 2 ) β 0 : β 1 : Cse V: ε N(0, 1) with outliers in y diretion β 0 : β 1 : Cse VI: ε N(0, 1) with high leverge outliers β 0 : β 1 : Tle 2: MSE of Point Estimtes for Exmple 1 with n = 100 TRUE OLS M H M T LMS LTS S MM REWLSE Cse I: ε N(0, 1) β 0 : β 1 : Cse II: ε t 3 β 0 : β 1 : Cse III: ε t 1 β 0 : β 1 : Cse IV: ε 0.95N(0, 1) N(0, 10 2 ) β 0 : β 1 : Cse V: ε N(0, 1) with outliers in y diretion β 0 : β 1 : Cse VI: ε N(0, 1) with high leverge outliers β 0 : β 1 :

22 Tle 3: MSE of Point Estimtes for Exmple 2 with n = 20 TRUE OLS M H M T LMS LTS S MM REWLSE Cse I: ε N(0, 1) β 0 : β 1 : β 2 : β 3 : Cse II: ε t 3 β 0 : β 1 : β 2 : β 3 : Cse III: ε t 1 β 0 : β 1 : β 2 : β 3 : Cse IV: ε 0.95N(0, 1) N(0, 10 2 ) β 0 : β 1 : β 2 : β 3 : Cse V: ε N(0, 1) with outliers in y diretion β 0 : β 1 : β 2 : β 3 : Cse VI: ε N(0, 1) with high leverge outliers β 0 : β 1 : β 2 : β 3 :

23 Tle 4: MSE of Point Estimtes for Exmple 2 with n = 100 TRUE OLS M H M T LMS LTS S MM REWLSE Cse I: ε N(0, 1) β 0 : β 1 : β 2 : β 3 : Cse II: ε t 3 β 0 : β 1 : β 2 : β 3 : Cse III: ε t 1 β 0 : β 1 : β 2 : β 3 : Cse IV: ε 0.95N(0, 1) N(0, 10 2 ) β 0 : β 1 : β 2 : β 3 : Cse V: ε N(0, 1) with outliers in y diretion β 0 : β 1 : β 2 : β 3 : Cse VI: ε N(0, 1) with high leverge outliers β 0 : β 1 : β 2 : β 3 :

24 Tle 5: Cigrettes dt Country Per pit onsumption of igrette Deths rtes Austrli Cnd Denmrk Finlnd GretBritin Ielnd Netherlnds Norwy Sweden Switzerlnd USA Tle 6: Regression estimtes for Cigrettes dt Complete dt Dt without USA Estimtors Interept Slope Interept Slope LS MM REWLSE Tle 7: Brekdown Points nd Asymptoti Effiienies of Vrious Regression Estimtors Estimtor Brekdown Point Asymptoti Effiieny High BP LMS LTS S-estimtes GS-estimtes MM-estimtes REWLSE Low BP GM-estimtes(Mllows,Shweppe) 1/(p + 1) 0.95 Bounded R-estimtes < Monotone M-estimtes 1/n 0.95 LAD 1/n 0.64 OLS 1/n

25 MSE vs. Cses for interept MSE vs. Cses for slope MSE de de e d d e LMS LTS S MM REWLSE de de de MSE de de e d d e LMS LTS S MM REWLSE de de de Cse Cse Figure 1: Plot of MSE of interept (left) nd slope (right) estimtes vs. different ses for LMS, LTS, S, MM, nd REWLSE, for model 1 when n =

26 MSE vs. Cses for interept MSE vs. Cses for et1 MSE d e LMS LTS S MM REWLSE de de de de de de MSE de de de d e LMS LTS S MM REWLSE de de de Cse Cse MSE vs. Cses for et2 MSE vs. Cses for et3 MSE d e LMS LTS S MM REWLSE de d de e de de de MSE d e LMS LTS S MM REWLSE de de de de de de Cse Cse Figure 2: Plot of MSE of different regression prmeter estimtes vs. different ses for LMS, LTS, S, MM, nd REWLSE, for model 2 when n =

27 Deth rtes LS line MM line REWLSE line Per pit onsumption of igrettes Figure 3: Fitted lines for Cigrettes dt 27