M-Estimators in Regression Models

Transcription

1 M-Estimators i Regressio Models MuthukrishaR Departmet of Statistics, Bharathiar Uiversity Coimbatore , Tamiladu, Idia muthukrisha70@rediffmailcom RadhaM Departmet of Statistics, Bharathiar Uiversity Coimbatore , Tamiladu, Idia radhamyilsamy@gmailcom Abstract Regressio aalysis plays a vital role i may areas of sciece Almost all regressio aalyses rely o the method of least squares for estimatio of the parameters i the model But this method is usually costructed uder specific assumptios, such as ormality of the error distributio Whe outliers are preset i the data, this method of estimatio, results i parameter estimates that do ot provide useful iformatio for the majority of the data Robust regressio aalyses have bee developed as a improvemet to least square estimatio i the presece of outliers The mai purpose of robust regressio aalysis is to fit a model that represets the iformatio of the majority of the data May researchers have worked i this field ad developed methods for these problems The most commoly used robust estimators are Huber s M-estimator, Hampel estimator, Tukey s bisquare estimator etc I this paper, a attempt is made to review such type of estimators ad a simulatio study of these estimators i regressio models is carried out R code has bee writte for the purpose ad illustratios are provided Keywords: Regressio Model, Robust estimator, M-estimators, R software 1 Itroductio The theory of robustess developed by Huber ad Hampel (1960) laid the foudatio for fidig practical solutios too may problems, whe statistical cocepts were vague to serve the purpose Robust regressio aalyses have bee developed as a improvemet to least squares estimatio i the presece of outliers ad to provide us iformatio about what a valid observatio is ad whether this should be throw The primary purpose of robust regressio aalysis is to fit a model which represets the iformatio i the majority of the data Robust regressio is a importat tool for aalyzig data that are cotamiated with outliers It ca be used to detect outliers ad to provide resistat results i the presece of outliers May methods have bee developed for these problems May researchers have worked i this field ad described the methods of robust estimators The class of robust estimators icludes M-, L- ad R-estimators The M-estimators are most fleible oes, ad they geeralize straightforwardly to multiparameter problems, eve though they are ot automatically scale ivariat ad have to be supplemeted for practical applicatios by a auiliary estimate of scale ay estimate I this paper, a attempt has bee made to make a elaborate study of the some of the M-estimators Sectio 2 deals with the descriptios of the M-estimators The redescedig M-estimators are preseted i the sectio 3 A simulatio study of these estimators providig certai umerical illustratios by usig R software is preseted i the last sectio 2 M-estimator The class of M-estimator was itroduced by PJHuber i 1964; subsequetly, such estimators have bee discussed etesively by several authors, Adrews et al (1972), Buke ad Buke (1986), Hampel et al (1986), Lecoutre ad Tassi (1987), Robusseeuw ad Leroy (1987), Staudte ad Sheather (1990), Rieder (1994), Jureckova ad Se (1996), Atoch et al (1998), Dodge ad Jureckova (2000), Jureckova ad Picek (2006) ad others M-estimator T is defied as a solutio of the miimizatio problem, ρ(x i,θ):= mi, with respect to θ Θ E P [ρ(x,θ)] = mi, θ Θ (1) where ρ(, ) is a properly chose fuctio The class of M-estimator covers also the maimal likelihood estimator of parameter θ i the parametric model P = {P θ,θ Θ};if f (,θ), is the desity fuctio of P θ, the the MLE is a solutio of Published by Caadia Ceter of Sciece ad Educatio 23

2 the miimizatio ( log f (X i,θ)) = mi, θ Θ If ρ i (1) is differetiable i θ with a cotiuous derivative ψ(,θ) = θ ρ(,θ) the, T is a root or roots of the equatio ψ(x i,θ) = 0, θ Θ hece 1 ψ(x i, T ) = E P [(X, T )] = 0 T Θ (2) From (1) ad (2) that the M-fuctioal correspodig to T, is defied as a solutio of the miimizatio ρ(, T(P))dP() = E P [ρ(x, T(P)] = mi, T(P) Θ (3) or as the solutio of the equatio ψ(, T(P))dP() = E P [ψ(x, T(P)] = mi, T(P) Θ (4) The fuctio T(P) is Fisher cosistet, if the solutios of (3) ad (4) are uiquely determied 21 M-estimator of Locatio parameter A importat special case is the model with the shift parameter θ, where X 1, X 2,, X are idepedet observatios with the same distributio fuctio F( θ),θ R; the distributio fuctio F is geerally ukow M-estimator of locatio parameter T is defied as a solutio of the miimizatio ad if ρ( )isdifferetiable with absolutely derivative ψ( ), the T solves the equatio ρ( i θ) := mi (5) ψ( i θ) = 0 (6) The correspodig M-fuctioal T(F) is Fisher cosistet, provided the miimizatio ρ(x θ)dp() = mi (7) have a uique solutio θ = 0, ie, the solutio of the equatio is, ψ(x θ)dp() = 0 (8) 22 Asymptotic properties of M-estimator A fairly simple ad straightforward theory is possible if ψ(,θ) is mootoe i θ Assume that ψ(,θ) is measurable i ad decreasig i θ, from strictly positive to strictly egative values Put T = Sup t ψ( i ; t) > 0, Clearly, < T T T = If t ψ( i ; t) < 0, (9) <, ad ay value T satisfyig T T T ca serve as our estimate Note that { T < t } { ψ( i ; t) 0 } { T t }, 24 ISSN E-ISSN

3 hece { T < t } { ψ( i ; t) < 0 } { T t } (10) P { T < t } = P { ψ( i ; t) 0 }, P { T < t } = P { ψ( i ; t) < 0 }, (11) at the cotiuity poits t of the left-had side The distributio of the customary midpoit estimate 1/2(T + T )is somewhat difficult to work out, but the radomized estimate T, which selects oe of T or T at radom with equal probability, has a eplicitly epressible distributio fuctio P {T < t} = 1 2 P { ψ( i ; t) 0 } P { ψ( i ; t) < 0 } (12) It follows that the eact distributios of T, T, ad T ca be calculated from the covolutio powers of G = L(ψ( i ; t)) Let, λ(t) = λ(t, F) = E F ψ(x, t) (13) If λ eists ad is fiite for atleast oe value of t, the it eists ad is mootoe for all t Assume that there is a t 0 such that λ(t) > 0 for t < t 0 ad λ(t) < 0 for t > t 0 The both T ad T coverge i probability ad almost surely to t 0 Cosider the followig coditios, (C1) ψ(, t) is measurable i ad mootoe decreasig i t (C2) There is atleast oe t 0 for which λ(t 0 ) = 0 Let Γ 0 be the set of t-values for which λ(t) = 0 (C3) λ is cotiuous i a eighborhood of Γ 0 (C4) σ(t) 2 = E F [ψ(x, t) 2 ] λ(t, F) 2 is fiite, ozero, ad cotiuous i a eighborhood of Γ 0 Put σ 0 = σ(t 0 ) Uder the above coditios λ(t ) is a asymptotically ormal N(0,σ 2 0 ) 3 Redescedig M-estimator Redescedig M-estimators are very popular Ψ-type M-Estimator which has Ψ fuctios that are o-decreasig ear the origi, but decreasig toward 0 far from the origi Their Ψ fuctios ca be chose to redesced smoothly to zero, so that they usually satisfy Ψ() = 0 for all with X > k,where r is referred to as the miimum rejected poit Whe choosig a redescedig Ψ fuctios we must take care that it does ot desced too steeply, which may have a very bad ifluece o the deomiator i the epressio for the asymptotic variace Ψ 2 df ( (Ψ df)) 2 where F is the miture model distributio This effect is particularly harmful whe a large egative values of Ψ () combies with a large positive values ofψ 2 (), ad there is a cluster of outliers ear First we itroduce Hampel s three-part M-estimator, it has Ψ fuctios which are odd fuctios ad defied for ay by: ψ() =, 0 a (cetral segmet) asig() a b (high ad low f lat segmets) a(k ) k b sig() b k (ed slopes) 0, k (le f t ad right tails) Tukey s biweight or bisquare M-estimator have ψ fuctios for ay positive k, which defied by { } [(1 /k) ψ() = 2 ] 2 for k 0 for > k Huber proposed fuctio i 1964,that is { ψ() =, for k ksig, for > k } For regressio aalysis, some of the redecedig M-estimators ca attai the maimum breakdow poit Moreover, some of them are the solutios of the problem of maimizig the efficiecy uder bouded ifluece fuctio whe the regressio coefficiet ad the scale parameter are estimated simultaeously Hece redecedig M-estimators satisfy several outlier robustess properties Published by Caadia Ceter of Sciece ad Educatio 25

4 4 Simulatio Results This sectio presets the simulatio results to check the performace of Huber M-estimator as compared to other well kow redescedig M-estimators The simulatio study is carried out i three stages First stage is the ormal situatio; cosider the followig liear regressio model y i = β 0 + β 1 i + u i, i = 1, 2,, i which u i N(0, 1) ad the eplaatory variables are geerated as i N(100, 2) usig R software ad the the values of y i s are evaluated for the specified values of β 0 = 2 ad β 1 = 2 The, values β 0 ad β 1 are computed uder various methods of estimators by usig R software I the secod stage, 10% of the y i observatios are replaced by the values geerated from N(10,5), which are referred as outliers i y-directio After that, as usual, computatios performed to estimate the values of β 0 ad β 1 I the last stage, 20% of the i observatios are replaced by the values are geerated from N(10,5) which are also referred as outliers i -directio with the same observatios available i the secod stage The estimated values of β 0 ad β 1 i differet stages are summarized i Table 1 for the value of fied as 50 The same procedure is repeated for =100 ad =500, ad the results arrived by usig R software, are preseted i Table 2 ad 3 From these tables it is clear that the results of the redescedig M-estimator are very similar to that of ordiary least square estimator i ormal situatio The redecedig estimators are ot affected by the outliers i both secod ad third situatios while the ordiary least square estimator is affected i these situatios 5 Coclusio The performace of robust estimators has bee assessed i regressio model Estimators ad results are obtaied by usig R software It is iterestig to ote that the class of M-estimators is foud to yield essetially the same results as the method of least square estimator i ormal situatio Whe outliers are preset i the data; least square estimator does ot provide useful iformatio for the majority of the data but ot i the case of robust estimators That is, it is observed that the M-estimators are ot affected by outliers The study establishes the fact that the performace of M-estimators are almost same as the method of least squares i ormal situatios ad also i the presece of outliers Hece it is cocluded that the robust statistical procedures ca be cosidered as modificatio of the classical procedures ad such procedures may ot fail whe there are small deviatios from the assumed coditios Refereces Ali, A, Qadir, M F (2005) A Modified M-estimator for the detectio of outliers Pakista joural of Statistics ad Operatios Research, 1, Adrews, D F (1971) Sigificace test based o residuals Biometrika, 58, Ascobe, F J (1960) Rejectio of outliers Techometrics, 2, Buke, H & Buke, O (eds) (1986) Statistical iferece i Liear models Joh Wiley & Sos, Chichester, UK Huber, PJ (1964) Robust estimatio of locatio parameter The Aals of Mathematical Statistics, 35, Huber, PJ (1973) Robust regressio: Asymptotics, Cojectures ad Mote Carlo The Aals of Statistics, 1, Huber, PJ (1981) Robust Statistics Joh Wiley & Sos, New York Isha Ullah, Muhammd, F Quadir, Asad ali (2006) Isha s redecedig M-estimator for robust regressio: A comparative study Pakista Joural of Statistics ad Operatios research, Vol II, No2, 2006, pp Jureckova, J (1980) Asymptotic represetatio of M-estimators of locatio Math Operatiosforsch Ud Statistik, Ser Statistics, 11:1, Jureckova, J & Ja picek (2006) Robust statistical methods with R Chapma & Hall/CRC Lecoutre, JP ad Tassi, P (1987) Statistique o parametricque et robustesse Ecoomica, Paris RadRWilco (2005) Itroductio to Robust Estimatio ad Hypothesis Testig Academic Press Rieder (1994) Robust Asymptotic Statistics Spriger, New York Rousseeuw ad Leory, AM (1987) Robust regressio ad outlier detectio Joh Wiley & Sos, New York Staudte, WA ad Sheather, SJ (1990) Robust estimatio ad testig Joh Wiley & Sos, New York 26 ISSN E-ISSN

5 Table 1 Simulatio Results of Regressio with Itercept, for =50 Estimators Normal Outliers i y Outliers i Least Square Estimator Huber M-estimator Hampel M-estimator Tukey s M-estimator Table 2 Simulatio Results of Regressio with Itercept, for =100 Estimators Normal Outliers i y Outliers i Least Square Estimator Huber M-estimator Hampel M-estimator Tukey s M-estimator Table 3 Simulatio Results of Regressio with Itercept, for =500 Estimators Normal Outliers i y Outliers i Least Square Estimator Huber M-estimator Hampel M-estimator Tukey s M-estimator Published by Caadia Ceter of Sciece ad Educatio 27