A COMPARATIVE STUDY OF SOME ROBUST RIDGE AND LIU ESTIMATORS

Transcription

1 Scence World Journal Vol (No 4) 06 ISSN A COMPARATIVE STUDY OF SOME ROBUST RIDGE AND IU ESTIMATORS Ajboye S. Adegoke Emmanuel Adewuy Kayode Aynde, and Adewale F. ukman Full ength Research Artcle Deartment of Statstcs, Federal Unversty of Technology, Akure Deartment of Statstcs, adoke Akntola Unversty of Technology, P.M.B. 4000, Ogbomoso, Oyo State, Ngera. E-Mal: asajboye@futa.edu.ng, kaynde@lautech.edu.ng, olex.tawo@gmal.com, wale3005@yahoo.com ABSTRACT In multle lnear regresson analyss, multcollnearty and outlers are two man roblems. When multcollnearty exsts, based estmaton technques such as Rdge and u Estmators are referable to Ordnary east Square. On the other hand, when outlers exst n the data, robust estmators lke M, MM, TS and S Estmators, are referred. To handle these two roblems jontly, the study combnes the Rdge and u Estmators wth Robust Estmators to rovde Robust Rdge and Robust u estmators resectvely. The Mean Square Error (MSE) crteron was used to comare the erformance of the estmators. Alcaton to the roosed estmators to three (3) real lfe data set wth multcollnearty and outlers roblems reveals that the M-u and TS-u Estmator are generally most effcent.. Keywords: Ordnary east Squares, Rdge Regresson Estmator, u Estmator, Robust Estmator, Robust Rdge Regresson Estmator, Robust u Estmator.0. INTRODUCTION Regresson analyss s used to study the relatonsh between a sngle varable Y, called the resonse varable, and one or more X X by a lnear model. The exlanatory varable(s),,, method of Ordnary east Squares (OS) estmator of model arameters s best lnear unbased estmator (BUE) and most effcent under certan assumtons (Gujarat, 003). One of the assumtons of near Regresson model s that of ndeendence between the exlanatory varables (.e. no multcollnearty). Volaton of ths assumton arses most often n regresson analyss. Among methods used n detectng the resence of Multcollnearty s varance nflaton factor (VIF). The erformance of OS estmator s neffcent f ths assumton s not vald and the regresson coeffcents have large standard errors and sometmes have wrong sgn (Gujarat, 003). In ths stuaton, many estmators have been roosed to combat ths roblem among whch are: Sten Estmator by Sten (956), u Estmator by u (993) and Rdge Estmator roosed by Hoerl and Kennard (970). Other roblems n regresson analyss nclude the roblem of outler and leverage onts. An outler s an observaton that s dstant from other observatons. everage onts are onts that aear to be outlyng n the regressors. Methods such as studentzed deleted resdual and Mahalanobs dstance are used to detect the resence of outlers and leverage ont resectvely. Cooks D and DFFITS are often used to determne f ether the outlers or leverage onts nfluences the regresson coeffcents. Robust regresson estmator s commonly used to crcumvent the roblem of outler. Examles of these estmators are M-estmator roosed by Huber (964), east Trmmed Mean (TS) by Rousseeuw and Van Dressen (998), S estmator roosed by Rousseeuw and Yoha (984) and MM estmator by Yoha (987) among others. These two roblems may jontly exst n regresson analyss. Ths has attracted the attenton of some researchers. Holland (973) roosed robust M-estmator for rdge regresson to handle the roblem of multcollnearty and outlers. Askn and Montgomery (980) roosed rdge regresson based on the M-estmator. Walker (984) modfed Askn and Montgomery s aroach to allow the use of Generalzed M estmators nstead of M estmators. Smson and Montgomery (996) roosed a basedrobust estmator that uses a multstage Generalzed M estmator wth fully terated rdge regresson. ukman et al. (04) roosed and aled some Robust Rdge Regresson Estmators to the Hussen and Abdalla (0) data. Alu and Samkar (00) aled u estmator based on M estmator to a VO data. The am of ths study s to combne Rdge and u estmators wth some robust estmators to jontly handle the roblem of Multcollnearty and outlers. Also, to comare the erformances of these combned estmators wth ther ndvdual counterarts.0. MATERIAS AND METHOD.. Ordnary east Square Estmator Consder the standard regresson model: Y X where X s an n matrx wth full rank, Y s a n vector of deendent varable, β s a vector of unknown arameters, and s the error term such that E( ) 0 and E( ') I. X ' X s nvertble, the OS estmator s gven by Provded ˆ ( X ' X) ' X y () The regresson model n equaton can be wrtten n canoncal form y Z (3) A Comaratve Study of Some Robust Rdge and u Estmators 6

2 Scence World Journal Vol (No 4) 06 ISSN where Z XQ, Q ' X ' X. Then Z ' Z Q' X ' XQ dag (,,..., ), are ordered egenvalues of X X. wth columns that consttute the egenvectors of and Q s the orthogonal matrx where... Thus, the Ordnary east Square Estmator of α s ˆ OS Z' y (4).. Rdge Regresson Estmator If Z Z matrx s ll-condtoned, (esecally when there s a nearlnear deendency among the exlanatory varables), the OS estmator of tends to have a large varance. Rdge arameter s added to the Z Z matrx to reduce the collnearty effect. Hoerl and Kennard (970) defned the rdge regresson estmator of as: α (k) = (Z Z + ki) Z y (5) where k s the rdge arameter. The value of k used n ths study s the one roosed by ukman (05). K A (6) ˆ max( ˆ ) where σ s the estmated MSE calculated as σ = = e and n α s the estmated regresson coeffcent..3. u Estmator To overcome multcollnearty, u (993) roosed the u Estmator by combnng the Sten estmator wth Rdge estmator to form ˆ ( ' ) ( ' ) ˆ X X I X X di OS (7) where d s the Basng arameter and can be comuted accordng to u by d ( ) ˆ ˆ ( ) (8) ˆ Q ˆ and ˆ are Ordnary east Square where ' OS estmators of α and resectvely, and Q s the matrx of egenvectors corresondng to the egenvalues o the matrx X X. Now equaton (7) can be wrtten n canoncal form: ˆ ( ) ( ) ˆ I di OS (9) where IP s the -dentty matrx.4. Robust Regresson.4.. M-Estmator The most common method of robust regresson s M-estmaton, ntroduced by Huber (964). It s nearly as effcent as OS. Rather than mnmzng the sum of squared errors, M-estmate chooses β to mnmze ρ ( y x T β = ) σ (0) Possble choces of ρ are: ρ(x) = x s just least squares and ρ(x) = x s called the least absolute devatons regresson (AD). ρ(x) = x {, f x c c x c, otherwse () Dfferentatng the M-estmate crteron wth resect to β j and settng to zero, we get; ρ ( y j= x j β j = ) x j = 0, j = σ,, Now let u = y j= x j β j to get ρ (u ) = j= ) = x u j (y x j β j 0 (3) makng the dentfcaton of w(u) = ρ (u) and fnd w(u) for choces of ρ above: u. S: w(u)s constant.. AD: w(u) = / u, f u c 3. Huber: w(u) = { c u,.4.. MM-Estmator It was frst ntroduced by Yoha (987). It has become ncreasngly oular and erhas one of the most commonly emloyed robust regresson technque. The MM n the name refers to the fact that more than one M-estmaton rocedure s used to calculate the fnal estmates. Followng from the M-estmaton case, teratvely reweghted least squares (IRS) s emloyed to fnd estmates. The rocedure s as follows:. Intal estmates of the coeffcents β and corresondng resduals e are taken from a hghly resstant regresson (.e., a regresson wth a breakdown ont of 50%). Although the estmator must be consstent, t s not necessary that t be effcent. As a result, S-estmaton wth Huber or bsquare weghts (whch can be seen as a form of M-estmaton) s tycally emloyed at ths stage.. The resduals e from the ntal estmaton at Stage are used to comute an M-estmaton of the scale of the resduals, σ e. 3. The ntal estmates of the resduals from Stage and of the resdual scale σ e from Stage are used n the frst teraton of weghted least squares to determne the M-estmates of the regresson coeffcents = w (e /σ e) x = 0 (4) where the w are tycally Huber or bsquare weghts. 4. New weghts are calculated, w, usng the resduals from the ntal WS (Ste 3). 5. Keeng constant the measure of the scale of the resduals from Ste, Stes 3 and 4 are contnually reterated untl convergence. A Comaratve Study of Some Robust Rdge and u Estmators 7

3 Scence World Journal Vol (No 4) 06 ISSN S-Estmator In resonse to the low breakdown ont of M-estmators. Rousseeuw and Yoha (984) roosed S-estmates by consderng the scale of the resduals. S-estmates are the soluton that fnds the smallest ossble dserson of the resduals mnσ (e (β ),, e n (β )). Rather than mnmzng the varance of the resduals, robust S-estmaton mnmzes a robust M- estmate of the resdual scale n ρ (e = ) = b, (5) n σ e where b s a constant defned as b = E φ [ρ(e)]and φ reresents the standard normal dstrbuton. Dfferentatng Equaton 8 and solvng results n estmator (RE) whch are M-u, MM-u, TS-u and S-u estmators. These Robust u estmators can be comuted as: ˆ ˆ (9) RE ( I ) ( dri ) R where d R s the basng arameter obtaned from robust estmators, comuted as: d ˆ ( ) R Robust ˆ Robust ( ) (0) n n = ), (6) ψ (e σ e where ψ s relaced wth an arorate weght functon. As wth most M-estmaton rocedures, ether the Huber weght functon or the bweght functon s usually emloyed east Trmmed Squares (TS) Estmator Another method develoed by Rousseeuw (998) s least trmmed squares (TS) regresson. Extendng from the trmmed mean, TS regresson mnmzes the sum of the trmmed squared resduals. The TS estmator s found by; q mn e (), = where q = [n( α) + ] s the number of observatons ncluded n the calculaton of the estmator, and α s the roorton of trmmng that s erformed. Usng q = ( n ) + ensures that the estmator has a breakdownont of 50%..5. Robust Rdge Regresson (RRR) To solve the roblems of multcollnearty and outler smultaneously, the rdge estmator was combned wth some robust estmators (M, MM, TS and S estmators) to form robust rdge estmator (RRE) whch are M-Rdge, MM-Rdge, TS-Rdge and S-Rdge estmators (ukman et al., 04). These Robust rdge estmators can be comuted as: α RR = (Z Z + k Arobust I ) Z Y (7) where k Arobust s the robust rdge arameter. It s obtaned from the robust regresson methods nstead of the OS estmaton, and can be comuted as gven below; k Arobust ˆ Robust max( ˆ ) Robust.6. Robust u Estmator (Proosed) To solve the roblems of multcollnearty and outler smultaneously, u estmator combned wth some robust estmators (M, MM, TS and S estmators) to rovde robust u.7. Data Descrton Three datasets are used n ths study to examne the erformance of the estmators. The datasets are gven n detals below..7.. ongley Data ongley Data s a macroeconomc dataset whch rovdes a wellknown examle for a hghly collnear regresson. A data frame wth seven economc varables observed yearly from 947 to 96. The varables are: Emloyment, Prces, Unemloyed, Mltary, GNP, Poulaton Sze, Year. GNP s the Gross Natonal Product, Emloyment s the number of eole emloyed, Unemloyed s the number of unemloyed, Mltary s the number of eole n the armed forces, Poulaton sze s the nonnsttutonalzed oulaton of ersons at age 4 years, Prce s the GNP mlct rce deflator and year s the tme. ongley data have been dagnosed to suffer both roblems of multcollnearty and outler (Cook, 977; Besley et al., 980; and Jahufer, 03)..7.. Portland cement data Portland dataset traceable to Woods et al. (93) has been wdely analysed by Kacranlar et al. (999). The dataset contans four exlanatory varaables whch are trcalcum alumnate (X), trcalcum slcate (X), tetracalcum alumnoferrte (X3) and β- dcalcum slcate (X4). The heat evolved after 80 days of curng s the deendent varable (Y). The dataset suffers multcollnearty snce varance nflaton factors are greater than 0. Mahalanobs dstances of observatons 3 and 0 revealed that the observatons are leverage. Wth ths t s obvous that there s outler n the x- drecton and no outler n the y-drecton. As a result, t s observed that multcollnearty and leverage ont exsts jontly n the dataset Hussen and Abdalla data Ths dataset was used by Hussen and Abdalla (0) and t covered the roducts n the manufacturng sector of Iraq n the erod of 960 to 990. The varables used are the roduct value n the manufacturng sector(y), value of morted ntermedate (X), morted catal commodtes (X) and value of morted raw materals (X3). Hussen and Abdalla (0) showed that the dataset suffers the roblem of multcollnearty snce VIF > 0. ukman et al. (04) dentfed case number:, 4, 5, 6, 7, 8, 9, 0 and as outlers n the y-drecton and also dentfed case number, 4 and 5 as leverages. Therefore, outlers exst n the y and x drecton. A Comaratve Study of Some Robust Rdge and u Estmators 8

4 Scence World Journal Vol (No 4) 06 ISSN Crteron for nvestgaton To nvestgate whether the robust rdge estmator s better than the OS estmator, the MSE was calculated as follows: λ MSE(α Rdge) = σ OS α,os (λ +k OS ) = + (λ +k OS ) k OS = (0) MSE(α Robust Rdge ) = σ robust α,robust (λ +k R ) λ = + (λ +k R ) k R = MSE(α u ) = σ P ( +d) + (d P α ( +) = ( +) ) = () MSE(α Robust u ) = σ robust P P ( +d R ) = ( +) ) α,robust = (3) ( +) MSE(α OS ) = σ = λ + (d R (4) where λ, ( =,,, ) are the egenvalues of X X, k s the rdge arameter obtaned from OS and Robust estmates, α ( =,,, ) s the th element of the vector α = Q β RESUTS AND DISCUSSION Table 3: Estmates of Rdge, u, Robust Rdge and Robust u Estmators From Table, t s obvous that the data suffers a severe roblem of multcollnearty snce the Varance Inflaton Factors (VIF) are greater than 0 excet for X5. Also from Table, the data has outlers n the y-drecton, hence the data suffers both roblem of multcollnearty and outlers. It follows from Table 3 that the two roblems have been crcumvented wth the use of Robust u and Robust Rdge Regresson Estmators and t s found that the Robust u s the most effcent n term of MSE. 3.. Results for Portland Cement Data Table 4: Estmates of OS and Robust estmators 3.. Result for ongley Data Table : Summary of OS and Robust Estmates Table 5: Estmates of Rdge, u, Robust Rdge and Robust u estmators Table : Summary of the Influental and everage Ponts From Table 4 and 5, t can be seen that n terms of MSE crteron of the estmators, the TS-u, TS-Rdge n ths order, are more effcent than the OS. Thus, TS-u s most effcent Result for Hussen and Abdalla Data Table 6: Estmates of OS and Robust estmator A Comaratve Study of Some Robust Rdge and u Estmators 9

5 Scence World Journal Vol (No 4) 06 ISSN Table 7: Estmates of Rdge, u, Robust Rdge and Robust u estmators The result n Table 7 shows that robust u (TS-u) and robust Rdge (TS-Rdge) regresson estmators have least mean square error Concluson Ordnary east Square (OS), u Regresson and Ordnary Rdge Regresson (ORR) estmators could not erform well n term of ther Mean Squared Error (MSE) n the resence of multcollnearty and outler but ORR and u estmator erforms better than that of Ordnary east Square (OS) Estmator. It s observed that Robust Rdge Estmators (RRE) and Robust u estmators erform better than the ORR, RE and OS estmators when both roblems exst. Fnally, M-u and M-Rdge erform most n ths order when the outlers are n y-drecton, whle TS- Rdge and TS-u erform better when the outlers are n x- drecton (everage). REFERENCES Alu, O. and Samkar, H. (00). u Estmator based on an M Estmator. Turkye Klnkler Bostat, (): Askn, G. R., and Montgomery, D. C. (980). Augmented robust estmators. Techonometrcs,, Chatterjee, S. and Had, A. S. (988). Senstvty Analyss n near Regresson. Wley Seres n Probablty and Mathematcal Statstcs.Wley, New York. Cook, R.D. (977). Detecton of nfluental observatons n lnear regresson. Technometrcs, 9, 5-8. Belsley, D. A, Kuh, E. and Welsch, R. E. (980). Regresson Dagnotcs: Identfyng Influence Data and Source of Collnearty. Wley, New York. Gujarat, N. D. (003). Basc Econometrcs, New Delh: Tata McGraw-Hll, New York. Hoerl, A. E. and Kennard, R.W. (970). Rdge Regresson Based Estmaton for Nonorthogonal Problems.Technometrcs,, Holland, P. W. (973). Weghted rdge regresson: Combnng rdge and robust regresson methods. NBER Workng Paer Seres,, -9. Huber, P.H. (964). Robust estmaton of a locaton arameter. The Annals of Mathematcal Statstcs, 35, 0. Hussen, Y. A. and Abdalla, A. A. (0). Generalzed Two stages Rdge Regresson Estmator for Multcollnearty and Autocorrelated errors. Canadan Journal on Scence and Engneerng Mathematcs, 3(3), Jahufer, A. (03). Detectng Global Influental Observatons n u Regresson Model. Oen Journal of Statstcs, 3, 5-. Kacranlar, S., Sakalloglu, S., Akdenz, F., Styan., G. P. H., and Werner, H. J. (999). A new based estmator n lnear regresson and a detaled analyss of the wdelyanalysed dataset on ortland cement. Sankhya, Ser. B, Indan J.Statst., 6: u, K. (993). A new class of based estmate n lnear regresson. Communcaton n Statstcs. (): ongley, J. W. (967). An Arasal of east Squares Programs for Electronc Comuter for the Pont of Vew of the User. Journal of Amercan Statstcal Assocaton. 6(39), ukman, A. F, Arowolo, O. and Aynde, K. (04). Some Robust Rdge Regesson for Handlng Multcollnearty and Outlers. Internatonal Journal of Scences: Basc and Aled Research (IJSBAR). 6(), 9-0. ukman, A. F. (05). Revew and Classfcatons of the Rdge Parameter Estmaton Technques. UnublshedM.hl./ PhD thess. Rousseeuw, P.J. and Van Dressen, K. (998). Comutng TS Regresson for arge Data Sets, Techncal Reort, Unversty of Antwer, submtted. Rousseeuw, P.J., and Yoha. (984). Robust regresson by means of S estmators. In W. H. J. Franke and D.Martn, Robust and Nonlnear Tme Seres Analyss, Srnger-Verlag, New-York, Smson, J. R., and Montgomery, D. C. (996). A based robust regresson technque for combned outlermultcollnearty roblem. Journal of Statstcal Comutaton Smulaton, 56, -. Sten, C. (956). Inadmssblty of the usual estmator for the mean of a multvarate normal dstrbuton. In Proceedngs of the Thrd Berkeley symosum on mathematcal statstcs and robablty,, Walker, E. (984). Influence, collnearty and robust estmaton n regresson. Unublshed Ph.D. dssertaton, Deartment of Statstcs, Vrgna Polytechnc Insttute. Woods, H, Stenour, H. and Starke, H. (93). Effect of Comoston of Portland Cement on Heat Evolved Durng Hardenng, Industral and Engneerng Chemstry, 4, Yoha, V.J. (987). Hgh breakdown ont and hgh breakdownont and hgh effcency robust estmates for regresson. The Annals of Statstcs, 5, A Comaratve Study of Some Robust Rdge and u Estmators 0