Some Robust Ridge Regression for handling Multicollinearity and Outlier

Transcription

1 Internatonal Journal of Scences: Basc and Appled Research (IJSBAR) ISSN (Prnt & Onlne) Some Robust Rdge Regresson for handlng Multcollnearty and Outler Adewale Lukman a *, Olatunj Arowolo b and Kayode Aynde c a Department of Statstcs, Ladoke Akntola Unversty of Technology, P.M.B. 4000, Ogbomoso, Oyo State, Ngera. b Department of Mathematcs and Statstcs, Lagos State Polytechnc, Ikorodu, Lagos, Ngera. c Department of Statstcs, Ladoke Akntola Unversty of Technology, P.M.B. 4000, Ogbomoso, Oyo State, Ngera. a Emal: wale3005@yahoo.com b Emal: saka_197@yahoo.com c Emal: kaynde@lautech.edu.ng Abstract Rdge Regresson and Robust Regresson Estmators were proposed to deal wth the problem of multcollnearty and outler n a classcal lnear regresson model respectvely. Ths paper proposes a robust rdge regresson estmator (RRR) for solvng the problem of multcollnearty and outler n a classcal lnear regresson model smultaneously. The technque of the estmator requres usng the robust estmators (M, MM, S, LTS, LAD, LMS) to estmate the rdge parameter nstead of usng the Ordnary Least Squares (OLS) estmator. The Robust Rdge Estmators performed better than OLS and the Ordnary Rdge Regresson (ORR) estmator when data set suffers from both problems. Mean Square Error was used as a crteron for examnng the performance of these estmators. Result was acheved by the applcaton of the proposed estmator to a data set havng the two problems. Keywords: Ordnary Least Square Estmator, Rdge Regresson Estmator, Robust Regresson Estmator, Robust Rdge Regresson Estmator * Correspondng author. E-mal address: wale3005@yahoo.com 19

2 1. Introducton Lnear regresson model routnely assesses the degree of relatonshp between one dependent varable and a set of explanatory varables. The Ordnary Least Squares (OLS) Estmator s most popularly used to estmate the parameters of regresson model. The estmator has some very attractve statstcal propertes whch have made t one of the most powerful and popular estmators of regresson model. The performance of OLS estmator s neffcent n the presence of multcollnearty. The regresson coeffcents possess large standard errors and some even have the wrong sgn Gujarat [8]. In lterature, there are varous methods exstng to solve ths problem. Among them s the rdge regresson estmator frst ntroduced by Hoerl and Kennard [11]. Rdge Regresson Estmator has a smaller MSE than OLS estmator. Consder the standard regresson model: YY = XXXX + εε (1) X s an n pp matrx wth full rank, Y s a n 1 vector of dependent varable, ββ s a p 1 vector of unknown parameters, and εε s the error term such that E(εε) = 0 and EE(εεεε )=σσ II. The OLS estmator s defned as: ββ = (XX XX) 1 XX YY () Whle the rdge estmator s defned as: ββ = (XX XX + KKKK) 1 XX YY (3) Where K s a scalar rdge parameter. Another common problem n a regresson model s problem of outler and non-normalty of error term. Robust regresson estmator s an mportant estmaton technque for analyzng data that are contamnated wth outlers or data wth non normal error term. It can be used to detect outlers and to provde resstant (stable) results n the presence of outlers. These nclude M estmaton proposed by Huber [13], LTS estmaton by Rousseeuw [4], S estmaton by Rousseeuw and Yoha [3], and MM estmaton proposed by Yoha [9]. Inevtably, these two problems can exst together n a data set; see for nstance [1]. When both problems exst then robust rdge regresson (RRR) estmator proposed n ths paper s suggested. Ths has also attracted the attenton of some researchers. Holland [1] proposed robust M-estmator for rdge regresson to handle the problem of multcollnearty and outlers. Askn and Montgomery [3] proposed rdge regresson based on the M- estmates. It s computed usng weghted least squares procedures. Walker [7] modfed Askn and Montgomery s approach to allow the use of GM estmators nstead of M estmators. Smpson and Montgomery [6] proposed a based-robust estmator that uses a multstage GM estmator wth fully terated rdge regresson 193

3 to control both nfluence and collnearty n the regresson data set. Pfaffenberger & Delman [] combnes least absolute value estmator wth Rdge to proposed Rdge Least Absolute Value Estmator. Slvapulle [5] proposed a new class of rdge type M estmators obtaned by usng M estmators nstead usng OLS estmators. Arslan & Bllor [4] proposed two alternatve rdge type GM estmators to handle multcollnearty and outlers smultaneously. Authors n [8] ntroduced a robust regresson estmator that performs well regardless of the quantty and confguraton of outlers. Habshah and Marna [9] proposed Rdge MM estmator (RMM) by combnng the MM estmator and rdge regresson. Hatce and Ozlem [10] proposed robust rdge regresson methods based on M, S, MM and GM estmators. Maronna [19] proposed robust MM estmator n rdge regresson for hgh dmensonal data. In ths study, rdge regresson methods based on M, S, MM, LTS, LAD and LMS estmators are examned n the presence of both outlers and multcollnearty. Mean Square Error was used as a crteron for examnng the performances of these estmators. The data sets used n ths study was extracted from the study of Hussen and Ahmed [14].. Materals and Methods.1 Rdge Regresson The concept of rdge regresson was ntroduced by Hoerl and Kennard [11]. Rdge regresson s a method of based lnear estmaton whch has been shown to be more effcent than the OLS estmator when data exhbt multcollnearty. It reduces multcollnearty by addng a rdge parameter, K, to the man dagonal elements of X'X, the correlaton matrx. The rdge estmator s defned n (3) such that KK 0. In lterature several technques for estmatng the Rdge parameter K have been suggested by dfferent researchers. Among them are [11,18,17,6,7,15,16,1,,1,0]. In ths study, the rdge parameter by Kbra [15] s used. It s defned as: KK GGGG = σσ pp αα =1 1 pp (4) σσ and αα are generally unknown. Hoerl and Kennard n [11] suggested the replacement of σσ and αα by ther correspondng unbased estmators σσ aaaaaa αα where σσ = nn =1 ee. nn pp. Robust Estmators..1 M Estmators The most common general method of robust regresson s M-estmaton, ntroduced by Huber [13]. It s nearly as effcent as OLS. Rather than mnmze the sum of squared errors as the objectve, the M-estmate mnmzes a functon ρ of the errors. The M-estmate objectve functon s 194

4 mn n n e ˆ y X β ρ = mn ρ s s = 1 = 1 (5) Where s s an estmate of scale often formed from lnear combnaton of the resduals. The functon ρ gves the contrbuton of each resdual to the objectve functon. A reasonable ρ should have the followng propertes: ρ(e) 0, ρ(0)=0,ρ(e)=ρ(-e), and ρ( e) ρ ( e ) for e e the system of normal equatons to solve ths mnmzaton problem s found by takng partal dervatves wth respect to β and settng them equal to 0, yeldng, n y ˆ X β ψ X = 1 s =0 (6) Where ψ s a dervatve of ρ. The choce of the ψ functon s based on the preference of how much weght to assgn outlers. Newton-Raphson and Iteratvely Reweghted Least Squares (IRLS) are the two methods to solve the M-estmates nonlnear normal equatons. IRLS expresses the normal equatons as: X WX ˆ β = X Wy (7).. S Estmator Rousseeuw and Yoha [3] ntroduced S estmator whch s derved from a scale statstcs n an mplct way, correspondng to s (θ) where s(θ) s a certan type of robust M-estmate of the scale of the resduals ee 1 (θθ),, ee nn (θθ). They are defned by mnmzaton of the dsperson of the resduals: mnmze S (ee 1 (θθ),, ee nn (θθ ) wth fnal scale estmate σσ = S( ee 1 (θθ),, ee nn (θθ )). The dsperson ee 1 (θθ),, ee nn (θθ ) s defned as the soluton of 1 nn ρρ nn ee =1 = kk (8) ss K s a constant and ρρ ee s the resdual functon. Rousseeuw & Yoha suggested Tukey s bweght functon ss gven by: 4 6 x x x + for x c 4 c 6c ρ( x) = c for x > c 6 (9) Settng c= and K= gves 50% breakdown pont (Rousseeuw & Leroy, 1984). 195

5 ..3 MM Estmator MM-estmaton s specal type of M-estmaton developed by Yoha [9]. MM-estmators combne the hgh asymptotc relatve effcency of M-estmators wth the hgh breakdown of class of estmators called S- estmators. It was among the frst robust estmators to have these two propertes smultaneously. The MM refers to the fact that multple M-estmaton procedures are carred out n the computaton of the estmator. Yoha descrbed the three stages that defne an MM-estmator: Stage 1 A hgh breakdown estmator s used to fnd an ntal estmate, whch we denote ββ. the estmator need to be effcent. Usng ths estmate the resduals, rr (ββ) = yy xx TT ββ are computed. Stage Usng these resduals from the robust ft and 1 nn ρρ nn rr =1 = kk where k s a constant and the objectve functon ρρ, an M-estmate of scale wth 50% BDP s computed. Ths s (rr 1 ββ,, rr nn ββ ) s denoted ss nn. The objectve functon used n ths stage s labeled ρρ 0. ss Stage 3 The MM-estmator s now defned as an M-estmator of ββ usng a redescendng score functon, φφ 1 (uu) = ρρ 1(uu ), and the scale estmate ss nn obtaned from stage. So an MMestmator ββ defned as a soluton to nn =1 xx φφ 1 ( yy xx TT ββ ss nn )=0 j=1,, p. (10)..4 LTS Estmator Rousseeuw [4] developed the least trmmed squares (LTS) estmaton method. Extendng from the trmmed mean, LTS regresson mnmzes the sum of trmmed squared resduals. Ths method s gven by, ββ LLLLLL = aaaaaaaaaaaaqq LLLLLL (ββ) (11) where QQ LLLLLL (ββ) = h =1 ee such that ee (1) ee () ee (3) ee (nn) are the ordered squares resduals and h s defned n the range nn + 1 h 3nn+pp+1, wth n and p beng sample sze and number of parameters 4 respectvely. The largest squared resduals are excluded from the summaton n ths method, whch allows those outler data ponts to be excluded completely. Dependng on the value of h and the outler data confguraton. LTS can be very effcent. In fact, f the exact numbers of outlyng data ponts are trmmed, ths method s computatonally equvalent to OLS. 196

6 ..5 LMS Estmator The least medan of squares (LMS) estmator s defned as the p-vector ββ LLLLLL = aaaaaaaaaaaaqq LLLLLL (ββ) (1) Where QQ LLLLLL (ββ) = ee h such that ee (1) ee () ee (3) ee (nn) are the ordered squares resduals and h s defned n the range nn + 1 h 3nn+pp+1. The breakdown value for the LMS estmate s also nn h nn. However the LTS estmate has several advantages over the LMS estmate LAD Estmator Least Absolute Value (LAV) regresson s also known by several other names, ncludng Mnmum Absolute Devaton regresson, Least Absolute Devaton (LAD) regresson, and Mnmum Sum of Absolute Errors regresson Delman [5] developed the LAD estmator whch mnmzes the sum of the absolute values of the resduals wth respect to the coeffcent vector b: mmmmmm nn =1 yy xx bb. (13) A property of the LAD estmator s that there are K resduals that are exactly zero. LAD s robust to an outler n the y-drecton. However, LAD estmator does not protect aganst outlyng x (leverages)..3 Robust Rdge Regresson Ths s a combnaton of rdge and robust regresson to handle the problem of multcollnearty and outlers smultaneously. Ths wll dampen the effects of both problems n a classcal lnear regresson model. To compute Robust Rdge Estmator, the formula used s: ββ RRRRRRRRRRRRRRRRRRRRRR = (XX XX + KK RR II) 1 XX YY (14) Where KK RR s called the robust rdge parameter. It s obtaned from robust regresson methods nstead of usng OLS. Ths wll be computed as gven above, only that respectvely. σσ aaaaaa αα are replaced wth σσ rrrrrrrrrrrr aaaaaa αα rrrrrrrrrrrr.4 Data Used n ths Study Data set taken from Hussen and Ahmed [14] was used to examne the performance of the consdered estmators. Ths contans three (3) regressors and one (1) response varable. 197

7 .5 Crteron for Investgaton To nvestgate whether the rdge estmator s better than the OLS estmator, the MSE was calculated usng the followng equaton: MMMMMM ββ rrrrrrrrrrrr rrrrrrrrrr = σσ rrrrrrrrrrrr pp tt =1 (tt +KK) pp αα + KK =1 (15) (tt +KK) pp 1 MMMMMM ββ OOOOOO = σσ =1 (16) tt Where tt 1, tt,, tt pp are the egenvalues of XX XX, K s the rdge parameter obtaned from OLS and robust estmates. αα s the th element of the vector αα = QQ ββ. 3. Results and Dscusson The model does not nclude the ntercept term because the data was standardzed. The results of robust dagnostc check of the data as shown n Table 1 revealed the presence of outlers and leverages. The followng observatons: 1, 14, 15, 16, 17, 18, 19, 0, 1 were dentfed as the outlyng ponts n the X-space whle observatons 1, 13, 14,15,30,31 as the outlyng ponts n the Y-space. Observaton 1, 14, 15 are bad leverages. Ths necesstated the use of the robust regresson and ths provdes a more stable regresson estmates than OLS as seen n Table. The result n Table shows that the estmates of LTS, S and MM estmator are farly close and provdes more stable regresson estmates when compared wth other robust estmators. It s also observed that the scale estmates (σσ ) of LTS, MM, S are more effcent than others. Though the scale estmates of M estmators s not too dfferent from the frst three estmators mentoned. The VIF n Table revealed the presence of multcollnearty snce VIF s > 10. Also, the coeffcent of ββ 3 has a negatve sgn whch s not consstent wth the pror expectaton, ths ndcated the presence of multcollnearty and hence necesstate the use of Rdge regresson estmator rather than usng OLS. It could therefore be nferred from Table 1 and that the data set suffered the problem of multcollnearty and outlers smultaneously. Hence, snce the data set suffered both problem of multcollnearty and outler, the rdge parameter K s computed from the estmates of the followng robust estmators: (M, MM, S, LTS, LAD, and LMS) and the performance s compared wth the rdge parameter computed usng OLS (Ordnary rdge regresson). The rdge regresson estmates based on the robust estmators n ths study s called robust rdge regresson. The results were presented n Table 3. The negatve sgn n ββ 3 s corrected and found to be consstent wth pror expectaton whch shows that the effect of multcollnearty has been handled. The regresson estmates of robust rdge estmates based on LTS, MM, S and M are farly closed than those obtaned based on OLS, LAD and LMS. Table 4 revealed that the problem of multcollnearty has been solved usng all the estmators but n terms of the MSE of the coeffcents robust rdge estmates based on LTS, S and MM performs better than other estmators. OLS has the least performance among the estmators wth a large MSE. 198

8 Table 1: Robust Regresson Dagnostcs Observaton Index Mahalanobs Robust MCD Leverage Standardzed Robust Outler Dstance Resdual * * * * * * * * * * * * * * * Table : OLS and Robust Estmates Coeffcent OLS LTS S MM M LAD LMS VIF ββ ββ ββ σσ K GM Table 3: Ordnary Rdge and Robust Rdge Estmates Coeffcent ORR LTS S MM M LAD LMS ββ ββ ββ MSE(β)

9 Table 4: VIF OF OLS and Robust Rdge Estmates Coeffcent ORR LTS S MM M LAD LMS ββ ββ ββ Concluson The OLS and the robust estmators could not perform well n the presence of multcollnearty and outler. The estmators could not correctly estmate the regresson coeffcents. The performance of OLS, LAD and LMS were close because of the presence of bad leverages. M estmaton s a commonly used method for outler detecton and robust regresson when contamnaton s manly n the response drecton. Its performance cannot be compared wth the hgh breakdown value estmators (LTS, S, and M). These hgh breakdown estmators are good estmators especally when data sets have bad leverages. In ths study, the performance of LTS, S and MM were not statstcally dfferent n terms of ther scale (σσ ) and MSE(β). The mean square error revealed that rdge regresson estmated based on OLS s the least compared to rdge regresson based on the robust estmators. Rdge estmates based on LTS perform better than all other estmators followed by Rdge estmates based on MM and S snce they have the smallest mean square error n ther order. The other rdge estmates based on M, LAD and LMS also perform better than the rdge estmates based on OLS. In concluson, t has been seen that when data set exhbt both problem of multcollnearty and outler the robust rdge regresson estmator are better than OLS and the counterparts rdge or robust estmators. References [1] Alkhams, M., Khalaf, G. and Shukur, G. Some modfcatons for choosng rdge parameters. Communcatons n Statstcs- Theory and Methods, 35(11), , 006. [] Alkhams, M., and Shukur, G. Developng rdge parameters for SUR model. Communcatons n Statstcs- Theory and Methods, 37(4), , 008. [3] Askn, G. R., & Montgomery, D. C. Augmented robust estmators. Techonometrcs,, , (1980). [4] Arslan, O., & Bllor, N. Robust rdge regresson estmaton based on the GM estmators. Journal of Mathematcal and Computatonal Scence, 9(1), 1-9, [5] Delman, T.E. Least absolute value estmaton n regresson models: An annotated bblography. Communcatons n Statstcs - Theory and Methods. 4, ,

10 [6] Dempster, A.P., Schatzoff, M. and Wermuth, N. A smulaton study of alternatves to ordnary least squares. Journal of the Amercan Statstcal Assocaton, 7, 77-91, 1977 [7] Gbbons, D. G. A smulaton study of some rdge estmators. Journal of the Amercan Statstcal Assocaton, 76, , [8] Gujarat, N. D. Basc Econometrcs (4th Ed.). New Delh: Tata McGraw-Hll, 748, 807, 003. [9] Habshah Md & Marna Zahar. A Smulaton Study on Rdge Regresson Estmators n the Presence of Outlers and Multcollnearty. Jurnal Teknolog. 47(C): 59-74, 007. [10] Hatce, Samkar and Alpu, Ozlem."Rdge Regresson Based on Some Robust Estmators."Journal of Modern Appled Statstcal Methods: Vol. 9: Iss., Artcle 17, 010. [11] Hoerl, A. E. & Kennard, R.W. " Rdge Regresson Based Estmaton for Nonorthognal Problems",Technometrcs,Vol.1, 55-67, [1] Holland, P. W. Weghted rdge regresson: Combnng rdge and robust regresson methods. NBER Workng Paper Seres, #11, 1-19, [13] Huber, P.H Robust estmaton of a locaton parameter. The Annals of Mathematcal Statstcs, [14] Hussen Yousf Abd. Eledum, Abdalla Ahmed Alkhalfa (01). Generalzed Two stages Rdge Regresson Estmator for Multcollnearty and Autocorrelated errors. Canadan Journal on Scence and Engneerng Mathematcs, Vol. 3 No. 3, March 01 [15] Kbra, B. M. G. Performance of some new rdge regresson estmators. Communcatons n Statstcs- Smulaton and Computaton, 3, , 003. [16] Khalaf, G. and Shukur, G. Choosng rdge parameters for regresson problems. Communcatons n Statstcs- Theory and Methods, 34, , 005. [17] Lawless, J.F. and Wang, P. A smulaton study of rdge and other regresson estmators. Communcatons n Statstcs A, 5, , [18] McDonald, G. C. and Galarneau, D. I. A Monte Carlo evaluaton of some rdge-type estmators. Journal of the Amercan Statstcal Assocaton, 70, , [19] Maronna, R.A. Robust Rdge Regresson for Hgh-Dmensonal Data. Technometrcs, 53(1): 44-53,

11 [0] Mansson, K., Shukur, G. and Kbra, B. M. G. A smulaton study of some rdge regresson estmators under dfferent dstrbutonal assumptons. Communcatons n Statstcs-Smulatons and Computatons, 39(8), , 010. [1] Munz, G. and Kbra, B. M. G. On some rdge regresson estmators: An emprcal comparson. Communcatons n Statstcs-Smulaton and Computaton, 38, , 009. [] Pfaffenberger, R. C., & Delman, T. E. A comparson of regresson estmators when both multcollnearty and outlers are present. In Robust regresson: Analyss and applcatons, K. Lawrence & J. Arthur (Eds.), New York: Marcel Dekker, [3] Rousseeuw P.J., and Yoha, Robust regresson by means of S estmators. In W. H. J. Franke and D.Martn (Edtors.), Robust and Nonlnear Tme Seres Analyss, Sprnger-Verlag, New-York, (pp. 56-7), [4] Rousseeuw, P.J. and Van Dressen, K. Computng LTS Regresson for Large Data Sets, Techncal Report, Unversty of Antwerp, submtted, [5] Slvapulle, M. J. Robust rdge regresson based on an M estmator. Australan Journal of Statstcs, 33, , [6] Smpson, J. R., & Montgomery, D. C. A based robust regresson technque for combned outlermultcollnearty problem. Journal of Statstcal Computaton Smulaton, 56, 1-, [7] Walker, E: Influence, collnearty and robust estmaton n regresson. Unpublshed Ph.D. dssertaton, Department of Statstcs, Vrgna Polytechnc Insttute, [8] Wsnowsk, J. W., Smpson, J. R., & Montgomery, D. C. An mproved compound estmator for robust regresson. Communcatons n Statstcal Smulatons, 31(4), , 00. [9]Yoha, V.J. "Hgh breakdown pont and hgh breakdown-pont and hgh effcency robust estmates for regresson. The Annals of Statstcs, 15, ,