The Efficiency of Some Robust Ridge Regression. for Handling Multicollinearity and Non-Normals. Errors Problems

Transcription

1 Applied Mathematical Scieces, Vol. 7, 203, o. 77, HIKARI Ltd, The Efficiecy of Some Robust Ridge Regressio for Hadlig Multicolliearity ad No-s Errors Problems Moawad El-Fallah Abd El-Salam Departmet of Statistics & Mathematics ad Isurace Faculty of Commerce, Zagazig Uiversity, Egypt Copyright 203 Moawad El-Fallah Abd El-Salam. This is a ope access article distributed uder the Creative Commos Attributio Licese, which permits urestricted use, distributio, ad reproductio i ay medium, provided the origial work is properly cited. Abstract A commo problems i multiple regressio models are multicolliearity ad o-ormal errors, which produce udesirable effects o the least squares estimators. So, it would seem importat to combie methods of estimatio desiged to deal with these problems. I this paper, a comparative ivestigatio was doe experimetally for some differet estimatio methods, which amely the ordiary least squares ( LS), Ridge Regressio ( R ), Ridge least Absolute Deviatio (RLAD), Weighted Ridge (WR), Robust M(M) ad Robust Ridge regressio based o M-estimatio (RM). From a simulatio study, the resultig robust ridge regressio estimator (RM)is efficiet tha other estimators, usig the mea squared error criterio for may combiatios of error distributio ad degree of multicolliearity. Keywords: No-ormal errors; Multicolliearity; Ridge regressio; Robust M- estimatio

2 3832 Moawad El-Fallah Abd El-Salam. Itroductio Two importat problems are cosidered i regressio aalysis; multicolliearity ad oormal error distributios. The ordiary least squares estimators (LS) of coefficiets are kow to possess certai optimal properties whe explaatory variables are ot correlated amog themselves, ad the disturbaces of the regressio equatio are idepedet, idelically distributed ormal radom variables. The presece of correlatio amog the explaatory variables may result i imprecise iformatio beig available about the regressio coefficiets. I additio, the least squares estimator may produce extremely poor estimates i the presece of oormal disturbace distributios. Thus, various remedial techiques have bee suggested for these problems separately. Oe such remedial techique is ridge regressio to deal with multicolliearity, ad the robust estimatio techiques are ot as strogly affected by oormal disturbaces. However, although, we usually thik of these two problems separately, but i practical situatios, these problems occur simultaeously. Motgomery ad peck (982), have suggested that either robust or ridge estimatio methods aloe may be sufficiet for dealig with the combied problem. To remedy these two problems simultaeously, several robust ridge regressio estimators have bee put forward that are much less iflueced by oormality ad multicolliearity. Aski ad Motgomery (980), suggested combiig the ridge ad the least absolute deviatio (LAD) robust regressio techiques. I this paper, we take the iitiative to develop a more robust techique to remedy these two problems. We proposed combiig the ridge regressio with the highly efficiet ad high breakdow poit estimator, amely the M-estimator. We call this modified method, the robust ridge regressio based o M-estimatio (RM). We expect that, the modified method would be less sesitive to the presece of oormality ad multicolliearity. So, the aim of this paper is devoted to examie some estimators which are resistat to the combied problems of multicolliearity ad oormality. Exactly, ca the ridge estimators ad some robust estimatio techiques be combied to produce a robust ridge regressio estimator? I sectio (2), the ridge estimator will be discussed, ad a search for the robust estimatio techiques will be discussed i sectio (3). I sectio (4), the alterative combied estimators of ridge ad robust regressio are discussed. Sectio (5) presets the results of a Mote Carlo simulatio study to ivestigate how such estimators perform well, ad some cocludig remarks are preseted i sectio (6). 2. Ridge Regressio Estimators Cosider the followig liear regressio model : Y = Xβ + e, ()

3 Efficiecy of some robust ridge regressio 3833 where : y is a ( ) vector of observatios o the depedet variable, X is a ( p) matrix of observatios o the explaatory variables, β is a ( p ) vector of regressio coefficiets to be estimated, ad e is a ( ) vector of disturbaces. The least squares estimator of β ca be writte as : ˆ β ( X' X )- X 'Y LS = (2) This method gives ubiased ad miimum variace amog all ubiased liear estimators provided that the errors are idepedet ad idetically, ormally distributed. However, i the presece of multicolliearity, the sigularities preset i ( X ' X ) matrix ad this ill-coditioed X matrix ca result i very poor estimates. The degree of multicolliearity is ofte idicated by coditioed umber (CN) of the matrix X ( or X ' X ). CN is defied as the ratio of the largest sigular values of X to the smallest, λ CN ( X ) = max, (3) λ mi where : λ are the eigevalues of the matrix ( X ' X ). Belsley et al. (980) have empirically show that weak depedecies are liked to CN aroud 5 to 0, whereas moderate to strog relatios are liked to CN of 30 to 00. Hoerl ad Keard (970) poited out that addig a small costat to the diagoal of a matrix, will improve the coditioig of a matrix as this would dramatically reduced its CN. The ridge regressio is defied as follows : ˆ β = ( X' X + KI )- X' Y, (4) R where : I is the ( p p) idetity matrix ad K is the biasig costat. Various methods for determiig K value bee itroduced i the literature such as described by Hoerl ad Keard (970) ad Gibbos (98) as: PS 2 Kˆ LS H = β' βˆ, (5) LS LS where, S ( Y - Xβˆ )' ( Y - Xβˆ ) LS LS - p ˆ β ˆ R = β, whe K > 0, R K, ˆ β 0 LS 2 = (6) Whe k = 0, LS precise, tha LS estimator ad whe βˆ is biased but more stable ad R. Hoerl ad Keard

4 3834 Moawad El-Fallah Abd El-Salam (970) have show that, there always exist a value K > 0 shuch that MSE( ˆ β ) R is Less tha MSE( ˆ β ). LS 3. Robust Regressio Estimators Robust regressio estimators have bee prove to be more reliable ad efficiet tha least squares estimator especially whe disturbaces are oormal. " Noormal disturbaces" are disturbace distributios that have heavy or fatter tails tha the ormal distributio ad are proe to produce outliers. Sice outliers greatly ifluece the estimated coefficiets, stadard errors ad test statistics, the usual statistical procedure may be most iefficiet as the precisio of the estimator has bee affected. Several differet classificatios of robust regressio exist. Two of the most commoly cosidered group are : L-estimators ad M- estimators. The L-estimators is the earliest oe. Oe importat member of regressio L-estimators is called the least absolute deviatio estimator (LAD). The LAD estimator, βˆ, ca be defied as the solutio to the followig LAD miimizatio problem : mim i= Y - X' i β LAD (7) Rather tha miimizig the sum of squared residuals as i least squares estimatio, the sum of the absolute values of the residuals is miimized. Thus, the effect of outliers o the LAD estimates will be less tha that o LS estimates. The procedure of Weighted Least Squares ca be used to compute the LAD estimates. The Weighted Least Squares estimator ca be writte as : ˆ β = ( X ' WX ) X ' WY WLS () where W is a diagoal matrix with diagoal elemets of W matrix are set equal to : (8) w. The diagoal elemets ii w = ii eˆ i If If eˆ i eˆ i zero = zero, (9) () Aski, R.G., ad D.C. Motgomey (980)" Augmeted robust estimators". Techometrics, 22, p.338

5 Efficiecy of some robust ridge regressio 3835 where the ê are residuais from a iitial LS fit to the data. The weights w are ii applied to the observatios ad are iteded to dowweight the extreme observatios. Thus, the Weighted least Squares estimates ca be computed by applyig least squares to the trasformed observatios w ii y i ad w ii x i. This produces a estimate equivalet to βˆ i equatio (8). I this case, the WLS estimatio procedure ca be iterated to produce what are called the iteratively reweighted least squares estimates. See Motgomery ad Peck (982, p.368). 4. Robust Ridge Regressio Estimators Aski ad Motgomery (980) discussed augmeted robust estimators as a way of combiig biased ad robust regressio techiques. The combied procedure is based o the fact that robust estimates ca be combuted usig weighted least squares procedure. Whe, both outliers ad multicolliearity occur i a data set, it would seem preferred to combie methods for dealig with these problems simultaeously. I this sectio, we preset some combiatios of ridge ad robust regressio estimatio discussed i sectios (2) ad (3) respectively. I this respect, the robust ridge estimator, called weighted ridge estimator βˆ, ca WR be computed usig the followig formula : ˆβ = (X' WX + KI) - X'WY WR, (0) where the value of K is defied by (5) ad the weighits w are determied from ii equatio (9). I additio, aother robust ridge estimator is based o the LAD ad ridge estimators. This estimator will be deoted by the RLAD estimator ad ca be writte as : ˆβ = (X' X + K* I) - X' Y RLAD () where the value of K * is determied from data usig : PS 2 K * = LAD βˆ βˆ, (2) LAD LAD ad (Y - Xβˆ )'(Y - Xβˆ ) S 2 = LAD LAD, (3) p

6 3836 Moawad El-Fallah Abd El-Salam βˆ is the LAD estimator defied as the solutio to equatio (7). It be oted RLAD * K is the estimator of K preseted by equatio (5) with two that the value of chages. First, the LAD estimator of β is used rather tha LS estimator. Secod, the estimator of σ 2 used i equatio (3) is modified by the LAD coefficiet estimates rather tha the least squares estimates. These chages are aimed to reduce the effect of extreme poits o the value chose for the biasig parameter. Fially, the M-estimatio ca be used to determie the biasig parameter K as : PS K = M, (4) M ˆ ˆ β β ' M 2 M where the M-estimatio procedure of β is used rather tha LS estimator i computig the K ad S 2 values i order to reduce the effect of oormality o the value chose for K, ad the S 2 value ca be writte as : M S 2 = ( Y - Xβˆ )'( Y - Xβˆ ) (5) M M M I this case, the ridge M-estimator of the parameter β is give by : ˆ β ( X' X - KI )- X' Y, (6) RM = where, K is give i equatio (4). 5. Simulatio Study (5.) Desig of the Experimet : We carry out a Mote Carlo simulatio study to compare the performace of some alterative combied estimators uder cocem. The simulatio is desiged to allow both multicolliearity ad oomality simultaeously. Varyig degrees of multiocolliearity are allowed. Also, the oormal distributios are used to geerate outliers. The study cotais six estimators which are : () The least squares estimator (LS). (2) The ridge regressio estimator (R). (3) The least absolute deviatio estimator (LAD).

7 Efficiecy of some robust ridge regressio 3837 (4) The weighted ridge regressio estimator (WR). (5) The ridge least absolute deviatio estimator (RLAD). (6) The ridge M-estimator (RM). The LS estimator was defied i equatio (2). The R estimator was defied i equatio (4) usig the K value i equatio (5). The LAD estimator was defied as the solutio to the miimizatio problem i equatio (7) ad the iteratively reweighted least squares procedure was used to compute the LAD estimates. The WR estimator was defied i equatio (0). The RLAD estimator was defied i equatio () with K of equatio (2) used to determie a value for K, ad the RM estimator was defied i equatio (6) with K of equatio (4) used to determie a value of K. Suppose, we have the followig liear regressio model : y = β + β x + β x + e, where i =,2,..., (7) i o i 2 i2 i The parameter values β o, β ad β 2 are set equal to oe ( ). The explaatory variables x i ad x i2 are geerated as : x = ( ρ 2 ) z + ρ z, i =,2,...,, j =, 2 (8) ij ij ij where, z are idepedet stadard ormal radom umbers geerated by the ij IMSL subroutie GGNPM, ρ represets the correlatio betwee the two explaatory variables ad its values were chose as : 0.0, 0.5, 0.8, 0.9, 0.95, ad Oce, for a give sample size, the explaatory variables values were geerated. The sample sizes which will be examied i this study are : 20, 40 ad 80. Oe importat factor i this study is the disturbace distributio. The followig three disturbace distributios are used : Stadard ormal distributio. distributio with mea zero ad variace two. distributio with media zero ad scale parameter oe. I geeral, all the obtaied radom umbers are geerated usig the IMSL subrouties as : Stadard ormal radom umbers are geerated usig the GGNPM subroutie. radom umbers are geerated usig the GGUBFS subroutie. radom umbers are geerated usig the GGCAY subroutie. () Dempster, A.P. Schatzoff, M., ad N. Wermuth (977) "A simulatio study of altematives to Ordiary Least Squares " J.A.S.A., 72, p.99.

8 3838 Moawad El-Fallah Abd El-Salam I all cases, disturbaces are geerated idepedetly of the explaatory variables. The simulatios were performed o a IBM 434 Model 2. Programs were writte i double-precisio FORTRAN. For each of the = 54 treatmets i the three factor experimet, ( ρ, sample size, umber of distributios), 500 Mote Carlo trials are used. For each of the 54 treatmets, the followig statistics are computed. () The average of the estimates. (2) The mea squared error (MSE) ad the 5 pairwise MSE ratios where: MSE = ( βˆ i - β i ) 500 i = (9) (3) The mea absolute deviatios (MAD) ad the 5 pairwise MAD ratios, where : MAD = i = βˆ i - β i (20) (4) The 5 pairwise comparisos of " closeess " to the actual parameter values. The pairwise comparisos are : ˆ β with ˆ LS β ),( ˆ β with ˆ LS βlad ),( ˆ β with ˆ LS βwr ), ˆ β with ˆ LS β ),( ˆ β with ˆ LS βrm ),( ˆ β with ˆ R βlad ), ˆ β with ˆ R β ), ( ˆ β with ˆ R βrlad ), ( ˆ β with ˆ R βrm ), ˆ β with ˆ LAD β ), ( ˆ β with ˆ LAD βrlad ), ( ˆ with ˆ LAD βrm ) ˆ β with ˆ β ), ˆ β with ˆ β ), ˆ β with ˆ β ). ( R ( RLAD ( WR ( WR ( WR RLAD ( WR RM β, ( RLAD RM (5.2) The Results of comparisos : We cosider the compariso of the two robust ridge estimators WR ad RLAD. Table () presets the umber of times that the RLAD estimates are closer tha the WR estimates to the true value of the parameter β oly. While, Table (2) presets the results of the mea squared estimatio error ratios. These ratios represet the efficiecy of RLAD relative to WR. It be oted that, values less tha oe idicate that RLAD is more efficiet, while values greater tha oe idicated that WR is more efficiet.

9 Efficiecy of some robust ridge regressio 3839 Table () : Number of times RLAD is closer tha WR to the true parameter value Error Distributio β. Values of ρ = = 40 = Table (2) : Mea Squared Error Ratios of RLAD to WR for Estimatio of Error Distributio Values of ρ β * = = 40 = * Value less tha oe idicate RLAD is more efficiet tha WR; while, values greater tha oe idicate WR is more efficiet tha RLAD.

10 3840 Moawad El-Fallah Abd El-Salam From the results of Table (), we see that the RLAD estimator performs better tha WR estimator over a wide rage of combiatios betwee ρ ad the error distributio. These results are supported by the mea squared estimatio error ratios preseted i Table (2). Therefore, as the RLAD estimator clearly is superior to the WR estimator, the remaiig comparisos will be restricted to RLAD to coserve space. Tables (3), (5) ad (7) show the umber of times that the RLAD estimates are closer tha the, R, LAD, ad LS estimates, respectively, to the true value of the parameter β. Also, the MSE ratios of RLAD to each of these estimators R, LAD ad LS are give i Tables (4), (6) ad (8) respectively. Table (3) : Number of times RLAD is closer tha R to the true parameter value β. Error Distributio Values of ρ = = 40 =

11 Efficiecy of some robust ridge regressio 384 Table (4) : Mea Squared Error Ratios of RLAD to R for Estimatio of Error Distributio Values of ρ β * = = 40 = * Value less tha oe idicate RLAD is more efficiet tha R; while, values greater tha oe idicate R is more efficiet tha RLAD. Table (5) : Number of times RLAD is closer tha LAD to the true parameter value β. Values of ρ Error Distributio = = 40 =

12 3842 Moawad El-Fallah Abd El-Salam Table (6) : Mea Squared Error Ratios of RLAD to LAD for Estimatio of Error Distributio Values of ρ β * = = 40 = * Value less tha oe idicate RLAD is more efficiet tha LAD; while, values greater tha oe idicate LAD is more efficiet tha RLAD Table (7) : Number of times RLAD is closer tha LS to the true parameter value β. Values of ρ Error Distributio = = 40 =

13 Efficiecy of some robust ridge regressio 3843 Table (8) : Mea Squared Error Ratios of RLAD to LS for Estimatio of Error Distributio Values of ρ β * = = 40 = * Value less tha oe idicate RLAD is more efficiet tha LS; while, values greater tha oe idicate LS is more efficiet tha RLAD. From Tables (3) ad (4), we see that the R estimator margially is superior tha RLAD whe disturbaces are ormal ad the correlatio is high. Otherwise RLAD is superior. From Tables (5) ad (6), LAD is superior whe the correlatio is zero or low ad disturbaces are oomral. Otherwise, RLAD is superior. From Tables (7) ad (8), LS is superior whe there is o correlatio except for disturbaces. Otherwise, RLAD is superior. To coclude, the results from comparisos of RLAD estimator to the R, LAD ad LS estimators are ot etirely uexpected, give the properties of the various estimators. Therefore, the most importat result from these comparisos is, the RLAD estimator is superior to LAD estimator over a wide rage values of ρ for the give disturbace distributios as the ridge regressio, i some cases, is expected to perform well. 6. Cocludig Remarks Noormality ad multicolliearity are cosidered tow of the more frequet problems i regressio aalysis. Although, we usually thik of these two problems separately, but i applied situatios, these problems occur simultaeously. A Mote Carlo simulatio was desiged to compare the performace of some combiig ridge ad robust regressio estimators for

14 3844 Moawad El-Fallah Abd El-Salam dealig with these two problems. The results of comparisos idicate estimator is superior to WR estimator for may combiatios of error distributio type ad degree of multicolliearity (Table (2)). Oly, this estimator RLAD is less efficiet that the RID estimator whe disturbaces are ormal. However, the loss i efficiecy is small; at most 6 % whe = 20, 4 % whe = 40 ad 3% whe = 80 (see Table (4)). I additio, RLAD outperforms both LAD ad LS estimators whe the degree of multicolliearity is high. Therefore, the RLAD estimator appears to be a suitable altemative to other estimators whe both multicolliearity ad oormal disturbaces are preset. There are limitatios to the preset study, however. First, sice this is a simulatio study, its limitatios must be recogized. Data have bee geerated to try ad allow geeralizatio to practical situatios, however. Secod, oe specific form of the weighted ridge estimator (WR) was compared to the RLAD estimator. May other possible weightig forms could be used to costruct the WR estimator. Some of these forms are discussed i Aski ad Motogomery (980). Refereces [] D.F. Adrews, A robust method for multiple liear regressio, Techometrics, 6, (974), [2] R.D. Armstrog, E. Frome ad D. Kug.,A revised simplex algorithm for absolute deviatio curve-fittig problem, Commu. Stat. B Simul. Comput. 8, (979), [3] R.G. Aski, ad D.C. Motgomery. Augmeted robust estimators. Techometrics, 22, (980), [4] G.W. Bassett ad R.W. Koeker.a empirical quatile fuctio for liear models with iid errors. J.A.S.A, 77, (982), [5] D., Kuh, E. Belsley ad R.E. Welsh. Regressio diagostics. Wiley, New York, (980). [6] G. Casella. Coditio umbers ad miimax ridge-regressio estimators. J.A.S.A, 80, (985), [7] R.D. Cook ad S.Weisberg. Itroductio to Regressio Graphics. Wiley, New York, (994).

15 Efficiecy of some robust ridge regressio 3845 [8], A.P. Dempster, M. Schatzoff ad N. Wermuth.A simulatio study of alteratives to Ordiary Least Squares. J.S.A. 72, (977), [9] R.W. Fareborthe. A examiatio of recet criticisms of ridge regressio simulatios desigs. Commu. Stat. S, Theory Methods, 2, (983), [0] D. Gibbos. A simulatio study of some ridge estimators" J.S.A.76, (98), [] R.L. Gust, Maso. Biased estimatio i regressio : A evaluatio usig mea squared error. J.S.A. 76, (977), [2] A.E. Hoerl ad dr.w. Keard. Ridge regressio : Iterative estimatio of the biasig parameter. Commu. Stat. S, Theory Methods, 5, (976), [3] R.W. Koeker. Robust methods i ecoometrics. Ecoometric Rev.,, (982), [4] J.F. Lawless ad P.Wag. A simulatio study of ridge ad other regressio estimators. Commu. Stat. A Theory Methods, (76), [5] T.S. Lee ad D.B. Campbell. Selectig the optimum- K i ridge regressio. Commu. Stat.A, Theory Methods, 4, (985), [6] G.C. McDoald ad D.I.Glamea. Mote-Carlo evaluatio of some ridge-type estimators. J.S.A. 70, (975), [7] D.C.Motghomery ad E.A. Peck. Itroductio to Liear Regressio Aalysis. Wiley, New York, (982). [8] J.O. Ramsay. A comparative study of several robust estimates of slope, itercept, ad scale i liear regressio. J.S.A. 72, (977), [9] K.D. Lawrece ad J.I. Arthur. Robust Regressio Aalysis ad Applicatio. New York : Marcel Dekker, (990). [20] A.E. Hoerl ad R. W.Keard. Ridge Regressio : Iterative Estimatio of the Biasig parameter. Commuicatios i statistics : A Theory Methods, 5, (970a), [2] D. Gibbos. A Simulatio Study of some Ridge Estimators. Joural of America Statistical Associatio, 76,.(98), 3-39.

16 3846 Moawad El-Fallah Abd El-Salam Received: Jue, 203