A Modification of the Ridge Type Regression Estimators

Transcription

1 Amercan ournal of Aled Scences 8 (): 97-0, 0 ISSN Scence Publcatons A Modfcaton of the Rdge Tye Regresson Estmators Moawad El-Fallah Abd El-Salam Deartment of Statstcs and Mathematcs and Insurance, Faculty of Commerce, Zagazg, Unversty, Zagazg, Egyt Abstract: Problem statement: Many regresson estmators have been used to remedy multcollnearty roblem. The rdge estmator has been the most oular one. However, the obtaned estmate s based. Aroach: In ths stuyd, we ntroduce an alternatve shrnage estmator, called modfed unbased rdge (MUR) estmator for cong wth multcollnearty roblem. Ths estmator s obtaned from Unbased Rdge Regresson (URR) n the same way that Ordnary Rdge Regresson (ORR) s obtaned from Ordnary Least Squares (O). Proertes of MUR estmator are derved. Results: The emrcal study ndcated that the MUR estmator s more effcent and more relable than other estmators based on Matrx Mean Squared Error (MMSE).Concluson: In order to solve the multcollnearty roblem, the MUR estmator was recommended. Key words: Multcollnearty, Ordnary Least Squares (O), Ordnary Rdge Regresson (ORR), Unbased Rdge Regresson (URR), Modfed Unbased Rdge (MUR), Matrx Mean Squared Error (MMSE), Cumulatve Densty Functon (CDF), rdge arameter, alternatve shrnage estmator, harmonc mean INTRODUCTION Consder the followng lnear regresson model: Y Xβ+Є () wth the usual notaton. The Ordnary Least Squares (O) estmator: β (X'X) X'Y () follows N( βσ, (X'X) ). If X X s sngular or near sngular, we say that there s multcollnearty n the data. As a consequence, the varances of elements of β are nflated. Hence, alternatve estmaton methods have been roosed to elmnate nflaton n the varances of β. Hoerl and Kennard (970) roosed Ordnary Rdge Regresson (ORR) as: β + β (X'X KI ) X'Y, K 0 () [I K(X'X KI ) ] + (3) Usually 0 < K <. Ths estmator s based but reduces the varances of the regresson coeffcents. Subsequently, several other based estmators of β have been roosed (Swndel, 976; Sarar, 996; Batah and Gore, 008; Batah et al., 009; Arayesh and Hossen, 00; Aseunowo et al., 00; Hrun and Srsoonsl, 00; Rana et al., 009). Swnded (976) defned Modfed Rdge Regresson (MRR) estmator as follows: (,b) (X'X KI ) (X'Y Kb), K 0 β + + (4) where, b s a ror estmate of β. As K ncreases ndefntely, the MRR estmator aroaches b. Crouse et al. (995) defned the Unbased Rdge Regresson (URR) estmator as follows: (, j) (X'X KI ) (X'Y Kj), K 0 β + + (5) σ where, ~ N β, I for K>0. They also roosed K the followng estmator of the rdge arameter K: K Pσ f ( β )'( β ) > ( β )( β ) σ tr(x'x) σ tr(x'x) Pσ ( β )'( β ) Consder sectral decomoston of X X, namely X X 97 Otherwse (Y X β )'(Y X β) where, σ s an unbased (n P) estmator of σ. They further noted that K s a generalzaton of K Pσ of Hoerl et al. (98). β β

2 TAT, where TT T T. In ths case, Eq. can be wrtten as: Y XTT' β+ ZL+ (6) wth Z XT, L T'β where Z Z T X XT A dag (λ, λ,.., λ P ). The dagonal elements of A are the egenvalues of X X and T conssts of corresondng the egenvalues of X X. Hence O, ORR and URR of L are wrtten as LO A Z'Y,L (K) (A+ KI ) Z'Y and L (A KI ) (Z'Y K) (K,) + +, resectvely. In ths study, we ntroduce an alternatve shrnage estmator, called Modfed Unbased Rdge (MUR) estmator. Ths estmator s obtaned from URR n the same way that ORR s obtaned from O. It s observed that O s unbased but has nflated varances under multcollnearty. Smlarly, URR suffers from nflated varances whle elmnatng the bas. The constructon of MUR s based on the logc that just as ORR avods nflatng the varances at the cost of bas, MUR would have smlar roertes. Wth re-multle the matrx [I K (X'X + KI ) ] to reduce the nflated varances n O, so that we exect the same effect wth URR. so that we exect the same effect wth URR. Ths s our motvatng the alternatve modfed estmator. In ths resect, the man object of ths aer s that the MUR estmator erforms well under the condtons of multcollnearty. The roertes of ths alternatve modfed estmator are studed, and some condtons for ths estmator to have smaller MMSE than ORR and URR are derved also. In addton, as the value of K must be secfed for K n MUR n the same way as n ORR and URR, so three dfferent ways for determnng K are comared usng smulated data. MATERIA AND METHODS The roosed estmator: We roose the followng estmator of β: β + β [I (X'X+ KI ) ](X'X+ KI ) (X'Y+ K) () [I (X'X KI ) ] (,) Am.. Aled Sc., 8 (): 99-0, 0 (7) σ where, ~ N( β,( I ) ) and >0. Ths estmator s called Modfed Unbased Rdge Regresson (MUR) because t s develoed from URR. The MUR n model (6) becomes: + (8) L () [I (A KI ) ]L(,) 98 Bas: The MUR estmator has the followng roertes. Bas ( β ()) E( β ()) βs β (9) where, S X X and S (S + KI). Varance: Var ( β ()) E[( β () E( β ()))( β ())] σ WS W ' where, W [I KS ]. Matrx Mean Squared Error (MMSE): MMSE( β ()) Var( β ()) + [bas( β ())] [bas( β ())] σ WS W' + S ββ's Scalar Mean squared Error (SMSE): SMSE( β ()) E[( β() β)'( β() β)] tr(mmse( β ())) where, tr denotes the trace. Then: SMSE(L ()) σ ( λ + ) ( λ + ) λ ( λ + )L (0) () () where, {λ } are egenvalues of X X. β β s the O estmator: ( 0) (X'X) X'Y lm β () β 0 Comarson wth other estmators: MUR s based and t s therefore comared wth other estmators n terms of MMSE. We obtan condtons for MUR to have smaller MMSE than another estmator. Comarson wth ORR: The MMSE of ORR s (Ozale and Kaçranlar, 007): MMSE( β ()) σ WS W ' + S ββ 'S (3) so that: SMSE(L()) λ σ + ( λ + ) ( λ + ) (4) L

3 Am.. Aled Sc., 8 (): 99-0, 0 Consder: Δ MMSE( β()) MMSE( β ()) σ W(S S )W' σ H (5) β'[ I S ] β σ The condton of result () s verfed by testng: Snce S S KI s ostve defnte (.d.), t s easy to show that S S s.d. whenever > 0. Hence we have the followng result. Result : MUR has smaller MMSE than ORR when > 0. Comarson wth URR: The MMSE of the URR estmator s (Ozale and Kaçranlar, 007): MMSE( β (,)) σ (6) and hence: S SMSE( β (,)) tr (MMSE( β (,))) (7) Then: SMSE(L(,)) σ (8) ( λ + ) From (): Δ MMSE( β(,)) MMSE( β ()) σ [S WS W '] S ββ's S [ σ ( I S ) ββ']s Now, Δ s non-negatve defnte (n.n.d.) (assumng > 0) f and only f Φ S Δ S s n.n.d. Further: Φσ ( I S ) ββ ' (9) H : β'[ I S ] β σ 0 Aganst: H : β'[ I S ] β>σ Snce Λ-Λ * () s ostve sem defnte, the * condton n Result () becomes β't Λ () T' β σ f β'tλ T' β σ. Under the assumton of normalty: * * () T' () N( () * Λ β Λ Λ σ Λ β σ Λ (I )T', () (I ) ) and the test statstcs: β ()'TΛ T' β () / β'tλ T' β ' /n σ F F,n, under H 0. The concluson s that MUR has a smaller MMSE than URR f H 0 s acceted and hence Result () holds. Otmal rdge arameter: Snce the MMSE of MUR deends on the rdge arameter, the choce of s crucal for the erformance of MUR. Hence we fnd condtons on the values of for MUR to be better than other estmators n terms of SMSE. Result 3: We have: SMSE (L ()) < SMSE (L(, )),for 0 < < SMSE (L ()) > SMSE (L(,)),for < < Snce the matrx I S s ostve defnte (Farebrother, 976), Φ s n.n.d. f and only f: β'[ I S ] β σ (0) Where: ( σ λl ) ( σ λl ) σ λ > L 4L L Proof: Result (3) can be roved by showng that: Hence we have the followng result. 3 ( λ + ) [SMSE (L ()) SMSE (L(,))] Result : MUR has smaller MMSE than URR f: [L ( σ λl) λσ ] 99 / ()

4 Am.. Aled Sc., 8 (): 99-0, 0 whch s obtaned from () and (8). Ths comletes the roof. Next, we comare SMSE of (L ()) wth that of O comonent-wse. Notce that the MUR estmator reduced to O when 0. The -th comonent for SMSE of L of O s gven by: σ SMSE (L ),,,, λ We state the followng result. Result 4: We have: If λ L σ 0, then the: SMSE (L ()) < SMSE (L ),for 0 < < () If λl σ > 0, then there exsts a ostve, such that: and: SMSE (L ()) > SMSE (L ),for 0 < < λ 6σ [( ( )) ] (4) () L From (), (3) and (4), t can be easly verfed that < () < f λl σ > 0. In case, we can obtan as the harmonc mean of () n (4). It s gven by: σ () 4 Lλ 6Lλ λ L + 4 [L / [( ) ]] 4σ σ σ (5) Usng an argument from Hoerl et al. (98), t s reasonable to adot the harmonc mean of the regresson coeffcents. Note that () n (5) deends on unnown arameters L and σ and hence has to be estmated. Estmatng the rdge arameter : In ths secton, we roose to construct MUR by usng the oeratonal rdge arameter roosed by Hoerl et al. (98) and Crouse et al. (995). Frst, snce the harmonc mean of otmal rdge arameter values, (see (4)) deends on the unnown arameters L and σ, we use ther O estmates. The oeratonal rdge arameter n (5) s: SMSE (L ()) < SMSE (L ),for < < Where: σ [L / [( ) ]] 4σ σ σ () 4 Lλ 6Lλ λ L + 4 (6) λ σ λ σ λ σ ( λ L 3 σ λ) 3 λ σ ( λ L 3 σ λ) + > 0 (3) 4( L ) ( L ) ( L ) Proof: Result (4) can be roved by showng that: Ths s called the () rdge arameter. Second, the rdge arameter (Hoerl et al., 98) s: σ (7) L' L λ ( λ + ) [SMSE (L ()) SMSE (L )] [ L )] L 3 ) 3 ] 3 λ σ +λ σ λ λσ whch s obtaned from () and (). Ths comletes the roof. Furthermore, dfferentatng SMSE (L ()) wth resect to and equatng to zero, we have the followng equaton: Thrd, rdge arameter (Crouse et al., 995) s: σ f ( β )' ( β )'( β ) σ tr(x'x) ( β ) >σ tr(w'x) SMSE (L ()) λ L + λl 3σλ 0 O 4 ( λ + ) ORR usng the rdge arameter (ORR ()) Thus, the otmal value of the rdge arameter s: ORR usng the rdge arameter (ORR ()) 00 σ ( β )'(( β )) Otherwse Usng these three oeratonal rdge arameters, we comare the followng ten estmators:

5 Am.. Aled Sc., 8 (): 99-0, 0 Table : Values of estmates and SMSE for 0.033, and where SMSE shows the SMSE for estmators β β β 3 β 4 β 5 SMSE Poulaton O β ( ) β ( ) β ( ) β (, ) β (, ) β (, ) β ( ) β ( ) β ( ) ORR usng the rdge arameter (ORR ()) URR usng the rdge arameter (URR ()) URR usng the rdge arameter (URR ()) URR usng the rdge arameter (URR ()) MUR usng the rdge arameter (MUR ()) MUR usng the rdge arameter (MUR ()) MUR usng the rdge arameter (MUR ()) RESULTS We analyze the data generated by Hoerl and Kennard (98). The data set s generated by tang a factor structure a real data set and choosng β 9.069, β , β , β and β at random wth constrant β β 300 and a standard normal error s added to form the observed resonse varable β, β, β 3, β 4 and β 5 are random wth the constrant β' β 300 and normal error e has zero mean and σ. The resultng model s YXβ+Є and Є s normally dstrbuted as N(0,σ I). The data was then used by Course et al. (995) to comare SMSE erformance of URR, ORR and O. Recently, Batah et al. (009) used the same data to llustrate the comarsons among O and varous rdge tye estmators. We now use ths data to llustrate the erformance of the MUR estmator to the O, ORR and URR estmators to comare the MMSE erformance of these estmators. DISCUSSION Table shows the estmates and the SMSE values of these estmators. The egenvalues of X'X matrx are , 0.940, 0.549, , the rato of the largest to the smallest egenvalue s 33.5 whch mles the exstence of multcollnearty n the data set. The comarson between SMSE ( β ) and SMSE ( β ( )) show that the magntude of shrnage s not enough. When based and unbased estmators are avalable, we refer unbased estmator. Crouse et al. (995) 5 suggested [ β /5] 5 as a realstc emrcal ror nformaton where s the vector of ones. URR wth leads to smaller SMSE than wth and and correct the wrong sgn. We thus fnd that s suffcent. MUR has smaller SMSE than ORR. Table summarzes the erformance of estmators for secal values of. we observe that MUR estmator wth (6.7437, , , , ) s not always better than other estmators n terms of havng smaller SMSE. Also we can see that MUR s better than ORR for all, and under the MMSE crteron, whch s result (). The value of ' β [ I S ] β gven n result () s obtaned as for, for and 6.64 for whch are not smaller than the O estmate of σ.48. Therefore, URR estmator s better than the MUR estmator for, and n terms of MMSE as n Table. The value of the F test n Result () s F Cul , the non-central F arameter value calculated s wth numerator degrees of freedom 5, denomnator degrees of freedom 0 by usng the Cumulatve Densty Functon (CDF) Calculator for the Non-central-F Dstrbuton (see webste htt://wwwdanelsoer.com/statcalc/calc06.asx). Here, the non-central F CDF s equal to Then H 0 s

6 Am.. Aled Sc., 8 (): 99-0, 0 acceted and the condton n Result () holds. That s, MUR has smaller MMSE than URR. CONCLUSION In ths study artcle we have ntroduced Modfed Unbased Rdge (MUR) estmator. Comarson of ths estmator to that ORR and URR has been studed usng the MMSE. Condtons for ths estmator to have smaller MMSE than other estmators are establshed. The theortcal results ndcate that MUR s not always better than other estmators n terms of MMSE. MUR s best and deends on the unnown arameters β, σ and also usng the rdge arameter. for sutable estmates of these arameters, MUR estmator mght be consdered as one of the good estmators usng MMSE. REFERENCES Arayesh, B. and S.. Hossen, 00. Regresson analyss of effectve factor on eole artcaton n rotectng, revtalzng, develong and usng renewable natural resources n Ilam rovnce from the vew of users. Am.. Agrc. Bo. Sc., 5: DOI: /ajabss Aseunowo, V.O., G. Olutunla and A.G. Daramola, 00. Seeng an alternatve modalty to the management of Ngera's fertlzer subsdy schemean emrcal aroach to the case study of ondo state ( ).. Soc. Sc., 6: DOI: /jss Batah, F.S.B, M.R. Özale and S.D. Gore, 009. Combnng unbased rdge and rncal comonent regresson estmators. Commun. Statstcs-Theory Methods, 38: DOI: 0.080/ Batah, F.S.M, S.D. Gore and M.R.Verma, 008. Effect of jacnfng on varous rdge tye estmators. Model Asss. Stat. Al., : 0-0. Crouse, R.H., C. n and R.C. Hanumara, 995. Unbased rdge estmaton wth ror nformaton and rdge trace. Commun. Statstcs-Theory Methods, 4: DOI: 0.080/ Farebrother, R.W.,976. Further results on the mean square error of rdge regresson.. Royal Stat. Soc. B., 38: Hrun, W. and S. Srsoonsl, 00. Analyss of nterregonal commodty flows. Am.. Eng. Aled Sc., 3: DOI: /ajeass Hoerl, A. and R. Kennard, 970. Rdge regresson: Based estmaton for nonorthogonal roblems. Technometrcs, : Hoerl, A. and R. Kennard, 98. Rdge regresson- 980: Advances, algorthms and alcatons. Am.. Math. Manage. Sc., : Rana, M.S., M. Habshah and A.H.M.R. Imon, 009. A robust rescaled moment test for normalty n regresson.. Math. Stat., 5: DOI: /jmss Sarar, N., 996. Mean square error matrx comarson of some estmators n lnear regressons wth multcollnearty. Stat. Probabl. Lett., 30: DOI: 0.06/067-75(95)00- Swndel, B.F., 976. Good rdge estmators based on ror nformaton. Commun. Statstcs-Theory Methods, 5: DOI: 0.080/