Ridge Regression Estimators with the Problem. of Multicollinearity

Transcription

1 Appled Mathematcal Scences, Vol. 7, 2013, no. 50, HIKARI Ltd, Rdge Regresson Estmators wth the Problem of Multcollnearty Mae M. Kamel Statstc Department, Faculty of Commerce Tanta Unversty, Tanta, Egypt Sarah F. Aboud Egyptan Academy of Computers, Informaton & Management Technology Mnstry of Hgh Educaton, Tanta, Egypt Copyrght 2013 Mae M. Kamel and Sarah F. Aboud. Ths s an open access artcle dstrbuted under the Creatve Commons Attrbuton Lcense, whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgnal work s properly cted. Abstract The study ams to llustrate the negatve effect of the Multcollnearty problem upon the specmen, dentfy the way of Rdge Regresson as a way to deal wth the Multcollnearty problem, focus on some of the estmators of Rdge regresson as (James and Sten, Bhattacharya, Heurstc) and dentfy whch estmator from the prevously mentoned estmators s hghly preferable to be used, to estmate the parameters of a model whch faces the Multcollnearty problem. Mnmum mean-square error (MSE) has been used as the best measure for estmator. Applcaton has been done on specfc data for return on total assets of a bank after makng sure that ths data faces the Multcollnearty problem. Also, smulaton method was used to generate fabrcated data sets, whch gave more space n the applcaton. Accordng to the study we can see that James and Sten s estmator has got the mnmum mean square error (MSE). Consequently the study recommends ts usage to estmate model parameters whch face the Multcollnearty problem.

2 2470 Mae M. Kamel and Sarah F. Aboud Keywords: Rdge Regresson Multcollnearty Heurstc Bhattacharya James and Sten Smulaton 1-Introducton Several years ago some researchers n Econometrc feld used ordnary least square (OLS) method to estmate the parameters of lnear regresson model. The Ordnary least square method s based on a number of assumptons; one of these assumptons says that there s no lnear relatonshp between the explanatory varables, n case of droppng ths hypothess the multcollnearty problem wll appear. The multcollnearty problem threatens both the assumpton and usage of ordnary least square (OLS). In case the model s not full ranked,t won t be possble to fnd the nverse matrx of explanatory varables;consequently, we wll get nfnte solutons accordng to (OLS) method. On the other hand f the relatonshp degree between the explanatory varables s much enough and we use the (OLS) method we wll fnd that: The estmatons of model parameters have dfferent results n addton to that the standard errors for these estmatons ncrease when the relatonshp between explanatory varables ncreases. More standard errors of estmators mean a breadth of confdence nterval n the parameters of pobulaton. Increasng rsk of type two errors (acceptng wrong assumptons) because of the wdth of confdence nterval due to an ncrease n the standard error of the estmaton. If the lnear relatonshp s hgh, we get hgher R 2 however;most parameters of the relatonshp are non-sgnfcant statstcs by usng t test. The exstence or nonexstence of the lnear relatonshp between the varables sn t the pont but the strength degree s.we prevously mentoned that there s a lnear relatonshp between the explanatory varables (mathematcal varables) not a random relatonshp, so ths phenomenon concerns wth the sample not the socety for whom the sample was selected.so we don t test the exstence of these relatonshps but we measure the strength degree n any sample.due to the negatve mpact caused by Multcollnearty problem, there are many ways to deal wth ths problem and one of these methods s Rdge regresson (RR).

3 Rdge regresson estmators Rdge Regresson Rdge Regresson represents one of the methods whch deal wth multcollnearty problem and rdge regresson estmators as Horel and Kennard represented t n It s presented n low values from the dstnctve values (Egen Values) of the matrx (X X) n the model (Y = Xb + u).these values are added to the elements of the man dameter n the matrx (X X) n the estmators of least squares to gan the rdge regresson estmators whch represent based estmators dependng on addng low postve values whch called low added values (k) wth (rdge parameters) or (basng factor). It could be clarfed by the followng lnear regresson equatons: Y = Xb + e (1) If the model parameters were estmated accordng to the (OLS) method, we wll fnd that (b 0 ) wll be estmated as follows: b o = (2) But f the model parameters are estmated accordng to (RR) method, we wll fnd that the ordnary rdge regresson estmator (b orr ) wll be estmated accordng to the followng formula: B orr = (3) There are several rdge regresson estmators due to the lack of consstent law to dentfy the value (k), whereas (k > 0) denotes to basng factor.hence the dea of ths research s represented to llustrate and show some of rdge regresson estmators as James and Sten s parameter, Bhattacharya s parameter and Heurstc s parameter. In tryng to understand whch one of these parameters s able to help us n determnng the value of basng factor whch s less than MSE. 3-Rdge Regresson Estmators General Rdge Regresson estmator (b GRR ) b GRR = (XX + GKG) -1 X Y (4) where: (G) Represents the dstnctve columns n the matrx (X X). (K) Represents orthogonal desgn matrx. Suppose that all K values are equal n formula no. (4) and has K value, t means that (k = k). So we fnd that general regresson rate estmator (GRRE) denotes to ordnary regresson rate estmator(orre) whch s estmated by the followng formula: b ORR = (5)

4 2472 Mae M. Kamel and Sarah F. Aboud James and Sten estmator (b S ) In the followng regresson model Y = H b (Orth) + e (6) Ths model denotes to an orthogonal model Where (Y) symbolzes vector of dependant varables. and (H) symbolzes the matrx of explanatory varables for an orthogonal model. also b (orth) symbolzes vector regresson coeffcents for an orthogonal model. and (e) denotes to random error vector. In 1961 James Sten represented ( ) estmator [(vector regresson coeffcents) n orthogonal model ] whch s estmated accordng to the followng formula: = (7) Denotes to (OLS) estmator for vector regresson coeffcents for the llustrated model n the formula no. (6). (S) had been llustrated before by James &Sten n 1961 as the followng: S = 1- (8) Where: (P) denotes to the number of regresson coeffcents n the model. (n) denotes to vews sample (sampler s sze). (v) denotes to the total sum squared error whch s llustrated n formula no.(6) where [ V= ]. The relatonshp between b o (orth) estmator and b o estmator. Accordng to (OLS) estmator whch s llustrated n regresson model and formula no.(6) we fnd that : b o (orth) =(H' H) -1 H' Y (9) where: H ' H= 1 = H ' Y ` by multplyng the two Sdes by by substtutng n the formula (X'X) = ' (10)

5 Rdge regresson estmators 2473 In 1961 James & Sten presented (b s ) estmator n case that vector regresson coeffcent has more than two compound (p 3) t can be llustrated n ths formula as follows: b s = Sb o (11) where: b o denotes to (OLS) as t was prevously explaned n formula (2) Bhattacharya Estmator (b BH ) Bhattacharya estmator can be obtaned by the followng steps: Frst: Bhattacharya defned (b BH (orth)) estmator whch denotes to an evaluaton for regresson coeffcents n an orthogonal regresson model as follows: Where s a dametrcal matrx, the man dameter elements s represented n. Second : The elements can be obtaned by (reverse of )as follows: = p-+1 (13) Note: the symbol (-) above the varable denotes to reverse relatonshp for ths varable. Thrd : values was llustrated wth the followng formula: = p j= j j p j= O ( b ) α f,ν α ( Orth) j Fourth : Bhattacharya consdered the non-negatve values ά as follows: j = λ λ O Ffth: Bhattacharya defned ( f ( b Orth ), v (12) (14) for =1,2,3, P (15) ( ) 1 = 1,2 O f - 2 ( b ( Orth), v) = ν 1- = 3,4,..., p (16) 2 n - p + 2 O b ( Orth) Where b O O 2 ( Orth) = ( b ( orth) ) (17) 2 j= 1 j Sxth:By reference to formula no.(10) whch connects between and (orth). (18) By substtutng n from formula no.(14) (19) By substtutng n n the accordng to formula nom. (10)

6 2474 Mae M. Kamel and Sarah F. Aboud (20) (21) Heurstc Estmator ) It represents an extenson to Bhattacharya estmator, t s also known that ths estmator matches between the results whch each one of (GRR) and (Bhattacharya) had reached to. It can also be possble evaluate Heurstc ( ) accordng to the followng formula: = G δ G' (22) δ δ δ = δ p (23) The elements of the man dameter represented n n the dametrcal matrx. δ p α g = 1 = p O ( b ( Orth), ν ) α O In the prevous formula g ( b ( Orth), v) denotes to reorder process, f ( 0,v ) n order to be non-ncreasng. (24) 4-Usng Mean Square of Error to Compare Among study Estmators Comparng between Ordnary Least Square and Rdge Regresson Estmators s beng made to evaluate model parameters n the presence of multcollnearty problem accordng to mean square of error crtera. The functon error of ordnary least square(ols) can be expressed as follows: MSE( )= E ' (25) The functon error for James and Sten s estmator for (b) can be expressed as follows: MSE(b s )=E ' (26) The functon error for Bhattacharya s estmator can be expressed as follows: MSE(b BH )=E ' (27) The functon error for Heurstc s estmator can be expressed as follows: MSE(b H )=E ' (28)

7 Rdge regresson estmators Smulaton Technque The researchers have used HKB (Hoerl, Kennard, and Baldwn) style for smulaton to generate dfferent collocatons from regresson coeffcents. Ths knd s dstnctvely known wth ts easy style n applcaton n addton to that t gves a certan range to choose regresson coeffcents by usng Sgnal to Nose Rato,and t s easy to llustrate the steps of (HKB) applcatons as follows: 1-Pcked up randomly a certan number ( say 10 ) n a certan range[1-2500] to represent the SNR (Sgnal to Nose Rato) whch also represents the total squared bas dvded on the contrast, and t can be expressed as follows: / (28) 1, t was also found that the hgh s gves a great chance to the other estmators to overcome the ordnary least square. SNR values were represented n (1,4,9,25,48,81,200,400,800,2500) 2- The values of matrx of explanatory varables ( X )are nserted as follows: X= From each value from Sgnal-to-nose rato (SNR) we pck some of random numbers(r) from the standard normal dstrbuton equal m. where: *(r) represents the numbers. *(m) represents the number of explanatory varables n the model and they are (7) and takes the symbol (w). * r = 500 n ths study. Fnally the number of samples whch have been generated are 5000 sample (10 x 500), as (10) represents the number of Sgnal-to-nose rato values, whch have been dentfed n step No. (1).

8 2476 Mae M. Kamel and Sarah F. Aboud 4-To calculate the regresson coeffcents b (orth) for each generated random sample (where b (orth) represents regresson coeffcent n the orthogonal regresson model.) (29) 5- A 500 dfferent groups wll be generated from the random varable error n the normal dstrbuton wth excepected value equal to zero and one standard devaton and ts symbol s (e) to represent the error element n the orthogonal regresson model. 6- Values of the dependant varable wll be calculated (Y) for each sample among 500 samples. 6-The study result Table (1) clarfes the results whch the study has reached to: 1) The frst column represents the one tenth value n SNR. 2) The second column represents the outcome of usng (OLS) method to evaluate data regresson coeffcents whch have been generated by usng smulaton technque. 3) The thrd column shows the results of usng (James and Sten) regresson coeffcents evaluaton for data whch has been generated by usng smulaton method. 4) The forth column shows the results of usng Bhattacharya estmator. 5) The ffth column shows the results of usng Heurstc estmator.

9 Rdge regresson estmators 2477 Table (1) (MSE for each Estmator) SNR OLS Sten BH H * * * * * * * * * * The sgn denotes to mean square estmator n comparson to all study estmators. - The sgn * denotes to lowest mean square error n rdge regresson estmators. The study results can be clarfed by the followng Chart.

10 2478 Mae M. Kamel and Sarah F. Aboud Lne Plot of OLS, Sten, BH, H Data SNR OLS Sten BH H Chart (1 ) - The horzontal lne n chart (1) represents the study estmator [(OLS), Sten, Heurstc, Bhattacharya) respectvely. - The vertcal lne represents mean square error& the ten llustrated lnes n the chart represent SNR estmators. - In comparson to the rest of rdge regresson estmators; mean square error was found to be the best estmator whch got the least estmaton by the appled study, n (SNR=1, 4, 9, 49, 400, 2500). - In comparson to the rest of rdge regresson estmators, Heurstc estmator has got the least mean square error by the appled study, n (SNR=25, 81, 200). - In comparson to the rest of rdge regresson estmators, (OLS) estmator has got the leastmean square error by the appled study, n (SNR=25, 81, 200, 900). Concluson Accordng to the data study and the results whch were found, James and Sten was the best estmator among the study estmators as t got the least mean square error. t s recommended to be used n estmatng regresson coeffcents for a model whch face lnear Multcollnearty problem. It s worth notng that Heurstc estmator was the second best

11 Rdge regresson estmators 2479 estmator whch got the least mean square error n regresson coeffcents, whch requres greater attenton whle preparng studes based on Heurstc estmator. References [1] Bhattacharya, K., Estmatng the mean of a multvarate normal populaton wth general quadratc loss functon, Journal of mathematcal statstcs, 37(1996), [2] Hoerl, A, Kennard, R. and Baldwn, K., "Rdge regresson: some smulaton" Communcatons n statstcs, 4 (1975), [3] Hoerl, A. and Kennard, R., "Rdge regresson: based estmaton for nonorthoganal problems", Technometrcs, 12 (1970), [4] James, W. and Sten, C., "Estmaton wth Ouadratc loss", fourth edton Berkeley; Calforna, [5] Mcdonald, G. and Galarneau,D., "A Monte Carlo evaluaton of some rdge type estmators," Journal of the Amercan statstcs assocaton, 70 (1975), [6] Meshal, S., "On the selecton of the best rdge", Master degree n statstcs Insttute of statstcal studes and research, Caro Unversty [7] Pash, G.R. and Shah, M. A., "Applcaton of rdge regresson to multcollnear data" Journal of research scence, 15 (2004), [8] Rao, C., "Lnear statstcal nference and ts applcaton", second edton Kohn Wley& sons; New York, [9] Roso, M., "Estmaton of genetc effects n the presence of multcollnearty n multbreed beef cattle evaluaton "Journal of anmal scence, 83 ( 2005 ),

12 2480 Mae M. Kamel and Sarah F. Aboud [10] Sten, C., "Inadmssblty of the usual estmator for the Mean of a multvarte normal dstrbuton", thrd edton Berkeley; Calforna [11] Sten, C., "Estmaton of the mean of a multvarte normal dstrbuton", Ann statstc, 9 (1981), [12] Thel, H. and Glejser, H., "Studes n mathematcal and manageral economcs", Sxth edton North Holland; New York.,1980. Receved: February 21, 2013