Robust Logistic Ridge Regression Estimator in the Presence of High Leverage Multicollinear Observations

Transcription

1 Mathematcal and Computatonal Methods n Scence and Engneerng Robust Logstc Rdge Regresson Estmator n the Presence of Hgh Leverage Multcollnear Observatons SYAIBA BALQISH ARIFFIN 1 AND HABSHAH MIDI 1, Faculty of Scence and Insttute for Mathematcal Research, Unverst Putra Malaysa, UPM Serdang, Selangor, MALAYSIA 1 syababalqsh@gmal.com, habshah@upm.edu.my Abstract: - he logstc rdge regresson (LRR) estmator under both condtons of multcollnearty and hgh leverage ponts s examned and a new robust logstc rdge regresson (RLRR) estmator s ntroduced. he performance of LRR and RLRR estmators are evaluated and compared to the maxmum lkelhood (ML) estmator based on bas and mean squared error (MSE). Fndngs sgnfy that the ML estmator s the most affected followed by the LRR estmator n the presence of both condtons evdent by the larger bases and MSEs. In all cases nvestgated, the RLRR estmator outperforms both the ML and the LRR estmators. Key-Words: - maxmum lkelhood, logstc rdge regresson, multcollnearty, hgh leverage ponts, robust estmator, weghtng functon 1 Introducton hs artcle s concerned wth the problem of hgh leverage pont n hghly correlated data. Logstc regresson model s often used to analyze epdemologc and medcal data. Although the methodology of maxmum lkelhood (ML) and logstc rdge regresson (LRR) estmators for ths model s well developed, there are stll lack of nvestgaton on the model assessment n the stuaton when both multcollnearty and hgh leverage ponts occur together.it s evdent that the ML estmator s severely nfluenced by the presence of multcollnearty [13] and hgh leverage ponts [16]. Although the LRR estmator serves as a better alternatve n dealng wth multcollnearty [5, 9], there s uncertanty that the LRR estmator performs equally good when multcollnearty and hgh leverage pont occur together Multcollnearty s defned as a strong correlaton among predctors. In the last decades, some dscussons appeared n the lterature on the multcollnearty problem n the logstc regresson model [10, 13, 14]. Others explored ths problem n generalzed lnear model (GLM) [7, 15, 17]. he exstence of multcollnearty nflates the varance of estmated regresson coeffcents [5, 9]. Another effect of multcollnearty s havng lack of statstcal sgnfcance of ndvdual predctor s test whle the overall model may be strongly sgnfcant [7]. Hgh leverage pont s referred as outlyng observaton n X-space and t may causes more dffcultes to multcollnear data [16]. Moreover, the hgh leverage ponts bas the estmated regresson coeffcents and obscure other good observatons. Lkewse the lnear regresson, dentfcaton of multcollnearty n logstc regresson s detected usng condton ndces (CI) and condton number (CN) by computng egen values from correlaton matrx and Fsher nformaton matrx [7, 10]. In order to have a standard of comparson, re-scalng the predctors and ntercept term to unt length are suggested [7]. Meanwhle, for detecton of hgh leverage ponts, we recommended to use robust logstc dagnostc (RLGD) method by Syaba and Habshah [16] whch s capable to detect hgh leverage ponts correctly. Statstcal practtoners also rely on dagnostc approaches when dealng wth multcollnearty or hgh leverage ponts. hey tend to delete nfluental predctors n model before makng nference, leavng only a few overlapped cases between Y = 0 and Y = 1, whch can also cause large estmated regresson coeffcents and standard errors [, 1, 15]. herefore, the LRR estmator offers an alternatve estmatng method wthout deletng predctors. We expect that the LRR estmates would be larger when multcollnear data s contamnated wth the hgh leverage ponts In order to mprove the performance of LRR estmators, we ncorporate a ISBN:

2 Mathematcal and Computatonal Methods n Scence and Engneerng robust weghted Banco and Yoha (WBY) estmator [1] nstead of usng the ML estmator to obtan robust ntal coeffcent and robust rdge parameter. he new robust logstc rdge regresson (RLRR) employs robust ntal coeffcent and robust rdge parameter n teratve updatng formula. So far, the robust WBY estmator serves the best estmates when dealng wth hgh leverage ponts compared to other exstng robust methods avalable n the lterature. he advantages of weghtng scheme n WBY estmator n down-weghtng the effect of hgh leverage ponts s extensvely nvestgated by Habshah and Syaba [3]. Methodology In ths secton, we descrbe frst the LRR estmator wth the best opton of rdge parameter [5, 9] and how ths method can be mproved by mplementng the robust WBY estmator to compute robust ntal coeffcents and robust rdge parameter. he logstc regresson wth bnary response can be formulated n lnk lnear logt functon: logt ( π ) = ln π ( 1 π ) = x β (1) { } or n probablty of occurrence of success π = exp( x β) 1+ exp( x β) () { } where x s the -th row of an n ( p 1) wth p predctors and β s a ( p + ) + matrx X 1 1 vector of regresson coeffcents. he obectve functon for estmatng β s to maxmze the log lkelhood functon: ( X; β) = y log( π) + ( 1 y) log( 1 π) (3) he ML estmates are obtaned by solvng the subsequent equaton: ( X; β ) = X ( y π ) = 0 (4) β he equaton (4) s a nonlnear form n β. We appled the Newton-Rapshon method to solve t. he teratve re-weghted least square (IRLS) scheme for maxmum log lkelhood estmaton can be expressed as: ˆ ˆ β ˆ ML = X WX X Wz (5) ( ) ( ) where W ˆ s a matrx wth dagonal element ˆ π ( 1 ˆ π ) and z = x β + { y ˆ π ˆ π ( 1 ˆ π ) }. he asymptotc of mean squared error (MSE) equals the trace of covarance matrx: E( L ) = E( β β ML ML ) ( β β ML ) J 1 = tr ( X WX ) = = 1 λ where λ s the -th egen values of the (6) X WX matrx. It s mportant to note that the effect of hghly correlated predctors nflate the asymptotc varances for the ML causng some of the egen values to be small [9]. Due to ths maor drawback of ML estmates on multcollnear data, the LRR offers as a remedal estmator to ths problem..1 Logstc Rdge Regresson Rdge regresson was frstly ntroduced by [4] and [13], who used rdge estmator n logstc regresson. [5, 9] mproved the LRR estmator by ntroducng selecton of rdge parameters and [6, 8] appled t n a probt regresson model. he ML estmator produces large standard errors and MSE when predctors are multcollnear whle the LRR estmator reduces the mpact of multcollnearty [13]. he successfulness of LRR estmator depends on the rdge parameters, k, but no unversally accepted method of choosng k exsts. he LRR s defned as: ˆ ˆ β = X WX + ki X WX ˆ ˆ β (7) LRR ( ) ( ML ) where Wˆ and β are from the ML estmates. he ˆML asymptotc MSE of the LRR equals: J λ E( LRR ) = + = 1 λ + k (8) β ( ) ( + ) ' k X WX ki where the frst term s the asymptotc varance and the second term s the squared bas. By replacng λ wth λ + k n the denomnator, the asymptotc varance wll not be nflated..1.1 Choosng Rdge Parameter Earler recommendaton for the rdge parameter s ' k = ( p+ 1) ˆ ββ where k > 0 and I s an dentty matrx [13]. But, f k ncreases, squared bas also ncreases. herefore, the choce of k s based on logcally balance between the decrease n the varance that should be larger than the ncrease of the squared bas [9]. In ths paper, we consder the best opton k for the LRR estmates as suggested by [5, 9], when the degree of correlaton s hgh. he recommended k s as follows: β ISBN:

3 Mathematcal and Computatonal Methods n Scence and Engneerng where Gven 1 k = max m ˆ σ y ˆ π y ˆ π m = and ˆ σ = α n p ˆmax ˆ α s an element of ( ) (9) ( ) ( ) γβ ˆ where λ are egen values and γ s egen vector of X WX ˆ. It s mportant to note that the computaton of rdge parameter, k do not nclude the ntercept term.. Robust Logstc Rdge Regresson Bascally, the algorthm for RLRR s smlar to the LRR. he only dfference s that we replace the ntal estmated values of β, k and W ˆ by the ˆML ML estmator wth those estmated by the WBY estmator. Fortunately, the WBY estmator s already mplemented n CRAN software statstcal programmng under lbrary or package namely robustbase...1 Incorporate Robust Estmator to Downweght Hgh Leverage Ponts he Weghted Banco and Yoha (WBY) estmator s a weghted verson of the Banco and Yoha (BY) estmator. he BY estmator s found to be much more resstant than the ML estmator, but t has a bounded nfluence functon. Hence, the WBY estmator wth an overall bounded nfluence functon s obtaned where the effect of hgh leverage ponts s reduced by assgnng weghts to each of observaton [1]. he WBY estmator can be defned as: n argmn t ˆ γ = ωϕ zγ; y (10) ML ( ) n BY γ = 1 where the functon of ϕ BY s for the BY estmator and the weghts are computed usng: 1 f RD χ p,0.975 ω = (11) 0 else Identfcaton of hgh leverage pont s determned by computng RD, robust Mahalanobs dstance, where the estmaton subset s determned by mnmum covarance determnant (MCD) or FASMCD by [11]. For detaled nformaton on algorthm and formula, we refer to [1]. hen, we defne teratve coeffcents update for the RLRR as follows: ˆ β RLRR ( ˆ * * * XWX ki) ( XWX ˆ ˆ βwby ) = + (1) 3 Monte Carlo Smulaton It s a demand to conduct a Monte Carlo smulaton when exact theoretcal solutons are not avalable to support our expectaton. In ths secton, the performance of RLRR estmator n logstc regresson s nvestgated to see whether we can rely on ts parameter estmates, when the predctors are hghly correlated n the presence of hgh leverage ponts. Most authors consdered varous degrees of correlaton, the number of sample sze and the number of predctors as mportant factors that may affect the propertes of the dfferent estmates by the ML and LRR estmators [5, 9]. o acheve our obectve, we added the number of hgh leverage pont as a fourth factor to evaluate the estmates of RLRR and to compare them wth the ML and LRR estmates. 3.1 Evaluaton Crteron Lkewse prevous works [9], the X matrx wth multcollnearty s generated wth varous degrees of correlaton usng the followng equaton: x = ( 1 ρ ) ( 1/ ) z + ρz (13) p = 1,..., n and = 1,..., p where ρ represent the correlaton between predctors wth z N( 0,1 ) whch are randomly generated from normal dstrbuton. Meanwhle, the responses y are generated usng the Bernoull dstrbuton y Bern ( π ), where we defne the probabltes as: π = exp x β 1+ exp x β (14) { } ( ) ( ) Here, true values are fxed as β = 0 and 0 ( p) ( 1/ ) β = 1. hree dfferent degrees of correlaton are consdered, ρ = (0.75,0.85,0.95) for a number of predctor, p = and sample szes, n = (70,90,110) In the contamnaton set, h = (0,1, 3) are the number hgh leverage ponts that are generated usng unform dstrbuton, h U( 10,15 k ) wth response y whch are fxed at Y = 0. hen, the k contamnated observatons h k are plugged on the last rows of x. ISBN:

4 Mathematcal and Computatonal Methods n Scence and Engneerng 3. Estmator s Assessment he best estmator s determned based on comparson of lowest MSE among the estmators [5, 9]. It s mportant to note that the estmator wth the lowest MSE may have larger bas, snce the reducton of varance n logstc rdge regresson estmates s larger than the ncrease n squared bas dependng on the number of sample sze and magntude of rdge parameter. herefore, the best estmator should have both lowest MSE and bas. he estmated coeffcents of all estmators are computed over R = 1000 replcatons and contan summary measures of MSE and bas combnng the ndvdual results for the estmated coeffcents ncludng ntercept term. Bas and MSE measures are computed as follows: R 1 Bas = ˆ β β (15) R = 1 and R 1 MSE = ˆ β β R where. ndcates the Eucldean norm. (16) = 1 4 Fndngs and Interpretaton It s generally beleved that the LRR estmator should always be preferred when dealng wth multcollnear data set, but the RLRR estmator s expected to be better, snce we are not able to ensure that our multcollnear data set s free from the hgh leverage ponts. In ths secton, we would lke to dscuss the advantages of ncorporatng the robust WBY estmator n the RLRR estmator. ables 1-3 present the results of our Monte Carlo experment concernng the MSEs and bases of the ML, LRR and RLRR estmators. We wll dscuss on how these estmators are related to the degrees of correlaton, to the number of hgh leverage ponts and to the number of sample szes. Let us frst focus on able 1, where the generated data set s free from the hgh leverage pont (h=0). he estmated MSEs and bases for the ML estmator are the most affected by the presence of multcollnearty as t becomes evdent by observng at the larger MSEs and bases as the degree of correlaton ncreases. However, the LRR and the RLRR estmators have smaller MSEs and bases compared to the ML estmator and both estmates are farly close to each others. Now, let s see the performance of these estmators wth sngle contamnaton of the hgh leverage ponts (h=1) as tabulated n able. As t can be expected, the ML estmator performs even worst wth both contamnatons, as t gves the largest MSEs and bases. It s nterestng to observe that n the presence of hgh leverage pont n multcollnear data, the LRR estmator do not perform well by lookng at the estmated MSEs and bases that are larger compared to the LRR estmator havng only multcollnearty for every degree of correlaton ncreases. On the other hand, the RLRR estmates gve the smallest MSEs and bases. he ML and the LRR estmators severely affected as we ncreased the number of contamnated hgh leverage pont to h=3 (we refer to able 3), whch becomes evdent by the ncreased n MSEs and bases compared to h=1. Conversely, the RLRR estmator stll produces the smallest MSEs and bases. We observe that, as the number of sample szes and hgh leverage ponts ncreases, the estmates of RLRR perform better as t becomes evdent by the smaller MSEs and by the bases compared to small sample sze and sngle hgh leverage pont. able 1: Estmated MSEs and bases of all estmators wth multcollnearty ML LRR RLRR b0=0, ρ=0.75, p=, h=0 70 MSE Bas MSE Bas MSE Bas b0=0, ρ=0.85, p=, h=0 70 MSE Bas MSE Bas MSE Bas b0=0, ρ=0.95, p=, h=0 70 MSE Bas MSE Bas MSE Bas ISBN:

5 Mathematcal and Computatonal Methods n Scence and Engneerng able : Estmated MSEs and bases of all estmators wth multcollnearty and hgh leverage pont (h=1) ML LRR RLRR b0=0, ρ=0.75, p=, h=1 70 MSE Bas MSE Bas MSE Bas b0=0, ρ=0.85, p=, h=1 70 MSE Bas MSE Bas MSE Bas b0=0, ρ=0.95, p=, h=1 70 MSE Bas MSE Bas MSE Bas able 3: Estmated MSEs and bases of all estmators wth multcollnearty and hgh leverage ponts (h=3) ML LRR RLRR b0=0, ρ=0.75, p=, h=3 70 MSE Bas MSE Bas MSE Bas b0=0, ρ=0.85, p=, h=3 70 MSE Bas MSE Bas MSE Bas b0=0, ρ=0.95, p=, h=3 70 MSE Bas MSE Bas MSE Bas Concluson hs artcle s concerned wth the performance of ML, LRR and RLRR estmators n logstc regresson to obtan relable estmates when the predctors are severely collnear n the presence of hgh leverage ponts. he assessment on these estmators by the Monte Carlo smulaton ndcates that multcollnearty serously affects the ML estmator by producng large and unstable estmates, whle the LRR estmator has potental for producng better estmates wth smaller MSEs and bases compared to the ML estmator n multcollnear data. In the presence of both hgh leverage ponts and multcollnearty, the LRR estmator s affected, evdent by the ncreased n MSEs and bases. Hence, the LRR estmator s not relevant to be used when both multcollnearty and hgh leverage pont exst n data, snce the hgh leverage pont leads to a hgh MSE and bas. he best opton s to use the RLRR estmator, where the effect of hgh leverage pont s reduced by the weghtng functon n the WBY robust estmator. he RLRR reduces the MSE and bas substantally even n a stuaton, where the degree of correlaton s hgh n the presence of hgh leverage ponts n data. References: [1] Croux, C. and Haesbroeck, G., Implementng the Banco and Yoha Estmator for Logstc Regresson, Computatonal Statstcs & Data Analyss, Vol.44, No.1-, 003, pp [] Chrstmann, A. and Rousseeuw, P. J., Measurng Overlap n Bnary Regresson, Computatonal Statstcs & Data Analyss, Vol.37, No.1, 001, pp [3] Habshah, M., Syaba, B. A., he Performance of Classcal and Robust Logstc Estmators n the Presence of Outlers, Pertanka Journal of Scence & echnology, Vol.0, No., 01, pp [4] Hoerl, A. E, and Kennard, R. W., Rdge Regresson: Applcatons to Non-Orthogonal Problems, echnometrcs, Vol.1, No.1, 1970, pp [5] Kbra. B. M. G., Månsson, K. and Shukur, G., Performance of Some Logstc Rdge Regresson Estmators, Computatonal Economcs, Vol.40, No.4, 01, pp [6] Kbra B. M. G. and Salleh A. K. E., Improved the Estmators of the Parameters of a Probt Regresson Model: A Rdge Regresson Approach. Journal of Statstcal Plannng and Inference, Vol.14, 01, pp ISBN:

6 Mathematcal and Computatonal Methods n Scence and Engneerng [7] Lesaffre, E. and Marx, B. D., Collnearty n Generalzed Lnear Regresson, Communcatons n Statstcs heory and Methods, Vol., No.7, 1993, pp [8] Lockng, H., Månsson, K and Shukur. G., Performance of Some Rdge Parameters for Probt Regresson: Wth Applcaton on Swedsh Job Search Data, Communcatons n Statstcs Smulaton and Computaton, Vol.4, No.3, 013, pp [9] Månsson, K. and Shukur, G., On Rdge Parameters n Logstc Regresson, Communcatons n Statstcs heory and Methods, Vol.40, No.18, 011, pp [10] Marx, B. D. and Smth, E. P., Weghted Multcollnearty n Logstc Regresson: Dagnostcs and Based Estmaton echnques wth an Example From Lake Acdfcaton, Canadan Journal of Fsheres and Aquatc Scences, Vol.47, No.6, 1990, pp [11] Rousseeuw, P. J and Van Dressen, K., A Fast Algorthm for the Mnmum Covarance Determnant Estmator, echnometrcs, Vol. 41, No. 3, 1999, pp [1] Rousseeuw, P. J. and Chrstmann, A., Robustness Aganst Separaton and Outlers n Logstc Regresson, Computatonal Statstcs & Data Analyss, Vol.43, No.3, 003, pp [13] Schaefer, R. L., Ro, L.D. and Wolfe, R. A., A Rdge Logstc Estmator, Communcatons n Statstcs heory and Methods, Vol.13, No.1, 1984, pp [14] Schaefer, R. L., Alternatve Estmators n Logstc Regresson When the Data are Collnear, Journal of Statstcal Computaton and Smulaton, Vol.5, No.1-, 1986, pp [15] Segerstedt, B. and Nyqust, H., On the Condtonng Problem n Generalzed Lnear Models, Journal of Appled Statstcs, Vol.19, No.4, 199, pp [16] Syaba, B. A. and Habshah, M., Robust Logstc Dagnostc for the Identfcaton of Hgh Leverage ponts n Logstc Regresson Model, Journal of Appled Scences, Vol.10, No.3, 010, pp [17] Wessfeld, L. A. and Sereka, S. M., A Multcollnearty Dagnostc for Generalzed Lnear Model, Communcatons n Statstcs heory and Methods, Vol.0, No.4, 1991, pp ISBN: