Logistic Regression Classification for Uncertain Data

Transcription

1 Abstract Research Joural of Matheatcal ad Statstcal Sceces ISSN Vol. 2(2), -6, February (204) Res. J. Matheatcal ad Statstcal Sc. Logstc Regresso Classfcato for Ucerta Data Abdallah Bashr Musa College of Matheatcs ad Coputer Scece, Hebe Uversty, Baodg 07002, CHINA Avalable ole at: Receved 7 th October 203, revsed 2 d Deceber 203, accepted 2 th February 204 Logstc regresso (LR) s a faous classfcato techque cooly used statstcs, ache learg, ad data g area of kowledge for learg a respose of bary ature. It assues that the data values are pre-detered precsely, but ths s ot true for all codtos. Ucertaty data arses ay applcatos because of data collecto ethodology as repeated easures, outdated sources ad precse easureet as physcal experets. Studyg ths ucertaty data becoes area of terest for researchers owadays. I ucertaty, the value of data te s ostly characterzed by a ultple values. So, ache learg techques are also requred to aage a ucerta data. hs paper studes the odfcato of LR techque to hadle data wth a ucertaty. Statstcal ferece ad theory of probabltes are used to obta sgle ubased estator that represets the ultple values suffcetly ad effcetly. he Maxu Lkelhood Estators (MLE) ad the Probabltes Desty Fucto (PDF) are used to capture the ucertaty. Results of the Experets o UCI data sets deostrated that the ucerta LR classfer ca be costructed successfully, ad ts accuracy ca be proved by takg to cosderato the ucertaty forato. Keywords: Logstc Regresso (LR), Ucerta data, classfcato, Maxu Lkelhood Estators (MLE), Probabltes Desty Fucto (PDF). Itroducto LR odel s cosdered as oe of ost reowed classfcato odels. LR s apprecated ad broadly used statstcs, ache learg, ad data classfcato coutes he advatages of ths odel clude a strog statstcal foudato ad probablstc odel whch helps aalyzg data 5. It s ostly used bary classfcato probles appled sceces such as edce, bology ad epdeology. It has bee wdely appled due to ts splcty ad great terpretablty 6. LR has bee cosdered to be a effcet classfer ad a powerful predcto ethod. However, as for ost of the ache learg algorths, t was costructed to hadle oly the data wth certaty where there s a sgle value each attrbute. Recetly ucertaty data are arsg ay applcatos Ucertaty ca orgate due to uerous reasos such as devses precso ltato, data saplg error, ad collecto data probles as repeated easures, whch t sees to be the coo type of data wth ucertaty, aog others etc,.i ucertaty data forato caot be dyllcally represeted by a sgle value. herefore, t s dffcult to acheve satsfactory results whe classfyg such data wthout aagg ucertaty. he error the data s geerally treated as a rado varable wth probablty dstrbuto 0. he ucerta attrbute value s ofte represeted by a terval wth a probablty dstrbuto fucto over ths terval, 2. Classfcato s a classcal proble ache learg ad data g 3. It s the process of buldg a odel that ca descrbe ad predct the class label of data based o features. May classfcato algorths such as support vector ache, decso tree, Bayesa ad eural etworks used the certaty data for perforg the classfcato tasks. A coo way to hadle the ucertaty s to represet t by ts expected value ad treat t as certaty data, so the classfcato algorth ca be drectly appled. However, ths ethod does ot effectvely utlze portat forato such as probablty fucto ad dstrbuto tervals. herefore, buldg classfer based o ucerta data s a great challege that eeds specfc classfcato algorths. here are sgfcat studes that proposed classfcato algorths to aalyze ucertaty data, such as support vector classfcato wth put data ucertaty 4, decso trees for ucerta data 3, a decso tree for classfyg ucerta Data 5, ave Bayes classfcato of ucerta Data 6 a Bayesa classfer for ucerta data 7, ad a eural etwork for ucerta data classfcato 0. o the best of preset kowledge, there s o study focus o buldg LR fro ucertaty data, Moreover, the ethod that propose ths paper s very dfferet fro those used prevous studes. I ths paper, the LR classfer algorth has bee exteded to hadle the ucertaty data he axu lkelhood estator (MLE) 8.9 whch s the best, ubased, suffcet ad effcet estator s used wth the Gaussa ad the ufor dstrbutos to represeted the ultple values, ulke others studes where average s used as a estator whch s ot always good as Iteratoal Scece Cogress Assocato

2 Research Joural of Matheatcal ad Statstcal Sceces ISSN Vol. 2(2), -6, February (204) es. J. Matheatcal ad Statstcal Sc. the case of the ufor dstrbuto. he kerel desty estato ethod 20 whch provdes a effectve represetato ad good approxato of the desty s used. he dea kerel desty estato s to provde a cotuous estate of the desty of data at a gve pot. he value of desty at a gve pot s estated as the su of soothed values of kerel fuctos assocated wth each pot data set. Each kerel fucto s assocated wth a kerel wdth whch deteres the level of soothg created by the fucto. he probabltes desty fucto (PDF) ethod of the Gaussa dstrbuto s used to capture the ucertaty. he purpose of ths study s to costruct the LR o the ucertaty data ad to vestgate the classfcato accuracy acheved by the average, MLE ad PDF. Logstc regresso: Logstc Regresso (LR) s a well kow statstcal classfcato ethod for odelg dchotoous (bary) data Let x R deote a vector of explaatory or feature varables, ad let y {, + } deote the assocated bary class label or outcoe. he logstc odel s defed as: exp(y( β x + α )) pr ( y / x) () + exp(-y( β x + α )) + exp(y( β x + α )) Where Pr(y/x) s the codtoal probablty of y gve x R. he logstc odel has paraeters α R represet the tercept te ad β R represet the weght vector. β x + α 0 defes a hyperplae the feature space, o whch P(y/x)0.5. he codtoal probablty Pr(y x) s larger tha 0.5 f β x + α has the sae sg as y, ad less tha 0.5 otherwse. Suppose we are gve a set of observed or trag data{, }, where x R deote the -th saple ad x y y {, + } deote the correspodg class label. hese saples are assued to be depedet saples. Accordg to the logstc odel, the vector of the codtoal probabltes assocated of these saples s: exp y ( ) pr (, ) p ( / ) β x + α α β y x,..., (2) + exp y ( β x + α ) he lkelhood fucto assocated wth the saples s pr ( α, β ), ad the log lkelhood fucto s: log pr (, ) f ( a y ) α β β + α (3) Where a x y R ad f s the logstc loss fucto that s: f ( z) log( + exp( z) (4) Usg (4),(3) ca be wrtte as log pr (, ) log( exp ( a y ))) α β + β + α he egatve of the log lkelhood fucto s called the (eprcal) logstc loss, ad dvdg by we obta the average logstc loss, lavg ( α, β ) log( + exp ( β a y ))) + α (6) he odel paraeters β ad α ca be detered by axu lkelhood estato fro the observed exaples, by solvg the covex optzato proble ze l avg ( α, β ) (7) he proble (7) s called the logstc regresso proble (LRP). hs LRP s a sooth covex optzato proble, ad ca be solved by a wde varety of ethods, such as gradet descet, steepest descet, Newto, quas-newto, or cougate-gradets (CG) ethods. I ths paper Newto ethod s used. Oce we fd axu lkelhood values of α ad β, that s, a soluto of (7), we ca predct the probablty of the two possble outcoes. Gve a ew features vector x R, by usg the assocated logstc regresso odel, the logstc regresso classfer s fored as: ϕ( x) sg( β x + α ) Where + z > 0 sg( z) z 0 Whch pcks the ore lkely outcoe, gve x, accordg to the logstc odel. Whe, the uber of trag saples s saller tha, the deso of the saples, drectly solvg the logstc regresso forulato (7) s ll-posed ad ay lead to overfttg. A stadard techque to avod overfttg s regularzato. l-regularzed Logstc Regresso: More recetly, l- regularzed logstc regresso has receved uch atteto as a prosg ethod for feature selecto he l-regularzed logstc regresso proble (l-regularzed LRP) ca be forulated as: ze lavg ( α, β) + λ β log( + exp( ( β a y )) + α + λ β (9) Where. deote the l-or.e., β β ad λ > 0 s a pre-specfed regularzato paraeter. he obectve fucto l-regularzed LRP (9) s covex, but ot dfferetable. he a otvato s that l-regularzed LR typcally yelds a sparse vector β, that s, β typcally has (5) (8) Iteratoal Scece Cogress Assocato 2

3 Research Joural of Matheatcal ad Statstcal Sceces ISSN Vol. 2(2), -6, February (204) es. J. Matheatcal ad Statstcal Sc. relatvely few ozero coeffcets. assocated logstc odel does ot use the th copoet of the feature vector, so sparse β correspods to a logstc odel that uses oly a few of the features, that s, copoets of the feature vector. Ideed, we ca thk of a sparse β as a selecto of the relevat features. he regularzato paraeter roughly cotrols the uber of those ozero coeffcets, wth larger typcally yeldg sparser weghted vector β. For solvg the l- regularzed LRP (9), geerc ethods for odfferetable covex probles such as the ellpsod ethod ca be used. I ths paper, the ethod of a precodtoed cougate gradet (PCG), whch kow as a best ethod, s represeted 2 ad s used ths paper. Hadlg Ucertaty Iforato: I ths secto, three techques for hadlg ucerta data LR classfcato task are proposed; these techques are average, axu lkelhood estator (MLE) ad the probabltes desty fucto (PDF). Averagg: he easy tutve ethod to deal wth ucertaty data classfcato s to copute the average of the ucertaty values for each obect ad used ths average as represeted value of those values. By usg ths ethod all the obects wll clude a sgle value. Cosequetly the data set wll be covertg to a certaty data ad hece the tradtoal logstc regresso classfer algorth ca be successfully reappled. Maxu lkelhood estator (MLE): he axu lkelhood estato (MLE) ethod 8,9 s a powerful ad a well-kow estato ethod statstcs. It estates the paraeters of a statstcal odel. Whe the axu lkelhood appled to a data set ad gve a statstcal dstrbuto, t provdes estates for the dstrbuto's paraeters. he cocept of the axu lkelhood s as follow: for a gve a rado saple x, x 2,.., x where x ~ f (, θ ), so that the lkelhood fucto s the o x desty fucto of the L( θ, x) f ( x, θ )... f ( x, θ ) By supposg that x s detcal, depedet dstrbuto (d), the axu lkelhood fucto ca be defed as: L( θ, x) f ( x, θ ) Whe β 0, the Most of lkelhood fuctos satsfy the regularty codtos, so axu lkelhood estator s the soluto of the followg equato: log L (, x,... x ) log ( θ ) θ f ( x, θ ) 0 (2) ( θ ) saple.e (0) () MLE obtaed by usg equato (2) s uforly u varace ubased estator (UMVUE) 9. he propertes of the axu-lkelhood estators are: suffcet : clude the axu forato that cluded the saple, ubased: the expected value of the MLE s ted to be the estated paraeter θ, the dstrbuto of the MLE teds to the Gaussa dstrbuto wth ea θ ad covarace atrx equal to the verse of the Fsher forato atrx ad effcecy,.e., t acheves the Craer Rao lower boud whe the saple sze teds to fty. hs eas that o asyptotcally ubased estator has lower asyptotc ea squared error tha the MLE, although the MLE s assued to be cosstet, however, t eed ot be cosstet s soe codto 23. More precsely the MLE for the Gaussa ad the ufor dstrbuto thatt we used ths paper are ubased, cosstet, suffcet ad effcet. he kerel desty estato s appled her to obta soothed values of the estated fucto, those values are used to copute the axu lkelhood estators (MLE) for the dstrbutos. Dstrbuto based: For the ucertaty odel, a feature value s represeted by PDF f, ot by a sgle value. A PDF f could be pleeted uercally by storg a set of s saple pots; x [a,b ] wth the assocated value f (x), effectvely approxatg f by a dscrete dstrbuto wth s possble values. he PDF s coputed for the Gaussa dstrbuto ths terval,.e. ( x µ, ( x) e dx, thus the 2 2σπ 2σ f b a ucertaty data s represeted the PDF value that terval usg the s saple pots. Usg ths approach for represet the ucertaty, the aout of forato avalable the ucertaty data s captured by the PDF, deftely, good classfcato odel ca be buld by usg ths rch forato. Methodology he data sets: he data sets that are used ths study are coposed of 5 uercal features. hese data sets are dowloaded fro the UCI Mache Learg repostory avalable at ftptocs.uc.edu/pub/a che learg databases; they occur frequetly the lterature of the feld. able- gves a uercal suary of the data sets. he experetal set up: he experets have bee perfored o the datasets lsted able-. hese datasets are chose because they all have uercal attrbutes. As the orgal data cota pot values wthout ucertaty, so the followg procedures are used to ake data values havg ucertaty: For the AVG ethod, the orgal pot value data are used as the expected value ad the experets are perfored o the orgal datasets. For the axu lkelhood estators (MLE) ethod, the Gaussa ad the ufor dstrbutos are used as the ucertaty odels. he orgal pot s used as the ea of the obect ( µ ) for the two ) 2 Iteratoal Scece Cogress Assocato 3

4 Research Joural of Matheatcal ad Statstcal Sceces ISSN Vol. 2(2), -6, February (204) es. J. Matheatcal ad Statstcal Sc. dstrbutos, for the Gaussa dstrbuto the stadard devato s set to be sga 0.25(ax v v)* w% ad for the ufor dstrbuto t set to be sga (axv v)/2* w%, where axv ad v are respectvely the axu ad u values aog the whole attrbutes of the feature respectvely, ad w s a percetage paraeter used to cotrol the ucertaty level of the obects. I ths experet, four values for w are used whch are %, 5%, 0% ad 20%. A saple of 00 data pots s geerated for each obect for the two dstrbutos usg the paraeters that have detered prevously. he kerel desty estato s appled to those s data pots to obta sooth cotuous estate of desty of data. he value of desty at a gve pot s estated as the su of soothed values of kerel fuctos K h( ) assocated wth each pot the data set. Each kerel fucto s assocated wth a kerel wdth h called badwdth whch deteres the level of soothg created by the fucto. For the badwdth paraeter h, a cooly used badwdth estato rule called the Slvera approxato rule 20 was used, whch suggests settg badwdth as (/ 5) wdthh.06*( sga ) *( ), where sga s the stadard devato of the obect ad s the uber of the data obects. he axu lkelhood estators(mle) s coputed fro the values that obtaed usg the kerel fucto, for the Gaussa dstrbuto the ea s the MLE, for the ufor dstrbuto the axu ordered value that obtaed after reorderg the values ascedg s represets the MLE. Usg these values of MLE, the data sets are aga recoverted to certaty data sets, effectvely; the tradtoal LR algorth s successfully appled. For the probabltes desty fucto (PDF ethod, the rage of the s saple data pots that s geerated prevously s oted; x a, b ], the PDF s [ geerated ths terval by cosderg the Gaussa ad the ufor dstrbutos usg the s saple data pots. Cosequetly, the data set wth ucertaty values s trasfored to certaty data value represeted by the PDF. Slarly, the tradtoal LR algorth s appled. he Newto ethod through l_logreg package s used to tra the LR. For all data sets 0 fold cross-valdatos ethod s used. able- Suary of the data sets Data sze Nuber of varables Page block Pa dabetes Breast cacer(orgal) heart oosphere Results ad Dscusso he results are obtaed usg l_logreg: A large-scale solver for l-regularzed logstc regresso proble package verso , avalable at ( /l_logreg/), uder atlab ( R2009a) terface, where the Newto ethod s used for estato the logstc regresso odel paraeters. Accuracy proveet: he classfcato results of applyg the axu lkelhood estators (MLE) for the data sets usg Gaussa ad the ufor dstrbutos are gve able -2 ad able -3 respectvely. For all data sets a 00 data pots (.e.s00) are used wth the varous values for w. Both the Gaussa ad the ufor dstrbutos are appled to all the data sets. able -4 shows the classfcato results of applyg the average ad the probabltes desty fucto (PDF) ethods to the data sets cosderg the Gaussa dstrbuto usg the sae varous levels of w ad 00 data pots for each data set. Because of the lted space of the paper oly the results of the accuracy are show for the classfcato ad the results of the others perforace easures are otted. he best value of the accuracy for each data set s detfed by bold fot. he relatoshp betwee the accuraces of MLE of Gaussa ad ufor dstrbutos ad the PDF of Gaussa dstrbuto usg ( w20% ) s depcted Fgure. usg the sae order able-. It shows that ostly, the accuraces values of the tree approaches of each data set le closely. Dscusso: Fro table -2, table -3 ad the results of average the frst colu table -4, t s clear that MLE buld ore accurate classfcato tha the average of the orgal data for both the Gaussa ad the ufor dstrbutos. Usg ufor dstrbuto gves better accuracy 4 out of 5 data sets. Fro table-4, applyg PDF also gves ore accurate classfcato tha average cosderg Gaussa dstrbuto. O the other had coparg the results of MLE ad PDF for Gaussa, both ethods alost gve the sae accuracy for ost of data sets. Geerally, the results show that the accuracy s proved wth creasg the level of w, especally whe applyg the ethod of the PDF, whch ay ea that ore the ucertaty, ore the accurate classfcato. Coparg MLE for Gaussa ad ufor dstrbutos wth the PDF of the Gaussa dstrbuto, (fro fgure-) all the three approaches alost gve the sae perforaces, however MLE for the ufor dstrbutos gve better accuracy 4 data sets out of 5, copared to the other two approaches. he experet has bee repeated usg varetes of uber of saple pots ragg fro 00 to 000, keepg the level of w costat; o sgfcace proveet the accuracy was observed. Defatly, f the data sets that used ths study are orgaly ucertaty data, the proveet accuracy wll be better ad uch clear. Iteratoal Scece Cogress Assocato 4

5 Research Joural of Matheatcal ad Statstcal Sceces ISSN Vol. 2(2), -6, February (204) es. J. Matheatcal ad Statstcal Sc. able-2 he accuracy of the Maxu Lkelhood Estator (MLE) for Gaussa Maxu Lkelhood Estator(MLE) Gaussa w% W5% W0% W20% Page block Pa dabetes Breast cacer heart oosphere able-3 he accuracy of the Maxu Lkelhood Estator (MLE) for ufor Maxu Lkelhood Estator(MLE) Ufor w% W5% W0% W20% Page block Pa dabetes Breast cacer heart oosphere able-4 he accuracy of the probabltes desty fucto (PDF) Orgal PDF-Gaussa Dstrbuto (average) w% W5% W0% W20% Page block Pa dabetes Breast cacer heart oosphere Fgure- he relatoshp betwee MLE ad PDF for the data sets MLE(Gaussa) MLE(ufor) PDF Cocluso I ths study, the stadard tradtoal LR classfcato algorth whch s costructed to hadle the data wth sgle data pot s exteded to hadle data whose obects are uercal wth ucertaty. he MLE of the Gaussa ad the ufor dstrbutos are coputed ad used to covert the data to ucertaty. Accordg to the experetal results of the study, Iteratoal Scece Cogress Assocato 5

6 Research Joural of Matheatcal ad Statstcal Sceces ISSN Vol. 2(2), -6, February (204) es. J. Matheatcal ad Statstcal Sc. hadlg data ucertaty usg the MLE results a proveet the accuracy for both dstrbutos I the sae way, hadlg data ucertaty wth PDF results sgfcat proveet accuracy for LR classfcato. Fally t ca be cocluded that LR ca successfully hadle the case whe the data s ucerta. Aalyzg ucertaty data ca produce classfers wth hgher accuracy as copared to usg tradtoal LR algorth whch uses average as a represetato of ucertaty, therefore t s recoeded that the data should be collected ad stored as ucertaty data. Ackowledgeets hs work was supported by a grat fro Hebe Uversty, Baodg, Hebe, P.R.Cha. I lke to thak the PhD studets of the departets of coputer Sceces ad atheatcs for ther ecourageet, useful dscussos, ad terest. Note: ths work s copleted Hebe Uversty durg My PhD study perod. Refereces. Hoser D.W. ad Leeshow S., Appled logstc regresso, 2d ed. Wley seres probablty ad statstcs, Wley, Ic, New York, (2000) 2. Meard S., Appled logstc regresso aalyss, 2d ed.sage publcatos Ic, (2002) 3. Neter J., Kuter M.H., Nachtshe C.J. ad Wassera W., Appled lear statstcal odels, 4th ed. Irw, Chcago, (996) 4. hoas P. Rya, Moder Regresso Methods. 2d ed. Wley-Iter scece New York, NY, USA, (2008) 5. Brzezsk J.R. ad Kafl G.J, Logstc regresso odelg for cotext-based classfcato. Database ad Expert Systes Applcatos, 999. Proceedgs. eth Iteratoal Workshop o, , 999do: 0.09/DEXA ,(999) 6. Musa A.B., Coparatve study o classfcato perforace betwee support vector ache ad logstc regresso, It J Mach Lear Cyber, 4(), 3-24 (203) 7. Aggarwal C.C., O Desty Based rasfors for ucerta Data Mg. I ICDE Coferece Proceedgs, (2007) 8. Corode G. ad McGregor A., Approxato algorths for clusterg ucerta data, I Prcple of Data base Syste (PODS), M. Lezer ad D. Lebo, Eds. ACM, (2008) 9. Sgh S., Mayfeld C., Prabhakar S., Shah R., ad Habrusc S., Idexg categorcal data wth ucertaty, I ICDE, , (2007) 0. J. Ge, Y. Xa ad C. Nadugodage, UNN: A eural etwork for ucerta data classcato," PAKDD, (200). C.C. Aggarwal, A Survey of Ucerta Data Algorths ad Applcatos. I IEEE rasactos o Kowledge ad Data Egeerg, 2(5), (2009) 2. C.C. Aggarwal, O Desty Based rasfors for ucerta Data Mg. I ICDE Coferece Proceedgs, (2007) 3. sag S., Kao B., Yp K., Ho W. ad Lee S, Decso trees for ucerta data. I: Iteratoal Coferece o Data Egeerg (ICDE), (2009) 4. J. B ad. Zhag, Support vector classcato wth put data ucertaty," Advaces Neural Iforato Processg Systes (NIPS), 6-68 (2004) 5. B. Q, Y. Xa, ad F. L, DU, A decso tree for ucerta data," PAKDD, 4-5 (2009) 6. Jagtao Re, Sau Da Lee, Xalu Che, Be Kao, Reyold Cheg ad Davd Cheug, Nave Bayes Classfcato of Ucerta Data, Nth IEEE Iteratoal Coferece o Data Mg, (2009) 7. B. Q, Y. Xa ad F. L, A Bayesa classer for ucerta data, ACM Syposu o Appled Coputg, (200) 8. Elaso S., Maxu Lkelhood Estato: Model ad Practce (993) 9. Mood, Graybll, Itroducto to the heory of Statstcs, 3rd ed. McGraw Hll, New York, USA, , (974) 20. Slvera B.W., Desty estato for statstcs ad data aalyss, Lodo; chapa ad hall (986) 2. K S.J., Koh K., Lustg M., Boyd S. ad Gorevsky D., A teror-pot ethod for largescale l-regularzed least squares. IEEE Joural o Selected opcs Sgal Processg, (4), (2007) 22. Musa A.B., Coparso of l-regularzo, PCA, KPCA ad ICA for Desoalty Reducto Logstc Regresso, It J Mach Lear Cyber. do: 0.007_s , (203) 23. E.L. Leha George Casella, heory of Pot Estato, Secod Edto, Sprger, Sprger-Verlag New York, Ic, 83-4, (998) 24. Koh K., K S.J., Boyd S., l_logreg: A large-scale solver for l-regularzed logstc regresso probles Avalable at (2009) Iteratoal Scece Cogress Assocato 6