ROC ANALYSIS FOR PREDICTIONS MADE BY PROBABILISTIC CLASSIFIERS

Transcription

1 Proceedngs of the Fourth Internatonal Conference on Machne earnng and Cybernetcs, Guangzhou, 8- August 005 ROC ANAYSIS FOR PREDICTIONS MADE BY PROBABIISTIC CASSIFIERS ZENG-CHANG QIN Artfcal Intellgence Grou, Deartent of Engneerng Matheatcs, Unversty of Brstol, BS8 TR, UK E-MAI: Abstract: Recever Oeratng Characterstcs (ROC analyss was orgnated fro sgnal detecton theory and has been ntroduced to achne learnng county n recent years to evaluate the algorth erforance under recse envronent. ROC grahs have becoe ncreasngly oular n achne learnng, because they offer a ore robust fraewor for evaluatng classfer erforance than the tradtonal accuracy easure. In ths aer, we nvestgate the relaton between a robablstc classfer and ts corresondng redctor n a vew of ROC analyss. A ethod of generatng ROC curves for redcton (or regresson robles s roosed and soe roertes of ROC curves for redcton are dscussed wth eales. Keywords: ROC Analyss; ; raner; robablstc classfer. Introducton Tradtonally, the an crteron for evaluatng the erforance of a classfer s accuracy (ercentage of test eales that are correctly classfed or error (ercentage of sclassfed eales. However, n any stuatons, not every sclassfcaton has the sae consequences when sclassfcaton costs have to be taen nto account. Recent study shows that the accuracy of a classfer s also nfluenced by the class dstrbuton [7]. Recever Oeratng Characterstcs (ROC analyss, whch was orgnated fro sgnal detecton theory, has been ntroduced to evaluate achne learnng algorths [,, 6, 7] and t has becoe ncreasngly oular n achne learnng research. In addton to beng a generally useful erforance grahng ethod, they have roertes that ae the esecally useful for doans wth sewed class dstrbuton and unequal classfcaton error costs. For eale, gven a classfer whch has accuracy of 80%. The accuracy doesn t ae sense wthout nowng the class dstrbuton: If the database conssts of 90% ostve and 0% negatve eales. We can do better sly by classfyng all the data as ostve that wll gve 90% accuracy. So, ROC analyss s not ust about cost-senstve learnng, t consders the relatve ortance of negatve vs. ostve eales. Ths relatve ortance can be reresented by a sew rato by consderng both costs and class dstrbuton [3]. Many classfers not only gve dscreet redcted classes but also the estates of class ebersh robabltes (e.g., Nave Bayes. The forer are referred to as dscreet classfers and the latter as robablstc classfers or raners, because the ebersh robabltes can be used to ran nstances fro ost to least lely ostve. By settng a threshold, a raners can act as a classfer. Area under the curve ( of ROC s used to easure the qualty of ranng for a robablstc classfer [4, 8]. ng et al. roved that s statstcally consstent and ore dscrnatng than the accuracy easure [5]. So, t s far to use rather than accuracy to evaluate a learnng algorth. Currently, all the ROC analyss research are for classfcaton robles. However, n any real-world alcatons, data rangng fro fnancal analyss to weather forecastng are redcton robles. We are wonderng f we can etend to the ROC analyss to redctons? Ths s the otvaton of ths research. Here n ths aer, soe ntal nvestgatons s resented where we only consder the redctors based on defuzzfcaton wth two fuzzy labels on robablstc classfers. Ths aer s organzed as follows: we frst ntroduce the bascs of ROC analyss for classfcaton n secton. In secton 3, the ROC analyss s etended to redcton robles and the ethod for lottng ROC curves s roosed. In the secton 4, the ethod of calculatng values s roosed and soe secal roertes of ROC curves for redcton are dscussed wth eales.. ROC Analyss for Classfcaton Tradtonally, accuracy and error are wdely used easures for evaluatng erforance of a classfer. Usng accuracy as a erforance easure assues that the error /05/$ IEEE 39

2 Proceedngs of the Fourth Internatonal Conference on Machne earnng and Cybernetcs, Guangzhou, 8- August 005 costs are equal and the class dstrbuton s balanced. However, ths s not realstc f we consder robles such as edcal dagnoss or fraud detecton. We begn by consderng classfcaton robles usng only two classes (or bnary classfcaton roble. Usually, the nstances are dvded accordng to the followng contngency table or confuson atr: Fgure. Confuson atr for a bnary (.e., ostve and negatve classfcaton roble. If the nuber of ostves are negatves are denoted by P and N, resectvely, the redcted ostves and negatves are denoted by?p and N?, then, the classfcaton accuracy s defned as: TP TN Accuracy ( P N ROC analyss decooses erforance nto true and false ostve rates defned as follows: the true ostve rate (TPR of a classfer s: TPR TP /P and the false ostve rate (FPR of a classfer s: FPR FP /N. If we lot FPR on the X as and TPR on the Y as. A sngle classfcaton s then reresented by a ont n ths D coordnate sace whch s referred to as ROC sace. In the ROC sace, the uer left ont (0, s ost wanted because t gves 00% ercent of true ostves and zero false ostves. It can be called as ROC Heaven and, corresondngly, the ont (, 0 s the least wanted ont that can be called ROC Hell [3]. The dagonal lne reresents a rando classfer whch always gves 50% of true ostve rate and 50% false ostve rate. Each dscreet classfer can be resented by a sngle ont accordng to ts TPR and FPR n the ROC sace. Dfferent ROC rofles wll be ore or less desrable under dfferent class dstrbutons and dfferent error cost functons. More detals about basc ROC sace roertes are avalable n [3]. Consder a robablstc classfer wth two classes and -. We can sort the nstances accordng to the robabltes of belongng to class. Dfferent classfcaton Fgure. ROC curves for the classfers CS and CS. results wll be gven accordng to the varyng threshold T based on: { } f : P( T : { } Otherwse where we norally set T 0.5 when we calculate accuracy for a robablty estaton odel. If we vary the value of T through [0, ], t wll result a contnuous curve n ROC sace whch s referred to as a ROC curve. In another words, A classfer results n a ROC curve, whch aggregates ts behavor for all ossble decson thresholds. The qualty of the classfer can be easured by the area under the curve of ROC (, whch easures how well the classfer searates the two classes wthout reference to a decson threshold. In other words, reresents the qualty of ranng of eales by ths classfer. Gven nstances, there are only ossble thresholds. A ractcal ethod s as follows: ( ran test nstances on decreasng ebersh scores. ( Startng n (0, 0, f the net nstance n the ranng lst s ostve then ove /P u, f t s negatve then ove /N to the rght. Gven the two classfers CS and CS n table, the ROC curves drawn by the above ethod are shown n fgure. Accordng to Hand and Tll [4], the value for a bnary classfcaton roble wth two classes {, } can be calculated by: P r P(P / ( PN The otal threshold for a robablstc classfer deends on the class dstrbuton and sclassfcaton costs. The ebersh scores are not calbrated estate of robabltes n ost cases [9]. Therefore, assgnng T0.5 (e.g., for Naïve Bayes classfer s a sleadng n any achne learnng lteratures. 30

3 Proceedngs of the Fourth Internatonal Conference on Machne earnng and Cybernetcs, Guangzhou, 8- August 005 Table. Two classfers (CS and CS wth the sae accuracy but dfferent values. Ths table s nsred by a slar table n [5]. Eales CS r for CS CS r for CS where P and N are the nuber of ostve and negatve eales, resectvely. r s the ran of th ostve eale n the ranng lst accordng to the robabltes of the class. For eale, the values for classfer and lsted n table are: (CS (CS ( (5 / 5 5 ( (5 / We ay notce that both classfer and have the sae accuracy 80% (8 of 0 eales are correctly classfed and thus they are equally good n accuracy. However, the ntuton tells us that classfer s better than classfer snce classfer gves a better overall ranng. Ths can be seen fro easure but not the accuracy easure. ng et al. [5] atheatcally roved that the easure s consstent and ore dscrnatng than the accuracy easure. The ethod for calculatng for ult-class robles s gven n [4], however, n ths aer, only two-class robles are consdered. Fgure 3. Dfferent degrees of overlang between two fuzzy labels that are used as class labels. 3. ROC Analyss for Predcton target attrbute t s nuerc. For each nstance (a ultdensonal vector, t also wrtten as n equatons to redct ts target value t (.e. t. Suose we dscretze the outut unverse wth fuzzy sets: F,..., F. We can consder each fuzzy set as a sngle class label that has weghts denoted by and each nstance can be aed to a reresentat n as follows: o < F : ξ,..., F : ξ > where, ξ We then can use an arbtrary robablstc classfer to obtan a seres of condtonal robabltes on target fuzzy sets gven a test nstance : P(F,, P(F. The estate of t, denoted?t to be the eected value: t ˆ t( t dt (3 where: and Ω ( t (t F Ω t ( t F P( F M t (F M t (F dt where M (F s the ebersh of belongng to fuzzy set of labels F. So that we can obtan: t ˆ P( F E( t (6 where: F tm ( F dt (4 (5 t Ωt E ( t F (7 M t ( F dt Ωt where the rocess of calculatng E( t F s also called defuzzfcaton n soe other lteratures. Fro above we can see that, by fuzzfyng the contnuoustarget attrbute t nto ntervals that could be consdered as class labels, any robablstc classfers can be etended to a redcton odel. However, we need to notce that the class labels not dscreet but overlaed each other and there are any dfferent degrees of overlang. For eale, fgure 3 shows four dfferent ossble overlang. In ths aer, we only consder the slest case that, where one fuzzy label s reresented by and the other by. In the followng aer, unless otherwse stated, we wll use the fuzzy labels wth 50% overlang (fgure 3-d, t satsfes: : P( P( Consder a redcton roble that the outut sace or 3

4 Proceedngs of the Fourth Internatonal Conference on Machne earnng and Cybernetcs, Guangzhou, 8- August 005 Fgure 4. An llustraton of drawng ROC curve and calculaton by addng a new ont. The basc dfference between such redctors and noral robablstc classfers s that the class labels are overlaed each other. For a artcular nstance, t has orgnal ebersh robabltes of ostves fro fuzzy dscretzaton P( and redcted class robabltes?p ( fro classfers In the followng contet, unless otherwse stated, we wll focus on the ebersh robabltes of ostves and we wrte?p ( as and?p ( as?. For eale, gven the orgnal ebersh scores and redcted scores, how can we draw the ROC curve? A sle and ractcal ethod based on dscreet class labels s roosed as follows: Gven a test set of sze, ran the nstances on decreasng redcted ebersh scores of the ostve class?, where {,,,}. TP 0 0, FP 0 0 for :, Do: TP TP /n, FP FP ( /n Startng fro (0, 0, for :, draw the curve by onng (FP, TP and (FP, TP successvely. Fgure 5. ROC curves for a erfect-ranng redctor, a rando redctor, and two redctors obtaned by erfect ranng redctor corruted by dfferent levels of nose. where n s the su of ostve arts on all eales, t s obtaned by: n Slarly, we can obtan: ( ( n ( ( n Fgure 5 shows a set ROC curves on a real-world redcton roble: the ared curve s a erfect ranng, whch eans that gven a raned lst n a decreasng anner based on? (.e. ˆ... ˆ, the relaton... holds. The curves ared wth α 0. reresents a erfect ranng redctor corruted by a unfor dstrbuted nose n the range of [0, 0.], denoted by U[0, 0.]. So that the redcted robabltes are: ˆ ε ε U[ 0, α] The rando classfer s a rando guess that follows: ˆ U[0, ] As we can see fro those curves, they ehbt slar roertes as wth dscreet labels, the only dfference s that the au value for redcton s not. Ths wll be dscussed n detals n the net secton. (8 4. Value for Predcton Fgure 4 gves an llustraton of drawng ROC curve for such redcton robles. We need to notce that the otal ont s not (,0 for redctons (.e. value s always less than. The reason for ths s because we use 3

5 Proceedngs of the Fourth Internatonal Conference on Machne earnng and Cybernetcs, Guangzhou, 8- August 005 overlang fuzzy labels. ROC analyss reflects the searaton of ostve and negatve eales by a classfer. In ths case, no atter how good a classfer s, t stll cannot coletely searate the ostves and negatves because they are overlaed to each other. The dfferent overlang degrees wll result n dfferent au values. Fgure 6 dects the ROC curves wth au values on the fuzzy labels wth dfferent overlang degrees shown n fgure 3. In the legend, the values that are calculated by the ethod that wll be dscussed n the followng art of ths secton. Consder the ranng lst on decreasng ebersh scores n the way we draw the ROC curves. The frst eale of the ranng lst s the one wth the hghest redcted score wth orgnal score of. By addng ths eale to the ROC sace, the area under the ROC curve s a trangle wth sde lengths of ( /n and /n, resectvely (see fgure 4. So that ( (9 n By addng a new eale wth score etended and a new area n traezodal the current becoes: ( ( ( n Slarly, by addng the thrd ont: ( ( 3 ( n ( ( 3 ( 3 By successvely addng the th eale (, we can obtan: ( (0 n Equaton 0 can be rearranged and the value for redcton on a test set wth eales s: ( C ( n where, C ( ( Fgure 6. ROC curve wth au values gven fuzzy labels on dfferent degrees of overlang n fgure 3. Consder the equaton, the frst ter ( s nvarant to dfferent ranngs. Now we only consder the ter C to nvestgate the relaton between value and eale ranng. Ter C can be searated nto two ters A and B ( so that C A B. Suose we have the followng ranng of eales accordng to the B ters: R :, f we swa the ostons of these two eales to: R :, Suose, such swang s referred to as bad swang, because R s ore desrable than R for a better ranng. The swang wll result n a change n values, f we defne: D( R D( R A B ( T ( A B ( ( T T ( ( T where T and accordng to equaton we obtan: C ( R D( R C ( R D( R ( ( 33

6 Proceedngs of the Fourth Internatonal Conference on Machne earnng and Cybernetcs, Guangzhou, 8- August 005 Table. vales wth dfferent ranngs by echangng eales fro the erfect ranng, where n -. n n n 3 n 4 n 5 n R R R R The latter ters for C(R and C(R have dentcal values. Therefore, accordng to equaton, we can calculate the change of values by echange these two eales as follows: ( R ( R [ C( R C( R ] n [ D( R ( ] D R 0 n n where the equalty holds when. If we suose, such a swang then becoes a good swang, the values wll be ncreased by the sae value. For eale, we start fro a erfect ranng R shown n table 4. We can obtan: n n and n 5 3. By swang 0.8 and 0.6, we obtan the change n as follows: ( 0.6 ( n 3 So that the new value for the rearranged lst s ( R ( R Based on the new ranng lst then can obtan a new ranng lst n n R, swa 0.8 and 0.4, we R ( 0.4 ( ( R ( R , such that: Slarly, we can obtan another bad ranng lst such swang (.e. bad swang and the values wll ee decreasng. 5. Conclusons In ths aer, we etended ROC analyss that s coonly used n classfcaton to redcton. A ethod of drawng ROC curves for redcton s roosed and soe of ortant roertes of such ROC are dscussed. By ntroducng the ethod for calculatng values for redcton, we also nvestgate the relaton between the R 3 by values and the ranng of eales. In artcular, a quanttatve analyss of value by swang two neghborng nstances s gven. However, n ths aer, we only consder a very sle case that the redctor s obtaned by defuzzfcaton of robablstc classfers. The future research focus on etendng ths fraewor to ult-classes (ore than fuzzy labels and study the relaton between and soe other easures used n redcton such as ean squared error and average error. Acnowledgeents The author thans Prof. Peter Flach for useful dscussons on ROC analyss that nsred the research resented n ths aer. References [] T. Fawcett, ROC grahs: notes and ractcal consderatons for data nng researchers, HP Techncal Reort HP-003-4, HP aboratores, 003. [] P. A. Flach, The geoetry of ROC sace: understandng achne learnng etrcs through ROC soetrcs, Proceedngs of the ICM-04, 004. [3] P. A. Flach, The any faces of ROC analyss n achne learnng, ICM-004 Tutoral, Notes avalable at:htt:// [4] D. Hand and R. J. Tll, A sle generalsaton of the area under the ROC curve for ultle class classfcaton robles, Machne earnng, Vol. 45, 7-86, 00. [5] C. X. ng, J. Huang and H. Zhang, : a statstcally consstent and ore dscrnatng easure than accuracy, Proceedngs of IJCAI-03, 003. [6] F. Provost, T. Fawcett, and R. Kohav, The case aganst accuracy estaton for coarng nducton algorths, n J. Shavl, edtor, Proceedngs of ICM-98, , 998. [7] F. Provost and T. Fawcett, Robust classfcaton for recse envronents, Machne earnng. Vol. 4, 03-3, 00. [8] F. Provost and P. Dongos, Tree nducton for robablty-based ranng,machne earnng. 5, 99-5, 003. [9] [9] B. Zadrozny and C. Elan, Obtanng calbrated robablty estates fro decson trees and nave Bayesan classfers, Proceedngs of ICM-0,