THREE-WAY ROC ANALYSIS USING SAS SOFTWARE

ACTA UNIVERSITATIS AGRICULTURAE ET SILVICULTURAE MENDELIANAE BRUNENSIS Volum LXI 54 Numbr 7, 03 http://d.doi.org/0.8/actau0360769 THREE-WAY ROC ANALYSIS USING SAS SOFTWARE Juraj Kapasý, Marti Řzáč Rcivd: August 8, 03 Abstract KAPASNÝ JURAJ, ŘEZÁČ MARTIN: Thr-way ROC aalysis usig SAS Softwar. Acta Uivrsitatis Agricultura t Silvicultura Mdliaa Brusis, 03, LXI, No. 7, pp. 69 75 Th most commoly usd masur of modl accuracy i mdici with thr catgoris of targt variabl is th volum udr ROC surfac (VUS), which is th tsio of th ara udr curv (AUC) for biary modls (L ad Lili, 03). This papr dals primarily with usag of th multiomial logistic rgrssio ad th thr way ROC aalysis i th fiacial sctor, spcially i th crdit risk maagmt. Morovr, SAS systm is vry oft usd softwar i th fiacial sctor. Thrfor this papr is focusd o ways of doig thr way ROC aalysis i this statistical softwar, i particular o stimatig th VUS. W propos a stimat of th VUS basd o th cofusio matri, which is compard to stimats basd o Ma-Whity statistic ad o mpirical distributio fuctios. W dvlopd thr SAS macros basd o ths approachs for computig th VUS. Furthr- mor, w dvlopd som logistic modls for thr-valu targt variabl basd o th Loss Giv Dfault (LGD). This was do o ral fiacial data. Rsults obtaid by th SAS macros o ths modls ar prstd a discussd i th papr. multiomial logistic rgrssio, LGD, Thr-way ROC aalysis, ROC surfac, VUS, SAS Softwar INTRODUCTION I mdici, thr ar oft usd modls with thr catgoris targt variabl ad thr way ROC aalysis with VUS is prov to b o of th bst masurmt tchiqus for ths modls. It is quit diffrt i fiacial mathmatics or bakig. May baks us classic biary logistic rgrssio for th prdictio of th probability of dfault so thy ar abl to us ROC curvs ad also AUC (s Řzáč ad Řzáč, 0). Howvr, som baks hav rctly startd usig also multiomial logistic rgrssio ad thrfor thy also had to start usig VUS ad thr-way ROC aalysis as th masur of quality (prdictiv powr) for thirs modls. This papr racts to ths vts ad prsts som ways of ruig thr way ROC aalysis i SAS softwar, that is softwar vry commoly usd i fiacial sctor, spcially i bakig. Scurfild (996) ad Mossma (999) proposd ad discussd th cocpt of th VUS. Frri t al. (003) discussd act computatio of th VUS compard with its approimatios. Ladgrb ad Dui (006) did som primts to fid th lowr boud of th VUS ad to ivstigat itgratio approach to stimatig th VUS. Wagma at al. (008) gav thortical ad primtal vidc of th advatags of th VUS compard to rror rat, ma absolut rror ad othr rakigbasd prformac masurs. Li ad Fi (008) dvlopd a mthod basd o stimatig th class probabilitis. Wa (0) proposd a mpirical liklihood cofidc itrval for th VUS. A typical ara of usag of th multiomial logistic rgrssio i bakig is th crdit risk maagmt. Withi this ara, a vry actual issu is th stimatio of th LGD. Wh all dfaultd clits ar dividd ito thr catgoris, first o aroud LGD = 0, scod catgory with LGD ad third o with LGD highr tha, th multiomial logistic rgrssio ca b usd. Th papr is dvotd to this topic. W dvlopd som logistic modls for thr-valu targt variabl basd o th LGD. Furthrmor, w dvlopd thr SAS macros 69

70 Juraj Kapasý, Marti Řzáč basd o diffrt approachs for computig th VUS. This was do o ral fiacial data. Rsults obtaid by th SAS macros o ths modls ar prstd ad discussd i th papr. Thory of multiomial logistic rgrssio Th thory ad all formulas i this sctio wr rtrivd from Hosmr ad Lmshow (000). Th mai diffrc btw multiomial logistic rgrssio ad classic biary logistic rgrssio is that i multiomial rgrssio with thr catgoris of targt variabl thr ar two logit quatios. Lt b th vctor of valus of idpdt variabls, β th vctor of paramtrs for vry prdictor plus itrcpt ad Y th targt variabl with valus 0, or th logit quatios ca b writt as PY ( ) g( ) l 0pp ' PY ( 0 ) () ad PY ( ) g( ) l 0pp ' PY ( 0 ) By solvig quatios () ad () togthr with P(Y = 0 ) + P(Y = ) + P(Y = ) =, w gt th solutios for probabilitis for ach catgory, which qual to P Y 0 g g () (3) ( ) ( ) usig logarithm o l(β), w gt a log liklihood fuctio i ( i i i) ( g( i) g( i) ) i L( ) y g ) y g ( l. (6) W gt our liklihood quatios as partial drivatios of fuctio (6) with rspct to all (p + ) paramtrs. To simplify ths quatios w us dotatio π ji = π j ( i ) = P(Y = j i ). Fial liklihood quatios ar i form L( ) jk ( y ki í ji ) ji for j =, ad k = 0,,,, p ad 0i = for vry obsrvatio i. W gt maimum liklihood stimatio βˆ by sttig quatios (7) qual to 0 ad solvig thm. To solv ths quatios th umrical mthod has to b mployd. It ca b foud i much statistical softwar. From classic ROC to Thr-way ROC aalysis I classic biary cas ROC aalysis is basd o cofusio matri of modl, corrspodig with that i th Tab. I. I: Cofusio matri Prdictd catgory Currt catgory (7) T F T Tru Positiv (TP) Fals Positivs (FP) F Tru Ngativs (FN) Tru Ngativs (TN) g ( ) PY ( ) (4) g( ) g( ) g ( ) PY (5) g( ) g( ) It ca b obsrvd that vry probability is fuctio of ad has (p + ) paramtrs β = (β, β ). whr p is th umbr of idpdt prdictors. Maimum liklihood stimatio is th usd to stimat valus of paramtrs β. For cratig a liklihood fuctio 3 dummy variabls y j ar cratd accordig to followig pattr: If Y = 0 th y 0 =, y = 0 ad y = 0; if Y = th y 0 = 0, y = ad y = 0; ad if Y = th y 0 = 0, y = 0 ad y =. Th liklihood fuctio ca b cratd as y0 ( ) 0( ) i y ( ) i y l i i ( i) i, í whr π j ( i ) quals to P(Y = j i ) ad y ji ar dummy variabls mtiod bfor for obsrvatio i. Usig th fact that yji for vry obsrvatio ad j0 Thigs ar bit mor complicatd i multiomial cas. Similarly it is also basd o cofusio matri, whos dimsio, howvr, is ow 3 3, thrfor thr is o quivalt of Fals Positiv Rat or Fals Ngativ Rat, but thr ar 6 diffrt kids of diagostic rrors i thr way logistic modl. I biary cas tru positiv rat ca b computd as ad fals positiv rat as TPR TP TP FN FPR FP FP TN ad wh TPR vs. FPR is plottd, w gt rvrsd ROC curv, which is startig at uppr lft corr ad bdig dow to rach lowr right corr. This approach ca b tdd to thr dimsios ad th thr positivs rats ca b plottd o orthogoal as. If obsrvd poits ar th coctd with li sgmts, th rsult is ROC surfac. Fig. provids a ampl of ROC surfac for o of ths multiomial logistic modls.

Thr-way ROC aalysis usig SAS Softwar 7 : ROC surfac for modl with thr catgoris dpdt variabl : ROC surfac for uslss tst This pictur was rtrivd from work of L ad Lili (03) ad symbols i o ach ais ar quivalt to tru positiv rat for catgory i. O ca s i th Fig., that ROC surfac has thr paks (, 0, 0), (0,, 0) ad (0, 0, ). For th first o holds tru, P(Y = 0targt = 0) =, P(Y = targt = ) = 0 ad P(Y = targt = ) = 0, whr targt is th targt variabl with thr catgoris. Similar holds tru for othr two poits. Ths poits ar part of vry ROC surfac for ach modl. I multiomial cas volum udr ROC surfac (VUS), which is a tsio of AUC for classic biary modl, ca b computd ad it is qual to probability that thr radomly slctd obsrvatios, o from ach class, ar assigd corrct classs. Cosqutly, VUS for uslss tst has to b smallr tha AUC ad it quals /6 0.667. If o obsrvatio from first catgory is slctd, thr is /3 chac of placig it i a right bi. If this is do corrctly, thr is / chac of placig obsrvatio from scod catgory to th right bi. Ths probabilitis ca b multiplid: /3 / = /6 ad that quals to volum udr ROC surfac for uslss tst, which is cratd by coctig poits (,0,0), (0,,0) ad (0,0,) ad ca b s i Fig.. This pictur was rtrivd from work of L ad Lili (03) ad symbols i o ach ais ar agai quivalt to tru positiv rat for catgory i.

7 Juraj Kapasý, Marti Řzáč Applicatio of multiomial logistic rgrssio I this sctio a logistic rgrssio modl with dpdt variabl with thr catgoris is cratd usig SAS softwar ad ral fiacial data. Data To crat th modl th data from o of th biggst baks i Czch Rpublic wr usd. Ths data cosist of i variabls which ar id variabl, lgd ad sv prdictors (idpdt variabls) ad 8050 obsrvatios. Th variabl lgd cotais a valu of LGD for ach customr ad a w variabl LGD_targt is costructd whr vry obsrvatio is assigd a class of our targt variabl 0,, or, hc multiomial logistic rgrssio ca b usd to prdict th catgory of LGD for w customrs if thy dfault. Aftr computig tst for all variabls it was foud out, that thy all hav statistically sigificat rlatioship with th targt variabl, so it has to b carfully dcidd which to us i our modl. Bfor th data is usd for th modl, it has to b dividd ito traiig part ad validatio part. Oc this is do, th similarity of ths tabls ca b masurd by PSI (s Řzáč, 009) which ca b foud i Tab. II. I gral, PSI valu udr 0. is cosidrd as vry good for two similar tabls. Idtical tabl ca b cratd for vry variabl i th datast. Trasformatio of Variabls Thr wr may missig valus i th datast, so first thig to do was to rplac thm with valu NA i cas of catgorical variabls ad with mdia i cas of umrical itrval variabl. Liar rlatioship btw prdictors ad dpdt variabl wr lookd for, so it ca b said that if a clit movs from o catgory to aothr, his LGD will rais. To fid this rlatio quivalt of WOEs (wights of vidc) wr usd. A w variabl bad was cratd followig th pattr: bad = 0 if LGD is lss tha 0.8 ad bad = if LGD is mor tha 0.8. Th WOEs for idpdt prdictors with rspct to w variabl bad wr computd ad all idpdt variabls wr ordrd by thir WOEs to crat 7 w prdictors. Modllig I this sctio th data from prvious sctio wr usd ad a multiomial logistic rgrssio modl with targt variabl with thr catgoris was built. Thr multiomial logistic rgrssio modls wr cratd, first o (Modl ) with all idpdt variabls as prdictors, scod without th prdictors which p valu was biggr tha 0.05 (this mas that just statistically sigificat prdictors wr usd) i at last o of partial logistic rgrssio ad third without th prdictors which p valu was biggr tha 0.05 i both partial logistic rgrssios. Th rsults ca b foud i Tab. III. I th followig sctio thy ar amid to dtrmi, which o is th bst usig thr way ROC aalysis. β cofficits show i Tab. III wr usd for scorig of validatio data ad modls wr compard by valus of VUS for traiig ad validatio data i sctio dotd rsults. A importat fatur of th multiomial logistic modl is that it stimats modls, whr j is umbr of catgoris of targt variabl. I this cas th catgory zro was tratd as th rfrt o ad modls for catgory o rlativ to zro ad catgory two rlativ to zro wr stimatd. Accordig to Brui (0) th stadard itrprtatio of multiomial logistic rgrssio is that for a uit chag i th prdictor variabl, th logit of outcom m rlativ to rfrt group, i this cas logit of outcom ad rlativ to 0, is pctd to chag by its rspctiv paramtr stimat. III: β cofficits for computd modls # # #3 Variabl targt Itrcpt Itrcpt Prdictor4_WOE Prdictor4_WOE Prdictor_WOE Prdictor_WOE Prdictor5_WOE Prdictor5_WOE Prdictor_WOE Prdictor_WOE Prdictor3_WOE Prdictor3_WOE Prdictor6_WOE Prdictor6_WOE Prdictor7_WOE Prdictor7_WOE.09.886 0.59 0.804 0.0700 0.363.334 0.4089 0.0808 0.0 0.067 0.697 0.069 0.865 0.033 0.097.0868.53 0.35 0.3008 0.074 0.4.89 0.703 0.04 0.00 0.8655.4885 0.60 0.80 0.075 0.34.337 0.435 0.9 0.636 0.0707 0.87 0.0335 0.096 II: PSI id for variabl LGD_targt trai data validatio data LGD_targt # of clits ratio # of clits ratio psi 0 000 0.355 830 0.344 0.00036 668 0.96 75 0.300 0.00006 967 0.349 860 0.356 0.0004 total 5635 45 0.00056

Thr-way ROC aalysis usig SAS Softwar 73 Computig VUS i SAS Softwar I this sctio som ways for stimatig VUS ar prstd. Ths mthods wr applid o modls from Sctio 4 usig SAS. Lt S, S ad S 3 dot th scors rsultig from logistic modl for obsrvatios from catgoris o, two ad thr rspctivly. Thy idicat th probability that obsrvatio is assigd to appropriat catgory, thrfor if thr was a idal modl, S, S ad S 3 would always satisfy th coditio S S S 3. Paramtric approach Accordig to L ad Lili (03) th simpl paramtric approach is to assum that S i N (μ i, i ), i =,, 3. Th VUS udr ormality assumptio ca b prssd as VUS ( as b) ( cs d) ( s) ds, (8) whr b, c, 3 d, ϕ(.) a, 3 is stadard ormal distributio fuctio ad ϕ(.) is stadard ormal dsity fuctio. Th maimum liklihood stimat of VUS ca b obtaid by substitutig sampl μˆi ad sampl stadard dviatios σˆi ito (8). For som data this approach could achiv a vry prcis stimatio of VUS. Howvr, as ca b s i Fig. 3, i our data ormal assumptios ar ot satisfid so this approach caot b usd. Similar plots could b show for S ad S 3. Nvrthlss, du to th fact that VUS is ivariat udr mootoic trasformatio, Bo Co typ 3 trasformatio ca b applid to th data ad th th ormality-basd mthod for stimatio of th VUS ca b usd o th trasformd data (s L ad Lili, 03). Sic w obtaid vry poor rsults by this approach, w focusd o oparamtric approachs. Noparamtric approachs Ma Whity U statistic I biary cas AUC ca b computd as Wilcoo Ma Whity statistic (s Corts ad Mohri, 003) giv by AUC U I, aibj i j whr a i ar rsults assigd by logistic rgrssio to positiv obsrvatios ad b j ar rsults assigd to gativ os, +, ar rspctivly sampl sizs for positiv ad gativ obsrvatios ad I is idicator fuctio. This is qual to probability that two radomly chos obsrvatios, o from ach class, ar assigd with corrct class. Th first approach of stimatig VUS is basd o th fact that, lik i biary cas, VUS ca b stimatd by Ma Whity U statistic (s L ad Lili, 03). Lt th sampl sizs for catgoris o, two ad thr b, ad 3, rspctivly. Th th stimatio of th VUS is giv by 3 VUS U I( Si Sj S3 k ) (9) i j k 3: Q-Q plot for S

74 Juraj Kapasý, Marti Řzáč Approach basd o th cofusio matri Th scod approach is basd o th cofusio matri, which is i cas of thr catgoris of th targt variabl i form prstd i th Tab. IV. IV: Cofusio matri for modl with targt variabl with thr catgoris Prdictd catgory Supposdly th clit from catgory o is first to start with, th thr is TP TPF F3 chac that h will b assigd a corrct class. That lavs th pair from catgoris two ad thr. Clit from catgory two is t to b cotiud with, so thr is TP TP F 3 Currt catgory 3 TP F F3 F TP F3 3 F3 F3 TP3 chac that h will b assigd a corrct class ad fially if this is do proprly, th clit from catgory thr will b assigd to catgory thr with probability qual to. This approach lads us to VUS for this mthod, which is markd as VUS ad quals to TP TP VUS. (0) TPFF3 TPF 3 Thr ar si diffrt ways of fidig a corrct class ad ach o of thm is markd VUS VUS 6 rspctivly. Each of thm ca b chos with probability /6. Thrfor w propos th stimat VUS, which is giv by formula () 6 6 VUS VUS VUS ( VUS VUS ). 6 6 6 () Approach basd o mpirical distributio fuctios As mtiod svral tims bfor i prvious sctios, VUS quals to th probability that radomly slctd thr obsrvatios, ach from o class, will b corrctly sortd. Thrfor VUS ca b writt as VUS = P(S < S < S 3 ). It is kow that S, S ad S3 ar mutually idpdt thrfor this approach ca b basd o th followig calculatio from L ad Lili (03): VUS = P(S < S < S 3 ) = E S, S, S 3 I(S < S < S 3 ) = = E S E (S, S 3 ) [I(S < S < S 3 )S = s] = = E S E (S, S 3 ) [I(S < S ) I(S 3 > S )S = s] = F( s) F3( s) f( s) ds. Wh stimats of ths distributio fuctios F, F 3 ad f ar usd, VUS ca b stimatd usig formula () = E S P(S < s)p(s 3 > s) = VUS F( s) F3( s) f( s) ds. () Istad of krl stimats of distributio fuctios, which is rcommdd i L ad Lili (03), mpirical distributio fuctios wr usd. F ( s) I( S stargt), i whr is sampl siz for catgory o, I is idicator fuctio ad targt is targt variabl. This approach still lads to ubiasd stimat of th VUS, but computatioal dmads ar much lowr. Fuctios F ad F 3 ca b stimatd similarly. Dsity fuctio f was stimatd usig th stimat of th fuctio F. RESULTS Valus of th VUS stimatd with tchiqus mtiod i prvious sctios ca b foud i Tab. V. VUS is th valu obtaid by approach V: Estimatd VUS usig SAS Softwar for cratd modls Modl Modl Modl 3 Estimat Trai Validatio Trai Validatio Trai Validatio U 0.70 0.803 0.68 0.0 0.06 0.849 VUS 0.956 0.678 0.85 0.64 0.936 0.76 VUS 0.90 0.40 0.030 0.907 0.57 0.95

Thr-way ROC aalysis usig SAS Softwar 75 with formula (), VUS ar obtaid by approach with formulas () ad U is obtaid by dirct computatio of Ma Whity U statistic giv by (9). Thr ca b s that accordig to VUS th modl was slightly strogr. It was causd by th simplicity ad roughss of th stimat basd o cofusio matri. Bttr rsults wr achivd by Ma Whity U statistic, but computatio of this statistic, spcially for big data, has vry high computatioal dmads. Thrfor approach basd o formula () looks lik th bst choic for stimatio of th VUS. Accordig to this mthod th bst modl is th modl #, bcaus it has th biggst valu of th VUS usig both traiig ad validatio data. SUMMARY I this papr it has b showd o of may uss of multiomial logistic rgrssio, stimatio of LGD, which owadays is vry importat task. Masur of quality of ths modls ca b asily do with SAS softwar with cratd macros for stimatio of th VUS. Popl ad compais who ar usig SAS for prdictio modllig should cosidr ths mthods i cas of usig multiomial logistic rgrssio. Two approachs wr usd ad both lads to similar rsults. Our stimats usig first approach wr aroud 0.8 ad scod approach aroud 0.. Ths valus hav show that ths modls ar quit bttr tha uslss modl, but i th futur som improvmts could b do to gt v highr VUS. This papr was maily cocrd with which mthod us i SAS to comput th VUS, what is good complmt to misclassificatio rat i trms of masurig quality. Howvr, ths mthods ca b usd oly with thr catgoris of targt variabl thrfor i futur th rsarch could b mad to tt this articl to v mor tha thr dimsios bcaus multiomial logistic rgrssio with mor tha thr catgoris is also bcomig vry popular. REFERENCES BRUIN, J., 0: Nwtst: commad to comput w tst, http://www.ats.ucla.du /stat/stata/ado/aalysis/. Accssd o August 5, 03. CORTES, C., MOHRI, M., 003: AUC Optimizatio vs. Error Rat Miimizatio, http://machilarig. wustl.du/mlpaprs/papr_fils/nips003_ AA40.pdf. Accssd o July 6, 03. FERRI, C., HERNÁNDEZ ORALLO, J., SALIDO, M. A., 003: Volum Udr th ROC Surfac for Multi class Problms. Eact Computatio ad Evaluatio of Approimatios, http://usrs.dsic.upv.s/~jorallo/ scrits/vus-mor-tha-j.pdf. Accssd o Ju 4,03. HOSMER, D. W. ad LEMESHOW, S., 000: Applid logistic rgrssio. d d. Nw York: Joh Wily & Sos Ic, 375 p. ISBN 978-0-47-3563-5. LANDGREBE, T., DUIN, R. P. W., 006: A simplifid tsio of th Ara udr th ROC to th multiclass domai. [oli] 6. 7. 03. Availabl o: http://citsr.ist.psu.du/viwdoc/dowload?- doi=0...3.455&rp=rp&typ=pdf. LE K., LILI T., 03: Estimatio of th volum udr th ROC surfac with thr ordial diagostic catgoris. Computatioal Statistics & Data Aalysis, Volum 6, Ju 03, Pags 39 5, ISSN 067-9473. LI J., FINE J. P., 008: ROC aalysis with multipl classs ad multipl tsts: mthodology ad its applicatio i microarray studis. Biostatistics, Volum 9, Issu 3, Pags 566 576, ISSN 57-486. MOSSMAN, D., 999: Thr-way ROCs. Mdical Dcisio Makig, Volum 9, Issu, ISSN 07-989X. ŘEZÁČ, M., 009: Prdictiv Modl Moitorig Populatio Stability Idics. FORUM STATISTICUM SLOVACUM, Volum 5, Issu, Pags 79 84. ISSN 336-740. ŘEZÁČ, M. ad ŘEZÁČ, F., 0. How to Masur th Quality of Crdit Scorig Modls. Fiac a úvěr Czch Joural of Ecoomics ad Fiac, 6, 5, Pags 486 507. ISSN 005-90. SCURFIELD, B. K., 996: Multipl/vt forcd/ choic tasks i th of sigal dtctability. Joural of Mathmatical Psychology, 40, pp. 53 69, ISSN 00-496. WAEGEMAN, W., DE BAETS, B., BOULLART, L., 008: ROC aalysis i ordial rgrssio larig. Pattr Rcogitio Lttrs, Volum 9, Issu, Pags 9, ISSN 067-8655. WAN, S., 0.: A mpirical liklihood cofidc itrval for th volum udr ROC Surfac. Statistics ad Probability Lttrs, 8, pp. 463 467. ISSN 067-75. Addrss Bc. Juraj Kapasý, Mgr. Marti Řzáč, Ph.D., Dpartmt of Mathmatics ad Statistics, Faculty of Scic, Masaryk Uivrsity, Kotlářská, 6 37 Bro, Czch Rpublic, -mail: 35659@mail.mui.cz, mrzac@math. mui.cz