Constructing Decision Trees with Multiple Response Variables

Transcription

1 Coruc Deco Tree w ulple epoe Varable Seo-Ju Km Ka Bae ee Kau Naoal Uvery Deparme of Idural Eeer km@kau.ac.kr Caolc Uvery of Pua Scool of Bue Admrao kblee@cup.ac.kr Abrac Daa m a proce of dcover meaful paer lare daa e a are ueful for deco mak ad a recely receved a amou of aeo a wde rae of bue ad eeer feld. Deco ree alo kow a recurve paro or rule duco oe of e mo frequely ued meod for daa m. A deco ree o a dvde-ad-coquer ba provde a e of rule for clafy ample e lear daa e. o of work o deco ree ave bee coduced for e cae of le repoe varable. However uao were mulple repoe varable ould be codered are from may applcao for example maufacur proce moor cuomer maaeme ad clcal ad eal aaly. T arcle cocer coruc deco ree we ere are wo or more repoe varable e daa e. I arcle we veae ode omoeey crera uc a eropy ad G dex ad e pree ree approace o coruc deco ree w mulple repoe varable. To do o we fr decrbe exeo of eropy ad G dex o e cae wc mulple repoe varable are of cocer. A we meod for ode pl alo explaed. Nex we pree a deco ree mmz a expeced lo due o mclafcao. To llurae e procedure umercal example are ve w dcuo.. Iroduco Wdepread ue of ework ad formao ecoloe a made eaer o collec daa ad creaed lare daabae. Accordly e role of daa m become more mpora. Daa m e proce of dcover ere paer daabae a are elpful o deco mak[]. Daa m a bee ued for praccal applcao a varey of doma of bomedcal bue ad dural feld[]. Two prmary meodoloe for daa m are mace lear ad acal aaly. ace lear e udy of compuaoal meod o auomae e proce of kowlede acquo from example[3]. I kow a compared w paramerc acal aalye mace lear ecque are more uable for daa m of a lare complex daa e. T becaue uc daa e lkely o be uder dmeoaly mulcolleary ad o-omoeey[4]. aor caeore of mace lear ecque are deco ree eural ework cae-baed reao ad eec alorm[]. Accord o e rece udy by Boe ad aapara[] u deco ree mo popular e daa m of bue feld. Tey repored a abou a alf of daa m applcao baed upo deco ree meod. For more deal ee Boe ad aapara[]. Deco ree alo kow a recurve paro provde a e of rule for clafy ample e daa e o a dvde-ad-coquer ba. Deco ree are broadly dvded o clafcao ad rereo ree. Deco ree called clafcao ree we e repoe varable caeorcal ad rereo ree we e repoe varable umercal. Oe of e mo famou work o deco ree probably Clafcao ad ereo TreeCAT by Brema e al.[5]. I CAT a ode pl o wo offpr ode. So called a bary ree. CAT meod fr fd e maxmal ree by a pl procedure ad e e r zed ree by a pru procedure. Alo CHAID ID3 ad C4.5[4 6] are well-kow meod for buld deco ree. Alou ee ree-baed ecque ave bee wdely ued for daa m epecally for auomaed clafcao er applcao are maly rerced o e cae wc daa e a a le repoe varablesv. However o o dffcul o fd problem a mulple repoe varablev ould be uded. For example Za[7] a deal w explaaory varable ad 6 bary repoe o aalyze buld-relaed occupa compla ydromebocs clcal reearc. He a preeed a maxmzed lo-lkelood a a eeralzao of eropy ad a Hoell T ype ac for V-ode pl. Sclao ad ola[8] a alo ued 9 explaaory varable ad 3 caeorcal repoe o model e famly bude of bak cuomer. Tey ave propoed a predcably dex baed upo G dex a a V-ode pl

2 crero. I maufacur procee mulple repoe are rouely oberved a well. I wll be ere o udy deco ree w V for proce moor ad dao. T arcle deal w deco ree we e daa e a wo or more repoe varable all of wc are aumed o be caeorcal work. Te purpoe of paper o pree ode pl meod ak o accou mulple repoe varable. Te rema par of paper orazed a follow. Seco oule deco ree ad decrbe meaure of ode omoeey uc a eropy ad G dex. Alou deco ree a ome advaae over oer mace lear ecque ere are ll problem o olve. Fd a way o accommodae mulple repoe varable would be oe of em ad deal w Seco 3. We acually pree ree approace o produc a V-deco ree ad llurae ode pl procedure by example. I Seco 4 a ummary of our work ve w dcuo. mao a well a fuure dreco of reearc are alo aed.. A Overvew of Deco Tree. Deco Tree a a ace ear Tecque A meoed earler deco ree oe of meod o daa m. Accord o codo of explaaory varable or arbue wole lear ample are modeled o a deco ree. Te reula ree eveually provde a e of rule for clafy ee lear ample. T e wy deco ree baed clafcao called deco ree duco[] or rule duco[8]. odel reul by deco ree eay o expla ad erefore wdely ued o fd rule for clafy ew ample. Deco ree alo vewed a oe of e uperved lear meod becaue bul from lear ample w a kow clafcao. Te reul of lear repreeed a a ree e ode of wc pecfy arbue ad e brace pecfy arbue value. Fure ow a ypoecal deco ree. X 3 X 4 5 Fure. A ypoecal example of deco ree Tere are wo kd of ode a deco ree; eral ode ad ermal ode Termal ode are alo called leave. I e leraure of deco ree eral ode deoed by e crcle ad ermal ode by e box. Eac of all ode correpod a ube of e ere lear e. Te roo ode o e op repree all ample e lear e. Tee ample are dvded o wo do ubroup by e explaaory varable X. T proce of varable eleco ad ode pl coued ul eac ermal ode repree a dffere cla of ample. Eveually all ermal ode3 4 5 coue muually excluve ad exauve ube of e ere lear e. Te reul deco ree e appled o a e e of ample o evaluae accuracy clafy ew ample. To mprove e clafcao capably of deco ree fr mpora o cooe a approprae ode pl crero o a o maxmze omoeey of e offpr ode. Overf deco ree o e lear e ofe drop clafcao performace o ew ample. I uc cae ree pru requred o mae overf before e ree deployed a real lfe applcao. Cro valdao ca be alo ued o preve a deco ree from deped o a pecfc daa e.. Node Spl Crera A decrbed above we eed a ode pl crero order o meaure ode mpury ad o row e deco ree. Spl ode a o be uderake uc a ode mpury ca be mmzed or ode omoeey ca be maxmzed. Eropy ad G dex are maly ued for meaur ode omoeey were e daa e a a caeorcal repoe varable. Coder e follow fure o ee ow o pl ode.

3 + X Fure. A llurao of ode pl T fure ow a ypoecal uao a ode pl o wo ode ad accord o a arbue X. I e fure ad repecvely deoe e umber of lear ample ode ad. Suppo a e repoe varable Y a K ordered caeore we ca oba e follow frequecy able wc ow a lear ample are dvded o wo ubroup. Table. A ode pl reul Node Y K Sum K K K I e able ad are e umber of lear ample wc belo o cla a ode ad repecvely ad for K + Te a eropy o meaure ode omoeey a ode ca be expreed a K lo Noe a e maller eropy e er omoeey or a lower mpury. Oe ca ue G dex ead a a ode omoeey meaure a e follow. K Te amou of omoeey a aceved by pl ode ca be e obaed by η Tu a ree row by coo a pl a maxmze a eac ode. Suc ode pl ubequely coued ul a opp rule afed. For example we η maller a a predeermed value we op pl ode ad declare a a ermal oe.

4 .3 Dcuo ecely Boe ad aapara[] compared everal mace lear ecque for bue daa m. Accord o er udy deco ree a ome praccal advaae. Epecally erm of bo explaao capably ad applcably o lare daa e deco ree meod ouperform oer ecque uc a eural ework ad eec proramm. Eae of operao ould be alo meoed a oe of e re of ree-baed meod. I ee reao expeced a apply deco ree wll become wdepread a a varey of doma. However ere are ll problem o cope w deco ree meod. For example we e lear e a ome rreleva ample deco ree ed o dvde ode av few ample ad e e reula ree ed o be oo lare ad overpecfed[5]. T lead o a lear uably ad clafcao accuracy become uder queo. To avod uc drawback ree pru ca be codered. Aoer lmao le a ode pl procedure deped o a le arbue varable. T mple a a poed ou by Brow e al.[9] a adard deco ree ecque lkely o uffer from mul-modal problem. I order o overcome dffculy ey propoed a mulvarae ode pl baed upo lear combao of arbue. Oer a ee oe of mpora ue o deco ree cocer exeo o e cae of mulple repoe varablev. A aed earler Za[7] cluded 6 repoe varable clcal udy ad owever er work rerced o bary repoe lke ye or o. Sclao ad ola[8] alo deal w 3 repoe varable er reearc. Tey ued a we um of G dce compued from repecve repoe varable. Bu ow o coder varable mporace o uded. Ivea uc rerco of Za[7] ad Sclao ad ola[8] Seco 3 pree ree ode pl approace for V-uao. 3. Node Spl Crera w ulple epoe VarableV A aed Seco V-uao are ofe oberved bomedcal bue ad oer dural feld. Alou coa relavely more formao abou dde relaop V-daa e dffcul o deal w ow o problem complexy ad compuaoal burde. Tu mace lear w V wll be oe of opc worwle o udy furer. A doe SV-deco ree of prmary mporace rede ow ode omoeey ould be quafed w V. To awer e queo paper aemp o coder ree ode pl ceme. Te fr deal w exeo of eropy ad G dex. T doe by fd o frequecy drbuo of repoe varable a eac ode. Te covarace rucure of o drbuo are alo codered for ode pl. Te ecod oe o ue a we um of ode omoeee for repecve repoe varable. A doe Sclao ad ola[8] G dex ued o evaluae ode omoeey. Te rd approac cocered w mmz a expeced lo a a ode pl crero. 3. V Exeo of Eropy ad G Idex epoe varable are deoed by Y Y Y were e umber of repoe varable e daa e. Ad uppoe a repoe varable Y a K ordered clae for. Te a mulvarae eropy o repree omoeey of ode ca be defed by K K K lo were deoe e umber of lear ample woe repoe value are repecvely Y Y Y a ode. Smlarly a mulvarae G dex defed by K K K 3 We ere are wo or more repoe varable covarace marx obaed. T marx ow correlao bewee repoe varable wc are cocered w ode omoeey V-uao. Suppo a V deoe a ample covarace marx ode we ca ue e deerma of V a a areae meaure of omoeey ad correlao. Ta V 4 Po ou a V ca be erpreed a G dex u a le bary repoe Za[7] cluded V o crera for ode pl. I addo e recommeded o ue a Hoell T ype ac a e follow:

5 ' ] [ ] [ I V y y y y 5 were I V ad y repecvely repree a e of lear ample a ode covarace marx of ere lear ample ad ample mea vecor of repoe varable a ode. T ac a bee orally ued a mulvarae moor ac were repoe varable are couou raer a caeorcal. e S deoe a dex e of all poble pl a ode. Te e amou of omoeey a aceved by pl ode ve by } { } { } { } { η 6 ecall a e maller e er omoeey. A opmum pl for ode erefore coe by maxmz 6 ad wre by ar max * S η 3. We Sum of Node Homoeee A doe Sclao ad ola[0] we um of G dce ca be employed a a crero for V-ode pl. T meod a ome advaae. Eae of compuao wll be oe of em. T becaue G dce are obaed for repecve repoe varable. elave mporace bewee repoe varable are alo accommodaed crero. However a queo are a o ow o oba e mporace. Terefore for approac o be operaoal e awer o e queo a o be prepared. G dex for repoe varable Y a ode ve by K 7 were repree e umber of lear ample afy Y a ode. e w deoe e we of repoe varable Y we ca oba a we um of 7 a e follow: K w w / 8 were w w 0 ad Terefore a explaed e prevou eco we fd a opmum pl o a 6 ca be maxmzed. Baed upo e we um 8 Sclao ad ola[0] propoed o ue a predcably dex ead of 6 wc ve a / η τ

6 I oed a uforuaely ere o areed procedure o deerme we. Alou a daa-aalyc approac o fd a opmum we e omeme applcable ey are eeral ve by oer coderao for example maaeme polcy prory de ad operao aaly experece ad co rucure. 3.3 Node Spl by a Expeced o So far we explaed ode pl ceme of meaur ode omoeey. T eco roduce a expeced lo a a ode pl crero. I meod varable mporace ued o fd mclafcao co. e Yˆ deoe a predced cla of repoe varable for a lear ample. Te a lo fuco ca be defed by Y Y Y ; Y 9 0 Y Y were. Terefore a V-lo fuco of predco ca be rewre by w Y y Y ; 0 ˆ; y were y ad ŷ are vecor of rue clae ad of predced clae repecvely. Tak maemacal expecao o 9 we ca derve a expeced lo for repoe varable a Pr Y Y Pr Y Y Noe a e expeced lo elf a mclafcao probably. Tu ak expecao o 0 ad ubu aa produce a V-expeced lo wc wre by w Pr Y w Pr Y Y Y I pracce mclafcao probably for repoe varable a a ermal ode ca be emaed by max 3 p K were p /. T emao mple a all ample ode are clafed o a le cla e frequecy of wc e. Subu 3 o we ca oba a emaor of expeced lo a ode a e follow. w max p 4 K

7 Terefore oal lo over e ere ree ca be defed by T p 5 T were T repree a e of ermal ode e ree. Alo oe a p/n were N e oal umber of lear ample. Te amou of lo reduco aceved by pl ode e ve by { } { } λ { } { } Numercal Illurao I T eco llurae e ode pl procedure explaed Seco 3. u a ypoecal daa e wc a wo repoe varable. T daa e clude 600 lear ample ad e o drbuo of repoe varable a ow Table. Table. Jo drbuo of Y ad Y a e roo ode Y Y um Y Y Y um Aume a ere are wo explaaory varable X ad X w 3 caeore repecvely. Furer aume a X ordal caeorcal varable. To be w le u coder e follow pl codo; Take e lef ode f X ad ake e r ode elewere. Fure 3 llurae a paral ree produced by codo V X X or 3 V V Fure 3. A ode pl w mulple repoe varable I fure V V ad V are ample covarace marce of ode ad repecvely. A a reul of e ode pl codoal o drbuo of Y ad Y a ode ad are repecvely ve a e follow able. Table 3. Jo drbuo of Y ad Y ve X ad X or 3 a X b X or 3 Y Y um Y Y um Y Y Y Y Y Y um um U we ca fd mulvarae eropy of ode from Table a

8 lo lo lo lo lo lo Smlarly from Table ad.35 are obaed. Tu u 6 eropy a by pl ode ve by η ad correpod o mproveme of 0.79/.548.%. ulvarae G dex a ode alo calculaed a / by 3. Smlarly we ca ave ad 0.67 ad u e a of G dex η or 7.3%. Table 4 ow eropy G dex ad er a for 5 pl codo a ode. I example pl 4 produce e be oupu rrepecve of u eer eropy or G dex. Table 4. Node pl comparo of eropy ad G dex over S p p Eropy G Idex η η X X % % X X % % 3 X3 X % % 4 X X % % 5 X3 X % % Oer a eropy ad G dex e covarace marx deerma ad Hoell T ype ac ave bee ued Za[7]. Fr from Fure 3 V 0.36 V ad V are obaed. We ca ee a uder crero omoeey a η or 48.%. Now aum a ode e roo ode we ave V Tu from 6 Hoell T ype ac of ode ad are calculaed a I [ y [ y I [ y I ' y ] V y y ' ] V ' ] V [ y [ y [ y y ]/ y y ]/ ]/

9 repecvely. I example η0.446 or.3%. 3.5 Numercal Illurao II I eco ode pl meod decrbed Seco 3. ad 3.3 are lluraed by aoer coruced example. Fr uppoe a ode a 00 lear ample. Alo uppoe a ere are wo repoe varable ad wo explaaory varable. Te follow ow coecy able bewee explaaory ad repoe varable. Table 5. Cro abulao of X X Y ad Y a ode Y Y Y3 um Y Y Y3 um X X X X X3 8 0 X um um Y Y Y3 um Y Y Y3 um X X X X X X um um U 6 yeld G dce for Y ad Y a ode a follow Tu e weed G dex 7 ca be obaed by w + w aum e we are ve a w0.7 ad w0.3. However Sclao ad ola[8] ued ode mpure o oba we for Y a ode wc wre by U 7 produce wo we a ode a follow. w / 7 w 0.6 / w 0.64 / Tu weed G dex obaed by w + w

10 Te expeced lo 4 obaed a follow. Fr from 3 mclafcao probable for Y ad Y are ve by max max repecvely. Terefore e expeced lo 4 become w + w Table 6 ow a comparo reul of 6 cearo for ode pl. Spl 5 be we weed G dex ued. O e corary for e expeced lo pl 5 be. Alou o cluded able pl be uder e we ceme propoed by Sclao ad ola[8]. Table 6. Node pl by weed G dex ad by expeced lo we ww p p G Idex Expeced o η λ % % X X X3 X X X X X X3 X X X By ca e we we ca compare e above x cearo aa. Tee are depced e follow fure. Alou e wo pl crera produce dffere reul eac oer ard o ay wc oe beer or wore example. To awer e queo more rorou comparo are requred fuure. Neverele oe o uderle a we for repoe varable ave o be carefully coe. T becaue e be pl ca be alered by e we. A ow by e fure a are mooly crea or decrea alo e we. For example pl 3 yeld e lare a we w 0.6. Bu e lower rae of w pl 6 be for weed G dexfure 4a ad pl 5 be for expeced lofure 4b. We ca ee a parcular e preferece of pl 5 eve o e coce of we. O cora pl 4 look que robu aa w wc becaue a of ode pl oo lle. 4. Summary ad Cocluo Deco ree a a mace lear approac provde a prom way o buld clafcao model from a lare daa e. ay applcao are oberved e area of bomedce publc eal ad bue. Tree-baed approace would be alo elpful for oer dural applcao for example lke proce moor ad dao. From a acal vewpo ecque ca be rearded a oparamerc ac. I eeral we kowlede abou e populao uffce paramerc meod ave a dffculy deal w lare complex daa e[4]. I uc cae mace lear ecque ca be codered a a ueful alerave. However a e are populao more pecfc ad doma kowlede creae paramerc ac become more uable o uderad e populao. A aed Boe ad aapara[] deco ree are ueful o deal w a lare amou of daa ad ave a applcao capably. Neverele order o develop more relable model collaborave ue w paramerc meod uc a rereo ad dcrma aaly ould be purued. I may applcao we ca oberve a repoe varable are wo or more. edcal dao cuomer cred predco ad proce moor are uc example. clafcao rule baed upo mulple repoe varablev oe of e mo ere problem a V-daa e ca coa more formao for expla lae relaop. T arcle cocered w coruc deco ree we ere are wo or more repoe varable daa e. Bacally deco ree arcle are bary clafcao ree.

11 a e a % S S S3 S4 S5 S w b lam bda % S S S3 S4 S5 S w Fure 4. Ga of weed G dex ad expeced lo for x pl cearo Fr we provded a overvew o deco ree ad veaed ode pl crera w a le repoe varablesv. Te we propoed ree approace o V-ode pl. Te fr approac employ eropy or G dex o repree ode mpury w V. Alou ca be vewed a a aural exeo o V uao approac a a lmao a dvdual caracerc bewee repoe varable are o accommodaed. Te ecod oe a we meod: fr G dce are obaed for eac of repoe varable ad e ey are ummed by predeermed we. T approac defely a advaae a compuao load relavely mall ad a varable mporace are codered w eae. However more ude are requred a o ow we are deermed a well a ow e correlao rucure bewee repoe varable corporaed o e ode pl procedure. Te rd alo a we meod. Bu dffere from e ecod oe a a expeced lo employed a a ode pl crero ead of eropy or G dex. By u expeced lo crero we ca fd a deco ree a mmze mclafcao. Fally llurao of e ree approace are ve by example. However for more rorou comparo of e preeed approace coduc exeve expermeao wll be requred. Eve ou udy rerced o bary clafcao ree our framework o deal w V uao could be ll appled for oer clafcao ad rereo ree. Fuure work area clude ode pl procedure wc repoe varable are umercal. Collabora deco ree ad acal model meod would be alo a fruful ubec o udy furer. Ackowledeme: T work wa uppored by ra No from e Korea Scece ad Eeer Foudao. Te auor alo would lke o ackowlede e uppor of Bra Korea Proec 00. eferece [] Idral Boe ad ada K. aapara Bue Daa A ace ear Perpecve Iformao & aaeme Vol. 39 pp [] Jawe Ha ad cele Kamber Daa - Cocep ad ecque Sa Fracco CA: ora Kaufma 00. [3] P. aley ad H. A. Smo Applcao of ace ear ad ule Iduco Commucao of e AC

12 Vol. 38 No. pp [4] Kaara D. C. Sark ad Drk U Pfeffer Te Applcao of No-paramerc Tecque o Solve Clafcao Problem Complex Daa Se Veerary Epdemoloy A Example Ielle Daa Aaly Vol. 3 pp [5] eo Brema Jerome H. Fredma card A. Ole ad Carle J. Soe Clafcao ad ereo Tree Boca ao F: Capma & Hall/CC 998. [6] You. Cae Seu H. Ho Kyou W. Co Do H. ee ad Su H. J Daa Approac o Polcy Aaly a Heal Iurace Doma Ieraoal Joural of edcal Iformac Vol. 6 pp [7] Hep Za Clafcao Tree w ulple Bary epoe Joural of e Amerca Sacal Aocao Vol. 93 No. 44 pp [8] obera Sclao ad Fraceco ola ulvarae Daa Aaly ad odel Trou Clafcao ad ereo Tree Compuaoal Sac & Daa Aaly Vol. 3 pp [9] Doald E. Brow ad Clarece ou Pard Ha Park Clafcao Tree w Opmal ulvarae Deco Node Paer ecoo eer Vol. 7 pp