Generating Classification Rules According to User s Existing Knowledge *

Transcription

1 Generating Classifiatin Rles Arding t User s Existing Knwledge * Sh Chen and Bing Li, Shl f Cmpting, Natinal University f Singapre, 3 Siene Drive 2, Singapre lib@mp.ns.ed.sg Abstrat An imprtant prblem in applying lassifiatin rle indtin tehniqes t pratial appliatins is hw t prde rles that are related t the ser s existing knwledge abt the dmain and his/her rrent interests. Sh rles are interesting t the ser, and als easily nderstd and trsted by the ser. They an enhane the existing knwledge f the dmain and be relied pn in real-wrld perfrmane tasks. Past researh and appliatins have shwn this t be a rial reqirement in many real-life appliatins. Existing tehniqes fr dealing with this prblem typially se sphistiated methds t bias the rle indtin press in rder t prde rles that are nsistent with the existing knwledge. In this paper, we prpse a nvel and simple apprah. It nly needs t pre-press the data sing the ser s existing knwledge. It des nt make any mdifiatin t the rle indtin tehniqe. Pratial appliatins have shwn that this simple apprah is srprisingly effetive and flexible. It demnstrates that t btain sefl reslts, we d nt neessarily need t se sphistiated tehniqes. Smetimes simple apprahes may jst be sffiient. 1. Intrdtin The ser s bjetive in sing a knwledge disvery system (KDS) is t find interesting rles that are nderstandable [e.g., 5, 8, 9, 21, 22, 25, 20]. Rles are easy t nderstand if * This paper has been prepared in peratin with the Siety fr Indstrial and Applied Mathematis. 1

2 2 they nfrm t r are nsistent (flly r at least partially) with the prir knwledge f the dmain expert r the ser [20]. Rles are interesting if they are sefl (r atinable) [21] and/r nexpeted [25, 8, 9]. Hwever, fr a KDS t knw what rles are interesting and nderstandable t a ser is nt an easy task. A rle an be interesting t ne ser bt nt interesting t anther. Ths, whether a rle is interesting and nderstandable r nt is sbjetive. It depends n the ser s prir knwledge abt the dmain and his/her rrent interests. It is nw well regnized that applying standard lassifiatin rle learning tls in real-life appliatins is by n means a straight frward task [20] bease these tls ften prde rles that are mpletely nrelated t the ser s existing nepts abt the dmain and the ser s interests. These rles an be very diffilt t nderstand and/r ninteresting t the ser. The rt f the prblem is that in rder t prde a small set f arate rles t frm a mdel f the dmain [23] rle indtin systems se varis biases in its rle generatin press. These biases, hwever, may nt be in agreement with the existing knwledge f the hman ser, ths reslting in the rle nderstandability and interestingness prblems. That is, many f the rles generated d nt make sense t the ser and/r are nt interesting t the ser. The existing methd fr dealing with this prblem in mahine learning is t se the ser s knwledge as biases in rle indtin in rder t generate rles that nfrm t r are nsistent with the existing knwledge [1, 15, 17, 20]. This tehniqe invlves sphistiated mdifiatins t the rle indtin systems [1, 15, 17, 20]. They als d nt deal with the generatin f nexpeted rles. Unexpeted rles are, by definitin, interesting. In data mining researh, mst rrent apprahes fr dealing with the isse f finding interesting rles emply a pst-analysis mdle at the bak-end f a mining system [8, 9, 10, 11, 21]. This mdle ses the ser s inpt knwledge t help him/her identify interesting rles. Hwever, pst-analysis annt find the type f interesting rles stdied in this paper bease they are nt in the set f disvered lassifiatin rles. Althgh there are apprahes fr finding nexpeted rles sing the ser s existing knwledge, they are fr assiatin rle mining [18, 19]. Assiatin rle mining, hwever, annt handle nmerial attribtes, whih are very mmn in lassifiatin datasets. T perfrm assiatin rle mining, nmerial attribtes have t be disretized. Crrent disretizatin tehniqes, hwever, d nt nsider interatins amng the attribtes and ths tend t destry the inherent relatinships amng them. In this paper, we prpse a nvel and yet simple apprah t generate rles that are related t the ser s existing knwledge abt the dmain and his/her rrent interests. The essene f the prpsed tehniqe is t pre-press the database sing ser s existing knwledge and then se a existing rle indtin system (we se C4.5) t disver the interesting and nderstandable rles. Or tehniqe des nt need any mdifiatin t C4.5. The prpsed tehniqe has been sed in a nmber f real-life appliatins, three medial appliatins and tw edatin appliatins. The reslts shw that this simple apprah is srprisingly effetive and flexible. It demnstrated that t slve an imprtant prblem, we d nt neessarily need t design sphistiated tehniqes. Sme simple apprahes may well be sffiient. 2. Preliminary Disssins Sine we will se C4.5 [23] fr rle generatin, this setin first reviews the C4.5 rle generatin press. It then dissses the prblem that we are ging t address.

3 3 A database D fr C4.5 nsists f the desriptins f N bjets in the frm f tples. These N bjets have been lassified int q knwn lasses, C 1,, C m,..., C q. Eah bjet in the database is desribed by n distint attribtes, Attr 1,..., Attr l,..., Attr n. The bjetive f C4.5 is t find a set f harateristi desriptins (r lassifiatin rles) fr the q lasses. A lassifiatin rle in C4.5 has the fllwing frm: P 1,..., P i,..., P r C where, means and, and P i is a test n an attribte and C is a lass. T failitate nderstanding f the prpsed tehniqe, let s review the gemetri interpretatin f hw C4.5 wrks. C4.5 wrks by first bilding a deisin tree and then prding a set f lassifiatin rles frm the tree. Gemetrially, we an view the deisin tree as speifying hw a desriptin spae f the tples is t be arved p int regins assiated with the lasses. The regins prded by a deisin tree are all hyperretangles [23]. When the task at hand is sh that the lass regins are nt hyperretangles, the deisin tree will apprximate the regins with a set f hyperretangles. This is illstrated by a simple example in Figre 1(A) in whih 80 ases (r tples) f tw lasses (represented by and ) are desribed by tw ntins attribtes, X and Y. The intended divisin f the desriptin spae by an bliqe line is shwn in Figre 1(A), while Figre 1(B) displays the apprximatin t this divisin that is fnd by C4.5. We first disss the nderstandability isse. The main reasn that the generated rles are hard t nderstand is bease they are nt related t the ser s existing nepts. The nderstandability prblem, in general, an be explained as fllws: Let s se the previs example in Figre 1(A) t illstrate. The ser s nept spae is represented as the dark-shaded area in Figre 1(C). Nte that there are retangles frmed by C4.5 that d nt interset with the ser s nept spae. They represent things that are freign t the ser. Then the rles assiated with these retangles are mre diffilt t nderstand (bt they ld still be interesting, see belw). Nte als that there are ther retangles frmed by C4.5 that rss the bndaries f the ser s nept spae. These rssings als make the generated rles diffilt t nderstand bease they are nfsing t the ser. In this paper, we prpse a tehniqe that will gide the rle generatin press sh that the rles generated will be as lsely related t the bndaries f the ser s nept spae as pssible. This is ahievable bease f the bservatin that the divisin prded by C4.5 (e.g., in Figre 1(B)) is by n means niqe. Clearly, it is pssible t divide the spae in many ther ways. Fr example, Figre 1(D) shws anther pssible divisin. The prpsed tehniqe makes se f this bservatin t fre C4.5 t generate rles that are related t the ser s nepts. Y X Y X (A) (B)

4 4 Y X Y (C) (D) Figre 1. Real and apprximate divisin fr an artifiial task. Let s nw trn r attentin t the meaning f interesting rles. Past researh has prpsed tw main fatrs that ntribte t the sbjetive interestingness f a rle, nexpetedness and atinability [21, 8, 25]. Unexpetedness: Rles are interesting if they srprise the ser. Atinability: Rles are interesting if the ser an d smething with them t his/her advantage. In ther wrds, rles are sefl when they an help t ahieve sme rrent gals f the ser. These tw measres are nt mtally exlsive [25]. Ths, we an lassify sbjetively interesting rles int three ategries arding t the abve definitins: (1) rles that are bth nexpeted and atinable; (2) rles that are nexpeted bt nt atinable, and (3) rles that are atinable bt expeted. In this paper, we handle (1) and (2) by finding nexpeted rles and handle (3) by finding the rles that nfrm t the ser s expetatins. Whether a nfrming rle r an nexpeted rle is atinable r nt, it will be deided by the ser. We nw se a real example t illstrate the prblem we are ging t address. This example ses the redit sreening database reated by Chihar San in UCI mahine learning repsitry [14]. The database has 125 tples, 10 attribtes and tw lasses Yes and N representing whether redit has been granted. Witht nsidering any existing knwledge, the set f rles generated by C4.5 is as fllw: R1: Age > 25, Savings > 7, YR_Wrk > 2 Yes R2: Sex = Male, YR_Wrk > 2 Yes R3: Jbless = N, Bght = p Yes R4: Bght = medinstr, Age <= 34 Yes R5: Sex = Female, Age <= 25 N R6: Savings <= 7, M_LOAN > 7 N R7: YR_Wrk <= 2 N Hwever, the ser believes that the fllwing rle (alled the ser expeted rle) is tre frm experiene: Bght = Jewel, Sex = Female N. This rle nveys t imprtant messages: 1. The ser believes this rle shld be tre r it is the ser s hypthesis. 2. This rle is lsely related t the ser s rrent interests abt the dmain. By lking at the disvered rles abve, we have n idea hw rret the ser expeted rle is, and whether there are rles related t the ser s nept bease the generated X

5 5 rles have little relatinship with the ser s nept. T find the rretness f the expeted rle is easy, e.g., by testing the rle against the database. In this ase, the abve expeted rle is nly rret 28.6% f the time (14 tples satisfy the nditins, bt t f these 14 tples nly 4 f them satisfy the nlsin). The qestin is: an we haraterize these nfrming and nexpeted tples? This paper prpses sh a tehniqe. Fr the abve prblem, r prpsed tehniqe is able t fre C4.5 t prde the fllwing related rles and mre (see Setin 3.2 fr mre details): Rle1: Bght = jewel, Sex = Female, YR_Wrk > 2 Yes Rle2: Bght = jewel, Sex = Female, YR_Wrk <= 2 N Rle2 is a nfrming rle and Rle1 is an nexpeted rle. After seeing these tw rles, the ser will knw exatly what is wrng with his/her existing nept, and als find smething nexpeted, i.e., Rle1. These rles are easily nderstd by the ser bease they are lsely related t the ser s nept. The abve prblem an be illstrated gemetrially in Figre 1(C). Fr example, a ser expeted rle (e.g., its lass is ) is represented as the dark-shaded area in Figre 1(C). The area rsses the bndaries f 3 regins frmed by C4.5. Within the ser expeted rle, there are s whih represent nfrming tples, and s whih represent nexpeted tples. Obvisly, frm the divisin prded by C4.5 it is hard t knw what are the harateristis f the nfrming and nexpeted tples. Or prpsed tehniqe is able t make C4.5 t prde the nfrming and nexpeted rles. 3. The Prpsed Tehniqe The key pint f the prpsed tehniqe is t se the ser s nepts t help C4.5 disver interesting rles. The ser s nepts are expressed as a set f expeted rles, whih are in the same frmat as the generated rles. Using this representatin t represent ser nepts is natral bease when the ser is lking fr a partilar type f rles (e.g., lassifiatin rles), his/her expetatins are sally f the same type. Three main steps are invlved in the prpsed apprah: Step 1. The ser prvides a set f expeted rles E that he/she expets t find in the database. Eah E j E is expressed in the same frmat as disvered rles. Step 2. Pre-press the database D t prde D. This step (1) mptes the rretness f eah E j E, and (2) assigns a new lass CONFORM t eah tple that nfrms t E and assigns a new lass UNEXPECTED t eah tple that is nexpeted with respet t E. At the nlsin f this step, all the tples in D are lassified as nfrming, nexpeted r nrelated tples (Setin 3.1). Step 3. Rn C4.5 n D. C4.5 will prde three types f rles, nfrming rles, nexpeted rles and nrelated rles, whih give different types f interesting infrmatin t the ser. We will nt disss Step 1 frther in this paper as it simply asks the ser t express his/her previs nepts in the frm f a lassifiatin rle (see the example in Setin 2). (1) f Step 2 is als simple. The rretness (Crr j ) f eah E j E, an be mpted as fllwing:

6 6 Crr j = Ttal nmber f tples satisfying Ttal nmber f tples bth the nditins and nlsin satisfying the nditins f E j f E j 3.1 Pre-pressing the database and rnning C4.5 Dring the pre-pressing step, we intrde additinal lasses. A test is arried t n eah tple D k in the database D. Tples, that are fnd t be nfrming r nexpeted with respet t E, are assigned the Cnfrm r Unexpeted lasses respetively (i.e., their riginal lasses are replaed). The remaining tples (that are nrelated t E) will retain their riginal lasses. The press ses the fllwing definitins. Definitin: D k nfrms t an expeted rle E j E if D k satisfies bth the nditins and nlsin f E j. Definitin: D k is nexpeted with respet t E j E if D k satisfies the nditins f E j bt nt its nlsin. Definitin: D k is nrelated t E j E if D k des nt satisfies the nditins f E j. Definitin: D k nfrms t E if E j E, D k nfrms t E j. Definitin: D k is nexpeted with respet t E if E j E, D k des nt nfrm t E j, and E i E, D k is nexpeted with respet t E i. Definitin: D k is nrelated t E if E j E, D k is nrelated t E j. It mst be stressed that these definitins are nt niqe. See the explanatin t the algrithm belw. We are nw in the psitin t present the verall algrithm fr Step 2. 1 Initialize RleN j and CndN j t 0, 1 j E ; 2 fr eah tple D k D d 3 Cnfm FALSE; Unexp FALSE; Class the lass f D k ; 4 fr eah E j E d 5 if D k satisfies the nditins f E j then 6 Inrement CndN j 7 If D k satisfies the nlsin f E j then Inrement RleN j ; Cnfm TRUE 8 else Unexp TRUE 9 endif; 10 endif; 11 endfr; 12 if Cnfm then Change the lass f D k t <Class>CONFORM; 13 elseif Unexp then Change the lass f D k t <Class>UNEXPECTED; 14 endif; 15 endfr; 16 fr eah E j E d Crr j = RleN j CndN j endfr; Ntes abt the algrithm: Lines 1 and 3 are the initializatin. Line 5-7 prepare the vales RleN j and CndN j fr the mptatin f the rretness f E j in Line 16. Line 7 indiates that D k satisfies E j. We say that D k nfrms t E j. Line 8 indiates that D k satisfies nly the nditinal part f E j, bt nt the nlsin. We say that D k

7 7 is nexpeted with respeted t E j. Lines 12 and 13 assign new lasses t tples that nfrm t E and that are nexpeted with respet t E. It shld be nted that there may be ntraditry sitatins, i.e., D k nfrms t E j bt is nexpeted with respet t E m (j m). In sh sitatins, the abve algrithm and the definitins treat D k as a nfrming tple. Alternatively, we ld treat D k as an nexpeted tple, r assign it a CONTRADICTORY lass. All these variatins have been implemented in r system as ptins t the ser. In fat, this tehniqe is s flexible that it is als pssible nt t assign nfrming (r nexpeted) lasses if the ser is nt interested in the lasses (see the example in Setin 3.2 belw). Nte that the new lasses intrded are: <Class>CONFORM and <Class>UNEXPECTED. The interpretatin is as fllws: <Class> is the riginal lass f D k and CONFORM indiates that D k is fnd t be nfrming t E. Fr example, the riginal lass f D k is Yes and D k is nfrming, then the new lass f D k will be YesCnfrm. After the pre-pressing step, we btain the mdified D, dented by D. Next, we rn C4.5 n D t prde three types f rles: nfrming rles, nexpeted rles and nrelated rles. 3.2 Why des this simple tehniqe wrk? T answer this qestin, let s lk at the sitatin when there is nly ne expeted rle in E, all it R. When there are mltiple rles in E, the sitatin is similar bt mre mplex. Basially, the pre-pressing divides the tples in D int three grps: nfrming tples, nexpeted tples and nrelated tples. Then C4.5 will prde rles t distingish the new lasses <Class>CONFORM, <Class>UNEXPECTED and the riginal lasses (sed by nrelated tples) ths reslting in the nfrming, nexpeted and nrelated rles. T illstrate, let s se the gemetri interpretatin example in Figre 1(C) (reprded as Figre 2(A)). Sine the ser rle R has the same frmat and meaning as the generated rle, then R als represents a hyperretangle. Let the nditinal part f R be the dark-shaded retangle in Figre 2(A) and the lass f R be. Using the pre-pressing algrithm abve, the new divisin prded by C4.5 is shwn in Figre 2(B). Then, regin 1 and 3 represents tw nfrming rles (vering nfrming tples, s), and regin 2 represents an nexpeted rle (vering nexpeted tples, s). The rest f the regins represents nrelated rles. Obvisly, the nfrming and nexpeted rles are easy t nderstand by the ser bease they are related t the ser nepts (their bndaries are related), s are the nrelated rles. The prpsed tehniqe is very flexible as it an fre C4.5 t prde many interesting rles (as mentined in Setin 3.1). Fr example, if the ser is nt interested in nfrming rles bt nly nexpeted rles, the sitatin in Figre 2(C) is prded by C4.5. In this ase, there is nly ne nexpeted rle (regin 1). This implies that we are able t smmarize and present nly the relevant rles that are f interest t the ser.

8 8 X Y X Y (A) (B) Y X 1 (C) Figre 2. The new divisins bease f the ser s rle 4. Illstratins We have tested the system sing a nmber f pbli dmain databases. The system has als been sed in 3 real-life medial appliatins, and 2 edatin appliatins. Sine there is n reprted tehniqe that is able t perfrm this task, we are nable t d any mparisn. Here, we give sme test rns f r system sing a pbli dmain database and a real-life disease database t illstrate its se. 4.1 Example rns The first tw example rns se the redit sreening database reated by Chihar San in UCI mahine learning repsitry [14]. We hse this database bease it is easy t nderstand. This database has 125 tples, 10 attribtes and 2 lasses Yes and N representing whether redit has been granted. The meanings f the attribtes appeared belw are self-explanatry. Witht nsidering any ser expeted rle, the set f rles generated by C4.5 is as fllw (the vale in [ ] fllwing eah rle is the predited aray f the rle prded by C4.5): R1: Age > 25, Savings > 7, YR_Wrk > 2 Yes [94.8%] R2: Sex = Male, YR_Wrk > 2 Yes [93.4%] R3: Jbless = N, Bght = p Yes [84.1%] R4: Bght = medinstr, Age <= 34 Yes [82.0%] R5: Sex = Female, Age <= 25 N [56.8%] R6: Savings <= 7, M_LOAN > 7 N [54.6%] R7: YR_Wrk <= 2 N [54.4%] Assme the ser believes that the fllwing rle shld be tre frm his/her experiene:

9 9 Age > 23, M_LOAN <= 7 Yes By lking at the generated rles abve, we have n idea hw rret the ser expeted rle is, and whether there are nfrming rles and nexpeted rles, et., bease the generated rles have little relatinship t the ser s nept. This als makes the generated rles hard t nderstand with respet t the ser s nept. Hwever, if we lk at the rles (belw) prded sing the prpsed tehniqe, these qestins an all be answered easily. Crretness f the expeted rle: Crretness : 73.3% Frm the indiated rretness, we see that the rle is valid abt 73.3% f the time. Generated rles sing the prpsed tehniqe: Rle 1: Sex = Female, Age > 23, Age <= 26 NUnexpeted [75.8%] Rle 2: Jbless = Yes, Age <= 40, M_LOAN <= 7 NUnexpeted [75.8%] Rle 3: Age > 65, M_LOAN <= 7 NUnexpeted [63.0%] Rle 4: Jbless = N, Age > 26, Age <= 65, M_LOAN <= 7 YesCnfrm [95.0%] Rle 5: Jbless = N, Sex = Male, Age > 23, Age <= 65, M_LOAN <= 7 YesCnfrm [87.9%] Rle 6: Jbless = N, Age > 21, Savings > 9, M_LOAN > 7 Yes [81.9%] Rle 7: Bght = p, Age <= 23 Yes [70.7%] Rle 8: Age <= 23, Savings > 10, M_LOAN <= 6 Yes [61.2%] Rle 9: Savings <= 9, M_LOAN > 7 N [61.2%] Rle 10: Age > 32, M_LOAN > 7, YR_Wrk <= 2 N [58.7%] Rle 11: Age <= 23, M_LOAN > 6 N [56.6%] Rle 12: Age <= 23, Savings <= 30, M_LOAN <= 7, Mnths > 8 N [47.5%] Let s make sme bservatins abt this set f rles: The first three rles shw the sitatins where the expeted nditins an lead t nexpeted nlsin. Fr example, fr the tples that satisfy Age > 23 and M_LOAN <= 7.5, a persn is nt granted redit if the persn is female and her age is between 23 and 26 (Rle 1). This is ertainly nexpeted. The next tw rles shw the nfrming sitatins, bt with mre restritive nditins. Fr example, in Rle 4, there are extra nditins n Jbless and Age. After analyzing the first 5 rles, the ser wld have a mh learer pitre abt the database with respet t his/her expetatin. The last 7 rles shw the harateristi desriptins f the tples that are nrelated t the ser expetatin. They may als be interesting bease they are nknwn t the ser. This rle set is relatively easy t nderstand bease the rles are related t the ser s nept. We knw that the first 5 rles all satisfy the nditins f the ser rle, and the last 7 rles d nt. Finally, the new tehniqe tends t prde mre rles and mre nditins in eah rle. This is bease f the extra lasses, whih reqire C4.5 t make finer lassifiatin. Evalatin n the database prdes the fllwing statistis. Used means hw many tples satisfy the rle s nditins, and Wrng means hw many tples are lassified wrngly when they satisfy the rle s nditins.

10 10 Rle Used Wrng Class NUnexpeted NUnexpeted NUnexpeted YesCnfrm YesCnfrm Yes Yes Yes N N N N Nte that these statistis are different frm thse prded by C4.5. In r ase, we are mre interested in eah individal rle rather than hw the whle rle set perfrms in its prediatin as in C4.5. Frthermre, in C4.5, the rdering f rles is imprtant, bt fr s, the rdering is irrelevant. We test every rle against every tple in the database. Frm the table, we bserve that bth the nexpeted rles and the nfrming rles are qite arate. With these rles, the ser may deide t take sme atins. Fr example, he/she may want t find t why yng females are nt granted redit and if this is ndesirable, he/she an prpse rretive predres t ensre that it des nt happen in the ftre. Let s next shw an example (sing the same database) that has mre than ne expeted rle. Sppse the ser has tw expeted rles. Expeted Rle 1: Sex = Female, Bght = Jewel N Expeted Rle 2: Age >= 48, Sex = Female N Crretness f the expeted rles: Crretness: Expeted Rle 1: 28.6% Expeted Rle 2: 46.7% Generated rles sing the prpsed tehniqe: There are 11 generated rles altgether. Belw, we nly list the nfrming and nexpeted rles: Rle 1: Sex = Female, Age > 52 NCnfrm [82.0%] Rle 2: Sex = Female, Age > 47, Age <= 52 YesUnexpeted [84.1%] Rle 3: Bght = Jewel, Sex = Female, Age <= 52 YesUnexpeted [77.7%] Evalatin n the database prdes the fllwing statistis: Rle Used Wrng Class NCnfrm YesUnexpeted YesUnexpeted Rle 1 says that nly when Sex = Female and Age > 52, the persn will nt be granted redit. When the age is between 48 (Age > 47) and 52 (Rle 2), the persn is granted redit, whih is nexpeted. When Bght = Jewel, Sex = Female, and Age <= 52 (Rle 3), the persn is als given redit, whih is als nexpeted. Finally, we shw an example rn sing ne f r real-life disease databases. This database has 713 tples, 9 attribtes, and 2 lasses, YES and NO, representing whether the persn has the disease. Witht nsidering any ser expeted rle, the set f rles generated by C4.5 is as fllw:

11 11 Rle 1: Sex = FEMALE, Age > 50, DBP <= 72 Class = YES [91.7%] Rle 2: Sex = FEMALE, Age > 50, LDL > 4.58, SBP <= 160, DBP <= 82 Class = YES [90.6%] Rle 3: Age > 41, HDL > 1.01, DBP <= 59 Class = YES [79.4%] Rle 4: Age > 62 Class = YES [77.8%] Rle 5: Sex = MALE, Age > 49, LDL > 6.73 Class = YES [75.8%] Rle 6: Sex = FEMALE, Age > 49, LDL > 3.91, LDL <= 4.17 Class = YES [75.8%] Rle 7: Age > 41, LDL > 5.52, SBP > 112, GLUC <= 4.84 Class = YES [75.6%] Rle 8: Age > 49, Age <= 50 Class = YES [75.6%] Rle 9: Age > 49, LDL > 3.24, LDL <= 3.73 Class = YES [70.7%] Rle 10: Age > 32, HDL <= 0.39, TG <= 2.58 Class = YES [64.5%] Rle 11: Age <= 32 Class = NO [99.2%] Rle 12: Age <= 62, LDL <= 3.24 Class = NO [98.1%] Rle 13: Ethni = CHINESE, Age <= 41, HDL > 0.39 LDL <= 5.57, GLUC > 3.85 Class = NO [98.0%] Rle 14: Age <= 49, DBP > 59, TG > 0.61, TG <= 0.84 Class = NO [96.0%] Rle 15: Sex = MALE, HDL <= 0.8, LDL <= 6.73 Class = NO [94.7%] Nw, the dtr believes that the fllwing rle shld be tre frm his/her experiene: Age >= 43, SBP >= 148 Class = YES Again, by lking at the generated rles, we have n idea hw rret the expeted rle is, and whether there are nfrming and nexpeted rles. The rnning reslts sing the prpsed tehniqe are listed belw: Crretness f the expeted rle: Crretness : 44% Frm the indiated rretness, we see that the rle is valid abt 44% f the time. Generated rles sing the prpsed tehniqe: The nfrming and nexpeted rles are listed belw. The ttal nmber f rles prded is 15 (inlding nrelated rles). Rle 1. Age > 61, LDL > 3.42, SBP > 147 Class = YESCnfrm [82.0%] Rle 2. LDL > 3.42, SBP > 168, GLUC <= 5.28 Class = YESCnfrm [79.4%] Rle 3. Age > 42, Age <= 61, LDL > 3.7,SBP > 148, SBP <= 168 Class= NOUnexpeted [79.4%] Rle 4. Age > 42, Age <= 61, SBP > 147, GLUC > 5.28 Class = NOUnexpeted [66.2%] Evalatin n the database prdes the fllwing statistis. Rle Used Wrng Class YESCnfrm YESCnfrm NOUnexpeted NOUnexpeted Frm the abve, we an see that Rle 3 and Rle 4 shw the sitatins where expeted nditins an lead t nexpeted nlsin. The rles are qite arate. Fr example, in Rle 3, a persn that satisfies SBP >= 148 is nt likely t sffer frm the disease if he/she is less than 61 years ld and the LDL measre is greater than 3.7. This is nexpeted. Rle 1 and Rle 2 shw the nfrming sitatins, bt with mre restritive

12 12 nditins. Fr example, in Rle 1, it is nly when the persn s age is greater than 61 years ld and his/her LDL measre is greater than 3.42, then he/she is likely t sffer frm the disease. After analyzing the 4 interesting rles, the ser has a mh better nderstanding f the database with respet t his/her expetatin Disssins Belw, we make sme bservatins abt the prpsed tehniqe and its se. The prpsed tehniqe wrks best if in eah rn there is nly ne expeted rle r all the expeted rles are mtally exlsive (i.e., n tw expeted rles ver a set f mmn tples in the database). In this sitatin, the ser an learly see the nfrming and nexpeted rles with respet t eah individal expeted rle. This may be ineffiient if the database is very large. Hwever, in many appliatins f lassifiatin rle indtin the databases are nt that large. In ertain sitatins, the nmber f nfrming (r nexpeted) rles prded fr eah expeted rle an be large and/r their aray may als be qite lw. This means that the nfrming (r nexpeted) tples vered by the expeted rle may be qite randm, i.e., sattered in a nmber f areas. It may als mean that the nditins sed by the expeted rle are nt disriminating. The nrmal methds [23] an still be sed t test the aray f the nfrming and nexpeted rles n nseen tples if the prpse f finding these rles is fr ftre preditin rather than fr simply nderstanding the reglarities in the existing data. 5. Related Wrk Classifiatin rle indtin is an imprtant tpi in mahine learning. Hwever, the fs f mahine learning has been n generating shrt and arate rles [e.g., 23, 2, 15, 17, 20]. Systems, that emply existing dmain knwledge r thery in the indtin press [e.g., 17, 20], mainly se the knwledge t generate mre arate rles r t imprve the explainability f the rles [1]. Clearly, this is different frm r wrk, where r primary nern is t generate sbjetively interesting rles. A branh f researh in mahine learning that is related t r wrk is thery refinement [12, 15, 24]. In thery refinement, the initial knwledge given is a arse, perhaps inmplete r inrret thery f the prblem dmain. Training examples are sed t shape this initial thery int refined, mre arate thery. Refinement peratrs and hill-limbing searh are nrmally sed t mdify the existing rles. Operatrs an speialize rles (e.g., adding new nditins), generalize rles (e.g., deleting sme seless nditins), remve ld rles and add new rles [24]. There are als ther apprahes t thery refinement, e.g., neral netwrks and integrated methds [e.g., 12, 3]. In r ntext, the ser expeted rles an be seen as the initial thery. Hwever, r wrk is different frm thery refinement bease r fs is n finding sbjetively interesting rles (e.g., nexpeted rles) with regard t the ser s nepts rather than a set f arate rles that frms an effetive thery f the appliatin dmain. The rles prded after thery refinement may n lnger represent the riginal ser s nept spae bease f the refinement peratins (and the errrs in the ser s nepts). These rles still have the same prblem as the ne we disssed in Setin 2 (may be t a lesser degree), i.e., the ser s nept may interset a nmber f rles. Or tehniqe aims t help the ser nderstand his/her existing nepts by arving p his/her existing nept spae int nfrming and/r nexpeted regins (r rles). The prpsed pre-pressing

13 methd, whih simply assigns new lasses t tples (an be in different ways fr different prpses, see Setin 3.1 and 3.2), is als different frm the peratr-based rle mdifiatin apprah and the neral netwrk apprah t thery refinement. In the field f data mining, sbjetive interestingness [e.g., 21, 25, 8, 9, 10, 11, 18, 19] has lng been identified as an imprtant prblem. A nmber f data mining systems have been bilt with pst-analysis mdles [e.g., 8, 9] t help the ser identify interesting rles. Hwever, pst-analysis annt prde the type f interesting rles stdied in this paper bease pst-analysis des nt generate new rles bt simply evalates the interestingness f the disvered rles arding t sme riteria. The interesting rles stdied in this paper need t be generated diretly frm the database. There are als a nmber f apprahes that help the ser finding interesting assiatin rles. [7] prpses a template-based apprah fr finding interesting assiatin rles. In this apprah, the ser speifies interesting and ninteresting assiatin rles sing templates. A template desribes a set f rles in terms f items rred in the nditinal and the nseqent parts. The system then retrieves the mathing rles frm the set f disvered rles. [26] prpses an assiatin rle mining algrithm that an take item nstraints speified by the ser in the rle mining press s that nly thse rles that satisfy the nstraints are generated. [16] extends this apprah frther t allw mh mre sphistiated nstraints t be speified by the ser. It als ses the nstraints t ptimize the assiatin rle mining press. The idea f sing nstraints in the rle mining press is imprtant as it avids generating irrelevant rles. [18, 19] prpses a methd f disvering nexpeted assiatin rles that takes int nsideratin a set f expetatins r beliefs abt the prblem dmain. The methd disvers nexpeted patterns sing these expetatins t seed the searh fr patterns in data that ntradit the beliefs. All the abve methds fr finding interesting assiatin rles annt be sed in r ntext bease assiatin rle mining ld nt se nmeri attribtes, whih ften exist in lassifiatin datasets. Althgh disretizatin an be applied t partitin eah nmeri attribte int intervals befre sing an assiatin rle miner, disretizatin an destry the inherent relatinships f the attribtes sine eah attribte is disretized independently. Sh disretizatin may als prde intervals that make n sense t the ser (ths hard t nderstand). Additinally, assiatin rle mining is mre sitable fr sparse data sh as spermarket transatins. It is nt s sitable fr lassifiatin datasets, whih ften ntain a hge nmber f assiatins and an reslt in mbinatrial explsin [10, 28]. When we have t many rles, it bemes very hard, if nt impssible, fr hman t analyze. [25] prpses t se belief systems t desribe nexpetedness. A nmber f frmal apprahes t the belief systems are presented, e.g., Bayesian prbability and Dempster- Shafer thery. These apprahes reqire the ser t prvide mplex belief infrmatin, sh as nditinal prbabilities, whih are diffilt t btain in pratie. There are als existing tehniqes that wrk in the ntexts f speifi dmains. Fr example, [21] stdies the isse f finding interesting deviatins in a health are appliatin. Its data mining system, KEFIR, analyzes health are infrmatin t nver key findings. A dmain expert system is nstrted t evalate the interestingness (in this ase, atinability) f the key findings. The apprah is, hwever, appliatin speifi. It als des nt deal with assiatin rles. Or methd is general. It des nt make any dmain-speifi assmptins. [27, 11] desribes a tehniqe fr disvering exeptinal knwledge based n infrmatin thery. This tehniqe des nt se any ser s existing nepts. Clearly, it 13

14 14 is different frm r wrk bease we are interested in disvering interesting knwledge with respet t the ser s existing nepts. Rles disvered by r methd may nt be exeptins, bt general fats, whih the ser des nt knw abt r has wrng existing nepts f them. 6. Cnlsin In this paper, we prpsed a nvel and yet simple apprah t inrprating ser s existing nepts in the learning press t disver a speifi type f sbjetively interesting rles. This tehniqe nly invlves pre-pressing f the database, i.e., intrding extra lasses. It des nt mdify the nderlying rle disvery system. The rle indtin pwer f the nderlying rle disvery system itself is emplyed t generate interesting rles. This apprah has been fnd sefl in a nmber f real-life medial appliatins. Referenes [1]. P. Clark and S. Matwin. Using qalitative mdels t gide indtin learning, Preedings f ICML-93, pp , [2]. P. Clark and T. Niblett. The CN2 indtin algrithm. Mahine Learning 3, pp , [3]. S. K. Dnh and L. A. Rendell. Representing and restrtring dmain theries: a nstrtive indtin apprah, Jrnal f Artifiial Intelligene Researh, vl. 2, pp , Jly, [4]. R. R. Evans and D. Fisher, Overming press delays with deisin tree indtin, IEEE Expert, 9(1), pp , [5]. U. Fayyad, G. Piatesky-Shapir and P. Smyth. Frm data mining t knwledge disvery in databases. AI Magzine, pp , [6]. M. Kamber, and R. Shinghal. Evalating the interestingness f harateristi rles, KDD-96, pp , [7]. M. Klemetinen, H. Mannila, P. Rnkainen, H. Tivnen, and A.I. Verkam. Finding interesting rles frm large sets f disvered assiatin rles, Preedings f the Third Internatinal Cnferene n Infrmatin and Knwledge Management, pp , [8]. B. Li and W. Hs. Pst-analysis f learned rles, AAAI-96, pp , [9]. B. Li, W. Hs and S. Chen. Analyzing disvered lassifiatin rles sing general impressins, KDD-97, [10]. B. Li, W. Hs, and Y. Ma. Prning and smmarizing the disvered assiatins. KDD-99, pp , [11]. B. Li, M. H and W. Hs. "Mlti-level rganizatin and smmarizatin f the disvered rles," KDD-2000, [12]. J. J Mahney, and R. J. Mney. Cmparing methds fr refining ertainty-fatr rle bases, Preedings f ICML-94, pp , [13]. J. Majr, and J. Mangan. Seleting amng rles inded frm a hrriane database, KDD-93, pp , [14]. C. J. Merz, and P. Mrphy. UCI repsitry f mahine learning database. [ [15]. R. J. Mney. Indtin ver the nexplained: sing verly-general theries t

15 aid nept learning, Mahine Leaning, vl. 10, pp , [16]. R. Ng. L. Lakshmanan, J. Han, and A. Pang. Explratry mining and prning ptimizatins f nstrained assiatin rles. SIGMOD-98, [17]. J. Ortega and D. Fisher. Flexibly expliting prir knwledge in empirial learning, Preedings f IJCAI-95, pp , [18]. B. Padmanabhan, and A. Tzhilin. A belief-driven methd fr disvering nexpeted patterns. KDD-98, 1998, pp [19]. B. Padmanabhan, and A. Tzhilin. Small is Beatifl: Disvering the Minimal Set f Unexpeted Patterns. KDD-2000, [20]. M. Pazzani and D.Kibler. The tility f knwledge in indtive learning, Mahine learning, vl. 9, pp , [21]. G. Piatesky-Shapir and C. Mathes. The interestingness f deviatins, KDD-94, pp , [22]. G. Piatetsky-Shapir, C. Mathes, P. Smyth, and R. Uthrsamy. KDD-93: prgress and hallenges..., AI magazine, Fall (1994), pp , [23]. J. R. Qinlan. C4.5: prgram fr mahine learning. Mrgan Kafmann, [24]. B. L. Rihard and R. J. Mney. Atmated refinement f first-rder hrn-lase dmain theries, Mahine Learning, vl 19, n. 2, pp , [25]. A. Silbershatz & A. Tzhilin. What makes patterns interesting in knwledge disvery systems, IEEE Transatins n Knwledge and Data Engineering, vl. 8, n. 6, pp , [26]. R. Srikant, Q. V, Q. and R. Agrawal. Mining assiatin rles with item nstraints. KDD-97, 1997, [27]. E. Szki & M. Shimra. Exeptinal knwledge disvery in databases based n infrmatin thery. KDD-96, pp , [28]. M. Zaki, Generating Nnredndant Assiatin Rles, Pr. ACM Int l Cnf. Knwledge Disvery & Data Mining, ACM Press, New Yrk, 2000, pp