Classification: Rules. Prof. Pier Luca Lanzi Laurea in Ingegneria Informatica Politecnico di Milano Polo regionale di Como

Transcription

1 Metodologie per Sistemi Intelligenti Clssifiction: Prof. Pier Luc Lnzi Lure in Ingegneri Informtic Politecnico di Milno Polo regionle di Como Rules

2 Lecture outline Why rules? Wht re clssifiction rules? Which type of rules? Sequentil Covering Algorithms

3 Why rules? One of the most expressive nd most humn redle representtion for hypotheses is sets of IF-THEN rules

4 The Wether dtset gin! No True High Mild Riny Flse Norml Hot Overcst True High Mild Overcst True Norml Mild Sunny Flse Norml Mild Riny Flse Norml Cool Sunny No Flse High Mild Sunny True Norml Cool Overcst No True Norml Cool Riny Flse Norml Cool Riny Flse High Mild Riny Flse High Hot Overcst No True High Hot Sunny No Flse High Hot Sunny Ply Windy Humidity Temp Outlook

5 Clssifiction rules for the Wether dtset Rule 1: outlook = overcst -> clss Ply [70.7%] Rule 2: outlook = rin windy = flse -> clss Ply [63.0%] Rule 3: outlook = sunny humidity = high -> clss Don't Ply [63.0%] Rule 4: outlook = rin windy = true -> clss Don't Ply [50.0%] Defult clss: Ply

6 Wht re clssifiction rules? They re IF-THEN rules The IF prt sttes condition over the dt The THEN prt includes clss lel Which type of conditions? Propositionl, with ttriute-vlue comprisons First order Horn cluses, with vriles

7 Wht re the methods? Method 1: lern decision tree, then convert to rules Method 2: sequentil covering lgorithm

8 Sequentil covering lgorithms Consider the set E of positive nd negtive exmples Repet Lern one rule with high ccurcy, ny coverge Remove positive exmples covered y this rule Until ll the exmples re covered

9 Sequentil covering lgorithms

10 Exploring the Hypothesis Spce Generl to Specific Strt with the most generl hypothesis nd then go on through speciliztion steps Specific to Generl Strt with the set of the most specific hypothesis nd then go on through generliztion steps

11 Lern-one-rule

12 Exmple: generting rule y x If true then clss =

13 Exmple: generting rule, II y x y x 1 2 If x > 1.2 then clss = If true then clss =

14 Exmple: generting rule, III y x y 1 2 x y x If true then clss = If x > 1.2 nd y > 2.6 then clss = If x > 1.2 then clss =

15 Exmple: generting rule, IV y x y 1 2 x y x If true then clss = If x > 1.2 nd y > 2.6 then clss = If x > 1.2 then clss = Possile rule set for clss : If x 1.2 then clss = If x > 1.2 nd y 2.6 then clss = More rules could e dded for perfect rule set

16 Rules vs. trees Corresponding decision tree: (produces exctly the sme predictions) But: rule sets cn e more cler when decision trees suffer from replicted sutrees Also: in multi-clss situtions, covering lgorithm concentrtes on one clss t time wheres decision tree lerner tkes ll clsses into ccount

17 Rules vs. trees Sequentil covering genertes rule y dding tests tht mximize rule s ccurcy Similr to sitution in decision trees: prolem of selecting n ttriute to split on But decision tree inducer mximizes overll purity Ech new test reduces rule s coverge spce of exmples rule so fr rule fter dding new term

18 Lern-one-rule Strt from the most generl rule, consisting of n empty condition Add tests on single ttriutes until the performnce (the ccurcy) improves

19 Lern-one-rule

20 Exploring the hypothesis spce The lgorithm to explore the hypothesis spce is greedy nd might tend to locl optim To improve the explortion of the hypothesis spce, we cn em serch: t ech step k cndidte hypotheses re considered.

21 Another exmple: contct lens dt Rule we seek: Possile tests: If? then recommendtion = hrd Age = Young Age = Pre-presyopic Age = Presyopic Spectcle prescription = Myope Spectcle prescription = Hypermetrope Astigmtism = no Astigmtism = yes Ter production rte = Reduced Ter production rte = Norml 2/8 1/8 1/8 3/12 1/12 0/12 4/12 0/12 4/12

22 Modified rule nd resulting dt Rule with est test dded: Instnces covered y modified rule: Age Young Young Young Young Pre-presyopic Pre-presyopic Pre-presyopic Pre-presyopic Presyopic Presyopic Presyopic Presyopic Spectcle prescription Myope Myope Hypermetrope Hypermetrope Myope Myope Hypermetrope Hypermetrope Myope Myope Hypermetrope Hypermetrope Astigmtism If stigmtism = yes then recommendtion = hrd Ter production rte Reduced Norml Reduced Norml Reduced Norml Reduced Norml Reduced Norml Reduced Norml Recommended lenses None Hrd None hrd None Hrd None None None Hrd None None

23 Further refinement Current stte: If stigmtism = yes nd? then recommendtion = hrd Possile tests: Age = Young Age = Pre-presyopic Age = Presyopic Spectcle prescription = Myope Spectcle prescription = Hypermetrope Ter production rte = Reduced Ter production rte = Norml 2/4 1/4 1/4 3/6 1/6 0/6 4/6

24 Modified rule nd resulting dt Rule with est test dded: If stigmtism = yes nd ter production rte = norml then recommendtion = hrd Instnces covered y modified rule: Age Young Young Pre-presyopic Pre-presyopic Presyopic Presyopic Spectcle prescription Myope Hypermetrope Myope Hypermetrope Myope Hypermetrope Astigmtism Ter production rte Norml Norml Norml Norml Norml Norml Recommended lenses Hrd hrd Hrd None Hrd None

25 Further refinement Current stte: If stigmtism = yes nd ter production rte = norml nd? then recommendtion = hrd Possile tests: Age = Young Age = Pre-presyopic Age = Presyopic Spectcle prescription = Myope Spectcle prescription = Hypermetrope Tie etween the first nd the fourth test We choose the one with greter coverge 2/2 1/2 1/2 3/3 1/3

26 The result Finl rule: If stigmtism = yes nd ter production rte = norml nd spectcle prescription = myope then recommendtion = hrd Second rule for recommending hrd lenses : (uilt from instnces not covered y first rule) If ge = young nd stigmtism = yes nd ter production rte = norml then recommendtion = hrd These two rules cover ll hrd lenses : Process is repeted with other two clsses

27 Selecting test Gol: mximize ccurcy t totl numer of instnces covered y rule p positive exmples of the clss covered y rule t p numer of errors mde y rule Simple pproch: select test tht mximizes the rtio p/t We re finished when p/t = 1 or the set of instnces cn t e split ny further

28 *Test selection criteri Bsic covering lgorithm: keep dding conditions to rule to improve its ccurcy Add the condition tht improves ccurcy the most Mesure 1: p/t t totl instnces covered y rule p numer of these tht re positive Produce rules tht don t cover negtive instnces, s quickly s possile My produce rules with very smll coverge specil cses or noise? Mesure 2: Informtion gin p (log(p/t) log(p/t)) P nd T the positive nd totl numers efore the new condition ws dded Informtion gin emphsizes positive rther thn negtive instnces These interct with the pruning mechnism used

29 *Missing vlues, numeric ttriutes Common tretment of missing vlues: for ny test, they fil Algorithm must either use other tests to seprte out positive instnces leve them uncovered until lter in the process In some cses it s etter to tret missing s seprte vlue Numeric ttriutes re treted just like they re in decision trees

30 *Pruning rules Two min strtegies: Incrementl pruning Glol pruning Other difference: pruning criterion Error on hold-out set (reduced-error pruning) Sttisticl significnce MDL principle Also: post-pruning vs. pre-pruning

31 The RISE lgorithm It works in specific-to-generl pproch Initilly, it cretes one rule for ech trining exmple Then it goes on through elementry generliztion steps until the overll ccurcy does not decrese

32 The RISE lgorithm Input: ES is the trining set Let RS e ES Compute Acc(RS) Repet For ech rule R in RS, find the nerest exmple E not covered y R (E is of the sme clss s R) R = MostSpecificGenerliztion(R,E) RS = RS with R replced y R if (Acc(RS ) Acc(RS)) then RS=RS if R is identicl to nother rule in RS then,delete R from RS Until no increse in Acc(RS) is otined

33 Inferring rudimentry rules 1R: lerns 1-level decision tree I.e., rules tht ll test one prticulr ttriute Bsic version One rnch for ech vlue Ech rnch ssigns most frequent clss Error rte: proportion of instnces tht don t elong to the mjority clss of their corresponding rnch Choose ttriute with lowest error rte (ssumes nominl ttriutes)

34 Pseudo-code for 1R For ech ttriute, For ech vlue of the ttriute, mke rule s follows: count how often ech clss ppers find the most frequent clss mke the rule ssign tht clss to this ttriute-vlue Clculte the error rte of the rules Choose the rules with the smllest error rte Note: missing is treted s seprte ttriute vlue

35 Evluting the wether ttriutes Outlook Sunny Sunny Overcst Riny Riny Riny Overcst Sunny Sunny Riny Sunny Temp Hot Hot Hot Mild Cool Cool Cool Mild Cool Mild Mild Humidity High High High High Norml Norml Norml High Norml Norml Norml Windy Flse True Flse Flse Flse True True Flse Flse Flse True Ply No No No No Attriute Outlook Temp Humidity Windy Rules Sunny No Overcst Riny Hot No* Mild Cool High No Norml Flse True No* Errors 2/5 0/4 2/5 2/4 2/6 1/4 3/7 1/7 2/8 3/6 Totl errors 4/14 5/14 4/14 5/14 Overcst Overcst Mild Hot High Norml True Flse * indictes tie Riny Mild High True No

36 Deling with numeric ttriutes Discretize numeric ttriutes Divide ech ttriute s rnge into intervls Sort instnces ccording to ttriute s vlues Plce rekpoints where the clss chnges (the mjority clss) This minimizes the totl error

37 Deling with numeric ttriutes Exmple: temperture from wether dt Outlook Temperture Humidity Windy Ply Sunny Flse No Sunny True No Overcst Flse Riny Flse No No No No No

38 The prolem of overfitting This procedure is very sensitive to noise One instnce with n incorrect clss lel will proly produce seprte intervl Also: ID -like ttriutes will hve zero errors Simple solution: enforce minimum numer of instnces in mjority clss per intervl

39 Discretiztion exmple Exmple (with min = 3): No No No No No Finl result for temperture ttriute No No No No No

40 With overfitting voidnce Resulting rule set: Attriute Rules Errors Totl errors Outlook Sunny No 2/5 4/14 Overcst 0/4 Riny 2/5 Temperture /10 5/14 > 77.5 No* 2/4 Humidity /7 3/14 > 82.5 nd 95.5 No 2/6 > /1 Windy Flse 2/8 5/14 True No* 3/6

41 Discussion of 1R 1R ws descried in pper y Holte (1993) Contins n experimentl evlution on 16 dtsets (using cross-vlidtion so tht results were representtive of performnce on future dt) Minimum numer of instnces ws set to 6 fter some experimenttion 1R s simple rules performed not much worse thn much more complex decision trees Simplicity first pys off!

42 The Monks 1 dtset A1 A2 A3 A4 A5 A5 CLASS

43 Monks 1: decision tree ttriute#5 = 1: 1 (29.0/1.4) ttriute#5 = 2: 0 (31.0/13.4) ttriute#5 = 3: ttriute#6 = 1: 0 (13.0/4.7) ttriute#6 = 2: ttriute#3 = 1: 1 (7.0/3.4) ttriute#3 = 2: 0 (10.0/4.6) ttriute#5 = 4: ttriute#1 = 1: 0 (14.0/2.5) ttriute#1 = 2: ttriute#2 = 1: 0 (6.0/1.2) ttriute#2 = 2: 1 (4.0/1.2) ttriute#2 = 3: 0 (1.0/0.8) ttriute#1 = 3: ttriute#2 = 1: 1 (0.0) ttriute#2 = 2: 0 (3.0/1.1) ttriute#2 = 3: 1 (6.0/1.2)

44 Monks 1: clssifiction rules Rule 1: ttriute#5 = 1 -> clss 1 [95.3%] Rule 20: ttriute#1 = 3 ttriute#2 = 3 -> clss 1 [92.2%] Rule 17: ttriute#1 = 2 ttriute#2 = 2 -> clss 1 [91.2%] Rule 7: ttriute#1 = 1 ttriute#2 = 1 -> clss 1 [85.7%] Rule 14: ttriute#1 = 1 ttriute#5 = 4 -> clss 0 [82.2%] Rule 16: ttriute#1 = 2 ttriute#2 = 1 ttriute#5 = 4 -> clss 0 [79.4%] Defult clss: 0

45 Monks 1: nother solution (A1 = A2) OR (A5=1) Decision trees nd clssifiction rules do not include vriles, ut only propositions

46 Wht re Horn cluses? An Horn cluse is n expression of the form: H L 1 L 2 L 3 L n Where H, L 1 L n re positive literls (predictes pplied to set of terms) H is clled hed or consequent L 1 L 2 L 3 L n is clled ody or ntecedents Exmple: dughter(x,y) fther(y,x) femle(x)

47 Lerning First Order Rules: FOIL Extends the typicl sequentil covering lgorithms to the lerning of first order rules, or Horn cluses The most known lgorithm is FOIL tht employs n pproch very similr to the sequentil-covering nd lern-one-rule lgorithms. FOIL rules re more restricted thn generl Horn cluses (literls cnnot contin functions) FOIL rules re more expressive thn Horn cluses ecuse the literls ppering in the ody my e negted

48 The FOIL lgorithm

49 Monks 1: FOIL Cluse 0: is_0(a,b,c,d,e,f) :- A<>B, E<>A1.

50 Other pproches Sequentil-covering lgorithms re one of the possile pproches Clssifiction rules cn e lso derived from other representtions such s Decision trees Assocition Rules Neurl Networks Alterntively, clssifiction rules cn e derived through other serch pproches, such s Genetic Algorithms nd Genetic Progrmming.

51 Summry Clssifiction rules re used ecuse more expressive nd more humn redle Most of the lgorithms re sed on sequentil covering which cn e used oth to derive propositionl nd first order rules Other pproches exist Specific to generl explortion (RISE) Post processing of neurl networks, ssocition rules, decision trees, etc.

52 References Roert C. Holte, Very Simple Clssifiction Rules Perform Well on Most Commonly Used Dtsets. Computer Science Deprtment, University of Ottw Two-Wy Induction. Proceedings of the Seventh IEEE Interntionl Conference on Tools with Artificil Intelligence (pp ), Herndon, VA: IEEE Computer Society Press. Rule Induction nd Instnce-Bsed Lerning: A Unified Approch. Proceedings of the Fourteenth Interntionl Joint Conference on Artificil Intelligence (pp ), Montrel, Cnd: Morgn Kufmnn. The RISE System: Conquering Without Seprting. Proceedings of the Sixth IEEE Interntionl Conference on Tools with Artificil Intelligence (pp ), New Orlens, LA: IEEE Computer Society Press.