Decision Tree Learning. Decision Tree Learning. Example. Decision Trees. Blue slides: Mitchell. Olive slides: Alpaydin Humidity

Transcription

1 Decision Tree Learning Decision Tree Learning Blue slides: Michell Oulook Olive slides: Alpaydin Huidiy Sunny Overcas Rain Wind High ral Srong Weak Learn o approxiae discree-valued arge funcions. Sep-y-sep decision aking: I can learn disjuncive expressions: Hypohesis space is copleely expressive, avoiding proles wih resriced hypohesis spaces. Inducive ias: sall rees over large rees. 2 Exaple Decision Trees Day Oulook Teperaure Huidiy Wind PlayTennis D Sunny Ho High Weak D2 Sunny Ho High Srong D3 Overcas Ho High Weak D4 Rain Mild High Weak D5 Rain Cool ral Weak D6 Rain Cool ral Srong A popular inducive inference algorih. Algorihs: ID3, ASSISTANT, C4.5, ec. Applicaions: edical diagnosis, assess credi risk of loan applicans, ec. D7 Overcas Cool ral Srong D8 Sunny Mild High Weak D9 Sunny Cool ral Weak D0 Rain Mild ral Weak D Sunny Mild ral Srong D2 Overcas Mild High Srong D3 Overcas Ho ral Weak D4 Rain Mild High Srong 3 4

2 Decision Trees: Operaion Oulook 3 Tree Uses des and Leaves Sunny Overcas Rain Huidiy Wind High ral Srong Weak Each insance holds ariue values. Insances are classified y filering he ariue values down he decision ree, down o a leaf which gives he final answer. Inernal nodes: ariue naes or ariue values. Branching occurs a ariue nodes. 5 4 Divide and Conquer Inernal decision nodes Univariae: Uses a single ariue, x i Nueric x i : Binary spli : x i > w Discree x i : n-way spli for n possile values Mulivariae: Uses all ariues, x Leaves Classificaion: Class laels, or proporions Regression: Nueric; r average, or local fi Learning is greedy; find he es spli recursively (Breian e al, 984; Quinlan, 986, 993) Decision Trees: Wha They Represen Sunny Oulook Overcas Rain Huidiy Wind High ral Srong Weak Each pah fro roo o leaf is a conjuncions of consrains on he ariue values. (Oulook = Sunny Huidiy = ral) (Oulook = Overcas) (Oulook = Rain W ind = W eak) 6

3 Appropriae Tasks for Decision Trees Consrucing Decision Trees fro Exaples Good a classificaion proles where: Insances are represened y ariue-value pairs. The arge funcion has discree oupu values. Disjuncive descripions ay e required. Given a se of exaples (raining se), oh posiive and negaive, he ask is o consruc a decision ree ha descries a concise decision pah. Using he resuling decision ree, we wan o classify new insances of exaples (eiher as yes or no). The raining daa ay conain errors. The raining daa ay conain issing ariue values. 7 8 Consrucing Decision Trees: Trivial Soluion Finding a Concise Decision Tree A rivial soluion is o explicily consruc pahs for each given exaple. In his case, you will ge a ree where he nuer of leaves is he sae as he nuer of raining exaples. The prole wih his approach is ha i is no ale o deal wih siuaions where, soe ariue values are issing or new kinds of siuaions arise. Consider ha soe ariues ay no coun uch oward he final classificaion. Meorizing all cases ay no e he es way. We wan o exrac a decision paern ha can descrie a large nuer of cases in a concise way. In ers of a decision ree, we wan o ake as few ess as possile efore reaching a decision, i.e. he deph of he ree should e shallow. 9 0

4 Finding a Concise Decision Tree (con d) Decision Tree Learning Algorih: ID3 Basic idea: pick up ariues ha can clearly separae posiive and negaive cases. These ariues are ore iporan han ohers: he final classificaion heavily depend on he value of hese ariues. Main loop:. A he es decision ariue for nex node 2. Assign A as decision ariue for node 3. For each value of A, creae new descendan of node 4. Sor raining exaples o leaf nodes 5. If raining exaples perfecly classified, Then STOP, Else ierae over new leaf nodes ID3 sands for Ieraive Dichooizer 3 2 Choosing he Bes Ariue A=? [29+,35-] [29+,35-] f A2=? f Choosing he Bes Ariue o Tes Firs Use Shannon s inforaion heory o choose he ariue ha give he axiu inforaion gain. A or A2? [2+,5-] [8+,30-] [8+,33-] [+,2-] Wih iniial and final nuer of posiive and negaive exaples ased on he ariue jus esed, we wan o decide which ariue is eer. Pick an ariue such ha he inforaion gain (or enropy reducion) is axiized. Enropy easures he average surprisal of evens. Less proale evens are ore surprising. How o quaniaively easure which one is eer? 3 4

5 Inforaion Theory (Inforal Inro) Given wo evens, H and T (Head and Tail): Inforaion Theory (Con d).0 Rare (uncerain) evens give ore surprise: H ore surprising han T if P (H) < P (T ) H ore uncerain han T if P (H) < P (T ) Enropy(S) 0.5 How o represen ore surprising, or ore uncerain? Surprise(H) > Surprise(T ) if P (H) < P (T ) P (H) > P (T ) ( ) ( ) log > log P (H) P (T ) log (P (H)) > log (P (T )) p + S is a saple of raining exaples p is he proporion of posiive exaples in S p is he proporion of negaive exaples in S Enropy easures he average uncerainy in S Enropy(S) p log 2 p p log 2 p 6 log (P (X)) as a easure of uncerainy. Uncerainy and Inforaion By perforing soe query, if you go fro sae S wih enropy E(S ) o sae S 2 wih enropy E(S 2 ), where E(S ) > E(S 2 ), your uncerainy has decreased. The aoun y which uncerainy decreased, i.e., E(S ) E(S 2 ), can e hough of as inforaion you gained (inforaion gain) hrough geing answers o your query. Enropy and Code Lengh Enropy(S) = expeced nuer of is needed o encode class ( or ) of randoly drawn eer of S (under he opial, shores-lengh code) Inforaion heory: opial lengh code assigns log 2 p is o essage having proailiy p. Encode wih shor sring for frequen essages (less surprising), and long sring for rarely occurring essages (ore surprising). So, expeced nuer of is o encode or of rando eer of S: p ( log 2 p ) + p ( log 2 p ) Enropy(S) p log 2 p p log 2 p 7 8

6 Enropy and Inforaion Gain Exaple Day Oulook Teperaure Huidiy Wind PlayTennis Enropy(S) = i C P i log 2 (P i ) Gain(S, A) = Enropy(S) C: caegories (classificaions) S: se of exaples A: a single ariue v V alues(a) S v : se of exaples where ariue A = v. X : cardinaliy of arirary se X. S v S Enropy(S v) D Sunny Ho High Weak D2 Sunny Ho High Srong D3 Overcas Ho High Weak D4 Rain Mild High Weak D5 Rain Cool ral Weak D6 Rain Cool ral Srong D7 Overcas Cool ral Srong D8 Sunny Mild High Weak D9 Sunny Cool ral Weak D0 Rain Mild ral Weak D Sunny Mild ral Srong D2 Overcas Mild High Srong D3 Overcas Ho ral Weak D4 Rain Mild High Srong Which ariue o es firs? 9 20 Choosing he Bes Ariue S: [9+,5-] Which ariue is he es classifier? E =0.940 E =0.940 Huidiy S: [9+,5-] Wind Parially Learned Tree {D, D2,..., D4} [9+,5 ] Oulook Sunny Overcas Rain High ral Weak Srong {D,D2,D8,D9,D} {D3,D7,D2,D3} {D4,D5,D6,D0,D4} [2+,3 ] [4+,0 ] [3+,2 ]?? [3+,4-] [6+,-] [6+,2-] [3+,3-] E =0.985 E =0.592 E =0.8 E =.00 Gain (S, Huidiy ) Gain (S, Wind) = (7/4) (7/4).592 =.5 = (8/4).8 - (6/4).0 =.048 +: # of posiive exaples; : # of negaive exaples Iniial enropy = 9 4 log log 5 4 = You can calculae he res. e: 0.0 log even hough log 0.0 is no defined. 2 Which ariue should e esed here? S sunny = {D,D2,D8,D9,D} Gain (Ssunny, Huidiy) =.970 (3/5) 0.0 (2/5) 0.0 =.970 Gain (S sunny, Teperaure) =.970 (2/5) 0.0 (2/5).0 (/5) 0.0 =.570 Gain (S sunny, Wind) =.970 (2/5).0 (3/5).98 =.09 Selec nex ariue, ased on he reaining exaples. 22

7 Hypohesis Space Search in ID3 Hypohesis Space Search in ID3 + + Hypohesis space is coplee! Targe funcion surely in here A A Oupus a single hypohesis (which one?) Can play 20 quesions... A2 A2 ack racking + + A A4 Local inia Saisically-ased search choices A each ranch, we ake a decision regarding a paricular ariue. Choice of an ariue direcs he search oward a cerain final hypohesis. Rous o noisy daa... Inducive ias: approx prefer shores ree Inducive Bias in ID3 Accuracy of Decision Trees ID3 is iased: ecause of he resricion on he hypohesis space, u Because of he preference for a paricular hypohesis. Such an inducive ias is called Occa s razor: The os likely hypohesis is he siples one ha is consisen wih all oservaions. Divide exaples ino raining and es ses. Train using he raining se. Measure accuracy of resuling decision ree on he es se

8 Issue: Overfiing Issue: ise Oulook Accuracy Huidiy Sunny Overcas Rain Wind On raining daa On es daa High ral Srong Weak Size of ree (nuer of nodes) Overfiing: Given a hypohesis space H, a hypohesis h H is said o overfi he raining daa if here exiss soe alernaive hypohesis h H such ha h is worse han h on he raining se u h is eer han h over he enire disriuion of insances. Can e due o noise in daa. 27 Overcoing Overfiing Sop early. Allow overfiing, hen pos-prune ree. Use separae se of exaples no used in raining o onior perforance on unoserved daa (validaion se). Use all availale daa, u perfor saisical es o esiae chance of iproving. Use explici easure of coplexiy of encoding, and pu a ound on ree size Wha if Oulook = Sunny, T ep = Ho, Huidiy = ral, W ind = Srong, P lay = was added as a raining exaple? Furher elaoraion of he aove ree ecoes necessary. The resuling ree will fi he raining daa plus he noise, u i ay perfor poorly on he rue insance disriuion. 28 Regression Trees Error a node : E x N Afer spliing: j E' x if xx : xreachesnode 0 oherwise N 2 r g x g 2 r g j j j j x g x r x if xxj : x reachesnode and ranch j 0 oherwis e j x r x j

9 Model Selecion in Trees 0 Pruning Trees Reove surees for eer generalizaion (decrease variance) Prepruning: Early sopping Pospruning: Grow he whole ree hen prune surees ha overfi on he pruning se Prepruning is faser, pospruning is ore accurae (requires a separae pruning se) 9 Rule Exracion fro Trees Learning Rules 2 C4.5Rules (Quinlan, 993) Rule inducion is siilar o ree inducion u ree inducion is readh-firs, rule inducion is deph-firs; one rule a a ie Rule se conains rules; rules are conjuncions of ers Rule covers an exaple if all ers of he rule evaluae o rue for he exaple Sequenial covering: Generae rules one a a ie unil all posiive exaples are covered IREP (Fürnkranz and Wider, 994), Ripper (Cohen, 995)

10 Oher Issues Coninuous-valued ariues: dynaically define new discree-valued ariues Muli-valued ariues wih large nuer of possile values: Use easures oher han inforaion gain. Training exaples wih issing ariue values: Assign os coon value, or assign wih he occurring frequency. Ariues wih differen cos/weighing: Scale using he cos. 30