Decision Tree Learning. Decision Tree Learning. Decision Trees. Decision Trees: Operation. Blue slides: Mitchell. Orange slides: Alpaydin Humidity

Size: px

Start display at page:

Download "Decision Tree Learning. Decision Tree Learning. Decision Trees. Decision Trees: Operation. Blue slides: Mitchell. Orange slides: Alpaydin Humidity"

Kelly Price
6 years ago
Views:

1 Decision Tree Learning Decision Tree Learning Blue slides: Michell Oulook Orange slides: Alpaydin Huidiy Sunny Overcas Rain ral Srong Learn o approxiae discree-valued arge funcions. Sep-by-sep decision aking: I can learn disjuncive expressions: Hypohesis space is copleely expressive, avoiding probles wih resriced hypohesis spaces. Inducive bias: sall rees over large rees. 2 Decision Trees Decision Trees: Operaion Oulook A popular inducive inference algorih. Algorihs: ID3, ASSISTANT, C4.5, ec. Applicaions: edical diagnosis, assess credi risk of loan Sunny Huidiy Overcas Rain applicans, ec. ral Srong Each insance holds aribue values. Insances are classified by filering he aribue values down he decision ree, down o a leaf which gives he final answer. Inernal nodes: aribue naes or aribue values. Branching occurs a aribue nodes. 3 4

2 Tree Uses des, and Leaves Divide and Conquer Inernal decision nodes Univariae: Uses a single aribue, x i Nueric x i : Binary spli : x i > w Discree x i : n-way spli for n possible values Mulivariae: Uses all aribues, x Leaves Classificaion: Class labels, or proporions Regression: Nueric; r average, or local fi Learning is greedy; find he bes spli recursively (Breian e al, 984; Quinlan, 986, 993) ecure es for E Alpaydın 2004 Inroducion o Machine Learning The MIT Press (V.) 3 Lecure es for E Alpaydın 2004 Inroducion o Machine Learning The MIT Press (V.) 4 Decision Trees: Wha They Represen Appropriae Tasks for Decision Trees Oulook Sunny Overcas Rain Good a classificaion probles where: Huidiy Insances are represened by aribue-value pairs. The arge funcion has discree oupu values. ral Srong Disjuncive descripions ay be required. Each pah fro roo o leaf is a conjuncions of consrains on he aribue values. (Oulook = Sunny Huidiy = ral) (Oulook = Overcas) (Oulook = Rain W ind = W eak) The raining daa ay conain errors. The raining daa ay conain issing aribue values. 5 6

3 Consrucing Decision Trees fro Exaples Consrucing Decision Trees: Trivial Soluion Given a se of exaples (raining se), boh posiive and negaive, he ask is o consruc a decision ree ha describes a concise decision pah. Using he resuling decision ree, we wan o classify new insances of exaples (eiher as yes or no). A rivial soluion is o explicily consruc pahs for each given exaple. In his case, you will ge a ree where he nuber of leaves is he sae as he nuber of raining exaples. The proble wih his approach is ha i is no able o deal wih siuaions where, soe aribue values are issing or new kinds of siuaions arise. Consider ha soe aribues ay no coun uch oward he final classificaion. 7 8 Finding a Concise Decision Tree Finding a Concise Decision Tree (con d) Meorizing all cases ay no be he bes way. We wan o exrac a decision paern ha can describe a large nuber of cases in a concise way. In ers of a decision ree, we wan o ake as few ess as possible before reaching a decision, i.e. he deph of he ree should be shallow. Basic idea: pick up aribues ha can clearly separae posiive and negaive cases. These aribues are ore iporan han ohers: he final classificaion heavily depend on he value of hese aribues. 9 0

4 Main loop: Decision Tree Learning Algorih: ID3. A he bes decision aribue for nex node Choosing he Bes Aribue A=? [29+,35-] [29+,35-] f A2=? f 2. Assign A as decision aribue for node [2+,5-] [8+,30-] [8+,33-] [+,2-] 3. For each value of A, creae new descendan of node 4. Sor raining exaples o leaf nodes 5. If raining exaples perfecly classified, Then STOP, Else ierae over new leaf nodes A or A2? Wih iniial and final nuber of posiive and negaive exaples based on he aribue jus esed, we wan o decide which aribue is beer. How o quaniaively easure which one is beer? 2 Choosing he Bes Aribue o Tes Firs Use Shannon s inforaion heory o choose he aribue ha give he axiu inforaion gain. Pick an aribue such ha he inforaion gain (or enropy reducion) is axiized. Enropy easures he average surprisal of evens. Less probable evens are ore surprising. 3 Inforaion Theory (Inforal Inro) Given wo evens, H and T (Head and Tail): Rare (uncerain) evens give ore surprise: H ore surprising han T if P (H) < P (T ) H ore uncerain han T if P (H) < P (T ) How o represen ore surprising, or ore uncerain? Surprise(H) > Surprise(T ) if P (H) < P (T ) P (H) > P (T ) ««log > log P (H) P (T ) log (P (H)) > log (P (T )) 4

5 Inforaion Theory (Con d).0 Uncerainy and Inforaion Enropy(S) 0.5 By perforing soe query, if you go fro sae S wih enropy E(S ) o sae S 2 wih enropy E(S 2 ), where E(S ) > E(S 2 ), your uncerainy has decreased p + S is a saple of raining exaples p is he proporion of posiive exaples in S p is he proporion of negaive exaples in S Enropy easures he average uncerainy in S The aoun by which uncerainy decreased, i.e., E(S ) E(S 2 ), can be hough of as inforaion you gained (inforaion gain) hrough geing answers o your query. Enropy(S) p log 2 p p log 2 p 5 6 Enropy and Code Lengh Enropy(S) = expeced nuber of bis needed o encode class ( or ) of randoly drawn eber of S (under he opial, shores-lengh code) Inforaion heory: opial lengh code assigns log 2 p bis o essage having probabiliy p. Encode wih shor sring for frequen essages (less surprising), and long sring for rarely occurring essages (ore surprising). So, expeced nuber of bis o encode or of rando eber of S: p ( log 2 p ) + p ( log 2 p ) Enropy and Inforaion Gain Enropy(S) = X i C P i log 2 (P i ) Gain(S, A) = Enropy(S) C: caegories (classificaions) S: se of exaples A: a single aribue X v V alues(a) S v : se of exaples where aribue A = v. X : cardinaliy of arbirary se X. S v E Enropy(S v) Enropy(S) p log 2 p p log 2 p 7 8

6 Exaple Day Oulook Teperaure Huidiy PlayTennis Choosing he Bes Aribue Which aribue is he bes classifier? D Sunny Ho D2 Sunny Ho Srong D3 Overcas Ho D4 Rain Mild D5 Rain Cool ral D6 Rain Cool ral Srong S: [9+,5-] E =0.940 E =0.940 Huidiy ral S: [9+,5-] Srong D7 Overcas Cool ral Srong D8 Sunny Mild D9 Sunny Cool ral D0 Rain Mild ral D Sunny Mild ral Srong D2 Overcas Mild Srong D3 Overcas Ho ral D4 Rain Mild Srong Which aribue o es firs? 9 [3+,4-] [6+,-] [6+,2-] [3+,3-] E =0.985 E =0.592 E =0.8 E =.00 Gain (S, Huidiy ) Gain (S, ) = (7/4) (7/4).592 =.5 = (8/4).8 - (6/4).0 =.048 +: # of posiive exaples; : # of negaive exaples Iniial enropy = 9 4 log log 5 4 = You can calculae he res. e: 0.0 log even hough log 0.0 is no defined. 20 Parially Learned Tree Hypohesis Space Search in ID3 {D, D2,..., D4} [9+,5 ] Oulook + + Sunny Overcas Rain {D,D2,D8,D9,D} {D3,D7,D2,D3} {D4,D5,D6,D0,D4} [2+,3 ] [4+,0 ] [3+,2 ]?? A A Which aribue should be esed here? S sunny = {D,D2,D8,D9,D} Gain (Ssunny, Huidiy) =.970 (3/5) 0.0 (2/5) 0.0 =.970 Gain (S sunny, Teperaure) =.970 (2/5) 0.0 (2/5).0 (/5) 0.0 =.570 Gain (S sunny, ) =.970 (2/5).0 (3/5).98 =.09 A2 + + A3 + A2 + + A Selec nex aribue, based on he reaining exaples. A each branch, we ake a decision regarding a paricular aribue. Choice of an aribue direcs he search oward a cerain final hypohesis. 2 22

7 Hypohesis Space Search in ID3 Inducive Bias in ID3 Hypohesis space is coplee! Targe funcion surely in here... Oupus a single hypohesis (which one?) Can play 20 quesions... back racking Local inia... ID3 is biased: because of he resricion on he hypohesis space, bu Because of he preference for a paricular hypohesis. Such an inducive bias is called Occa s razor: The os likely hypohesis is he siples one ha is consisen wih all observaions. Saisically-based search choices Robus o noisy daa... Inducive bias: approx prefer shores ree Accuracy of Decision Trees Divide exaples ino raining and es ses. Train using he raining se. Measure accuracy of resuling decision ree on he es se. Accuracy Issue: Overfiing On raining daa On es daa Size of ree (nuber of nodes) Overfiing: Given a hypohesis space H, a hypohesis h H is said o overfi he raining daa if here exiss soe alernaive hypohesis h H such ha h is worse han h on he raining se bu h is beer han h over he enire disribuion of insances. Can be due o noise in daa

8 Issue: ise Oulook Overcoing Overfiing Sunny Overcas Rain Sop early. Huidiy Allow overfiing, hen pos-prune ree. ral Srong Wha if Oulook = Sunny, T ep = Ho, Huidiy = ral, W ind = Srong, P lay = was added as a raining exaple? Use separae se of exaples no used in raining o onior perforance on unobserved daa (validaion se). Use all available daa, bu perfor saisical es o esiae chance of iproving. Use explici easure of coplexiy of encoding, and pu a bound on ree size. Furher elaboraion of he above ree becoes necessary. The resuling ree will fi he raining daa plus he noise, bu i ay perfor poorly on he rue insance disribuion Regression Trees Model Selecion in Trees: Error a node : if x X : x reaches node b( x) = 0 oherwise E E' = = N N 2 ( r g ) b ( x ) g = 2 ( r gj ) bj ( x ) gj = j b b ( x ) r ( x ) Afer spliing: if x Xj : x reaches node and branch j bj ( x) = 0 oherwise b j b ( x ) r ( x ) j 8 9

Alpaydın 2004 Inroducion o Machine Learning The MIT Press (V.) 0 Lecure es for E Alpaydın 2004 Inroducion o Machine Learning The MIT Press (V.

9 Pruning Trees Rule Exracion fro Trees Reove subrees for beer generalizaion (decrease variance) Prepruning: Early sopping C4.5Rules (Quinlan, 993) Pospruning: Grow he whole ree hen prune subrees which overfi on he pruning se Prepruning is faser, pospruning is ore accurae (requires a separae pruning se) ecure es for E Alpaydın 2004 Inroducion o Machine Learning The MIT Press (V.) 0 Lecure es for E Alpaydın 2004 Inroducion o Machine Learning The MIT Press (V.) Oher Issues Learning Rules Coninuous-valued aribues: dynaically define new discree-valued aribues Rule inducion is siilar o ree inducion bu Muli-valued aribues wih large nuber of possible values: Use ree inducion is breadh-firs, easures oher han inforaion gain. rule inducion is deph-firs; one rule a a ie Rule se conains rules; rules are conjuncions of ers Rule covers an exaple if all ers of he rule evaluae o rue for he exaple Training exaples wih issing aribue values: Assign os coon value, or assign wih he occurring frequency. Aribues wih differen cos/weighing: Scale using he cos. Sequenial covering: Generae rules one a a ie unil all posiive exaples are covered IREP (Fürnkranz and Wider, 994), Ripper (Cohen, 995) 2 29

Decision Tree Learning. Decision Tree Learning. Example. Decision Trees. Blue slides: Mitchell. Olive slides: Alpaydin Humidity

Decision Tree Learning. Decision Tree Learning. Example. Decision Trees. Blue slides: Mitchell. Olive slides: Alpaydin Humidity Decision Tree Learning Decision Tree Learning Blue slides: Michell Oulook Olive slides: Alpaydin Huidiy Sunny Overcas Rain Wind High ral Srong Weak Learn o approxiae discree-valued arge funcions. Sep-y-sep