Decision Tree Learning

Topics Decision Tree Learning Sattiraju Prabhakar CS898O: DTL Wichita State University What are decision trees? How do we use them? New Learning Task ID3 Algorithm Weka Demo C4.5 Algorithm Weka Demo Implementation 2/6/2006 ML_S2006_DecisionTreeLearning 2 What are Decision Trees? Elements of Decision Tree Attribute value Attribute Yes Yes No 2/6/2006 ML_S2006_DecisionTreeLearning 3 2/6/2006 ML_S2006_DecisionTreeLearning 4 1

Example: PlayTennis Example Decision Tree 2/6/2006 ML_S2006_DecisionTreeLearning 5 2/6/2006 ML_S2006_DecisionTreeLearning 6 Example: Contact Lens data Example: Labour Negotiations (a) (b) 2/6/2006 ML_S2006_DecisionTreeLearning 7 2/6/2006 ML_S2006_DecisionTreeLearning 8 2

Exercise: Animal Decision tree for a simple disjunction. Example Vertibral Column Make_So und Legs Take_Food Animal Man Yes Talking 2 Cultivation Yes Mango Tree No Branch Moveme nt 1 Using_ Chlorophyll No Lizard Yes Using 4 Catches Yes Tongue Flies Parameciu m No No Sound 0 Absorbs Cells Yes Flytrap No No Sound 0 Catches Flies No 2/6/2006 ML_S2006_DecisionTreeLearning 9 2/6/2006 ML_S2006_DecisionTreeLearning 10 Structure of a Decision Tree New Learning Task Structure: Is a disjunction of conjunctions on the attribute values of instances Each path from tree root to a leaf corresponds to a conjunction of attribute tests Tree itself is a disjunction of thee conjunctions Example: PlayTennis Decision Tree (Outlook=Sunny Humidity=Normal) (Outlook = Overcast) (Outlook = Rain Wind = Weak) 2/6/2006 ML_S2006_DecisionTreeLearning 11 2/6/2006 ML_S2006_DecisionTreeLearning 12 3

Learner + Performance Element Using Decision Tree by Performance Element Training Examples Test Examples Learner Learnt Model: Decision Tree Performance Element Yes/ No Decision New Situation or example Decision Tree Classifier Yes/ No Decision 2/6/2006 ML_S2006_DecisionTreeLearning 13 2/6/2006 ML_S2006_DecisionTreeLearning 14 Characterizing Learning Task Strengths of DTL It should be possible to represent instances by attribute-value pairs Target Function should have discrete values It should be possible to represent Target Functions as disjunctive expressions Training data may contain errors Even when some attribute value pairs are missing in instance descriptions, we can use them 2/6/2006 ML_S2006_DecisionTreeLearning 15 2/6/2006 ML_S2006_DecisionTreeLearning 16 4

Decision Tree Learning Algorithm Decision Tree Learning Algorithm Topics: Different ways of partitioning instance space Entropy and Information Gain ID3 algorithm Topics: Different ways of partitioning instance space Entropy and Information Gain ID3 algorithm 2/6/2006 ML_S2006_DecisionTreeLearning 17 2/6/2006 ML_S2006_DecisionTreeLearning 18 Different ways of partitioning the instance space. Decision Tree stumps for the weather data. (a) (b) (a) (b) (c) (d) (c) (d) 2/6/2006 ML_S2006_DecisionTreeLearning 19 2/6/2006 ML_S2006_DecisionTreeLearning 20 5

Expanded tree stumps for the weather data. Decision tree for the weather data. (a) (b) (c) 2/6/2006 ML_S2006_DecisionTreeLearning 21 2/6/2006 ML_S2006_DecisionTreeLearning 22 Operation of a covering algorithm; decision tree for the same problem Example of building a partial tree. y b a b b b a b b a a a b b b a b b b b b y b a b b a b b b a a a b b b a b b b b b y 2 6 b a b b b a b b a a a b b b a b b b b b (a) (c) x 1 2 x 1 2 x (a) (b) (b) 2/6/2006 ML_S2006_DecisionTreeLearning 23 2/6/2006 ML_S2006_DecisionTreeLearning 24 6

Decision Tree Learning Algorithm Attribute Selection Topics: Different ways of partitioning instance space Attribute Selection: Entropy and Information Gain ID3 algorithm To build a decision tree, we need to select an attribute Selection of different attributes at different points lead to different decision trees Two measures are important for selection of attributes: Entropy Information Gain 2/6/2006 ML_S2006_DecisionTreeLearning 25 2/6/2006 ML_S2006_DecisionTreeLearning 26 Entropy Definition (1) Entropy Definition (2) Definition: It is a measure of (im)purity of a collection of examples. Formalization of Measure: Let S is a collection of instances Let a Target Concept, T divides the S into positive and negative examples Then entropy if S with respect to T is: In the entropy formula: p is the portion of positive examples in S p Θ is the portion of negative examples in S We define 0 log 2 0 to be 0 Entropy(S, T) -p log 2 p - p Θ log 2 p Θ 2/6/2006 ML_S2006_DecisionTreeLearning 27 2/6/2006 ML_S2006_DecisionTreeLearning 28 7

Example of Entropy S is a set of 14 examples +ve = 9 -ve = 5 Case1: Pure Samples: Entropy[14+, 0] = -1 log 2 1 0 = 0 Entropy[0, 14-] = -0 (14/ 14) log 2 1 = 0 Case2: Most Impure Samples: Entropy[7, 7] = -(7/14)log 2 (7/14) (7/14)log 2 (7/14) = 1 Case3: Other samples: Entropy [9+, 5-] = - (9/14) log 2 (9/14) (5/14) log 2 (5/14) = 0.940 CONCLUSION: Entropy is larger for larger impurity Entropy: Variation with Positive Example Ratio 2/6/2006 ML_S2006_DecisionTreeLearning 29 2/6/2006 ML_S2006_DecisionTreeLearning 30 Interpretation of Entropy (1) Interpretation of Entropy (2) Interpretation: Given an instance from S What is the (additional) information needed to tell the classification of that instance If p is 1, Let us say we arbitrarily choose an example No additional information is needed to classify it That is, it can be perfectly be classified Then entropy of that S is 0 If p is 0.5: Then S has equal number of + ve and -ve examples For any example picked up arbitrarily from S We cannot say it is positive or negative for sure To say the given example is positive or negative, we need to see S where all examples are all either positive or negative If p is 0.8: Then we may not be able to correctly classify an instance But we know the instance is more likely to be positive than negative At this point, we need less (additional) information to show the instance is positive We need more (additional) information to show that the instance is negative 2/6/2006 ML_S2006_DecisionTreeLearning 31 2/6/2006 ML_S2006_DecisionTreeLearning 32 8

Generalizing Entropy Criterion for Attribute Selection Entropy( S) = pilog pi c i= 1 c = number of values an attribute can take 2 2/6/2006 ML_S2006_DecisionTreeLearning 33 2/6/2006 ML_S2006_DecisionTreeLearning 34 Information as measure of purity of a subset Expected Amount of Information Information: Represents the expected amount of information needed to classify a new instance as yes or no. Examples of information: Gain([2,3])=0.971bits Gain([4,0])=0.0bits Gain([3,2])=0.971bits Expected Amount of Information required to classify a new instance correctly at this node is: The average value taking all the branches into account: Gain([2,3], [4,0], [3,2])=(5/15) x Gain([2,3]) + (4/14) x Gain([4,0]) + (5/14) x Gain([3,2]) Gain([2,3], [4,0], [3,2]) = (5/14) x 0.971 + (4/14) x 0 + (5/14) x 0.971 = 0.693 bits This is the amount of information required to specify the class of a new instance. 2/6/2006 ML_S2006_DecisionTreeLearning 35 2/6/2006 ML_S2006_DecisionTreeLearning 36 9

Gain as Expected Reduction in Entropy Attribute Selection based on Gain Gain is the Expected reduction in entropy caused by partitioning the examples according to this attribute G(S, A) = Entropy(S) Expected_Entropy (S, A) Expected_Entropy(S, A) =? vvaluesa ( ) Sv EntropyS ( v) S Larger the Gain, better it is for classification In the following example, Humidity is a better classifier than Wind Reason: Better Gain implies, reduction in Entropy as one goes down the tree Smaller Entropy means less information required to correctly classify an instance 2/6/2006 ML_S2006_DecisionTreeLearning 37 2/6/2006 ML_S2006_DecisionTreeLearning 38 Which attribute is best for classifier? Decision Tree Learning Algorithm Topics: Different ways of partitioning instance space Entropy and Information Gain ID3 algorithm 2/6/2006 ML_S2006_DecisionTreeLearning 39 2/6/2006 ML_S2006_DecisionTreeLearning 40 10

The Problem in Learning Decision Tree To find the simplest hypothesis To derive a decision tree that classifies all examples correctly. To be able to describe a large number of examples in a concise way Ockham s Razor: The most likely hypothesis is the simplest one that is consistent with all observations (examples). Simplest hypotheses have simple structure Decision tree with smallest number of levels Simplest hypothesis has the smallest decision tree In general, finding the smallest decision tree is an intractable problem. We use heuristics To test the most important attribute first. The most important attribute is the one that classifies all examples significantly. 2/6/2006 ML_S2006_DecisionTreeLearning 41 2/6/2006 ML_S2006_DecisionTreeLearning 42 Algorithm: Behavior Partially Learnt Decision Tree Step1: Root node is created Which attribute to be tested first in the tree? ID3 determines the information gain for each candidate attribute Select the one with the highest information gain Gain(S, Outlook) = 0.246 Gain(S, Humidity) = 0.151 Gain(S, Wind) = 0.048 Gain(S, Temperature) = 0.029 2/6/2006 ML_S2006_DecisionTreeLearning 43 2/6/2006 ML_S2006_DecisionTreeLearning 44 11

ID3 Functional Algorithm ID3 Algorithm Part I ID3(Examples, Target_attribute, Attributes) Create a Root node for the tree If Examples all positive? Return Single Node Tree Root, with label = + If Examples all negative? Return Single node Tree Root, with label = - If Attributes is empty Return single-node tree Root, label = most common value of Target_attribute in Examples 2/6/2006 ML_S2006_DecisionTreeLearning 45 2/6/2006 ML_S2006_DecisionTreeLearning 46 ID3 Algorithm Part II Computing Information Gain Otherwise A Best_Attribute (Attributes, Examples ) Root A For each value v i of A Add a new tree branch Examples_svi is a subset of Examples for vi If Examples_svi is empty? Add leaf node label = most common value of Target_attribute Add a new sub tree: ID3(Examples_svi, Target_attribute, Attributes {A}) 2/6/2006 ML_S2006_DecisionTreeLearning 47 2/6/2006 ML_S2006_DecisionTreeLearning 48 12

Continuing to Split Final Decision Tree 2/6/2006 ML_S2006_DecisionTreeLearning 49 2/6/2006 ML_S2006_DecisionTreeLearning 50 Analysis Points on Information Gain A single example cannot provide information on information gain Information Gain is a mapping from a distribution to a property of classified groups homogeneity Information gain is a function Domain: Distribution of examples Range: Homogeneity property 2/6/2006 ML_S2006_DecisionTreeLearning 51 2/6/2006 ML_S2006_DecisionTreeLearning 52 13

A Measure for Homogeneity: Ratio Information Gain in Information Theory A simple measure: Ratio of positive examples Max = 1 Min = 0 Impure = 0.5 1 Ratio Sender Information Receiver Disadvantages: 0 0 1 +-ex Examples Homogeneity 1. Does not tell us directly about examples 2. Does not give us the impurity Information 2/6/2006 ML_S2006_DecisionTreeLearning 53 2/6/2006 ML_S2006_DecisionTreeLearning 54 Information Examples Colored Balls Example + + + + Entropy = 0 ---- Entropy = 0 No code required No code required Entropy(S) = - p blue log2p blue p red log2 p red p green log2 p green p yellow log2 p yellow Maximum value of entropy = log 2 c +++++ ----- Entropy = 1 Maximum Information (1 bit) represents space 2/6/2006 ML_S2006_DecisionTreeLearning 55 2/6/2006 ML_S2006_DecisionTreeLearning 56 14

Explanation Example Values(A) = set of all values of A S v = subset of S with attribute A has value v Entropy(S) = Entropy of original collection Second Term = Expected value of entropy after S is partitioned using attribute A A = Wind Values(A) = {Weak, Strong} S Weak = [6+, 2-] S Strong = [3+, 3-] To compute Gain (S, Wind) Entropy (S) = 0.940 - S Weak = - (8/14) - S Strong = -(6/14) Gain (S, Wind) = 0.048 2/6/2006 ML_S2006_DecisionTreeLearning 57 2/6/2006 ML_S2006_DecisionTreeLearning 58 DTL as Search Main Issue: Select best attribute that classifies examples Each possible attribute points to a search state Search: Top_Down: Start with a concept that represents all examples Greedy Search: Select attribute that classifies maximum number of examples Algorithm never backtracks to reconsider earlier choices Systems ID3 and C4.5 2/6/2006 ML_S2006_DecisionTreeLearning 59 2/6/2006 ML_S2006_DecisionTreeLearning 60 15

Algorithm: Functional Modules If positive and negative examples present Choose best attribute to split them If all remaining examples are positive (or negative) Say yes (or no ) If no examples left Return default value (majority classification) If no attributes left, but there are unclassified examples There is noise in the data 2/6/2006 ML_S2006_DecisionTreeLearning 61 2/6/2006 ML_S2006_DecisionTreeLearning 62 Decision Tree Learning Algorithm Exercise See figure R18.7 Example: See figure R18.8 Learn the restaurant concept from the examples Try various best attributes 2/6/2006 ML_S2006_DecisionTreeLearning 63 2/6/2006 ML_S2006_DecisionTreeLearning 64 16