Decision Tree Learning

Topics Decision Tree Learning Sattiraju Prabhakar CS898O: DTL Wichita State University What are decision trees? How do we use them? New Learning Task ID3 Algorithm Weka Demo C4.5 Algorithm Weka Demo Implementation 9/14/2004 ML2004_DecisionTreeLearning 2 What are Decision Trees? Elements of Decision Tree Attribute value Attribute Yes Yes No 9/14/2004 ML2004_DecisionTreeLearning 3 9/14/2004 ML2004_DecisionTreeLearning 4 1

Example: PlayTennis Example Decision Tree 9/14/2004 ML2004_DecisionTreeLearning 5 9/14/2004 ML2004_DecisionTreeLearning 6 Example: Contact Lens data Example: Labour Negotiations (a) (b) 9/14/2004 ML2004_DecisionTreeLearning 7 9/14/2004 ML2004_DecisionTreeLearning 8 2

Exercise: Animal Decision tree for a simple disjunction. Example Vertibral Column Make_So und Legs Take_Food Animal Man Yes Talking 2 Cultivation Yes Mango Tree No Branch Moveme nt 1 Using_ Chlorophyll No Lizard Yes Using 4 Catches Yes Tongue Flies Parameciu m No No Sound 0 Absorbs Cells Yes Flytrap No No Sound 0 Catches Flies No 9/14/2004 ML2004_DecisionTreeLearning 9 9/14/2004 ML2004_DecisionTreeLearning 10 Structure of a Decision Tree New Learning Task Structure: Is a disjunction of conjunctions on the attribute values of instances Each path from tree root to a leaf corresponds to a conjunction of attribute tests Tree itself is a disjunction of thee conjunctions Example: PlayTennis Decision Tree (Outlook=Sunny Humidity=Normal) (Outlook = Overcast) (Outlook = Rain Wind = Weak) 9/14/2004 ML2004_DecisionTreeLearning 11 9/14/2004 ML2004_DecisionTreeLearning 12 3

Learner + Performance Element Using Decision Tree by Performance Element Training Examples Test Examples Learner Learnt Model: Decision Tree Performance Element Yes/ No Decision New Situation or example Decision Tree Classifier Yes/ No Decision 9/14/2004 ML2004_DecisionTreeLearning 13 9/14/2004 ML2004_DecisionTreeLearning 14 Characterizing Learning Task Strengths of DTL It should be possible to represent instances by attribute-value pairs Target Function should have discrete values It should be possible to represent Target Functions as disjunctive expressions Training data may contain errors Even when some attribute value pairs are missing in instance descriptions, we can use them 9/14/2004 ML2004_DecisionTreeLearning 15 9/14/2004 ML2004_DecisionTreeLearning 16 4

Decision Tree Learning Algorithm Decision Tree Learning Algorithm Topics: Different ways of partitioning instance space Entropy and Information Gain ID3 algorithm Topics: Different ways of partitioning instance space Entropy and Information Gain ID3 algorithm 9/14/2004 ML2004_DecisionTreeLearning 17 9/14/2004 ML2004_DecisionTreeLearning 18 Different ways of partitioning the instance space. Decision Tree stumps for the weather data. (a) (b) (a) (b) (c) (d) (c) (d) 9/14/2004 ML2004_DecisionTreeLearning 19 9/14/2004 ML2004_DecisionTreeLearning 20 5

Expanded tree stumps for the weather data. Decision tree for the weather data. (a) (b) (c) 9/14/2004 ML2004_DecisionTreeLearning 21 9/14/2004 ML2004_DecisionTreeLearning 22 Operation of a covering algorithm; decision tree for the same problem Example of building a partial tree. y b a b b b a b b a a a b b b a b b b b b y b a b b a b b b a a a b b b a b b b b b y 2 6 b a b b b a b b a a a b b b a b b b b b (a) (c) x 1 2 x 1 2 x (a) (b) (b) 9/14/2004 ML2004_DecisionTreeLearning 23 9/14/2004 ML2004_DecisionTreeLearning 24 6

Decision Tree Learning Algorithm Attribute Selection Topics: Different ways of partitioning instance space Attribute Selection: Entropy and Information Gain ID3 algorithm To build a decision tree, we need to select an attribute Selection of different attributes at different points lead to different decision trees Two measures are important for selection of attributes: Entropy Information Gain 9/14/2004 ML2004_DecisionTreeLearning 25 9/14/2004 ML2004_DecisionTreeLearning 26 Entropy Definition (1) Entropy Definition (2) Definition: It is a measure of (im)purity of a collection of examples. Formalization of Measure: Let S is a collection of instances Let a Target Concept, T divides the S into positive and negative examples Then entropy if S with respect to T is: In the entropy formula: p is the portion of positive examples in S p Θ is the portion of negative examples in S We define 0 log 2 0 to be 0 Entropy(S, T) -p log 2 p - p Θ log 2 p Θ 9/14/2004 ML2004_DecisionTreeLearning 27 9/14/2004 ML2004_DecisionTreeLearning 28 7

Example of Entropy S is a set of 14 examples +ve = 9 -ve = 5 Case1: Pure Samples: Entropy[14+, 0] = -1 log 2 1 0 = 0 Entropy[0, 14-] = -0 (14/ 14) log 2 1 = 0 Case2: Most Impure Samples: Entropy[7, 7] = -(7/14)log 2 (7/14) (7/14)log 2 (7/14) = 1 Case3: Other samples: Entropy [9+, 5-] = - (9/14) log 2 (9/14) (5/14) log 2 (5/14) = 0.940 CONCLUSION: Entropy is larger for larger impurity Entropy: Variation with Positive Example Ratio 9/14/2004 ML2004_DecisionTreeLearning 29 9/14/2004 ML2004_DecisionTreeLearning 30 Interpretation of Entropy (1) Interpretation of Entropy (2) Interpretation: Given an instance from S What is the (additional) information needed to tell the classification of that instance If p is 1, Let us say we arbitrarily choose an example No additional information is needed to classify it That is, it can be perfectly be classified Then entropy of that S is 0 If p is 0.5: Then S has equal number of + ve and -ve examples For any example picked up arbitrarily from S We cannot say it is positive or negative for sure To say the given example is positive or negative, we need to see S where all examples are all either positive or negative If p is 0.8: Then we may not be able to correctly classify an instance But we know the instance is more likely to be positive than negative At this point, we need less (additional) information to show the instance is positive We need more (additional) information to show that the instance is negative 9/14/2004 ML2004_DecisionTreeLearning 31 9/14/2004 ML2004_DecisionTreeLearning 32 8

Generalizing Entropy Criterion for Attribute Selection Entropy( S) = pilog pi c i= 1 c = number of values an attribute can take 2 9/14/2004 ML2004_DecisionTreeLearning 33 9/14/2004 ML2004_DecisionTreeLearning 34 Information as measure of purity of a subset Expected Amount of Information Information: Represents the expected amount of information needed to classify a new instance as yes or no. Examples of information: Gain([2,3])=0.971bits Gain([4,0])=0.0bits Gain([3,2])=0.971bits Expected Amount of Information required to classify a new instance correctly at this node is: The average value taking all the branches into account: Gain([2,3], [4,0], [3,2])=(5/15) x Gain([2,3]) + (4/14) x Gain([4,0]) + (5/14) x Gain([3,2]) Gain([2,3], [4,0], [3,2]) = (5/14) x 0.971 + (4/14) x 0 + (5/14) x 0.971 = 0.693 bits This is the amount of information required to specify the class of a new instance. 9/14/2004 ML2004_DecisionTreeLearning 35 9/14/2004 ML2004_DecisionTreeLearning 36 9

Gain as Expected Reduction in Entropy Attribute Selection based on Gain Gain is the Expected reduction in entropy caused by partitioning the examples according to this attribute G(S, A) = Entropy(S) Expected_Entropy (S, A) Expected_Entropy(S, A) =? vvaluesa ( ) Sv EntropyS ( v) S Larger the Gain, better it is for classification In the following example, Humidity is a better classifier than Wind Reason: Better Gain implies, reduction in Entropy as one goes down the tree Smaller Entropy means less information required to correctly classify an instance 9/14/2004 ML2004_DecisionTreeLearning 37 9/14/2004 ML2004_DecisionTreeLearning 38 Which attribute is best for classifier? Decision Tree Learning Algorithm Topics: Different ways of partitioning instance space Entropy and Information Gain ID3 algorithm 9/14/2004 ML2004_DecisionTreeLearning 39 9/14/2004 ML2004_DecisionTreeLearning 40 10

The Problem in Learning Decision Tree To find the simplest hypothesis To derive a decision tree that classifies all examples correctly. To be able to describe a large number of examples in a concise way Ockham s Razor: The most likely hypothesis is the simplest one that is consistent with all observations (examples). Simplest hypotheses have simple structure Decision tree with smallest number of levels Simplest hypothesis has the smallest decision tree In general, finding the smallest decision tree is an intractable problem. We use heuristics To test the most important attribute first. The most important attribute is the one that classifies all examples significantly. 9/14/2004 ML2004_DecisionTreeLearning 41 9/14/2004 ML2004_DecisionTreeLearning 42 Algorithm: Behavior Partially Learnt Decision Tree Step1: Root node is created Which attribute to be tested first in the tree? ID3 determines the information gain for each candidate attribute Select the one with the highest information gain Gain(S, Outlook) = 0.246 Gain(S, Humidity) = 0.151 Gain(S, Wind) = 0.048 Gain(S, Temperature) = 0.029 9/14/2004 ML2004_DecisionTreeLearning 43 9/14/2004 ML2004_DecisionTreeLearning 44 11

ID3 Functional Algorithm Functional Algorithm If positive and negative examples present Choose best attribute to split them If all remaining examples are positive (or negative) Say yes (or no ) If no examples left Return default value (majority classification) If no attributes left, but there are unclassified examples There is noise in the data 9/14/2004 ML2004_DecisionTreeLearning 45 9/14/2004 ML2004_DecisionTreeLearning 46 ID3 Algorithm Part I ID3 Algorithm Part II ID3(Examples, Target_attribute, Attributes) Create a Root node for the tree If Examples all positive? Return Single Node Tree Root, with label = + If Examples all negative? Return Single node Tree Root, with label = - If Attributes is empty Return single-node tree Root, label = most common value of Target_attribute in Examples Otherwise A Best_Attribute (Attributes, Examples ) Root A For each value v i of A Add a new tree branch Examples_svi is a subset of Examples for vi If Examples_svi is empty? Add leaf node label = most common value of Target_attribute Add a new sub tree: ID3(Examples_svi, Target_attribute, Attributes {A}) 9/14/2004 ML2004_DecisionTreeLearning 47 9/14/2004 ML2004_DecisionTreeLearning 48 12

Computing Information Gain Continuing to Split 9/14/2004 ML2004_DecisionTreeLearning 49 9/14/2004 ML2004_DecisionTreeLearning 50 Final Decision Tree Analysis 9/14/2004 ML2004_DecisionTreeLearning 51 9/14/2004 ML2004_DecisionTreeLearning 52 13

DTL as Search DTL as Search Main Issue: Select best attribute that classifies examples Each possible attribute points to a search state Search: Top_Down: Start with a concept that represents all examples Greedy Search: Select attribute that classifies maximum number of examples Algorithm never backtracks to reconsider earlier choices Systems ID3 and C4.5 9/14/2004 ML2004_DecisionTreeLearning 53 9/14/2004 ML2004_DecisionTreeLearning 54 Exercise Learn the Cell concept from the examples Try various best attributes Properties of ID3: Hypothesis Space What is a hypothesis in DTL? Each hypothesis is the disjunction of a set of paths starting from an attribute What is a hypothesis space in DTL? It is all the possible trees that are considered by ID3 algorithm How does the ID3 consider different possible hypotheses? Comparing attributes, from a set of attributes, in order to add to a node. 9/14/2004 ML2004_DecisionTreeLearning 55 9/14/2004 ML2004_DecisionTreeLearning 56 14

Search in Hypothesis Space Properties of Hypothesis Space Search (1) How do we characterize the search in ID3? Simple-to-complex, hill-climbing search through hypothesis space Evaluation function for hill-climbing: Information gain Decision Trees are completely expressive They can express all possible hypotheses The search is in this complete space If the target function is in the hypothesis space, ID3 will find it Search Strategy is incomplete Searches from simple to complex hypotheses using hill climbing until termination condition is reached It maintains a single current hypothesis 9/14/2004 ML2004_DecisionTreeLearning 57 9/14/2004 ML2004_DecisionTreeLearning 58 Properties of Hypothesis Space Search (2) Performance Measurement May not find globally optimal solution Performs no backtracking Uses all training examples at each step of its search 9/14/2004 ML2004_DecisionTreeLearning 59 9/14/2004 ML2004_DecisionTreeLearning 60 15

Performance Measurement C4.5 9/14/2004 ML2004_DecisionTreeLearning 61 9/14/2004 ML2004_DecisionTreeLearning 62 Curve Fitting: Inductive Learning Method Inductive Learning Method 9/14/2004 ML2004_DecisionTreeLearning 63 9/14/2004 ML2004_DecisionTreeLearning 64 16

Inductive Learning Method Inductive Learning Method 9/14/2004 ML2004_DecisionTreeLearning 65 9/14/2004 ML2004_DecisionTreeLearning 66 Inductive Learning Method Inductive Learning Method 9/14/2004 ML2004_DecisionTreeLearning 67 9/14/2004 ML2004_DecisionTreeLearning 68 17

Overfitting Overfitting Definition: Given a hypothesis space H, a hypothesis h H is said to overfit the training data if there exists some alternative hypothesis h H, such that h has smaller error than h over the training examples, but h has a smaller error than h over the entire distribution of instances. See figure 3.6 in book Illustrates how the overfit hypothesis loses accuracy over test data How overfit hypotheses are generated? If training examples have errors Error Example: <Outlook = Sunny, Temperature = Hot, Humidity = Normal, Wind = Strong, PlayTennis = No> Because of this, ID3 will construct a complex decision tree (h) that is more complex than the decision tree(h ) The more complexh fits the training examples, better than h The simpler h fits the test examples better than h 9/14/2004 ML2004_DecisionTreeLearning 69 9/14/2004 ML2004_DecisionTreeLearning 70 How to avoid Overfitting? Subtree Replacement Post-prune the decision tree Generate the decision tree that overfits the data using all the training examples Prune out the subtrees from this tree Use a separate set of examples (Validation set) to evaluate the utility of post-pruning nodes from tree Subtree replacement: Replace subtrees by leaves Starts from the leaves and works back up toward the root Example: The left branch of left tree is replaced by the right tree To decide whether to replace a subtree by a leaf The error rate is estimated 9/14/2004 ML2004_DecisionTreeLearning 71 9/14/2004 ML2004_DecisionTreeLearning 72 18

Implementation of ID3 ARFF file for the weather data. % ARFF file for the weather data with some numeric features % @relation weather @attribute outlook { sunny, overcast, rainy } @attribute temperature numeric @attribute humidity numeric @attribute windy { true, false } @attribute play? { yes, no } @data % % 14 instances % sunny, 85, 85, false, no sunny, 80, 90, true, no overcast, 83, 86, false, yes rainy, 70, 96, false, yes rainy, 68, 80, false, yes rainy, 65, 70, true, no overcast, 64, 65, true, yes sunny, 72, 95, false, no sunny, 69, 70, false, yes rainy, 75, 80, false, yes sunny, 75, 70, true, yes overcast, 72, 90, true, yes overcast, 81, 75, false, yes rainy, 71, 91, true, no 9/14/2004 ML2004_DecisionTreeLearning 73 9/14/2004 ML2004_DecisionTreeLearning 74 19