Apprentissage automatique et fouille de données (part 2) Telecom Saint-Etienne Elisa Fromont (basé sur les cours d Hendrik Blockeel et de Tom Mitchell) 1
Induction of decision trees : outline (adapted from Tom Mitchell s Machine Learning book and Hendrik Blockeel s lecture, KULeuven, Belgium) What are decision trees? How can they be induced automatically? top-down induction of decision trees avoiding overfitting converting trees to rules alternative heuristics a generic TDIDT algorithm 2
What are decision trees? Cf. guessing a person using only yes/no questions: ask some question depending on answer, ask a new question continue until answer known A decision tree Tells you which question to ask, depending on outcome of previous questions Gives you the answer in the end Usually not used for guessing an individual, but for predicting some property (e.g., classification) 3
Example decision tree 1 Mitchell s example: Play tennis or not? (depending on weather conditions) Outlook Sunny Overcast Rainy Humidity Yes Wind Strong High Weak Normal No Yes No Yes 4
Example decision tree 2 tree for predicting whether C-section necessary Leaves are not pure here; ratio pos/neg is given 1 Previous_Csection Fetal_Presentation 0 1 [3+, 29-].11+.89- Primiparous + [55+, 35-].61+.39-2 - 3 - [8+, 22-].27+.73-5
Representation power Typically: examples represented by array of attributes 1 node in tree tests value of 1 attribute 1 child node for each possible outcome of test Leaf nodes assign classification Note: tree can represent any Boolean function - i.e., also disjunctive concepts (<-> VS: conjunctive concepts) tree can allow noise (non-pure leaves) 6
Representing boolean formulae E.g., A B true A false true B Exercises: 1. A B 2. ( A B) (A B) 3. (A B) (C D E) true true false false 7
Exercise (Decision Trees and Generality) True or false? If a Boolean decision tree D2 is an elaboration of D1 ( elaboration meaning that a leaf of D1 has been changed into a sub-tree in D2) then D1 is more general than D2. 8
Classification, Regression and Clustering trees Classification trees represent function X -> C with C discrete (like the decision trees we just saw) Hence, can be used for concept learning Regression trees predict numbers in leaves could use a constant (e.g., mean), or linear regression model, or Clustering trees just group examples in leaves Most (but not all) decision tree research in machine learning focuses on classification trees 9
Example decision tree 3 (from study of river water quality) "Data mining" application Given: descriptions of river water samples biological description: occurrence of organisms in water ( abundance, graded 0-5) chemical description: 16 variables (temperature, concentrations of chemicals (NH 4,...)) Question: characterize chemical properties of water using organisms that occur 10
Clustering tree yes T = 0.357111 ph = -0.496808 cond = 1.23151 O2 = -1.09279 O2sat = -1.04837 CO2 = 0.893152 hard = 0.988909 NO2 = 0.54731 NO3 = 0.426773 NH4 = 1.11263 PO4 = 0.875459 Cl = 0.86275 SiO2 = 0.997237 KMnO4 = 1.29711 K2Cr2O7 = 0.97025 BOD = 0.67012 abundance(tubifex sp.,5)? T = 0.0129737 ph = -0.536434 cond = 0.914569 O2 = -0.810187 O2sat = -0.848571 CO2 = 0.443103 hard = 0.806137 NO2 = 0.4151 NO3 = -0.0847706 NH4 = 0.536927 PO4 = 0.442398 Cl = 0.668979 SiO2 = 0.291415 KMnO4 = 1.08462 K2Cr2O7 = 0.850733 BOD = 0.651707 no abundance(sphaerotilus natans,5)? yes no abundance(...) <- "standardized" values (how many standard deviations above mean) 11
Inducing Decision Trees... In general, we like decision trees that give us a result after as few questions as possible We can construct such a tree manually, or we can try to obtain it automatically (inductively) from a set of data The latter is what this chapter is about Outlook Temp. Hum. Wind Play? Sunny 85 85 False no Sunny 80 90 True no Overcast 83 86 False yes............... Outlook Humidity Yes High Sunny Overcast Rainy Normal Wind Strong Weak No Yes No Yes 12
Top-Down Induction of Decision Trees Basic algorithm for TDIDT: (later more formal version) start with full data set find test that partitions examples as good as possible - good = examples with same class, or otherwise similar examples, should be put together for each outcome of test, create child node move examples to children according to outcome of test repeat procedure for each child that is not pure Main questions: how to decide which test is best when to stop the procedure 13
Example problem? Is this drink going to make us ill, or not? 14
Data set: 8 classified instances 15
Obs 1: Shape is important SHAPE 16
Obs 2: for some glasses, colour is important SHAPE COLOUR 17
The decision tree SHAPE Non-orange COLOUR orange? 18
A DT creates a decision surface + + + + + + + - - - - - - - - - - - - - - - - - - - - 19
Exercise : (Decision Surface) Consider a data set with two numeric attributes a1 and a2 and one nominal target attribute c with two possible values: + and The training examples are shown in the Figure < 1. Find a decision tree that classifies all training examples correctly. 2. Draw the decision surface of this tree on the Figure 20
Finding the best test (for classification trees) For classification trees: find test for which children are as pure as possible Purity measure borrowed from information theory: entropy is a measure of missing information ; more precisely, number of bits needed to represent the missing information, on average, using optimal encoding Given set S with instances belonging to class i with probability p i : Entropy(S) = - Σ i p i log 2 p i 21
Entropy For 2 classes if x = proportion of instances of class 1 in a given node e Entropy (e) = - x log 2 (x) (1-x) log 2 (1-x) Entropy in function of p (proportion of examples from class 1), for 2 classes: Entropy max when the 2 classes are not well separated 22
Information gain Heuristic for choosing a test in a node: test that on average provides most information about the class test that, on average, reduces class entropy most - on average: class entropy reduction differs according to outcome of test expected reduction of entropy = information gain Gain(S,A) = Entropy(S) - Σ v ( S v / S ) Entropy(S v ) S = set of instances in a given node A S v / S = proportion of instances of S which go in the v st child of A 23
Other purity measure/gain Gini coefficient : a measure of statistical dispersion (a low coefficient means a more equal distribution) Gini (S) = 1- Σ i p i2 = 2 Σ i<j p i p j Gain(S,A) = Gini(S) - Σ v ( S v / S ) Gini(S v ) 24
Exercise Assume S has 9 + and 5 - examples; partition (with entropy) according to Wind or Humidity attribute High S: [9+,5-] S: [9+,5-] E = E = Humidity Wind Normal Strong Weak S: [3+,4-] S: [6+,1-] S: [6+,2-] S: [3+,3-] E = E = E = E = Which attribute gives the best gain? 25
Example Assume Outlook was chosen: continue partitioning using info gain in child nodes [9+,5-] Outlook Sunny Overcast Rainy? Yes? [2+,3-] [4+,0-] [3+,2-] 27
Exercise (entropy, info gain) Consider the following table of training examples: 1. What is the entropy with respect to the classification attribute? 2. What is the information gain of a2 relative to these training examples? 28
Hypothesis space search in TDIDT Hypothesis space H = set of all trees H is searched in a hill-climbing fashion, from simple to complex... 29
Inductive bias in TDIDT Note: for e.g. boolean attributes, H is complete: each concept can be represented! given n attributes, we can keep on adding tests until all attributes tested So what about inductive bias? Clearly no restriction bias (H 2 U ) as for version spaces (conjunctive concepts) Preference bias: some hypotheses in H are preferred over others In this case: preference for short trees with informative attributes at the top 30
Occam s Razor Preference for simple models over complex models is quite generally used in machine learning Similar principle in science: Occam s Razor roughly: do not make things more complicated than necessary Reasoning, in the case of decision trees: more complex trees have higher probability of overfitting the data set Why? Somewhat controversial, see later 31
Avoiding Overfitting Phenomenon of overfitting: keep improving a model, making it better and better on training set by making it more complicated increases risk of modeling noise and coincidences in the data set may actually harm predictive power of theory on unseen cases Cf. fitting a curve with too many parameters............ 32
Overfitting: example + + + + + + + - - - - - - - - - + - - - - - - - - - - area with probably wrong predictions 33
Overfitting: effect on predictive accuracy Typical phenomenon when overfitting: training accuracy keeps increasing accuracy on unseen validation set starts decreasing accuracy accuracy on training data accuracy on unseen data overfitting starts about here size of tree 34
How to avoid overfitting when building classification trees? Option 1: stop adding nodes to tree when overfitting starts occurring need stopping criterion Option 2: don t bother about overfitting when growing the tree after the tree has been built, start pruning it again 35
Stopping criteria How do we know when overfitting starts? 1. use a validation set: data not considered for choosing the best test - when accuracy goes down on validation set: stop adding nodes to this branch 2. use some statistical test - significance test: is the change in class distribution significant? (χ 2 - test) [in other words: does the test yield a clearly better situation?] - MDL: minimal description length principle entirely correct theory = tree + corrections for specific misclassifications minimize size(theory) = size(tree) + size(misclassifications(tree)) Cf. Occam s razor 36
Post-pruning trees After learning the tree: start pruning branches away For all nodes in tree: - Estimate effect of pruning tree at this node on predictive accuracy e.g. using accuracy on validation set Prune node that gives greatest improvement Continue until no improvements Note : this pruning constitutes a second search in the hypothesis space 37
Effect of pruning accuracy accuracy on training data effect of pruning accuracy on unseen data size of tree 38
Comparison Advantage of Option 1: no superfluous work But: tests may be misleading E.g., validation accuracy may go down briefly, then go up again Therefore, Option 2 (post-pruning) is usually preferred (though more work) 39
Turning trees into rules From a tree a rule set can be derived Path from root to leaf in a tree = 1 if-then rule Advantage of such rule sets may increase comprehensibility - Disjunctive concept definition can be pruned more flexibly - in 1 rule, 1 single condition can be removed vs. tree: when removing a node, the whole subtree is removed - 1 rule can be removed entirely 40
Rules from trees: example Outlook Sunny Overcast Rainy Humidity Yes Wind Strong High Weak Normal No Yes No Yes if Outlook = Sunny and Humidity = High then No if Outlook = Sunny and Humidity = Normal then Yes 41
Pruning rules : possible method 1. convert tree to rules 2. prune each rule independently remove conditions that do not harm accuracy of rule 3. sort rules (e.g., most accurate rule first) before pruning: each example covered by 1 rule after pruning, 1 example might be covered by multiple rules therefore some rules might contradict each other therefore sorting is important 42
Pruning rules: example true true A false B Tree representing A B true true false false if A=true then true if A=false and B=true then true if A=false and B=false then false Rules represent A ( A B) A B 43
Handling missing values What if result of test is unknown for example? e.g. because value of attribute unknown Some possible solutions, when training: guess value: just take most common value (among all examples, among examples in this node / class, ) assign example partially to different branches - e.g. counts for 0.7 in yes subtree, 0.3 in no subtree When using tree for prediction: assign example partially to different branches combine predictions of different branches 44
Alternative heuristics for choosing tests Attributes with continuous domains (numbers) cannot have different branch for each possible outcome allow, e.g., binary test of the form Temperature < 20 same evaluation as before, but need to generate value (e.g. 20) - For instance, just try all reasonable values Attributes with many discrete values unfair advantage over attributes with few values - cf. question with many possible answers is more informative than yes/no question To compensate: divide gain by max. potential gain SI Gain Ratio: GR(S,A) = Gain(S,A) / SI(S,A) - Split-information SI(S,A) = - S v / S log 2 Sv / S - with v ranging over different results of test A 45
Continuous Valued Attributes Create a discrete attribute to test continuous Temperature = 24.5 0 C (Temperature > 20.0 0 C) = {true, false} Where to set the threshold? Temperature 15 0 C 18 0 C 19 0 C 22 0 C 24 0 C 27 0 C PlayTennis No No Yes Yes Yes No 46
Handling costs Tests may have different costs e.g. medical diagnosis: blood test, visual examination, have different costs try to find tree with low expected cost, instead of low expected number of tests alternative heuristics, taking cost into account, have been proposed e.g: - replace gain by: Gain 2 (S,A)/Cost(A) [Tan, Schimmer 1990] - 2 Gain(S,A) -1/(Cost(A)+1)w w [0,1] [Nunez 1988] 47
Properties of good heuristics Many alternatives exist ID3/C4.5 use information gain or gain ratio CART uses Gini criterion Q: Why not simply use accuracy (% of correct predictions of tree, if leaves predict their majority class) as a criterion? 80-, 20+ A1 40-,0+ 40-,20+ 80-, 20+ A2 40-,10+ 40-,10+ How would - accuracy - information gain rate these splits? 48
Heuristics compared 1 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0 0 0,2 0,4 0,6 0,8 1 p Accuracy Entropy Gini Curves show impurity as a function of p (proportion of positive), when using Accuracy, Entropy, Gini Good impurity measures are strictly concave 49
Why concave functions? Gain E 1 E E 2 (n 1 /n)e 1 +(n 2 /n)e 2 p 1 p p 2 Assume node with size n, entropy E and proportion of positives p is split into 2 nodes with n 1, E 1, p 1 and n 2, E 2 p 2. We have p = (n 1 /n)p 1 + (n 2 /n) p 2 and the new average entropy E = (n 1 /n)e 1 +(n 2 /n)e 2 is therefore found by linear interpolation between (p 1,E 1 ) and (p 2,E 2 ) at p. Gain = difference in height between (p, E) and (p,e ). 50
Generic TDIDT algorithm Many different algorithms for top-down induction of decision trees exist What do they have in common, and where do they differ? We look at a generic algorithm General framework for TDIDT algorithms Several parameter procedures - instantiating them yields a specific algorithm Summarizes previously discussed points and puts them into perspective 51
Generic TDIDT algorithm function TDIDT(E: set of examples) returns tree; T' := grow_tree(e); T := prune(t'); return T; function grow_tree(e: set of examples) returns tree; T := generate_tests(e); t := best_test(t, E); P := partition induced on E by t; if stop_criterion(e, P) then return leaf(info(e)) else for all E j in P: t j := grow_tree(e j ); return node(t, {( j, t j )}); j 52
For classification... prune: e.g. reduced-error pruning,... generate_tests : Attr=val, Attr<val,... for numeric attributes: generate val best_test : Gain, Gainratio,... stop_criterion : MDL, significance test (e.g. χ 2 - test),... Info (how do we label the node?) : most frequent class ("mode") Popular systems: C4.5 (Quinlan 1993), C5.0 (www.rulequest.com) 53
Simple algorithm : ID3 (no pruning, 2 classes) ID3 (Examples, Target_Attribute, Attributes) Create a root node for the tree If all examples are positive, Return the single-node tree Root, with label = +. If all examples are negative, Return the single-node tree Root, with label = -. If number of predicting attributes is empty, Return the single node tree Root, with label = most common value of the target attribute in the examples. Otherwise Begin A = The Attribute that best classifies examples. Decision Tree attribute for Root = A. For each possible value, vi, of A : - Add a new tree branch below Root, corresponding to the test A = vi. - Let Examples(vi), be the subset of examples that have the value vi for A IF Examples(vi) is empty Then below this new branch add a leaf node with label = most common target value in the examples Else below this new branch add the subtree ID3 (Examples(vi), Target_Attribute, Attributes {A}) End Return Root 54
Exercise : ID3 1. Show a decision tree that could be learned by ID3 assuming it gets the following examples : 2. Add this example: then show how ID3 would induce a decision tree for these 5 examples. 55
Reduced-error pruning (1/2) IDEA : Prune a tree at a node A if the (weighted) sum of the error of the child node C v of A > error of A Weight = proportion of instances in a given node compared to total number of instances Don t use only error but error within a confidence interval The proportion p (X/n) follows a normal distribution (= Gaussian ) with a mean µ = p = f n and a standard deviation σ = ((p(1-p)/n) 1/2 (σ 2 : variance) u : confidence level or confidence coefficient (quantile ~cumulative distribution function). 56
Example : confidence interval Toss a coin 10 times : 3 heads, 7 tails, is there a pb with the coin? (not 5/5???) Probability to give the wrong answer < 5%, u = 1.96 f n = p = ½ n = 10 Compute IC. If 3/10 and 7/10 are within IC, the coin is ok 57
Reduce error pruning (2/2) To compare the error (proportion of wrongly classified instances) of the child node and the error of the father node, we only use the upper bound of the confidence interval weight 58
Exercise Should we prune A? (u = 1.96) (10+,3-) A B (9+,1-) (1+,2-) C 59
For regression... change best_test: e.g. minimize average variance info: mean stop_criterion: significance test (e.g., F-test) {1,3,4,7,8,12} {1,3,4,7,8,12} A1 A2 {1,4,12} {3,7,8} {1,3,7} {4,8,12} 60
Exercise : regression tree Consider the following set S of training examples (with a numeric target attribute). Compute the heuristic value (weighted average variance of the subsets: H(A) = Σ v Values(A) ( ( Sv / S ) *Var[Sv] )) for each attribute A. NB variance : Var[Sv] = i [Sv pi(xi-µ) 2 (µ :mean, pi: proportion of example i in Sv, xi : value of the target variable for example i ) Which attribute (a1 or a2) will be put in the top node of the regression tree? 61
CART Binary classification and regression trees [Breiman et al., 1984] Classification: info: mode, best_test: Gini Regression: info: mean, best_test: variance prune: cost-complexity" pruning penalty α for each node the higher α, the smaller the tree will be optimal α obtained empirically (cross-validation) 62
Pruning in CART (details) CART determines a pruning sequence t 0, t 1,., t p : the exact order in which each node should be removed sequence determined from the originally learned tree (t0) all the way back to root node tp The "weakest link" is pruned away (= the nodes that add least to overall accuracy of the tree) contribution to overall tree : a function of both accuracy (on training set) and size of node accuracy gain is weighted by the size of the considered sample small nodes tend to get removed before large ones If several nodes have the same contribution they are all pruned away simultaneously more than two terminal nodes could be cut off in one pruning The sequence is determined all the way back to root node if target variable is unpredictable we will want to prune back to root... the no model solution 63
CART pruning : evaluate a tree Once the sequence is computed, for each tree in the sequence, evaluate the tree : Take a test data set and drop it down the largest tree in the sequence and measure its predictive accuracy - how many cases right and how many wrong - measure accuracy overall and by class CART procedure requires test data (validation set) to guide tree evaluation 64
CART : pruning sequence example 24 Terminal Nodes 21 Terminal Nodes 20 Terminal Nodes 18 Terminal Nodes 65
n-dimensional target spaces Instead of predicting 1 number, predict vector of numbers info: mean vector best_test: variance (mean squared distance) in n- dimensional space stop_criterion: F-test mixed vectors (numbers and symbols)? use appropriate distance measure -> "clustering trees" 66
Clustering tree yes T = 0.357111 ph = -0.496808 cond = 1.23151 O2 = -1.09279 O2sat = -1.04837 CO2 = 0.893152 hard = 0.988909 NO2 = 0.54731 NO3 = 0.426773 NH4 = 1.11263 PO4 = 0.875459 Cl = 0.86275 SiO2 = 0.997237 KMnO4 = 1.29711 K2Cr2O7 = 0.97025 BOD = 0.67012 abundance(tubifex sp.,5)? T = 0.0129737 ph = -0.536434 cond = 0.914569 O2 = -0.810187 O2sat = -0.848571 CO2 = 0.443103 hard = 0.806137 NO2 = 0.4151 NO3 = -0.0847706 NH4 = 0.536927 PO4 = 0.442398 Cl = 0.668979 SiO2 = 0.291415 KMnO4 = 1.08462 K2Cr2O7 = 0.850733 BOD = 0.651707 no abundance(sphaerotilus natans,5)? yes no abundance(...) <- "standardized" values (how many standard deviations above mean) 67
Model trees Make predictions using linear regression models in the leaves info: regression model (y=ax 1 +bx 2 +c) best_test:? variance: simple, not so good (M5 approach) residual variance after model construction: better, computationally expensive (RETIS approach) stop_criterion: significant reduction of variance A 68
Pluses of Decision Trees Easy to generate; simple algorithm Easy to read small trees; can be converted to simple rules Highly expressive Fast and easy to construct Good performance on many tasks A wide variety of problems can be recast as classification problems 69
Minuses of Decision Trees Not always sufficient to learn complex concepts Can be hard to understand if the trees are large Some problems with continuously-valued attributes or classes may not be easily discretized Methods for handling missing attribute values are somewhat clumsy 70
Try it out... Weka toolbox By the University of Waikato Contains many machine learning algorithms - Including several TDIDT algorithms: ID3, J48, M5,... Contains toy data sets (incl. play-tennis example) Easy-to-use interface See http://www.cs.waikato.ac.nz/ml/weka 71
To Remember Decision trees & their representational power Generic TDIDT algorithm and how to instantiate its parameters Search through hypothesis space, tree to rule conversion For classification trees: details on heuristics, handling missing values, pruning, Some general concepts: overfitting, Occam s razor 72