Machine Learning 2010

Machine Learning 2010 Decision Trees Email: mrichter@ucalgary.ca -- 1 -

Part 1 General -- 2 -

Representation with Decision Trees (1) Examples are attribute-value vectors Representation of concepts by labeled trees Leave nodes: classes (e.g. positive / negative) inner nodes: Attributes Edges: Attribute values from the domain of the attribute at the upper node... A i w i,1 w i,2 w i,n... T i = {w i,1, w i,2,..., w i,n }... + - -- 3 -

Decision Trees (2) Equivalent logical representation: formula in propositional logic Each path from the root to a leave represents a conjunction All paths that lead to the same class are connected by a disjunction Example: sm + size la shape ci tr + - In logic: Size = small (Size = large Shape = circle) -- 4 -

Decision Trees (3) Validation of the -relation: K1 K2 K1 K2 ( is logical implication) Concept descriptions in form of decision trees are not unique, i.e. is a quasi ordering Example: - A 1 2 B 1 2 - B 1 2 A 1 2 - + - + -- 5 -

Decision Trees (4) (Example) positive example: (small, circle), negative example: (large, triangle) complete and consistent concept descriptions: size small size large small large + - triangle shape circle + shape - size circle triangle large small + - - + shape circle triangle + - -- 6 -

Quality Criteria on Trees Minimal depth of the tree: Minimal number of queries If the computation of attribute values have different costs: Minimal costs. Problem: The construction of such a decision tree is a very complex process (it is NP-complete) Therefore: Heuristic construction of decision trees -- 7 -

Logical Representation with Several Concepts Each path becomes a rule Conjunction of conditions along the path becomes the precondition Class at the leave nodes becomes the conclusion temperature warm cold windy humidity yes no high low rain sun rain snow IF Temperature = warm AND windy = yes THEN rain IF Temperature = warm AND windy = no THEN sun IF Temperature =cold AND humidity = high THEN rain IF Temperature = cold AND humidity = low THEN snow -- 8 -

Learning of Decision Trees from Examples Examples: temperature windy humidity class warm yes high rain warm no high sun cold yes high rain cold no low snow warm no low sun complete, consistent Decision tree: yes rain temperature warm cold windy humidity no high low sun rain snow -- 9 -

Alternative Decision Tree Examples: temperature windy humidity class warm yes high rain warm no high sun cold yes high rain cold no low snow warm no low sun high humidity low temperature cold warm rain windy yes rain no sun temperature cold warm snow sun -- 10 -

Part 2 The TDIDT Algorithm and Extensions -- 11 -

TDIDT (Top-Down Induction of Decision Trees) Basic Algorithm Given: set of classified examples E Algorithm: createtree (E) IF E = {} THEN RETURN leave node that is labeled with a suitable class IF all examples in E belong to the same class C THEN RETURN leave node labeled with C. Select some EFFICIENT attribute with domain {w 1,...,w n } Partition E into E 1,...,E n depending on the values of A compute recursively: T 1 := createtree(e 1 ),..., T n := createtree(e n ) RETURN Tree with root A and the edges labeled with w 1,...,w n to the subtrees T 1,...,T n -- 12 -

Parameters of the Basic Algorithm Admissible domains Choosen class if the example set of the partition is empty What is an efficient attribute? -- 13 -

ID3 (Quinlan, 1979) Conditions: Only finite domains, i.e. discrete symbolic attributes Attribute selection based on information gain Reason: Narrowing the search space Information theory provides a heuristics for attribute selection with the goal to minimize the number of queries We consider the value of some attribute as the result of a random experiment -- 14 -

Auswahl von Attributen How to Select an Efficient Attribute?(1) Possibilities: choose a random attribute choose an attribute with a minimal number of values choose an attribute with a maximal number of values choose an attribute such that the following subtrees are possibly small -- 15 -

How to Select an Efficient Attribute?(2) Select an attribute such that the following subtrees are a small as possible it contains as much information about the class of the object as possible as little additional information is necessary for finding the class of the object. Example: Suppose we are at the north pole. The attribute temparature is not efficient because we already know that it is always cold, i.e. the value of this attribute provides no information. -- 16 -

How to Select an Efficient Attribute?(3) Suppose we have two boolean attributes A and B such that Prob(A= 0 ) = Prob(A =1) = 1/2 Prob(B = 0) = 0.9, Prob(B =1) = 0.1 Then the information given by A is higher than the information given by B. This view ignores, however, that we are concerned with the information with respect to the classification. If e.g. the class is already decided by the value of B then by contains more information about the classification. Next we introduce the basic information theoretic concepts. -- 17 -

Information Theory for Attribute Selection Information theory as a heuristics for attribute selection with the goal to keep the average number of tests small. We consider random experiment in the sense of probability theory The results of an experiment are different events Random experiments for attribute selection: Determination of the attribute A i. Events: A i = w ij Determination of the class of an example. Events: possible classes humidity = high class: snow temperature = low -- 18 -

Information Content Suppose an experiment A is performed. What is the information content I(A) of the result A? In generai I(A) = f(p(a)), i.e. It depends on the probability. If C is the result representing the joint of independent results A and B then I(C) = I(A) + I(B) The more likely a result is the smaller is its information content. For very likely results we almost know them and do not need to ask additional queries. Requirements: If C is the result corresponding to the result of two independent events A and B then I(C) = I(A) + I(B). Consequence: f(xy) = f(x) + f(y). Normalization (arbitrary): f(½) = 1 This implies already f(x) = -ld(x), where ld is the logarithm of base 2. -- 19 -

Entropy (1) Definition: For some experiment A with n possible disjoint results A 1,...,A n the entropy for experiment A is H( A) P( A ) ld( P( A )) Example: n i 1 i i (Expected information content) Experiment with 2 possible results A 1 and A 2 with the probabilities P(A 1 )=0.01 and P(A 2 ) = 0.99: Entropy H(A) = - (0.01 * ld(0.01) + 0.99 * ld(0.99)) = 0.08 [Bit] -- 20 -

Entropy (2) Consider the entropy for P(A 1 )=p and P(A 2 ) = 1-p: H(p,1 p) 1 For p = 1: H(p,1 p) = H(1,0) = 1ld(1) ld(0) = 0 For p = 0: H(p,1 p) = H(0,1) = H(1,0) = 0 For p = ½: H(p,1 p) = H(½, ½) = ½ld(½) ½ld(½) = ½ +½ = 1 For p = ¾: H(p,1 p) = H(¾, ¼) = ¾ld(¾) ¼ld(¼) = 0,811 For p = ¼: H(p,1 p) = H(¼, ¾) = H(¾, ¼) = 0,811 ½ 1 p -- 21 -

Conditional Information Content If two experiments A and B are given: When B is performed then the event B has happened Question: What is now the information content if event A happens at experiment A? I(A B) is the ínformation content of event A after B has happened: I(A B) = -ld( P(A B) ) -- 22 -

Conditional Entropy Definition: If in some earlier experiment B the event B happened then the conditional entropy for experiment A under the assumption of B is: H( A B) P( A B) ld( P( A B)) i 1 Definition: If experiment B has the m disjoint outcomes B 1,...,B m, then the conditional entropy of experiment A under the assumption B is: m A B j 1 n i H( ) P( B ) H( A B ) j i j -- 23 -

Experiment (1) Three attributes: Size (small, large) Hair colour (blond, red,dark); Eye colour (blue, brown) Two classes (positive, negative) Presented examples: Size Hair colour Eye colour Class small blond blue + large red blue + large blond blue + large blond brown - small dark blue - large dark blue - large dark brown - small blond brown - -- 24 -

Example (2) Experiment A : Select an example Events: A 1 = class + ; A 2 = class - Probability estimates on the basis of the examples: P(A 1 ) = 3/8 P(A 2 ) = 5/8 Entropy H(A) = - 3/8 * ld(3/8) - 5/8 * ld(5/8) = 0.954 Size HairC. EyeC. Cl. Size HairC. EyC cl. small blond blue + large blond brown - large red blue + small dark blue - large blond blue + large dark blue - large dark brown - small blond brown - -- 25 -

Example (3) First: Experiment B 1 : Determine attribute Size Conditional Entropy: n H( A B) P( A B) ld( P( A B)) H(A large) = -2/5 * ld(2/5) - 3/5 * ld(3/5) = 0.971 H(A small) = -1/3 * ld(1/3) - 2/3 * ld(2/3) = 0.918 Conditional Entropy: P(large) = 5/8 P(small) = 3/8 H(A B ) = 5/8 * 0.971 + 3/8 * 0.918 = 0.951 i 1 i i Size HairC. EyeC. Cl. Size HairC. EyC cl. small blond blue + large blond brown - large red blue + small dark blue - large blond blue + large dark blue - large dark brown - small blond brown - -- 26 -

Example (4) First: Experiment B 1 : Determine attribute Eye color n Conditional Entropy: H( A B) P( A B) ld( P( A B)) i 1 H(A blue) = -3/5 * ld(3/5) - 2/5 * ld(2/5) = 0.971 H(A brown) = -0/3 * ld(0) - 3/3 * ld(3/3) = 0 i i Conditional entropy : H(A B ) = 5/8 * 0.971 + 3/8 * 0 = 0.607 Size HairC. EyeC. Cl. Size HairC. EyC cl. small blond blue + large blond brown - large red blue + small dark blue - large blond blue + large dark blue - large dark brown - small blond brown - -- 27 -

Example (5) Earlier: Experiment B 1 : Determine attribute hair colour Conditional Entropy: H(A blond) = -2/4 * ld(2/4) - 2/4 * ld(2/4) = 1 H(A red) = -1 * ld(1) - 0 * ld(0) = 0 H(A dark) = -1 * ld(1) - 0 * ld(0) = 0 H( A B) P( A B) ld( P( A B)) Conditional entropy : H(A B ) = 4/8 * 1 + 1/8 * 0 + 3/8 * 0 = 0.5 n i 1 i i Size HairC. EyeC. Cl. Size HairC. EyC cl. small blond blue + large blond brown - large red blue + small dark blue - large blond blue + large dark blue - large dark brown - small blond brown - -- 28 -

Selection Strategy Def.: The information gain after experiment B is H(A) - H(A B) Maximal information gain for an attribute means that H(A B) is minimal. Def.: Suppose S is a set of examples and A an attribute with Val(A) = {w 1,...,w n }. Take S j = { a S A(a) =w j } and s j = S j / S. Then the information gain of A w.r.t. S is gain(s,a) = H(S) s 1 H(S 1 )... s n H(S n ) Choose the attribute with highest information gain -- 29 -

Example (3): Optimal Tree H(..) = 0.954 H(..) = 0.951 Size Hair colour H(..) = 0.5 Eye colour H(..) = 0.607 large small blond red dark blue brown Size H(..) = 1 + - Eye colour H(..) = 1 H(..) = 0 large small blue brown + -- 30 - -

Important Extension C4.5 Extension by allowing (Quinlan, ca.1992) very large domains numerical (continuous) domains Combining of attribute values to groups Using given groups automatic formation of groups -- 31 -

Combination of Attribute Values in Groups Observation: Attributes with large discrete domains lead to very broad trees Partially unecessary distinctions are introduced Problem: For each partition there is only a small number of examples Therefore: Often lack of representative example sets for building subtrees Consequence: Low classification quality for new examples Basic idea: Combine values to groups -- 32 -

Formation of Groups (1) Groups are given in the representation language Possible form: Taxonomic domains Example: light colors dark white yellow orange blue green black light = { white, yellow, orange }, dark = { blue, green, black} Given: A complete partition of the domain into subsets (groups) G 1,...,G k Introduction of tests A i G 1,..., A i G k Selection of the most efficient test by the criterion of highest information gain -- 33 -

Formation of Groups (2) Groups can be given can be constructed automatically. Search for grouping with highest information gain General procedure: Hill climbing search in the space of all possible groupings Take the grouping with highest information gain -- 34 -

Algorithm for Automatic Grouping Given: Attribute A i with domain {w 1,...,w n } Actual partition of the TDIDT algorithm Algorithm: Initializing the groups W := { {w 1 },..., {w n } } REPEAT compute the information gain G for testing A i based on the actual groups W. W := W; G := G FOR each pair (U,V) W x W of groups with U V DO W = W \ { U } \ { V} { U V } (*merge 2 groups *) compute the information gain G for W IF G > G THEN W := W ; G := G UNTIL G = G OR W = 2 RETURN W -- 35 -

Extension by Numerical Domains (1) Attributes with continuous, ordered domains (e.g. natural numbers integers, reals) Introduction of a test of the form and use it for grouping: a i Z and a i > Z For a fixed Z compute the decision tree : yes A i (a) Z no Compute the information gain as before Goal: Choose Z such that the information gain is maximal Problem: There are infinitely many possibilities for Z! -- 39 -

Part 3 More Extensions -- 40 -

Extension by Numerical Domains (2) Determination of the threshold Z Suppose {v 1 <v 2...,<v m } are the values occurring in the attribute A in the examples of the partition. Consider the m-1 possible thresholds: vi vi 1 2 i=1...m-1 Select the most efficient test using the criterion of highest information gain -- 41 -

Looking at the Data There are many kinds of "noise" that could occur in the examples: Two examples have the same attribute, value pairs, but different classifications Some values of attributes are incorrect because of errors in the data acquisition process or the preprocessing phase The classification is wrong (e.g., + instead of -) because of some error Some attributes are irrelevant to the decision-making process. For example, the color of a die is irrelevant to its outcome. Consequences: Large influence of individually falsified examples Concepts get too complex Low classification quality for new examples -- 42 -

Overfitting and Underfitting There are two kinds of danger for learning correctly. First: Overfitting what means that one wants to be too exact. This occurs when the data are not fully exact because of noise or errors. Second: One is not exact enough what is called under fitting. A method that is not sufficiently complex. The model performs poorly on new examples as it is too simplistic to distinguish between new examples. -- 43 -

Overfitting Avoid too special hypotheses ( overfitting ) What is the most correct classification of real data? We do not know... But we let play the test data the role of the real data When can the problem of overfitting occur? Case 1: False data Case 2: Training data are not sufficlently representative; for instance the training set is too small. -- 44 -

Treatment of Noisy Examples A frequently occurring practical problem: Experiences are faulty Noise is a falsification of information contained in examples Two kinds of noise: Attribute noise: Falsification of one or more attribute values class noise: Falsification of the classification of an example Noise denotes a non-systematic error Measuring th degree of noise e.g. as percentage of falsified examples -- 45 -

Irrelevant Attributes The last problem, irrelevant attributes, can result in overfitting the training example data. For example if the hypothesis space has many dimensions because there are a large number of attributes, then we may find meaningless regularity in the data that is irrelevant to the true, important, distinguishing features. Question: How to deal with such problems? Idea: Modify the tree construction -- 46 -

Impact on Learning Methods Turning away from learning complete and consistent concepts Attempt: Try to cover as many examples as possible by simple concepts. Example: - - - - - - - - + + + - - + - - -Falsely classified example + + - - - + + + + + - - - - - - - - - - - -- 47 -

Pruning Up to now: Construction of a tree that classifies all seen test data correctly. If the correctness of test data is not guaranteed this is no longer meaningful and leads to overfitting. Basic idea: Stopping the generation of the tree if this is no longer meaningful Modifying a generated tree. Extensions of the TDIDT method employ pruning methods Fix by pruning lower nodes in the decision tree. For example, if Gain of the best attribute at a node is below a threshold, stop and make this node a leaf rather than generating children nodes. -- 48 -

Approaches for Pruning Pre-Pruning: The construction of the decision tree is interrupted following a certain criterion even if the class is not yet determined uniquely. Replace a partial tree by a leave Post-Pruning: 1. The decision tree is completely constructed 2. Determine different variants of pruned trees 3. Select some variant that is best according to some chosen criterion (error analysis). -- 49 -

Example (1) Pruning of the decision tree: Deletion of a subtree and replacement by a leave node Example: 2 Attributes: size (small, large), weight (natural numbers) concept to be learned: size = small weight < 5 positive examples: (s,1), (s,3), (s,4), (s,10) negative examples: (s,5), (s,10), (l,1),(l,2),(l.5),(l, 9) weight large size small weight <10 10 <5 5 - + + - -- 50 -

Example (2) large size small weight weight <10 10 <5 5 Misclassification if: seize = large weight 10 - + + - Pruned tree: - large size small weight concept is no more complete and consistent. <5 5 Only one training + - example is false classified. -- 51 -

Different Criteria for Pruning Measure using the complexity of the decision tree (e.g. number of nodes) and classification error on training data Partitioning of the set of training examples in a) a set for constructing the decision tree and b) a set for testing the classification quality for different variations of the Prunings. Select the variant with highest classification quality. Statistical methods for estimating the classification error. -- 52 -

Remarks on Pruning Cost-complexity pruning usually gives best results in experimental studies. Pessimistic pruning is most efficient and quite robust. Reduce-error pruning is rarely used, because it consumes training data. Pruning has relatively little effect on the correctness of classification. There is only a small number of possible prunings of a tree, and usually the serious errors made by splitting on the wrong nodes cannot be repaired by pruning. -- 53 -

Missing Values (1) Situation: Some attribute values are not represented unknown, too difficult to acquire etc. Representation: A i (a) =? An approach: Construct a tree that is not totally correct even on the test data Extend the values by a new value? Extend the concept of information and use the previous tree generation methods Take into account the not all examples are equally reliable The last aspect can be realized by introducing a weight for the examples: w(a) [0, 1]. -- 54 -

Missing Values (2) Put for an attribute A and an example set S: S! = {a S I A(a)?} IISII = S a S w(a) Now we can define a modified gain mod (S,A) by: gain mod (S,A) = IIS! II /IISII*gain(S,A) This allows to proceed the construction of the tree as before and we can use the same algorithm. -- 55 -

Advantages: Evaluation of TDIDT Methods Disjunctive concepts can be learned Several disjoint classes can be learned Efficieny by heuristic search restriction decision trees provide a strategy for chosing tests Disadvantages: Only attribute-value vectors for example representation Restricted to decision trees for concept representation Not incremental There are further variations to avoide these disadvantages -- 56 -

C4.5 ( ID3 Extension ) (1) One of the solution to solve this shortcoming of ID3 is presented in C4.5 which is an extension of ID3. At each node of the tree, C4.5 chooses one attribute of the data that most effectively splits its set of samples into subsets enriched in one class or the other. Its criterion is the normalized information gain(difference in entropy) that results from choosing an attribute for splitting the data. The attribute with the highest normalized information gain is chosen to make the decision.

C4.5 (2) Introducing a parameter called a Split Information which is defined as: Split Information is a parameter that is sensitive to how broadly and uniformly the attribute splits the data. -- 58 -

C4.5 ( Continued ) Instead of using Information Gain to choose an attribute in ID3, C4.5 uses Gain Ratio, which is defined as:

Algorithm (Sketch) For each attribute a Find the normalized information gain from splitting on a Let A_best be the attribute with the highest normalized information gain Create a decision node that splits on Aa_best Recurse on the sublists obtained by splitting on A_best, and add those nodes as children of node -- 60 -

Example (1) When to play golf? prediction play? occurences sunny yes 2x sunny no 3x cloudy yes 4x rain yes 3x rainy no 2x -- 61 -

Example (2) Info = - 9/14 ld (9/14) - 5/14 ld (5/14) = 0,940 bit Info Si = 5/14 ( - 2/5 ld (2/5) - 3/5 ld (3/5)) + 4/14 (- 4/4 ld (4/4) - 0/4 ld (0/4)) + 5/14 ( - 3/5 ld (3/5) - 2/5 ld (2/5)) = 0,694 bit => gain= 0,940 bit - 0,694 bits = 0,246 bit split info = -5/14 ld (5/14) - 4/14 ld (4/14) - 5/14 ld (5/14) = 1,577 bit gain ratio = 0,246 bit / 1,577 bits = 0,156 bit -- 62 -

Improvements of C4.5 over ID3 C4.5 made a number of improvements to ID3. Some of these are: Handling both continuous and discrete attributes - In order to handle continuous attributes, C4.5 creates a threshold and then splits the list into those whose attribute value is above the threshold and those that are less than or equal to it. [3] Handling training data with missing attribute values - C4.5 allows attribute values to be marked as? for missing. Missing attribute values are simply not used in gain and entropy calculations. Handling attributes with differing costs.

Further Improvement: C5.0 Speed - C5.0 is significantly faster than C4.5 (several orders of magnitude) Memory usage - C5.0 is more memory efficient than C4.5 Smaller decision trees - C5.0 gets similar results to C4.5 with considerably smaller decision trees. Weighting - C5.0 allows to weight different attributes and misclassification types. C5. is a commercial and closed-source product, although free source code is available for interpreting and using the decision trees and rule sets it outputs. -- 64 -

Cross-Validation for Experimental Validation of Performance 1. Divide all examples into N disjoint subsets, E = E1, E2,..., EN 2. For each i = 1,..., N do * Test set = Ei * Training set = E - Ei * Compute decision tree using Training set * Determine performance accuracy Pi using Test set 3. Compute N-fold cross-validation estimate of performance = (P1 + P2 +... + PN)/N -- 65 -

Applications Any kind of classification or diagnostic support where supervised learning takes place, in particular symbolic descriptions, e.g: Classification of reliable or not reliable banking customers Classification of car faults Classification of holiday resources Product lines -- 66 -

Tools Palisade: All New Decision Trees Suite 5.0. http://www.palisade.com/pr/proctergamble_pr_0102.asp C 5.0: http://www.rulequest.com/ WEKA 3 Matlab -- 67 -

Summary One of the most widely used learning methods in practice Can out-perform human experts in many problems Strengths: Fast; simple to implement; can convert result to a set of easily interpretable rules; empirically valid in many commercial products; handles noisy data Weaknesses: "Univariate" splits/partitioning using only one attribute at a time so limits types of possible trees; large decision trees may be hard to understand; requires fixed-length feature vectors; non-incremental (i.e., batch method) C4.5 and C5.0 -- 68 -

Recommended References J.R. Quinlan: Learning Efficient Classification Procedures and their Applications to Chess End Games. In: Machine Learning - An Artificial Intelligence Approach, ed. R.S. Michalski et al. Tioga, 1983. -- 69 -