Decision ree Decision ree Learning s of systems that learn decision trees: c4., CLS, IDR, ASSISA, ID, CAR, ID. Suitable problems: Instances are described by attribute-value couples he target function has discrete values Disjunctive descriptions of concepts may be required he ting set may contain errors (noise) he ting set may contain incomplete data c4. c4. [Qui9b,Qui96]: evolution of ID, also by J. R. Quinlan Inspired to one of the first decision tree learning system, CLS (Concept Learning Systems) by E.B. Hunt Benchmark for many learning systems Instances: Saturday mornings Classes: Good day for playing tennis Bad day for playing tennis Attributes outlook, discrete, values={,,} temperature, continuous humidity, continuous windy, discrete, values={true, false} 4
ing set Decision ree o D emp ( ) Humid (%) Class =? D D D4 8 7 90 8 9 D D6 69 7 90 Humidity =? D7 8 78 D8 D9 64 8 6 true false true false D0 7 D D 6 D 68 D4 96 6 = Decision ree Humidity : Humidity > : =: = =rue: =alse: otation Let be the ting set, Let {C,C,,C k } be the set of classes; 7 8
ree Building Algorithm of build_tree build_tree() returns a tree: contains examples from the same class Return a leaf with label the class contains examples from more than one class is partitioned into subsets,,, n according to a test on an attribute Call the algorithm recursively on the subsets: child i =build_tree( i ) for i=, n Return a subtree with the root associated to the test and childs child,,child n. 9 = o D D D D4 D D6 D7 D8 D9 D0 D D D D4 emp ( ) 8 7 69 7 8 64 8 7 6 68 Humid (%) 90 8 9 90 78 6 96 Class 0 est on est on = = o D D D D4 D emp ( ) 8 7 69 Humid (%) 90 8 9 Class o D0 D D D D4 emp ( ) 7 6 68 Humid (%) 96 Class = o emp ( ) Humid (%) Class D7 D8 D9 8 64 8 78 6
build_tree( ) build_tree( ) est: Humidity =, Humidity o emp ( ) D D 69 Leaf, label, Humidity > Humid (%) Class o D7 D8 D9 Leaf, label emp ( ) 8 64 8 Humid (%) 78 6 Class o emp ( ) Humid (%) Class D D D4 8 7 90 8 9 Leaf, label 4 build_tree( ) Decision ree est: =?,true = o D0 D Leaf, label,false = o emp ( ) 7 6 emp ( ) Humid (%) Humid (%) Class Class Humidity true false =? =? true false D D D4 68 96 Leaf, label 6
ests on Attributes Discrete attribute X with n possible values x,,x n : Equality with a constant: X=cost, possible outcomes: yes, no Equality test: X=?, n possible outcomes Membership in a set: X S, possible outcomes: yes, no Membership in a set of a partition of {x,,x n }: one outcome per set est on Discrete Attributes of membership in a set of a partition: Attribute, partition of the set of values {{},{,}} {} =? {,} Continuous attribute X Comparison with a threshold X cost, possible outcomes: yes, no 7 8 ermination Condition c4. stops When an uniform set is found When an empty set is found A leaf is returned with label the most frequent class in the father When no test is such that at least two subsets contain a minimum number of cases. he minimum number of cases is a userdefined parameter assuming value by default Building the ree Search in space of all possible trees Once a test is assigned to a node it is possible to backtrack Infeasible Greedy search ests on nodes chosen irrevocably: once a test is assigned to a node it is not possible to backtrack Choice on the basis of a heuristic Most used heuristics Entropy Gini index 9 0
Choice of the est Choice of the attribute Discrete attributes: Choice of the type of test ossibly choice of the constant or partition Continuous attributes Choice of the threshold Usually only the equality test X=? is used for discrete attributes Only the attribute must be chosen Constts that the test must satisfy: at least two among,,, n must contain a minimum number of examples Entropy In information theory, entropy is a measure of the uncertainty associated with a discrete random variable. Random variable C with k possible values C,,C k, entropy H(C) is given by k H ( C) = ( C j )log ( C j ) = E[ log j= Also known as Shannon entropy ( C)] Information heory Interpretation Suppose you want to transmit a series of messages made of the values of random variables independent and identically distributed (i.i.d.) What is the minimum number of bits necessary to transmit these messages? Source coding theorem (Shannon 948) he minimum number of bits necessary to encode the values of i.i.d. random variables with negligible risk of information loss is H(X) as tends to infinity, where H(X) is the entropy of the random variables H(X) is the minimum number of bits to encode the value of a random variable X Application to Decision rees Random variable C: class of an instance randomly selected from a set In this case ( C j ) = where j is the set of examples from that belong to class C j We can define the entropy of as the entropy of the random variable C H()=H(C) j 4
Entropy H() measures the minimum number of bits necessary for encoding, without loss, a message of the form he present example, randomly selected from the ting set, belongs to class C j H() is also called info() Entropy for wo Classes In the case of two classes, + and -, with probabilities p + =(+) and p - =(-) H ( ) p+ log p+ p log = p Only one variable is independent: p - =-p + H ( ) = p+ log p+ ( p+ ) log( p+ ) 6 Entropy for wo Classes Entropy for wo Classes or p + =0.: p - =0. H ) = 0. log ( 0. 0. 0. = 0. ( ) 0. ( ) =. or p + =0: p - =.0 H ( ) = 0 log 0 log = 0 0 In this case, we define p + def log p+ = 0 = lim p+ log + So H()=0-0=0 Similarly, if p + = H()=0 p p = + p 0 + 0 7 H() 0.8 0.6 0.4 0. 0 0 0. 0. 0. 0.4 0. 0.6 0.7 0.8 0.9 p+ 8
Entropy Entropy measures also the non-uniformity or impurity of a set: It is minimal when the set is most pure, i.e., when it contains examples from only one class It is maximal when the set is most impure, i.e., when it contains an equal number of examples of the two classes Intuition for wo Classes If all the examples belongs to the same class, I do not need to communicate the class of a randomly drawn example If the examples are uniformly distributed over classes in the ting set, I need to use bit on average: message 0 encodes one class, message the other 9 0 lay tennis problem ing set : 4 examples, 9 positive, negative Entropy of Remember: log a x=log 0 x/log 0 a=ln x/ln a log 0 =0.0 H()=-(9/4)log (9/4)-(/4)log (/4)= =-(9/4)log 0 (9/4)/0.0-(/4)log 0 (/4)/0.0= =-0.64*(-0.9)/0.0-0.7*(-0.447)/0.0= =0.40+0.0=0.940 Entropy for hree Classes hree classes {C,C,C }: p =(C ) p =(C ) p =(C ) Only two independent variables: p =-p -p H ( ) = p ( p log p p log p ) log ( p p ) p
Entropy for hree Classes Maximum of the Entropy. 0. H() he maximum of the entropy is obtained for a uniform distribution of the examples over classes or three classes: p =p =p =/=0, H()=-*/*log /=log =,8 or k classes, we get the maximum for (C j )=/k H max k ( ) = log = k log k = log j= k k k k p 0. 0. 0. 0.7 0. 0. 0. 0.7 0 p 4 c4. Heuristic Information Gain Assign to a test on an attribute the value given by the decrease of entropy (information gain) due to the partitioning of the set of examples according to the test est X with n possible outcomes Set of examples is partitioned into n subsets,, n Entropy after the partitioning: weighted average entropy in the subsets n i H X ( ) = H ( ) i = i gain( X ) = H ( ) H ( ) It is equivalent to the number of bits necessary for encoding the class of examples that we save when we also know the result of the test X on the examples Entropy decreases as more and more uniform subsets are obtained X 6
Attribute Choice he attribute with the highest information gain is selected for splitting the current set of examples est on the attribute : = => contains 6 positive and negative examples = => contains positive and negative examples =[9,] =[6,] =[,] 7 8 otal lay 6 9 Don t lay H( )=-6/8*log (6/8)-/8*log (/8)=0.8 H( )=-/6*log (/6)-/6*log (/6)= gain()=h()-(8/4*h( )+6/4*H( ))= 0.940-(8/4)*0.8-(6/4)*=0.048 otal 8 6 4 est on the attribute outlook: ==> contains pos and neg ==> contains 4 pos and 0neg ==> contains pos and neg =[9,], =[,], =[,], =[4,0] 9 40
roblems of Information Gain Sunny Overcast Rain otal lay 4 9 Don t lay 0 otal 4 4 High tendency to favor tests with many results : test on an attribute that is a key: i =, n = [,, K, n] H ( ) 0 H ( ) = 0 i = i X gain()=h()-((/4)*h( )+(/4)*H( )+ (4/4)*H( ))= 0.940-0.7*0.97-0.7*0.97-0.86*0=0.46 So the gain is maximum: gain ( X ) = H ( ) : diagnosis problem, examples=patients, attribute= atient s name Useless attribute but has the highest gain 4 4 ormalized Gain he gain is normalized with respect to the entropy of the test itself he random variable in this case is the value of the attribute X n i i H ( X ) = = log i Gain ratio gain( X ) gainratio ( X ) = H ( X ) Gain Ratio in the Diagnosis example X=patient name gain( X ) = H ( ) log k n H ( X ) = log = n log n = log n n n i= log gainratio( X ) log If n>>k gain ratio(x) is small H(X) is also called splitinfo k n n 4 44
gainratio()=gain()/h() H()=-8/4*log (8/4)-6/4*log (6/4)=0.98 gainratio()=0.048/0.98=0.048 gainratio()=gain()/h() H()=-/4*log (/4)-/4*log (/4) - 4/4*log (4/4)=.77 gainratio()=0.46/.77=0.6 Other impurity measure or two classes gini Gini Index gini( ) = k j= ( C j ( ) = p+ ( p+ ) = p+ p+ ) 4 46 Gini Index Gini index he max of Gini index is achieved for p + =p - =0,: gini()=0, he min is achieved for p + =0 or p - =0: gini()=0 or three classes p ( p ) gini( ) = p p gini 0.6 0. 0.4 0. 0. 0. 0-0. 0 0. 0. 0. 0.4 0. 0.6 0.7 0.8 0.9 p+ 47 48
Gini Index for hree Classes Gini Index 0,7 0,6 0, 0,4 gini 0, 0, In general, the Gini index has a maximum for p =p =...p k =/k k gini ) = i= k = k k ( = k and a minimum for p =, p =...p k =0: gini()=0 0, p 0 0, 0,4 0,6 0,8 0 0, 0,4 0,6 0,8 0 p 49 0 Gini index Gini index after the split on attribute X n i gini X ( ) = gini( ) j = i he attribute X that gives the lowest gini X () is chosen for the split Gini index is used by CAR ests on Continuous Attributes Suppose continuous attribute X assumes m values in Let <V,V,.,V m > be the values ordered from the smallest to the largest he test X V i divides the m values into two groups: X V i : {V,,V i } X > V i : {V i+,,v m } So we have m- candidates for the threshold: V,..., V m- Each candidate must be evaluated
ests on Continuous Attributes ests on Continuous Attributes Evaluation of a test: Sort the examples on the basis of the attribute X Count the examples Let e j,i be the number of examples that have value V i for X and belong to class C j for j=,,k, i=,,m- Let e j be the number of examples that belong to class C j for k=,,k Each test has two possible outcomes: yes or no d j,yes =examples of class j in branch X V i d j,no =examples of class j in branch X > V i or j= to k d j,yes =0 d j,no =e j or i= to m- Let the test be X V i or j= to k d j,yes :=d j,yes +e j,i d j,no :=d j,no -e j,i he heuristic for attribute X, threshold V i can be computed from the values d,yes,,d k,yes,d,no,,d k,no 4 Attributes with Unknown Value If the ting set contains examples with one or more attributes unspecified or example <?,7,90,,> How do we take into account this example when computing the heuristic and when splitting the example set? Attributes with Unknown Value est evaluation: Consider a discrete attribute X with n values {x,,x n } Let be the set of examples of that have X known provides us with a distribution of examples into values and class (x i,c j ) We assume that the unknown values have the same distribution (x i,c j ) with respect to the attribute and the class herefore the examples with unknown values do not alter the value of the entropy or of the Gini index 6
Attributes with Unknown Value Entropy of the partitioning: the unknown value is considered as a value of its own Let n+ be the set of examples of with unknown value for X Attributes with Unknown Value c4. further penalizes attributes with unknown values by multiplying the gain by the probability that the attribute is known If is the set of examples of with X known = n + i i H ( X ) log = i gain( X ) = ( H ( ) H X ( )) 7 8 Suppose that in the example database the case D6 : =, emperature=7, Humidity=90, = is replaced by =?, emperature=7, Humidity=90, = he frequencies of over are Sunny Overcast Rain otal lay 8 Don t lay 0 otal 9 Sunny Overcast Rain otal lay 8 Don t lay H()=-8/*log (8/)-/*log (/)=0.96 H ()=/*(-/*log (/)-/*log (/)) +/*(-/*log (/)-0/*log (0/)) +/*(-/*log(/)-/*log(/)) =0.747 gain()=/4*(0.96-0.747)=0.99 (slightly less than before) 0 otal 60
H()=-/4*log (/4)-/4*log (/4)- /4*log (/4)-/4*log (/4)=.9 gainratio()=0.99/.9=0.0 artitioning In which subset of a node with test on X do we put an example with X unknown? artitioning is generalized in a probabilistic sense: Each example of is assigned a weight, initially Each example is inserted in every subset i with a weight If we partition on discrete attribute X If example e, with weight w in, has X=x i, e is put in i with weight w and in j with j i with weight 0 If example e, with weight w in, has X=?, e is put in i with weight w (x i ) (x i ) can be estimated by relative frequency in 6 6 artitioning according to o problem for the cases of D6 is assigned to sets, and with weights /, / e / respectively Consider o emp Humidity Class Weight D D 90 D 8 8 D4 7 9 D 69 D6? 7 90 / If is partitioned on Humidity with threshold, we get the following distribution: Humidity class, 0 class Humidity > / class, class 6 64
he second subset still contains examples from two classes but no test produces two subsets with at least two examples=> stop = Humidity : (.0) Humidity > : (.4/0.4) =: (.) = =rue: (.4/0.4) Wndy=alse: (.0) Output Interpretation umbers (A/B) associated to leaves A=total weight of examples associated to the leaf B=total weight of examples associated to the leaf that are misclassified or example (.4/0.4) Means that.4 cases belong to the leaf of which 0.4 do not belong to class 6 66 Classification of Unseen Cases with All Values Known An unseen (new) case e with all attributes known is classified by raversing the tree from the root by following the branches that correspond to the values of the case Suppose that the leaf os (A/B) is reached, then e is classified as os with probability (A-B)/A and eg with probability B/A In the case of more than two classes, the distribution of examples into classes at the leaf is used Classification of Unseen Cases with Unkown Values If e has X unknown e is associated with a weight w, initially set to When traversing the tree, if the attribute at the current node is unknown in e we explore all subtrees. Subtree i is explored by assigning e to i with weight w (x j ) (x j ) is estimated by considering by relative frequency over he information about relative frequencies are stored in the leaves that are descendants of 67 68
Classification of Unseen Cases with Unkown Values In the end more than one leaf will be reached Let L be the set of leaves that are reached Let w l be the weight of e that reaches l Let (C j l) be the relative frequency of examples in l belonging to class C j hen ( C e) = w ( C l) j l L l j Unseen example e =, emperature=, Humidity=?, = = => first subtree Humidity=?=> we cannot determine whether Humidity We follow both branches with weights Branch Humidity : w=.0/.4=0. Branch Humidity > : w=.4/.4=0.60 69 Leaf humidity : ( l)=00% ( l)=0% Leaf humidity > ( l)= /.4=88% ( l)= 0.4/.4=% Overall we get ( e)=0.*00%+0.60*%=44% ( e)=0.60*88%=6% Overfitting A hypothesis overfits the data when there exists a simpler hypothesis less accurate on the ting set but more accurate on the universe It may happen when the ting set contains errors or when the ting set is small 7 7
Overfitting Overfitting : consider the previous ting set plus <,,,strong,> which should be positive but it is erroneously classified as negative he tree h learned from the new ting set is more complex than the tree h learned from the original ting set However, since the new example is incorrect, h will make more mistakes than h on U 7 74 Approaches to Avoid Overfitting Stop the growth before reaching a tree that perfectly classifies the ting examples (prepruning) Grow the tree and prune it afterwards (postpruning) How to Determine the Optimal Dimension of the ree Using a separate example set for evaluating the tree ( ting and validation set approach) Using all the available examples for ting but using a statistical test to decide whether to keep a node or not Using an explicit measure of the complexity of encoding the tree and the examples that are exceptions and choose a tree that (locally) minimizes this measure (minimum description length principle) 76
Observation on c4. Search c4. performs a hill-climbing search in the space of possible trees from the simplest hypothesis towards more and more complex hypothesis he space of all possible trees is equivalent to the powerset of the set of examples so the target concept is surely included in the search space c4. mantains only a single hypothesis during the search and does not perform backtracking, so it may find a solution only locally optimal Observations on c4. Search c4. uses all the ting examples in each step for deciding how to refine the tree differently from other methods that make decisions in an incremental way on the basis of individual examples. As a result, c4. is less sensible to errors in single examples. 77 78 Useful Links C4. http://www.rulequest.com/ersonal/ C source code Weka: open source suite of machine learning algorithms written in Java http://www.cs.waikato.ac.nz/ml/weka/ 79 References [Mit97]. M. Mitchell, Machine Learning, McGraw-Hill, 997 [Qui9b] J. R. Quinlan, C4.: rograms for machine learning, Morgan Kaufmann ublishers, San Mateo, California, 99 [Qui96] J. R. Quinlan, Improved Use of Continuous Attributes in C4., Journal of Artificial Intelligence Research, 4, pag. 77--90, 996. ftp://ftp.cs.cmu.edu/ project/jair/volume4/quinlan96a.ps [Wit0] I.H. Witten, E. rank, Data Mining: ractical Machine Learning ools and echniques with Java Implementations, Second Edition, Morgan Kaufmann, 00.