Data Mining and Analysis: Fundamental Concepts and Algorithms

Transcription

1 Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA 2 Department of Computer Science Universidade Federal de Minas Gerais, Belo Horizonte, Brazil Chapter 19: Decision Tree Classifier Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 1 / 23

2 Decision Tree Classifier Let the training dataset D = {x i, y i } n i=1 consist of n points in a d-dimensional space, with y i being the class label for point x i. A decision tree classifier is a recursive, partition-based tree model that predicts the class ŷ i for each point x i. Let R denote the data space that encompasses the set of input points D. A decision tree uses an axis-parallel hyperplane to split the data space R into two resulting half-spaces or regions, say R 1 and R 2, which also induces a partition of the input points into D 1 and D 2, respectively. Each of these regions is recursively split via axis-parallel hyperplanes until most of the points belong to the same class. To classify a new test point we have to recursively evaluate which half-space it belongs to until we reach a leaf node in the decision tree, at which point we predict its class as the label of the leaf. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 2 / 23

3 Decision Tree: Recursive Splits h X 2 h 4 R 1 h 0 h 5 R 5 z R 6 h 3 R 3 R 4 R X 1 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 3 / 23

4 Decision Tree Yes X No Yes X X Yes No No c 1 44 c 2 1 R 1 Yes c 1 0 c 2 90 R 2 X Yes No X No c 1 1 c 2 0 R 3 c 1 0 c 2 6 R 4 c 1 5 c 2 0 R 5 c 1 0 c 2 3 R 6 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 4 / 23

5 Decision Trees: Axis-Parallel Hyperplanes A hyperplane h(x) is defined as the set of all points x that satisfy the following equation h(x): w T x+b = 0 where w R d dd is a weight vector that is normal to the hyperplane, and b is the offset of the hyperplane from the origin. A decision tree considers only axis-parallel hyperplanes, that is, the weight vector must be parallel to one of the original dimensions or axes X j : h(x): x j + b = 0 where the choice of the offset b yields different hyperplanes along dimension X j. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 5 / 23

6 Decision Trees: Split Points A hyperplane specifies a decision or split point because it splits the data space R into two half-spaces. All points x such that h(x) 0 are on the hyperplane or to one side of the hyperplane, whereas all points such that h(x) > 0 are on the other side. The split point is written as h(x) 0, i.e. X j v where v = b is some value in the domain of attribute X j. The decision or split point X j v thus splits the input data space R into two regions R Y and R N, which denote the set of all possible points that satisfy the decision and those that do not. Categorical Attributes: For a categorical attribute X j, the split points or decisions are of the X j V, where V dom(x j ), and dom(x j ) denotes the domain for X j. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 6 / 23

7 Decision Trees: Data Partition and Purity Each split of R into R Y and R N also induces a binary partition of the corresponding input data points D. A split point of the form X j v induces the data partition D Y = {x x D, x j v} D N = {x x D, x j > v} The purity of a region R j is the fraction of points with the majority label in D j, that is, { } nji purity(d j ) = max i where n j = D j is the total number of data points in the region R j, and n ji is the number of points in D j with class label c i. n j Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 7 / 23

8 Decision Trees to Rules Yes X No Yes X X Yes No No c 1 44 c 2 1 R 1 Yes c 1 0 c 2 90 R 2 X Yes No X No c 1 1 c 2 0 R 3 c 1 0 c 2 6 R 4 c 1 5 c 2 0 R 5 c 1 0 c 2 3 R 6 A tree is a set of decision rules; each comprising the decisions on the path to a leaf: R 3 : If X and X and X 1 4.7, then class is c 1, or R 4 : If X and X and X 1 > 4.7, then class is c 2, or R 1 : If X and X 2 > 2.8, then class is c 1, or R 2 : If X 1 > 5.45 and X , then class is c 2, or R 5: If X 1 > 5.45 and X 2 > 3.45 and X 1 6.5, then class is c 1, or R 6 : If X 1 > 5.45 and X 2 > 3.45 and X 1 > 6.5, then class is c 2 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 8 / 23

9 Decision Tree Algorithm The method takes as input a training dataset D, and two parameters η and π, where η is the leaf size and π the leaf purity threshold. Different split points are evaluated for each attribute in D. Numeric decisions are of the form X j v for some value v in the value range for attribute X j, and categorical decisions are of the form X j V for some subset of values in the domain of X j. The best split point is chosen to partition the data into two subsets, D Y and D N, where D Y corresponds to all points x D that satisfy the split decision, and D N corresponds to all points that do not satisfy the split decision. The decision tree method is then called recursively on D Y and D N. We stop the process if the leaf size drops below η or if the purity is at least π. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 9 / 23

10 Decision Tree Algorithm DECISIONTREE (D, η, π): 1 n D // partition size 2 n i {x j x j D, y j = c i } // size of class c { i ni } 3 purity(d) max i n 4 if n η or purity(d) π then// stopping condition c { arg max ni } 5 ci n // majority class 6 create leaf node, and label it with class c 7 return 8 (split point, score ) (, 0)// initialize best split point 9 foreach (attribute X j ) do 10 if (X j is numeric) then 11 (v, score) EVALUATE-NUMERIC-ATTRIBUTE(D,X j ) 12 if score > score then (split point, score ) (X j v, score) else if (X j is categorical) then (V, score) EVALUATE-CATEGORICAL-ATTRIBUTE(D,X j ) if score > score then (split point, score ) (X j V, score) D Y {x D x satisfies split point } D N {x D x does not satisfy split point } create internal node split point, with two child nodes, D Y and D N DECISIONTREE(D Y ); DECISIONTREE(D N ) Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 10 / 23

11 Split Point Evaluation Measures: Entropy Intuitively, we want to select a split point that gives the best separation or discrimination between the different class labels. Entropy measures the amount of disorder or uncertainty in a system. A partition has lower entropy (or low disorder) if it is relatively pure, that is, if most of the points have the same label. On the other hand, a partition has higher entropy (or more disorder) if the class labels are mixed, and there is no majority class as such. The entropy of a set of labeled points D is defined as follows: H(D) = k P(c i D) log 2 P(c i D) i=1 where P(c i D) is the probability of class c i in D, and k is the number of classes. If a region is pure, that is, has points from the same class, then the entropy is zero. On the other hand, if the classes are all mixed up, and each appears with equal probability P(c i D) = 1 k, then the entropy has the highest value, H(D) = log 2 k. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 11 / 23

12 Split Point Evaluation Measures: Entropy Define the split entropy as the weighted entropy of each of the resulting partitions H(D Y, D N ) = n Y n H(D Y)+ n N n H(D N) where n = D is the number of points in D, and n Y = D Y and n N = D N are the number of points in D Y and D N. Define the information gain for a split point as Gain(D, D Y, D N ) = H(D) H(D Y, D N ) The higher the information gain, the more the reduction in entropy, and the better the split point. We score each split point and choose the one that gives the highest information gain. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 12 / 23

13 Split Point Evaluation Measures: Gini Index and CART Measure Gini Index: The Gini index is defined as follows: k G(D) = 1 P(c i D) 2 i=1 If the partition is pure, then Gini index is 0. The weighted Gini index of a split point is as follows: G(D Y, D N ) = n Y n G(D Y)+ n N n G(D N) The lower the Gini index value, the better the split point. CART: The CART measure is CART(D Y, D N ) = 2 n Y n n N n k P(c i D Y ) P(c i D N ) This measure thus prefers a split point that maximizes the difference between the class probability mass function for the two partitions; the higher the CART measure, the better the split point. i=1 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 13 / 23

14 Evaluating Split Points: Numeric Attributes All of the split point evaluation measures depend on the class probability mass function (PMF) for D, namely, P(c i D), and the class PMFs for the resulting partitions D Y and D N, namely P(c i D Y ) and P(c i D N ). We have to evaluate split points of the form X v. We consider only the midpoints between two successive distinct values for X in the sample D. Let {v 1,...,v m } denote the set of all such midpoints, such that v 1 < v 2 < < v m. For each split point X v, we have to estimate the class PMFs: ˆP(c i D Y ) = ˆP(c i X v) ˆP(c i D N ) = ˆP(c i X > v) Using Bayes theorem, we have ˆP(c i X v) = ˆP(X v c i )ˆP(c i ) ˆP(X v) = ˆP(X v c i )ˆP(c i ) k j=1 ˆP(X v c j )ˆP(c j ) Thus we have to estimate the prior probability and likelihood for each class in each partition. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 14 / 23

15 Evaluating Split Points: Numeric Attributes The prior probability for each class in D can be estimated as ˆP(c i ) = 1 n n j=1 I(y j = c i ) = n i n where y j is the class for point x j, n = D is the total number of points, and n i is the number of points in D with class c i. Define N vi as the number of points x j v with class c i, where x j is the value of data point x j for the attribute X, given as N vi = n I(x j v and y j = c i ) j=1 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 15 / 23

16 Evaluating Split Points: Numeric Attributes We can estimate P(X v c i ) and ˆP(X > v c i ) as follows: ˆP(X v c i ) = N vi n i ˆP(X > v c i ) = 1 ˆP(X v c i ) = n i N vi n i Finally, we have ˆP(c i D Y ) = ˆP(c i X v) = N vi k j=1 N vj ˆP(c i D N ) = ˆP(c i X > v) = n i N vi k j=1 (n j N vj ) The total cost of evaluating a numeric attribute is O(n log n+nk), where k is the number of classes, and n is the number of points. Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 16 / 23

17 Algorithm EVALUATE-NUMERIC-ATTRIBUTE EVALUATE-NUMERIC-ATTRIBUTE (D, X): sort D on attribute X, so that x j x j+1, j = 1,...,n 1 M // set of midpoints for i = 1,...,k do n i 0 for j = 1,...,n 1 do if y j = c i then n i n i + 1// running count for class c i if x j+1 x j then v x j+1 + x j ; M M {v}// midpoints 2 for i = 1,...,k do N vi n i // Number of points such that x j v and y j = c i if y n = c i then n i n i + 1 v ; score 0// initialize best split point forall v M do for i = 1,...,k do ˆP(c i D Y ) ˆP(c i D N ) N vi kj=1 N vj n i N vi kj=1 n j N vj score(x v) Gain(D, D Y, D N ) if score(x v) > score then v v; score score(x v) 19 return (v, score ) Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 17 / 23

18 Iris Data: Class-specific Frequencies N vi Classes c 1 and c 2 for attributesepal length Frequency: Nvi other (c 2 ) v = 5.45 iris-setosa (c 1 ) Midpoints: v Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 18 / 23

19 Iris Data: Information Gain for Different Splits Information Gain sepal-width(x 2 ) sepal-length(x 1 ) X Split points: X i v Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 19 / 23

20 Categorical Attributes For categorical X the split points are of the form X V, where V dom(x) and V. All distinct partitions of the set of values of X are considered. If m = dom(x), then there are O(2 m 1 ) distinct partitions, which can be too many. One simplification is to restrict V to be of size one, so that there are only m split points of the form X j {v}, where v dom(x j ). Define n vi as the number of points x j D, with value x j = v for attribute X and having class y j = c i : n vi = n I(x j = v and y j = c i ) The class conditional empirical PMF for X is then given as ˆP ( ) X = v and c i ˆP(X = v c i ) = = n vi ˆP(c i ) n i j=1 We then have ˆP(c i D Y ) = v V n vi k j=1 v V n vj ˆP(c i D N ) = v V n vi k j=1 v V n vj Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 20 / 23

21 Algorithm EVALUATE-CATEGORICAL-ATTRIBUTE EVALUATE-CATEGORICAL-ATTRIBUTE (D, X, l): for i = 1,...,k do n i 0 forall v dom(x) do n vi 0 for j = 1,...,n do if x j = v and y j = c i then n vi n vi + 1// frequency statistics // evaluate split points of the form X V V ; score 0// initialize best split point forall V dom(x), such that 1 V l do for i = 1,...,k do v V ˆP(c i D Y ) n vi kj=1 ˆP(c i D N ) v V n vj v V n vi kj=1 v V n vj score(x V) Gain(D, D Y, D N ) if score(x V) > score then V V; score score(x V) return (V, score ) Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 21 / 23

22 Discretizedsepal length: Class Frequencies Bins v: values Class frequencies (n vi ) c 1 :iris-setosa c 2 :other [4.3, 5.2] Very Short (a 1 ) 39 6 (5.2, 6.1] Short (a 2 ) (6.1, 7.0] Long (a 3 ) 0 43 (7.0, 7.9] Very Long (a 4 ) 0 12 Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 22 / 23

23 Categorical Split Points forsepal length Best split: X {a 1 }. V Split entropy Info. gain {a 1 } {a 2 } {a 3 } {a 4 } {a 1, a 2 } {a 1, a 3 } {a 1, a 4 } {a 2, a 3 } {a 2, a 4 } {a 3, a 4 } Zaki & Meira Jr. (RPI and UFMG) Data Mining and Analysis Chapter 19: Decision Tree Classifier 23 / 23