Level Construction of Decision Trees in a Partition-based Framework for Classification

Transcription

1 Level Construction of Decision Trees in a Partition-base Framework for Classification Y.Y. Yao, Y. Zhao an J.T. Yao Department of Computer Science, University of Regina Regina, Saskatchewan, Canaa S4S 0A2 {yyao, yanzhao, jtyao}@cs.uregina.ca Abstract A partition-base framework is presente for a formal stuy of consistent classification problems. An information table is use as knowlege representation. Solutions to, an solution space of, classification problems are formulate in terms of partitions. Algorithms for fining solutions are moele as searching in a space of partitions uner a refinement orer relation. We focus on a particular type of solutions calle conjunctively efinable partitions. Two level construction methos for ecision trees are investigate. Experimental results are reporte to compare the two level construction methos. 1. Introuction Classification is one of the main tasks in machine learning, ata mining an pattern recognition [1, 3, 4]. It eals with classifying labele objects. Knowlege for classification can be expresse in ifferent forms, such as classification rules, iscriminant functions, ecision trees an ecision graphs. Classification by ecision trees is a popular metho. The typical algorithms for ecision tree learning are the ID3 algorithm [6] an its escenent, the C4.5 algorithm [7]. Typically, ID3-like algorithms buil a ecision in a top-own, epth-first moe. Furthermore, the noe splitting criteria are base on local optimization. When splitting a noe, an attribute is chosen base on only information about this noe, but not on any other noes in the same level. Consequently, ifferent noes in the same level may use ifferent attributes, an the same attribute may be use at ifferent levels. The use of local optimal criteria makes it ifficult to juge the overall quality of the partial ecision tree uring its construction process. The main objective of the paper is to stuy a topown, breath-first, level-wise moe for constructing ecision trees. Two types of algorithms are propose an stuie. One is base on local optimization noe splitting criteria, an the other is base on global optimization criteria. The former is referre to as the level construction version of ID3 an is enote by LID3. The latter is in fact a levelwise, reuct base methos an is enote by klr. The klr algorithm combines the methos for constructing ecision trees an the methos for searching for reucts [4, 5, 9]. The rest of the paper is organize in two layers. A formal framework for classification is presente in Section 2, which sets the stage for the algorithmic stuies. Level construction algorithms are iscusse in Section 3 an their experimental evaluations are reporte in Section 4. The propose methos offer a complementary approach to epthfirst ID3. The ecision trees obtaine from the algorithms enables us to see the ifferent aspects of knowlege embee in ata. 2. Consistent Classification Problems 2.1. Information tables An information table provies a convenient way to escribe a finite set of objects by a finite set of attributes. It eals with the issues of knowlege representation for classification problems [5, 11]. An information table S is the tuple: S = (U, At, L, {V a a At}, {I a a At}), where U is a finite nonempty set of objects, At is a finite nonempty set of attributes, L is a language efine by using attributes in At, V a is a nonempty set of values for a where a At, an I a : U V a is an information function. Formulas of L are efine by the following two rules: (i) An atomic formula φ of L is a escriptor a = v, where a At an v V a ; (ii) The well-forme formulas (wff) of L is the smallest set containing the atomic formulas an close uner,,, an.

2 If φ is a formula, the set m S (φ) efine by m S (φ) = {x U x = φ}, is calle the meaning of the formula φ in S. If S is unerstoo, we simply write m(φ). The meaning of a formula φ is the set of all objects having the properties expresse by the formula φ. A connection between formulas of L an subsets of U is thus establishe. The notion of efinability of subsets in an information table is essential to ata analysis. In fact, efinable subsets are the basic units that can be escribe an iscusse, upon which other notions can be evelope. A subset X U is calle a efinable granule in an information table S if there exists at least one formula φ such that m(φ) = X. For a subset of attributes A At, X is an A-efinable granule if there exists at least one formula φ A using only attributes from A such that m(φ A ) = X. In many classification algorithms, one is only intereste in formulas of a certain form. Suppose we restrict the connectives of language L to only the conjunction connective. A subset X U is a conjunctively efinable granule in an information table S if there exists a conjunctor φ such that m(φ) = X Partitions in an information table Classification involves the ivision of the set U of objects into many classes. The notion of partitions provies a formal means to escribe classification. Definition 1 A partition π of a set U is a collection of nonempty an pair-wise isjoint subsets of U whose union is U. The subsets in a partition are calle blocks. Different partitions may be relate to each other. A partition π 1 is a refinement of another partition π 2, or equivalently, π 2 is a coarsening of π 1, enote by π 1 π 2, if every block of π 1 is containe in some block of π 2. The refinement relation is a partial orer, namely, it is reflexive, antisymmetric, an transitive. It efines a partition lattice Π(U). A partition π is calle a efinable partition in an information table S if every block of π is a efinable granule. A partition π is calle a conjunctively efinable partition if every equivalence class of π is a conjunctively efinable granule. Consier two special families of conjunctively efinable partitions below. The first family is the uniformly conjunctively efinable partitions, which has been stuie extensively in atabases [2]. A partition π is calle a uniformly conjunctively efinable partition in an information table S if, given a subset A of attributes in a certain orer, the blocks are further refine in a process that the attributes are one-by-one ae to the conjunctor. π = U is the coarsest partition, an π At is the finest partition, for any A At, we have π At π A π. The secon family is the non-uniformly conjunctively efinable partitions, which has been use extensively in machine learning [6]. A partition π is calle a non-uniformly conjunctively efinable partition in an information table S if the partition blocks select their own optimal attributes to further ivision, accoring to a consistent selection criteria. This strategy results in that the partition blocks at the same level may use ifferent attributes for refinement partition. Let Π tree be the set of non-uniformly conjunctively efinable partitions. The partial orer can be carrie over to Π tree. Suppose π t Π tree. It can be easily verifie that π At π t π. 2.3 Solutions to consistent classification problems In an information table for classification problems, we have a set of attribute At = F {class}. The problem can be formally state in terms of partitions. Definition 2 An information table is sai to efine a consistent classification if objects with the same escription have the same class value, namely, for any two objects x, y U, I F (x) = I F (y) implies I class (x) = I class (y). Suppose π is an A-efinable partition, that is, each block of π is an A-efinable granule. We say that π is a solution to the consistent classification problem, if π π class. A solution π is calle a most general solution if there oes not exist another solution π such that π π π class, where π π stans for π π an π π. Suppose π is a solution to the classification problem, namely, π π class. For a pair of equivalence classes X π an C π class with X C, we can erive a classification rule Des(X) = Des(C), where Des(X) an Des(C) are the formulas that escribe sets X an C, respectively. Let Π sol be the set of all solutions calle solution space. The partition π F is the minimum element of Π sol. For two partitions with π 1 π 2, if π 2 is a solution, then π 1 is also a solution. For two solutions π 1 an π 2, π 1 π 2 is also a solution. The solution space Π sol contains the trivial solution π F, an is close uner meet. The solution space is a meet sub-lattice. In most cases, we are intereste in the most general solutions, instea of the trivial solution π F. In many practical situations, one is satisfie with an approximate solution of the classification problem, instea of an exact solution. Definition 3 Let ρ : π π R +, where R + stans for non-negative reals, be a function such that ρ(π 1, π 2 ) measures the egree to which π 1 π 2 is true. For a threshol

3 α, a partition π is sai to be an approximate solution if ρ(π, π class ) α. The measure ρ can be efine to capture various aspects of classification. Two such measures are iscusse below, they are the ratio of sure classification (RSC), an the accuracy of classification. Definition 4 For the partition π = {X 1, X 2,..., X m }, the ratio of sure classification (RSC) by π is given by: m i=1 ρ 1 (π, π class )= {X i π C j π class, X i C j }, U (1) where enotes that carinality of a set. The ratio of sure classification represents the percentage of objects that can be classifie by π without any uncertainty. The measure ρ 1 (π, π class ) reaches the maximum value 1 if π π class, an reaches the minimum value 0 if for all blocks X i π an C j π class, X i C j oes not hol. For two partitions with π 1 π 2, we have ρ 1 (π 1, π class ) ρ 1 (π 2, π class ). Definition 5 For the partition π = {X 1, X 2,..., X m }, the accuracy of classification by a partition is efine by: m i=1 ρ 2 (π, π class ) = X i C j(xi ), (2) U where C j(xi ) = arg max{ C j X i C j π class }. The accuracy of π is in fact the weighte average accuracies of iniviual rules. The measure ρ 2 (π, π class ) reaches the maximum value 1 if π π class, an reaches the minimum value C k0 / U, where C k0 is the class with the maximum number of objects. For two partitions with π 1 π 2, we have ρ 2 (π 1, π class ) ρ 2 (π 2, π class ). Aitional measures can also be efine base on the properties of partitions. For example, one may use information-theoretic measures [12] Classification as search Conceptually, fining a solution to a classification problem can be moele as a search in the space of A-efinable partitions uner the orer relation. A ifficulty with this straightforwar search is that the space is too large to be practically applicable. One may avoi such a ifficulty in several ways, for instance, only searching the space of conjunctively efinable partitions. In particular, in searching the conjunctively efinable partitions, two special cases eserve consieration, namely, the space of uniformly conjunctively efinable partitions, an the space of non-uniformly conjunctively efinable partitions. Searching solutions in the space of uniformly conjunctively efinable partitions can be achieve by rough set base classification methos [5, 9]. Suppose all the attributes of F have same priorities. The goal of solution searching is to fin a subset of attributes A so that π A is a most general solution to the classification problem. The important notions of rough set base approaches are summarize below. Definition 6 An attribute a A is calle a core attribute, if π F {a} is not a solution, i.e., (π F {a} π class ). Definition 7 A subset A F is calle a reuct, if π A is a solution an for any subset B A, π B is not a solution. That is, (i) π A π class ; (ii) for any proper subset B A, (π B π class ). Each reuct provies one solution to the classification problems. There may exist more than one reuct. A core attribute must be presente at each reuct, namely, a core attribute is in every solution to the classification problem. The set of core attributes is the intersection of all reucts. The bias of searching solutions in the space of uniformly conjunctively efinable partitions is to fin the reuct, a set of iniviually necessary an jointly sufficient attributes. The ID3-like algorithms search the space of nonuniformly conjunctively efinable partitions. Typically, a classification is constructe in a epth-first manner until the leaf noes are subsets that consist of elements of the same class with respect to class. By labeling the leaves by the class symbol of class, we obtain a ecision tree for classification. The bias of searching solutions in the space of non-uniformly conjunctively efinable partitions is to fin the shortest tree construction. One can combine these two searches together, i.e., construct a classification tree by using a reuct set of attributes erive from a reuct-base algorithm. 3. Level Construction of Decision Trees Two level construction methos are iscusse in this section. One is the ID3-like approach, an the other is the reuct-base approach The LID3 algorithm The ID3 algorithm [6] is perhaps one of the most stuie epth-first metho for constructing ecision trees. It starts with the entire set of objects an recursively ivies the set by selecting one attribute at a time, until each noe is a subset of objects belong to one class.

4 A level construction metho base ID3 is given below. LID3: A level construction version of ID3 1. Let k = The k-level, k > 0, of the classification tree is built base on the (k 1) th level escribe as follows: if a noe in (k 1) th level oes not consist of only elements of the same class, then 2.1 Choose an attribute base on a certain criterion β : At R; 2.2 Divie the noe base on the selecte attribute an prouce the k th level noes, which are the subsets of that noe; 2.3 Label the noe by the attribute name, an label the branches coming out from the noe by values of the attribute. The selection criterion use by ID3 is an informationtheoretic measures calle conitional entropy. Let S enote the set of objects in a particular noe at level (k 1). The conitional entropy of class given an attribute a is enote by: H S (class a) = P S (v)h(class v) v V a = P S (v) P S ( v) log P S ( v) v V a V class = P S (, v) log P S ( v), (3) v V a V class where the subscript S inicates that all quantities are efine with respect to the set S. An attribute with the minimum entropy value is chosen to split a noe. For the information Table 1, we obtain a ecision tree shown in Figure 1, an the analysis of RSC an accuracy is summarize in Table 2. A B C D class 1 a 1 b 1 c a 1 b 1 c a 1 b 2 c a 1 b 2 c a 2 b 1 c a 2 b 1 c a 2 b 2 c a 2 b 2 c a 3 b 1 c a 3 b 1 c a 3 b 2 c a 3 b 2 c Table 1. An information table Rules RSC Accuracy k = 1 b 1 5/ B b 2 5/6 + k = 2 b 1 a 1 2/ ABD b 1 a 2 2/2 + b 1 a 3 1/2 + b 2 1 5/5 + b 2 2 1/1 k = 3 b 1 a 3 c 1 1/ ABCD b 1 a 3 c 2 1/1 Table 2. Rules generate by ID3 a 1 a C + c A b B 1 2 D a c Figure 1. An ID3 ecision tree 3.2. The klr algorithm Recall that a reuct is a set of iniviually necessary an jointly sufficient attributes that correctly classify the objects. An algorithm for fining a reuct can be easily extene into a level construction metho for ecision trees calle klr. klr: A reuct-base level construction metho 1. Let k = The k-level, k > 0, of the classification tree is built base on the (k 1) th level escribe as follows: if there is a noe in (k 1) th level that oes not consist of only elements of the same class then 2.1 Choose an attribute base on a certain criterion γ : At R; 2.2 Divie all the inconsistent noes base on the selecte attribute an prouce the k th level noes, which are subsets of the inconsistent noes; 2.3 Label the inconsistent noes by the attribute name, an label the branches coming out from the inconsistent noes by the values of the attribute. Note that when choosing an attribute, one nees to consier all the inconsistent noes. In contrast, LID3 only con- b

5 Rules RSC Accuracy k = 1 b 1 5/ B b 2 5/6 + k = 2 b 1 1 2/ BD b 1 1 3/4 b 2 1 5/5 + b 2 2 1/1 k = 3 b 1 2 a 1 2/ ABD b 1 2 a 2 1/1 b 1 2 a 3 1/1 + Table 3. Rules generate by klr 1 D b 1 A + 2 B a 1 a 2 a 3 + b 2 D 1 2 Figure 2. A klr ecision tree siers one inconsistent noe at a time. Conitional entropy can also be use as the selection criterion γ. In this case, the subset of examples consiere at each level is the union of all inconsistent noes. Let A (k 1) be the set of attributes use from level 0 to level (k 1). The next attribute a for level k can be selecte base on the following conitional entropy: H(class A (k 1) {a}). (4) The use of A (k 1) ensures that all inconsistent noes at level k 1 are consiere in the selection of level k attribute [9]. The ecision trees generate by the klr algorithm are level-constructe trees. The iea is similar to oblivious ecision trees, in which all noes at the same level test the same attributes accoring to a given orer. While the orer is not given, we can use the function γ : At R to ecie an orer. The criterion γ can be one of the information measures, for example, conitional entropy (as shown above) or mutual information, which inicate how much information the attributes contribute to the ecision attribute class; or the statistical measures, for example, the χ 2 test or binomial istribution, which inicates the epenency level between the test attribute an the ecision attribute class. We can get such an orer by testing all the attributes. However, we nee to upate the orer level by level. There are two reasons for level-wise upating. First, in respect that some noes of a classification tree are intene to halt the partition when the solutions or the approximate solutions are foun. The search space is possibly change for ifferent levels. Secon, for each test that partitions the search space into uneven-size blocks, the value of function γ is the sum of the function value of γ for each block multiplies the probability istribution of the block. Consier the earlier example, base on the conitional entropy, the ecision tree built by klr algorithm is shown in Figure 2. The analysis of RSC an accuracy is given by Table 3. Comparing these two ecision trees in Figures 1 an 2, we notice that klr ecision tree can construct a tree possessing the same RSC an accuracy as the LID3 ecision tree with fewer attributes involve. In this example, the set of attributes {A, B, D} is a reuct, since π {A,B,D} π class, an for any proper subset X {A, B, D}, (π X π class ). 4. Experimental evaluations In orer to evaluate level construction methos, we choose some well-known atasets from UCI machine learning repository [10]. SGI s MLC++ utilities 2.0 [8] is use to iscretize the atasets into categorical attribute sets. Set the accuracy threshol α = 100%. Dataset 1: Creit training objects, 14 attributes an 2 classes. Using all the attributes, neither LID3 nor klr can 100% consistently classify all the training instances. klr iscovers that one attribute contributes little to the classification, namely, we can have the same RSC an accuracy values with only 13 attributes. klr generates more rules than its counterpart. However comparing the trees where the same number of attributes are use, klr obtains higher RSC an accuracy. Dataset 2: Vote training objects, 16 attributes an 2 classes. It achieves 100% of RSC an accuracy for classification. In the case of LID3, the total tree length is 8 levels, but 15 attributes were require for 100% of RSC an accuracy. In the case of klr, we can reach the same level of RSC an accuracy by a 9-level-tree with only 9 relate attributes. Dataset 3: Cleve training objects, 13 attributes an 2 classes. It cannot be 100% consistently classifie either by all its 13 attributes. In this ataset, the klr tree is short than the LID3 tree. The klr tree iscovere consists of 11 attributes. This number is less than that is use for constructing a full LID3 tree. This observation is shown in all the other experiments. The experimental results of the above three atasets are

6 Dataset 1 Goal LID3 klr Accu % k = 1 (1 attr) k = 1 (1 attr) Accu % k = 4 (14 attrs) k = 5 (5 attrs) Accu % k = 6 (14 attrs) k = 8 (8 attrs) Accu. = 98.55% k = 11 (14 attrs) k = 13 (13 attrs) RSC 85.00% k = 6 (14 attrs) k = 8 (8 attrs) RSC 90.00% k = 7 (14 attrs) k = 9 (9 attrs) RSC 95.00% k = 10 (14 attrs) k = 12 (12 attrs) RSC = 95.80% k = 11 (14 attrs) k = 13 (13 attrs) Dataset 2 Goal LID3 klr Accu % k = 1 (1 attr) k = 1 (1 attr) Accu. = % k = 8 (15 attrs) k = 9 (9 attrs) RSC 95.00% k = 5 (11 attrs) k = 6 (6 attrs) RSC = % k = 8 (15 attrs) k = 9 (9 attrs) Dataset 3 Goal LID3 klr Accu % k = 3 (6 attrs) k = 3 (3 attrs) Accu % k = 5 (11 attrs) k = 6 (6 attrs) Accu % k = 6 (12 attrs) k = 8 (8 attrs) Accu. = 98.35% k = 13 (12 attrs) k = 11 (11 attrs) RSC 80.00% k = 6 (12 attrs) k = 7 (7 attrs) RSC 85.00% k = 7 (12 attrs) k = 8 (8 attrs) RSC 90.00% k = 8 (12 attrs) k = 9 (9 attrs) RSC = 94.72% k = 13 (12 attrs) k = 11 (11 attrs) Table 4. Experimental results of the atasets use in this paper reporte in Table 4. From the results of experiments, we can have the following observations. The ifference of local an global selection causes ifferent tree structures. Normally, LID3 may obtain a shorter tree. On the other han, if we restrict the height of ecision trees, LID3 may use more attributes than klr. With respect to the RSC measure, LID3 tree is normally better than klr tree at the same level. With respect to the accuracy measure, LID3 tree is not substantially better than klr tree at the same level. With respect to ifferent levels of two trees with the same number of attributes, klr obtains much better accuracy an RSC than LID3. The main avantage of klr metho is that it uses fewer number of attributes to achieve the same level of accuracy. 5. Conclusion The contribution of this paper is twofol, the evelopment of a formal moel an algorithms for level construction methos for builing ecision trees. The formal framework is base on partitions in an information table. Within the framework, we are able to efine precisely an concisely many funamental notions. The concepts of solution an solution space are iscusse. The structures of several search spaces are stuie. Two level construction methos are suggeste: a breath-first version of ID3 calle LID3, which searches the space of non-uniformly conjunctively efinable partitions; an a reuct-base metho calle klr, which searches solutions in the space of uniformly conjunctively efinable partitions. Experimental results are reporte to compare these two methos. They show that one nees to pay more attention to the less stuie level construction methos. References [1] Dua, R.O. an Hart, P.E. Pattern Classification an Scene Analysis, Wiley, New York, [2] Lee, T.T. An information-theoretic analysis of relational atabases - part I: ata epenencies an information metric, IEEE Transactions on Software Engineering, SE-13, , [3] Michalski, J.S., Carbonell, J.G., an Mirchell, T.M. (Es.), Machine Learning: An Artificial Intelligence Approach, Morgan Kaufmann, Palo Alto, CA, , [4] Mitchell, T.M., Machine Learning, McGraw-Hill, [5] Pawlak, Z., Rough Sets: Theoretical Aspects of Reasoning about Data, Kluwer Acaemic Publishers, Dorrecht, [6] Quinlan, J.R., Inuction of ecision trees, Machine Learning, 1, , [7] Quinlan, J.R., C4.5: Programs for Machine Learning, Morgan Kaufmann Publishers, Inc., [8] SGI s MLC++ utilities 2.0: the iscretize utility. [9] Wang, G.Y., Yu, H. an Yang, D.C., Decision table reuction base on conitional information entropy, Chinese Journal of Computers, 25(7), [10] UCI Machine Learning Repository. mlearn/mlrepository.html [11] Yao, J.T. an Yao, Y.Y., Inuction of classification rules by granular computing, Proceeings of International Conference on Rough Sets an Current Trens in Computing, , [12] Yao, Y.Y., Information-theoretic measures for knowlege iscovery an ata mining, in: Entropy Measures, Maximum Entropy an Emerging Applications, Karmeshu (E.), Springer, Berlin, , 2003.