Clustering using Unsupervised Binary Trees: CUBT

Transcription

1 Clustering using Unsupervised Binary Trees: CUBT arxiv: v1 [stat.me] 11 Nov 2010 Ricardo Fraiman Universidad de San Andrés Badih Ghattas Université de la Méditerrannée Marcela Svarc Universidad de San Andrés November 12, 2010 Abstract We introduce a new clustering method based on unsupervised binary trees. It is a three stages procedure, which performs on a first stage recursive binary splits reducing the heterogeneity of the data within the new subsamples. On the second stage (pruning) adjacent nodes are considered to be aggregated. Finally, on the third stage (joining) similar clusters are joined even if they do not descend from the same node. Consistency results are obtained and the procedure is tested on simulated and real data sets. 1 Introduction Clustering is a method of unsupervised classification and a common technique for statistical data analysis with applications in many fields, such as medicine, marketing, economics among others. The term cluster analysis (first used by Tryon, 1939) includes a number of different algorithms and methods for grouping similar data into respective categories. The grouping is build up in a way that the degree of association between data is maximal if they belong to the same group and minimal otherwise. Cluster analysis or clustering is the assignment of a set of observations from R p into subsets (called clusters) such that observations in the same clus- 1

2 ter are similar in some sense. These definitions are quite vague, since there is not a clear population objective function to measure the performance of a clustering procedure. Each clustering algorithm has implicitly an objective function which varies from one method to another. It is important to notice that even though most of the clustering procedures require the number of clusters beforehand in practice this information is usually unknown. On the contrary, in supervised classification the number of groups is known and we have in addition a learning sample and a universal objective function: to minimize the number of miss-classifications, or in population terms to minimize Bayes error. Despite these facts there are many similarities between supervised and unsupervised classification. Specifically, there are many algorithms that share the same spirit for both problems. Supervised and unsupervised classification (Hastie, 2003) algorithms have two main branches: algorithms can be partitional or hierarchical. Partitional algorithms determine all groups at once. The most popular and studied partitioning procedure for cluster analysis is k-means. Hierarchical algorithms find successively groups splitting or joining previously established groups. These algorithms can be either agglomerative ( bottom-up ) or divisive ( top-down ). Agglomerative algorithms begin with each element as a separate group and merge them into successively larger groups. Divisive algorithms begin with the whole set and proceed to split it into successively smaller groups. Hierarchical algorithms create a hierarchy of partitions which may be represented in a tree structure. The best known hierarchical algorithm for supervised classification is CART (Breiman et al., 1984). CART has a relevant additional property. The partition tree is built up based on few binary conditions on the original coordinates variables of the data. In most cases, the interpretation of the results is summarized in a tree that has a very simple structure. The usefulness of such classification scheme is valuable not only for rapid classification of new observations, but also, it can often yield a much simpler model for explaining why the observations are classified in a particular group, and this fact is remarkably important in many applications. Moreover, it is important to highlight that the algorithm does not assume any kind of parametric model on the underlying distribution. Several different methods to obtain clusters based on decision trees have been already introduced. Liu el at. (2000) use decision trees to partition the data space into clusters and empty (sparse) regions at different levels of details. The method is based on the idea of adding an artificial sample of size N uniformly distributed on the space. With these N points added to the original data set, the problem is to obtain a partition of the space into dense and sparse regions. They treat it as a classification problem using a 2

3 new purity function adapted to the problem based on the relative density among regions. Chavent et al. (1999) obtain a binary clustering tree based on a particular variable and its binary transformation. They present two different procedures. In the first one the splitting variables are recursively selected using correspondence analysis and the factorial scores lead to the binary transformation. On the second one the candidate variables and their variables transformations are simultaneously selected by an optimization criterion which evaluates the resulting partitions. Basak et al. (2005) propose four different measures for selecting the most appropriate features for splitting the data at every node and two algorithms for partitioning the data at every decision node. Specifically for categorical data Andreopoulus et al. (2007) introduce HIERDENC which is an algorithm that searches dense subspaces on the cube distribution of the attributes values of the data set. Our objective is to propose a simple clustering procedure sharing the appealing properties of CART. We introduce a hierarchical top-down method called CUBT (Clustering using unsupervised binary trees), where the clustering tree is based on binary rules on the original variables, which will help to understand the clustering structure. One main difference with the most popular hierarchical algorithms, such as single linkage or complete linkage, is that our clustering rule has a predicting property since allows to classify a new observation. The procedure is done in three stages. The first one grows a maximal tree by applying a recursive partitioning algorithm. The second one prunes the tree using a minimal dissimilarity criterion. Finally, the third one aggregates leaves of the tree which do not necessarily share the same direct ascendant. The paper is organized as follows. In Section 2 we introduce some notation and we describe the empirical and population versions of our method. The latter describes the method in terms of the population, regarded as a random vector X R p with unknown distribution P. The consistency of our method is shown in Section 3. In Section 4 we present the results of a simulation study where we challenge our method considering several different models and compare it with k means. We also compare on a synthetic data set the tree structure provided by CART (using the training sample) and CUBT considering the same sample without the labels. A real data example is analyzed in Section 5. Concluding remarks are given in Section 6. All proofs are given in the Appendix. 3

4 2 Clustering a la CART We start by fixing some notations, let X R p be a random p-dimensional real vector with coordinates X(j), j = 1,..., p, such that E( X 2 ) <. The data are constituted by n random independent and identically distributed realizations of X, Ξ = {X 1,..., X n }. For the population version the space is R p, while for the empirical version the space is Ξ. We denote by t to the nodes of the tree. Each node t of the tree determines a subset of R p which will be also denoted as t R p. To the root node we assign the whole space. Even though our procedure shares in many aspects the spirit of CART two main differences should be pointed out. First, as we are in the case of unsupervised classification, only the information of the observations without labels is available. Thus the splitting criterion cannot be based on the labels as in CART. The other essential difference is that instead of having one final pruning stage our algorithm has two phases, first we prune the tree and then there is a final joining stage. The former procedure evaluates the merging of adjacent nodes and the latter one aims to aggregate similar clusters that do not share the same direct ascendant in the tree. 2.1 Forward step: maximal tree construction As we are defining a top-down procedure we begin by assigning the whole space to the root node. Let t be a node and ˆt = Ξ t the set of observations coming from the sample at hand. At each stage a terminal node is considered to be split in two subnodes, the left and right child, t l, t r, if it fulfills a condition. At the begging there is only one node, the root, which contains the whole space. The splitting rule has the form x(j) < a, where x(j) is a variable and a is a threshold level. Thus, t l = {x R p : x(j) a} and t r = {x R p : x(j) > a}. Let X t be the restriction of X to the node t, i.e. X t = X {X t}, and α t the probability of being in t, α t = P (X t). Then R(t) is an heterogeneity measure of t defined by, R(t) = α t trace(cov(x t )), (1) where, cov(x t ) is the covariance matrix of X t. Thus, R(t) roughly measures the mass concentration of the random vector X at the set t weighted by the 4

5 mass of the set t. In the empirical algorithm α t and Cov(X t ) are replaced by their empirical versions (estimates), and R(t) is called the deviance. Then, we denote n t the cardinal of the set t, n t = n i=1 I {X i t}, (where I A stands for the indicator function of set A) hence the estimated probability is α t = nt and the estimate of E ( X n t µ t 2) is X i X t 2 {X i t} n t, (2) where X t is the empirical mean of the observations on t. The best split for t is defined as the couple (j, a) {1,..., p} R, (the first element indicates the variable where the partition is defined and the second element is the threshold level) that maximizes, (t, j, a) = R(t) R(t l ) R(t r ). It is easy to verify that (t, j, a) 0 for every t, j, a, this property is also verified by all the splitting criteria proposed in CART. We start with the whole space assigned to the root node, then each node is split recursively until one of the following stopping rules is satisfied: All the observations within a node are the same. There are less than minsize observations in a node. The deviance s reduction is less than mindev R(S), where S is the whole space. minsize and mindev are tuning parameters that must be supplied by the user, we consider minsize = 5 and mindev = Once the algorithm stops a label is assigned to each leaf (terminal node); we call this tree the maximal tree. We consider the standard criteria for enumerating the nodes on a complete binary tree, let t be the node n, the left child is numbered 2n and the right one 2n + 1. At this point, we have obtained a partition of the space and consequently a partition of the data set, where each leaf is associated to a cluster. Ideally, this tree has at least the same number of clusters as the population, in practice this tree may have too many clusters, then an agglomerative stage must be applied as in CART. It is important to remark that even though the number of clusters will be known beforehand, it is not needed on this stage and that small values of mindev ensure a tree with many leaves. Moreover, if the tree has the same 5

6 Model M1: minsize=10, minsplit=25, mindev=0.01 x1 < 2.72 x1 < x2 < x2 < x2 < x1 < x2 < x1 < x1 < x2 < x1 < 4.979x1 < x1 < Figure 1: Illustration of maximal tree for Model M1 simulated data. number of leaves as the number of clusters then is not necessary to run the subsequent stages of the algorithm. An example of a maximal tree is given on Figure 1 for a simulated data set defined in section 4, model M1. The maximal tree has fourteen leaves while model M1 has a three groups structure, the number of observations that belong to each terminal node is indicated. 2.2 Backward step: pruning and joining In this step we use successively two algorithms to give the final grouping conformation: the first one prunes the tree and the second one merges non adjacent leaves, we call it joining. We introduce a pruning criterion named minimum dissimilarity pruning Minimum dissimilarity pruning On this stage we define a measure of dissimilarities between sibling nodes and collapse them if this measure is lower than a threshold. First, we consider 6

7 the maximal tree T 0 obtained on the previous stage. Let t l and t r be a pair of terminal nodes sharing the same direct ascendant. Next, define (in populations term) the random variables W lr = D(X tl, X tr ), as the Euclidian distances between the random elements of X tl and of X tr. Finally define the dissimilarity measure between the sets t l and t r as, lr = δ 0 q α (W lr )dα, where q α (.) stands for the quantile function and δ is a proportion, δ (0, 1). If lr < ɛ, we prune the tree, i.e. we replace t l and t r by t l t r in the partition. It is worth to acknowledge that lr is just a more resistant version of the distance among the supports of the random vectors X tl and X tr. The dissimilarity measure can be estimated as follows, let n l (resp. n r ) be the size of ˆt l (resp. ˆt r ). Consider for every x i ˆt l and y j ˆt r, the sequences, d l = min y ˆt l d(x i, y), dr = min x ˆt r d(x, y j ) and their ordered versions denoted d l and d r. For δ [0, 1], let d δ l = 1 δn i δn i 1 d l, dδ r = 1 δn j δn j d r. We compute the dissimilarity between t l and t r as, d δ (l, r) = d δ (t l, t r ) = max( d δ l, d δ r), and on each step of the algorithm the leaves, t l and t r, are merged into the ascendant node t if d δ (l, r) ɛ where ɛ > 0. The dissimilarity pruning depends on two parameters: δ and ɛ, which from now on we call mindist Joining The idea of the joining step is to aggregate nodes which do not necessarily share the same direct ascendant. It is based on the relative decrease of the deviance when two nodes are aggregated. For any tree T we define its deviance as, R(T ) = 1 R(t), n 7 t T 1

8 where T is the set of the leaves of T. Our goal is to find an optimal subtree of T. Here all pairs of non terminal nodes t i and t j are compared computing, d ij = R(t i t j ) R(t i ) R(t j ). R(t i t j ) For the empirical version we consider a plug in estimate for d ij following the definitions provided on Section 2.1. As in standard hierarchical clustering procedures pairs of terminal nodes are successively aggregated in decreasing order of d ij ( ˆd ij respectively), then on each step there is one cluster less. We may consider two different stopping rules for the joining procedure which correspond either to the case where the number of clusters k is known or to the case where k is unknown. If k is known repeat the following step until m k: For each pair of values (i, j), 1 i < j m, let (ĩ, j) = arg min i,j {d ij }. Replace tĩ and t j by its union t ĩ t j, put m = m 1 and proceed ongoing. If k is unknown, if dĩ j < η replace t ĩ and t j by its union t ĩ t j, where η > 0 is a given constant, continue until this condition it is not fulfilled. In the first case the stopping criterion is the number of clusters while in the second case a threshold η for d ij must be settled. 2.3 CUBT and k-means In this section we discuss someway informally when our procedure and the very well known k means algorithm should produce a reasonably good output. We shall consider those cases where there are nice groups strictly separated. More precisely, let A 1,..., A k be disjoint connected compact sets on R p such that A i = A 0 i for i = 1,..., k, and {P i : i = 1,..., k} their probability measures on R p with supports {A i : i = 1,..., k}. A typical case is obtained defining a random vector X with density f and then considering the random vector X = X {f > δ} for a positive level set δ, as in several hierarchical clustering procedures. On the one hand, an admissible family for CUBT will be a family of sets A 1,..., A k such that there exist another family of disjoint sets B 1,..., B k built up as the intersection of a finite number of half spaces delimited by hyperplanes which are orthogonal to the coordinate axis satisfying A i B i. 8

9 On the other hand, k-means is defined through the vector of centers (c 1..., c k ) minimizing ( ) E min X c j, j=1,...,k associated with each center c j is the convex polyhedron S j of all points in R p closer to c j than to any other center, called the Voronoi cell of c j. The sets in the partition S 1,..., S k are the population clusters for k means. Therefore, the population clusters for k means are defined by exactly k hyperplanes in an arbitrary position. Then, an admissible family for k-means will be a family of sets A 1,..., A k that can be separated by exactly k hyperplanes. Even though the hyperplanes for k-means can be in general position, one cannot use more than k of them. It is clear that in this sense CUBT is much more flexible than k means, since the family of admissible sets is more general. For instance, k-means will necessarily fail to identify nested groups, while this will not be the case of CUBT. Another important difference between k means and CUBT is that our proposal is less sensitive to small changes in the parameters that define the partition. Effectively, small changes on them will produce small changes on the partition. However, small changes on the centers (c 1..., c k ) defining the k-means partition can produce significant changes in the associated partition given by the Voronoi cells. 3 Consistency of CUBT In this section we give some theoretical results about the consistency of our algorithm. First we prove an important property, the monotonicity of the deviance when the tree size increases. A simple equivalent characterization of the function R(t) is given in the following Lemma. Lemma 3.1. Let t l and t r be disjoint compact sets on R p and denote by µ s = E(X ts ), s = l, r respectively. If t = t l t r we have that, R(t) = R(t l ) + R(t r ) + α t l α tr α t µ l µ r 2. (3) The proof is given in the Appendix. 9

10 Remark 3.1. Monotonicity of the function R(.) and geometric interpretation. Observe that Lemma (3.1) entails that for all disjoint compact sets t l, t r and t = t l t r, the function R(.) is monotonic in the sense that, R(t) R(t l ) + R(t r ). (4) Moreover, R(t) will be close to R(t l ) + R(t r ) when the last term on the right hand side of (3) is small. This will happen either if one of the sets t l, t r has a very small fraction of the mass of t and/or if the centers of the two subsets t l, t r are very close. In either cases we will not want to split the set t. The following results show the consistency of the empirical algorithm to its population version. We begin with the splitting algorithm and follow then with the pruning and joining. Theorem 3.1. Assume that the random vector X has distribution P and a density f fulfilling that x 2 f(x) is bounded. Let X 1,..., X n be iid random vectors with the same distribution as X and denote by P n the empirical distribution of the sample X 1,..., X n. Let {t 1n,..., t mnn} be the empirical binary partition obtained by the forward empirical algorithm, and {t 1,..., t m } the population version. Then, we have that m n = m ultimately and each pair (i jn, a jn ) {1,..., p} R determining the empirical partition converges a.s. to the corresponding one (i j, a j ) {1,..., p} R for the population values. In particular, it implies that, lim n m P (t in t i ) = 0, i=1 where stands for the symmetric difference. The proof is given in the Appendix. Theorem 3.2. Let {t 1n,..., t k nn } be the final empirical binary partition obtained after the forward and backward empirical algorithm and {t 1,..., t k } the population version. Under the assumptions of Theorem 3.1 we have that k n = k ultimately (k n = k n if k is known), and lim n k P (t in t i ) = 0. i=1 The proof is given in the Appendix. 10

11 4 Some experiments. In this Section, we present the results of a simulation study where we challenge our method for four different models. We compare these results with those of k-means. As it is well known the performance of k-means strongly depends on position of the initial centroids used to start the algorithm. Several proposals have been made to handle this effect (see Steinly, 2007). We follow the recommendations in this last reference, considering ten random initializations and keeping the one with minimum within-cluster sum of squares given by, ( n k ) X i c j 2 I {X i G j }, i=1 j=1 where G j is the j th group and c j is the corresponding center. We denote this version k-means(10). 4.1 Simulated data sets We consider four different models, with k = 3, 4, and 10 groups. The sample size of each group is n i = 100 for i = 1,..., k in all cases, then the total sample size is N = 100k data. Except in the last model where n i = 30 for i = 1,..., 10, then the sample size is N = 300. M1. Three groups in dimension 2. The data are generated according to the following distributions: N(µ 1, Σ), N((µ 2, Σ), N((µ 3, Σ), (5) with µ 1 = ( 4, 0), µ 2 = (0, 0), µ 3 = (5, 5) and Σ = ( The left panel of Figure 4.1 exhibits an example of data generated from model M1. M2. Four groups in dimension 2. The data are generated from : ). N(µ i, Σ), i = 1,..., 4 distributions with centers ( 1, 0), (1, 0), (0, 1), (0, 1) and covariance matrix Σ = σ 2 Id with σ = 0.11, 0.13,..., The right panel of Figure 4.1 gives an example of data generated from model M2 with σ =

12 Model M1 Model M2, sigma=0.15 X X X X1 Figure 2: Scatter plot corresponding to M1 (left) and M2 (right) M3. Three groups in dimension 10. The data are generated according to the distributions given in (5) with Σ = σ 2 Id, σ = 0.7, 0.75,..., 0.9, µ 1 = ( 2,..., 2), µ 2 = (0,..., 0), µ 3 = (3,..., 3). The left panel of Figure 3 shows an example of data generated from model M3 with σ = 0.8 projected over the two first dimensions. M4. Ten groups in dimension 5. The data are generated from N(µ i, Σ), i = 1,..., 10. The means of the first five groups µ i, are the vectors of the canonical basis e 1,..., e 5 respectively, while the five remaining groups centers are µ 5+i = e i, i = 1, In every case, the covariance matrix is Σ = σ 2 Id, σ = 0.11, 0.13,..., The right panel of Figure 3 gives an example of data generated from model M4 with σ = 0.19 projected on the two first dimensions. 4.2 Tuning the method We perform M = 100 replicates for each model and compare these results with those of k-means and k-means(10). Throughout the simulations k is assumed to be known. In order to perform CUBT we must fix the values for the parameters involved at each stage of the algorithm: For the maximal tree we use: minsize = 5 and mindev = For the pruning stage: mindist = 0.3, 0.5 and δ =

13 Model M3: sigma=0.8 Model M4: sigma=0.19 X X X X1 Figure 3: Two dimensional projection scatter plots corresponding to M3 (left) and M4 (right) For the joining stage, the value of k for each model has been stated previously. Since we are working with synthetical data sets, we know the actual label of each observation, then it is reasonable to measure the goodness of a partition by computing the number of misclassified observations, which is the analogous to the misclassification error for supervised classification procedures. Denote the original clusters r = 1,..., R and the predicted ones s = 1,..., S. Let y 1,...y n be the group label of each observation, and ŷ 1,...ŷ n the class label assigned by the clustering algorithm. Let Σ be the set of permutations over {1,..., S} and A the set of arrangements of S elements taken from {1,..., R}, then the misclassification error we use may be expressed as: MCE = Min σ Σ 1 n n i=1 1 {y i σ(ŷ i )} if S R 1 = Min n σ A n i=1 1 {σ(y i ) ŷ i } else If the number of clusters is large the assignment problem may be computed in polynomial time using Bipartite Matching and the Hungarian Method (Papadimitriou et al, 1982). It is important to remark that Equation (6) is given in a more general framework than we need because in our case as the (6) 13

14 Sigma (σ) CUBT (0.3) CUBT (0.5) k-means k-means(10) Model e Model e Model e e e 04 Model Table 1: Simulation results for models M1 to M4. number of cluster is known, then S is equal to R, nonetheless Equation (6) admits S different from R. 4.3 Results Table 4.3 shows the results for the four models. Except for Model 1, we varied the values of σ reflecting the level of overlapping between classes. We report the misclassification error obtained for each clustering procedure, CUBT considering mindist = 0.3 (CUBT (0.3)) and mindist = 0.5 (CUBT (0.5)), k-means and k-means(10). The results for the different values of mindist are practically the same and in almost every case the results for CUBT are between those of k-means and k-means(10). Moreover, it is important to notice that the average number of leaves for the maximal tree is much bigger than the actual number of clusters. For model M1 it equals 33.8, for M2 it ranges between 35.2 and 40.4, for M3 between 18.4 and 20.8 and for M4 between 45.2 and

15 4.4 A comparison between CART and CUBT We compare in a simple example the tree structure obtained by CART using the complete training sample (observations plus the group labels) versus the tree structure obtained by CUBT considering only the training sample without the labels. We generate three sub populations in a two-dimensional variable space. The underlying distribution of the vector X =(X 1, X 2 ) is a bivariate normal distribution where the variables X 1 and X 2 are independent and their distributions for teh three groups are given by X 1 N (0, 0.03), N (2, 0.03), N (1, 0.25), X 2 N (0, 0.25), N (1, 0.25), N (2.5, 0.03), The data are then rotated π/4. A difficulty of this problem is that the optimal partitions are not parallel to the axis. Figure 4 shows for a single sample of size 300 the partitions obtained by CART and by CUBT respectively. We performed 100 replicates and for each a training sample of size 300 is generated, where every group has the same size. We compute then the mean misclassification rate with respect to the true labels. For CUBT its value was 0.032, while for CART there are not classification errors since we use the sample for both purposes, growing the tree and classifying the observations. Moreover, in order to compare our procedure with the traditional CART classification method, we found the respective binary trees for CART and CUBT. The structure is very similar, both trees are presented in Figure 5. Notice that the structure is exactly the same for both of them, and that the different cutoff values in the different branches may be understood with the aid of Figure 4 which corresponds to the same data set. 5 A real data example. European Jobs The data set contains the percentage employed in different economical activities for several European countries during The categories are, agriculture (A), mining (M), manufacturing (MA), power supplies industries (P), construction (C), service industries (SI), finance (F), social and personal services (S), and transportation and communication(t). It is important to notice that these data where collected during the cold war. The aim is to allocate the observations in clusters, but the number of clusters is unknown, then we are going to study the data structure for several numbers of clusters. 15

16 Figure 4: Plot of the partitions of the space for a generated data set. The solid lines exhibit the partition for CUBT and the dashed lines the partition for CART We consider first a four groups structure. In this case only one variable - the percentage employed in agriculture - determines the tree structure. The four groups are given on Table 2 and the corresponding tree is plotted on the top panel of Figure 6. As it can be seen in the tree, the highest value of A corresponds to Turkey, which is an outlier an conforms a single observation cluster, it can be explained due to its social and territorial proximity to Africa. The countries conforming groups 2 and 3 are countries who were either under a communist government or going through difficult political situations, for instance Spain was leaving behind Franco s regime. The countries that belong to Group 2 were poorer than those on Group 3. Finally Group 4 has the lowest percentage of employment in agriculture and the countries conforming that group were the most developed and not controlled by a communist regime, with the exception of East Germany. If we consider a four group structure by k-means, Turkey and Ireland, each of them are isolated in one group, Greece, Portugal, Poland, Romania and Yugoslavia conform another group and the rest of the countries conform another cluster. If instead of four clusters we consider a five cluster structure we can see 16

17 Figure 5: Left: Tree corresponding to CUBT. Right: Tree considering CART. In both cases the left branches indicate the smaller value of the partitioning variable. Group 1 Group 2 Group 3 Group 4 Turkey Greece Ireland Belgium Poland Portugal Denmark Romania Spain France Yugoslavia Bulgaria W. Germany Czechoslovakia E. Germany Hungary USSR Italy Luxembourg Netherlands United Kingdom Austria Norway Sweden Switzerland Table 2: CUBT clustering structure for four groups. that the main difference is that Group 3 of the previous partition is divided into three groups (Groups 3, 4 and 5) and that the variables that explained those partitions are the percentage employed in mining and the percentage employed in agriculture. The third group of the previous partition remain 17

18 Figure 6: Tree structure considering four groups top and five groups bottom, in every case on the left branch we present the smaller values of the variable that is making the partition. stable and groups 1 and 2 of the previous partition collapsed on one group that we denoted Group 1. The percentage employed in agriculture was the variable that determined the partition between groups 1, 2 and the rest of them. If a five cluster structure is considered via k-means then Turkey and Ireland, each of them are isolated in one group, Greece, Portugal, Poland, Romania and Yugoslavia conform another group this is the same as in the four cluster structure. Switzerland and east and west Germany conform a new cluster. Group 1 Group 2 Group 3 Group 4 Group 5 Turkey Ireland Belgium W. Germany France Greece Portugal Denmark E. Germany Italy Poland Spain Netherlands Switzerland Luxembourg Romania Bulgaria United Kingdom Austria Yugoslavia Czechoslovakia Norway Hungary Sweden USSR Table 3: CUBT clustering structure for five groups. 18

19 6 Concluding Remarks A new clustering method, CUBT in the spirit of CART is presented, defining the clusters in terms of binary rules on the original variables. This approach shares with classification trees this nice property which is very important in many practical applications. As the tree structure is based on the original variables it helps to determine which variables are important to in the cluster conformation. Moreover, the tree allows to classify new observations. A binary tree is obtained in three stages. In the first stage the sample is split into two sub samples reducing the heterogeneity of the data within the new sub samples according to the objective function R( ). Then the procedure is applied recursively to each sub sample. In the second and third stages the maximal tree obtained at the first stage is pruned using two different criteria, one for the adjacent nodes and the other one for all the terminal nodes. The algorithm is simple and runs in a reasonable computing time. There are no restrictions on the dimension of the data. Our method is consistent under quite general assumptions and behaves quite well on the simulated examples that we have considered, as well as in a real data example. A robust version could be developed changing in the objective function given in (1), cov(x T, X T ) by a robust covariance functional robcov(x T, X T ) (see for instance Maronna et al. Chapter 6, for a review) and then proceed in the same way. However a detailed study of this alternative goes beyond the scope of this work. 7 Appendix 7.1 Proof of Lemma 3.1 Observe first that since t l and t r are disjoint, E(X tl t r ) = γµ l + (1 γ)µ r, where γ = P (X t l X t l t r ). Given j = 1,..., p, denote by M (j) 2i = t i x(j) 2 df (x), i = l, r, where F stands for the distribution function of the vector X. It follows easily that E(X tl t r (j) 2 ) = γm (j) 2l 19 + (1 γ)m (j) 2r,

20 and therefore var(x tl t r (j)) = γvar(x tr (j))+(1 γ)var(x tr (j))+γ(1 γ)(µ l (j) µ r (j)) 2. Finally summing up on j we get the desired result. 7.2 Proof of Theorem 3.1 Let T be the family of polygons in R d with faces orthogonal to the axes, and fix i {1,..., p} and t T. For a R denote by t l = {x t : x(i) a} and t r = t \ t l. Define r(t, i, a) = R(t) R(t l ) R(t r ), (7) and r n (t, i, a) = R n (t) R n (t l ) R n (t r ), (8) the corresponding empirical version. We start showing the uniform convergence sup a R sup r n (t, i, a) r(t, i, a) 0 a.s. (9) t T By Lemma 3.1, α t (t, i, a) = α tl α tr µ l (a) µ r (a) 2, (10) where α A = P (X A) and µ j (a) = E(X tj ), j = l, r. Then, the pairs (i jn, a jn ) and (i j, a j ) are the arguments that maximize the right hand of (10) with respect to the measures P n and P respectively. Observe that the right hand size of (10) equals α tr x t 2 dp (x) + α tl x 2 dp (x) 2 xdp (x), xdp (x). (11) l t r t l t r It order to prove (9) it suffices to show that: 1. sup a R sup t T P n (t j ) P (t j ) 0 a.s. j = l, r 2. sup a R sup t T t j x 2 dp n (x) t j x 2 dp (x) 0 a.s. j = l, r 3. sup a R sup t T t j x(i)dp n (x) t j x(i)dp (x) 0 a.s. j = l, r, i = 1,..., p. 20

21 Since T is a Vapnik Chervonenkis class, we have that (i) holds. Now observe that the conditions for uniform convergence over families of sets still hold if we are dealing with signed finite measures. Therefore if we consider the finite measure x 2 dp (x) and the finite signed measure given by x(i)dp (x) we also have that (ii) and (iii) holds. Since lim α t a l α tr µ l (a) µ r (a) 2 = lim α t a l α tr µ l (a) µ r (a) 2 = 0, we have that argmax a R r n (t, i, a) argmax a R r(t, i, a) a.s. At the first step of the algorithm, t = R p and we get that i n1 = i 1 for n large enough and a n1 a 1 a.s. For the next step we have that the empirical procedure will start working with t nl and t nr while the population algorithm will do it with t l and t r. However, we have that sup a R sup t T sup r n (t nj, i, a) r(t j, i, a) a R r n (t nj, i, a) r(t nj, i, a) + sup r(t nj, i, a) r(t j, i, a), (12) a R for j = l, r. We already know that the first term on the right hand side of (12) converges to zero almost surely. To show that the second term also converges to zero, it suffices to show that 1. sup a R P (t nj ) P (t j ) 0 a.s. j = l, r 2. sup a R t j x 2 dp (x) t nj x 2 dp (x) 0 a.s. j = l, r 3. sup a R t j x(i)dp (x) t nj x(i)dp (x) 0 a.s. j = l, r, i = 1,..., p, which follows from the assumption that x 2 f(x) is bounded. This concludes the proof. 7.3 Proof of Theorem 3.2 We need to show that we have consistency in both steps of the backward algorithm. (i) Convergence of the pruning step. Let {t 1n,..., t mn} the output of the forward algorithm. The pruning step partition of the algorithm converges to the corresponding population version from the conclusions of Theorem

22 the fact that the random variables W lr dl, d r are positive. the uniform convergence of the empirical quantile function to it s population version. and the Lebesgue dominated convergence Theorem. (ii) Convergence of the joining step. Let {t 1n,..., t mn } the output of the pruning algorithm. Since (i) and the conclusions of Theorem 3.1 hold, the proof will be complete if we show that for any pair t ln, t rn {t 1n,..., t mn }, converges almost surely to d ln,rn = R n(t ln t rn ) R n (t ln ) R n (t rn ) R n (t ln t rn ) d l,r = R(t l t r ) R(t l ) R(t r ) R(t l t r ) as n. Now observe that the following inequalities hold R n (t ln ) R(t l ) R n (t ln ) R(t ln ) + R(t ln ) R(t l ) sup R n (t) R(t) + R(t ln ) R(t l ). (13) t T Finally from a similar argument to that in Theorem 3.1 we obtain that lim R n(t sn ) = R(t s ), n a.s., for s = i, j and from which we derive which completes the proof. lim R n(t ln t rn ) = R(t l t r ), n lim d ln,rn = d l,r, n a.s., a.s., REFERENCES Andreopoulos, B. and An, A. and Wang, X., 2007, Hierarchical Density- Based Clustering of Categorical Data and a Simplification, Advances in Knowledge Discovery and Data Mining,

23 Basak, J. and Krishnapuram, R., Interpretable Hierarchical Clustering by Construction an Unsupervised Decision Tree, IEEE Transactions on Knowledge and Data Engineering, Blockeel, H. and De Raedt, L. and Ramon, J., Top-down induction of clustering trees, Proceedings of the 15th International Conference on Machine Learning, ed. Morgan Kaufmann, Breiman, L. and Friedman, J. and Stone, C. J. and Olshen, R. A., Classification and Regression Trees, ed. Chapman & Hall/CRC. Chavent, M. and Guinot, C. and Lechevallier, Y. and Tenenhaus, M., Méthodes Divisives de Classification et Segmentation Non Supervisée: Recherche d une Typologie de la Peau Humanine Saine, Rev. Statistique Appliquée, XLVII Hastie, T. and Tibshirani, R. and Friedman, J. H., The Elements of Statistical Learning, Springer. Liu, B. and Xia, Y. and Yu, P. S., Clustering through decision tree construction, CIKM 00: Proceedings of the ninth international conference on Information and knowledge management, ed. ACM,New York, NY, USA, Maronna, R. A and Martin, D. R. and Yohai, V. J., Robust Statistics: Theory and Methods (Wiley Series in Probability and Statistics), Wiley. Papadimitriou, C. and Steiglitz, K., Combinatorial Optimization: Algorithms and Complexity, Englewood Cliffs: Prentice Hall. F. Questier and R. Put and D. Coomans and B. Walczak and Y. Vander Heyden, The use of CART and multivariate regression trees for supervised and unsupervised feature selection, Chemometrics and Intelligent Laboratory Systems, Smyth C. and Coomans D. and Everingham Y. and Hancock T., Autoassociative Multivariate Regression Trees for Cluster Analysis, Chemometrics and Intelligent Laboratory Systems, Smyth C. and Coomans D. and Everingham Y., Clustering noisy data in a reduced dimension space via multivariate regression trees, Pattern Recognition, Steinly D. and Brusco M., Initializing K-means Batch Clustering: A Critical evaluation of Several Techniques, Journal of Classification, Tryon, R.C., Cluster Analysis, McGraw-Hill. 23