Soft Clustering on Graphs

Transcription

1 Soft Custering on Graphs Kai Yu 1, Shipeng Yu 2, Voker Tresp 1 1 Siemens AG, Corporate Technoogy 2 Institute for Computer Science, University of Munich kai.yu@siemens.com, voker.tresp@siemens.com spyu@dbs.informatik.uni-muenchen.de Abstract We propose a simpe custering framework on graphs encoding pairwise data simiarities. Unike usua simiarity-based methods, the approach softy assigns data to custers in a probabiistic way. More importanty, a hierarchica custering is naturay derived in this framework to graduay merge ower-eve custers into higher-eve ones. A random wak anaysis indicates that the agorithm exposes custering structures in various resoutions, i.e., a higher eve statisticay modes a onger-term diffusion on graphs and thus discovers a more goba custering structure. Finay we provide very encouraging experimenta resuts. 1 Introduction Custering has been widey appied in data anaysis to group simiar objects. Many agorithms are either simiarity-based or mode-based. In genera, the former (e.g., normaized cut [5]) requires no assumption on data densities but simpy a simiarity function, and usuay partitions data excusivey into custers. In contrast, mode-based methods appy mixture modes to fit data distributions and assign data to custers (i.e. mixture components) probabiisticay. This soft custering is often desired, as it encodes uncertainties on datato-custer assignments. However, their density assumptions can sometimes be restrictive, e.g. custers have to be Gaussian-ike in Gaussian mixture modes (GMMs). In contrast to fat custering, hierarchica custering makes intuitive senses by forming a tree of custers. Despite of its wide appications, the technique is usuay achieved by heuristics (e.g., singe ink) and acks theoretica backup. Ony a few principed agorithms exist so far, where a Gaussian or a sphere-shape assumption is often made [3, 1, 2]. This paper suggests a nove graph-factorization custering (GFC) framework that empoys data s affinities and meanwhie partitions data probabiisticay. A hierarchica custering agorithm (HGFC) is further derived by merging ower-eve custers into higher-eve ones. Anaysis based on graph random waks suggests that our custering method modes data affinities as empirica transitions generated by a mixture of atent factors. This view significanty differs from conventiona mode-based custering since here the mixture mode is not directy for data objects but for their reations. Custers with arbitrary shapes can be modeed by our method since ony pairwise simiarities are considered. Interestingy, we prove that the higher-eve custers are associated with onger-term diffusive transitions on the graph, amounting to smoother and more goba simiarity functions on the data mani-

2 fod. Therefore, the custer hierarchy exposes the observed affinity structure graduay in different resoutions, which is somehow simiar to the waveet method that anayzes signas in different bandwidths. To the best of our knowedge, this property has never been considered by other aggomerative hierarchica custering agorithms (e.g., see [3]). The paper is organized as foows. In the foowing section we describe a custering agorithm based on simiarity graphs. In Sec. 3 we generaize the agorithm to hierarchica custering, foowed by a discussion from the random wak point of view in Sec. 4. Finay we present the experimenta resuts in Sec. 5 and concude the paper in Sec Graph-factorization custering (GFC) Data simiarity reations can be convenienty encoded by a graph, where vertices denote data objects and adjacency weights represent data simiarities. This section introduces graph factorization custering, which is a probabiistic partition of graph vertices. Formay, et G(V, E) be a weighted undirected graph with vertices V = {v i } n i=1 and edges E {(v i, v j )}. Let W = {w ij } be the adjacency matrix, where w ij = w ji, w ij > 0 if (v i, v j ) E and w ij = 0 otherwise. For instances, w ij can be computed by the RBF simiarity function based on the features of objects i and j, or by a binary indicator (0 or 1) of the k-nearest neighbor affinity. 2.1 Bipartite graphs Before presenting the main idea, it is necessary to introduce bipartite graphs. Let K(V, U, F) be the bipartite graph (e.g., Fig. 1 (b)), where V = {v i } n i=1 and U = {u p } m p=1 are the two disjoint vertex sets and F contains a the edges connecting V and U. Let B = {b ip } denote the n m adjacency matrix with b ip 0 being the weight for edge [v i, u p ]. The bipartite graph K induces a simiarity between v 1 and v j [6] w ij = m p=1 b ip b ( jp = BΛ 1 B ) λ, Λ = diag(λ 1,..., λ m ) (1) p ij where λ p = n i=1 b ip denotes the degree of vertex u p U. We can interpret Eq. (1) from the perspective of Markov random waks on graphs. w ij is essentiay a quantity proportiona to the stationary probabiity of direct transitions between v i and v j, denoted by p(v i, v j ). Without oss of generaity, we normaize W to ensure ij w ij = 1 and w ij = p(v i, v j ). For a bipartite graph K(V, U, F), there is no direct inks between vertices in V, and a the paths from v i to v j must go through vertices in U. This indicates p(v i, v j ) = p(v i )p(v j v i ) = d i p(u p v i )p(v j u p ) = p p p(v i, u p )p(u p, v j ) λ p, where p(v j v i ) is the conditiona transition probabiity from v i to v j, and d i = p(v i ) the degree of v i. This directy eads to Eq. (1) with b ip = p(v i, u p ). 2.2 Graph factorization by bipartite graph construction For a bipartite graph K, p(u p v i ) = b ip /d i tes the conditiona probabiity of transitions from v i to u p. If the size of U is smaer than that of V, namey m < n, then p(u p v i ) indicates how ikey data point i beongs to vertex p. This property suggests that one can construct a bipartite graph K(V, U, F) to approximate a given G(V, E), and then obtain a soft custering structure, where U corresponds to custers (see Fig. 1 (a) (b)).

3 (a) (b) (c) Figure 1: (a) The origina graph representing data affinities; (b) The bipartite graph representing data-to-custer reations; (c) The induced custer affinities. Eq. (1) suggests that this approximation can be done by minimizing (W, BΛ 1 B ), given a distance (, ) between two adjacency matrices. To make the probem easy to sove, we remove the couping between B and Λ via H = BΛ 1 and then have ( min W, HΛH ), s. t. H,Λ n i=1 h ip = 1, H R n m +, Λ D m m +, (2) where D m m + denotes the set of m m diagona matrices with positive diagona entries. This probem is a symmetric variant of non-negative matrix factorization [4]. In this paper we focus on the divergence distance between matrices. The foowing theorem suggests an aternating optimization approach to find a oca minimum: Theorem 2.1. For divergence distance (X, Y) = ij (x ij og xij y ij x ij + y ij ), the cost function in Eq. (2) is non-increasing under the update rue ( denote updated quantities) w ij h ip h ip (HΛH λ p h jp, normaize s.t. h ip = 1; (3) ) j ij i w ij λ p λ p (HΛH h ip h jp, normaize s.t. λ p = w ij. (4) ) ij ij p ij The distance is invariant under the update if and ony if H and Λ are at a stationary point. See Appendix for a the proofs in this paper. Simiar to GMM, p(u p v i ) = b ip / q b iq is the soft probabiistic assignment of vertex v i to custer u p. The method can be seen as a counterpart of mixture modes on graphs. The time compexity is O(m 2 N) with N being the number of nonzero entries in W. This can be very efficient if W is sparse (e.g., for k-nearest neighbor graph the compexity O(m 2 nk) scaes ineary with sampe size n). 3 Hierarchica graph-factorization custering (HGFC) As a nice property of the proposed graph factorization, a natura affinity between two custers u p and u q can be computed as p(u p, u q ) = n i=1 b ip b ( ) iq = B D 1 B d, D = diag(d 1,..., d n ) (5) i pq This is simiar to Eq. (1), but derived from another way of two-hop transitions U V U. Note that the simiarity between custers p and q takes into account a weighted average of contributions from a the data (see Fig. 1 (c)).

4 Let G 0 (V 0, E 0 ) be the initia graph describing the simiarities of totay m 0 = n data points, with adjacency matrix W 0. Based on G 0 we can buid a bipartite graph K 1 (V 0, V 1, F 1 ), with m 1 < m 0 vertices in V 1. A hierarchica custering method can be motivated from the observation that the custer simiarity in Eq. (5) suggests a new adjacency matrix W 1 for graph G 1 (V 1, E 1 ), where V 1 is formed by custers, and E 1 contains edges connecting these custers. Then we can group those custers by constructing another bipartite graph K 2 (V 1, V 2, F 2 ) with m 2 < m 1 vertices in V 2, such that W 1 is again factorized as in Eq. (2), and a new graph G 2 (V 2, E 2 ) can be buit. In principa we can repeat this procedure unti we get ony one custer. Agorithm 1 summarizes this agorithm. Agorithm 1 Hierarchica Graph-Factorization Custering (HGFC) Require: given n data objects and a simiarity measure 1: buid the simiarity graph G 0 (V 0, E 0 ) with adjacency matrix W 0, and et m 0 = n 2: for = 1, 2,..., do 3: choose m < m 1 4: factorize G 1 to obtain K (V 1, V, F ) with the adjacency matrix B 5: buid a graph G (V, E ) with the adjacency matrix W = B D 1 B, where D s diagona entries are obtained by summation over B s coumns 6: end for The agorithm ends up with a hierarchica custering structure. For eve, we can assign data to the obtained m custers via a propagation from the bottom eve of custers. Based on the chain rue of Markov random waks, the soft (i.e., probabiistic) assignment of v i V 0 to custer v p () V is given by ( ) p v p () v i = v ( 1) V 1 v (1) V 1 p ( v () p v ( 1)) ) p (v (1) v i = ( D 1 1 B )ip, (6) where B = B 1 D 1 2 B 2D 1 3 B 3... D 1 B. One can interpret this by deriving an equivaent bipartite graph K (V 0, V, F ), and treating B as the equivaent adjacency matrix attached to the equivaent edges F connecting data V 0 and custers V. 4 Anaysis of the proposed agorithms 4.1 Fat custering: statistica modeing of singe-hop transitions In this section we provide some insights to the suggested custering agorithm, mainy from the perspective of random waks on graphs. Suppose that from a stationary stage of random waks on G(V, E), one observes π ij singe-hop transitions between v i and v j in a unitary time frame. As an intuition of graph-based view to simiarities, if two data points are simiar or reated, the transitions between them are ikey to happen. Thus we connect the observed simiarities to the frequency of transitions via w ij π ij. If the observed transitions are i.i.d. samped from a true distribution p(v i, v j ) = (HΛH ) ij where a bipartite graph is behind, then the og ikeihood with respect to the observed transitions is L(H, Λ) = og ij p(v i, v j ) πij w ij og(hλh ) ij. (7) ij Then we have the foowing concusion Proposition 4.1. For a weighted undirected graph G(V, E) and the og ikeihood defined in Eq. (7), the foowing resuts hod: (i) Minimizing the divergence distance (W, HΛH ) is equivaent to maximizing the og ikeihood L(H, Λ); (ii) Updates Eq. (3) and Eq. (4) correspond to a standard EM agorithm for maximizing L(H, Λ).

5 Figure 2: The simiarities of vertices to a fixed vertex (marked in the eft pane) on a 6- nearest-neighbor graph, respectivey induced by custering eve = 2 (the midde pane) and = 6 (the right pane). A darker coor means a higher simiarity. 4.2 Hierarchica custering: statistica modeing of muti-hop transitions The adjacency matrix W 0 of G 0 (V 0, E 0 ) ony modes one-hop transitions that foow direct inks from vertices to their neighbors. However, the random wak is a process of diffusion on the graph. Within a reativey onger period, a waker starting from a vertex has the chance to reach vertices faraway through muti-hop transitions. Obviousy, mutihop transitions induce a sowy decaying simiarity function on the graph. Based on the chain rue of Markov process, the equivaent adjacency matrix for t-hop transitions is A t = W 0 (D 1 0 W 0) t 1 = A t 1 D 1 0 W 0. (8) Generay speaking, a sowy decaying simiarity function on the simiarity graph captures a goba affinity structure of data manifods, whie a rapidy decaying simiarity function ony tes the oca affinity structure. The foowing proposition states that in the suggested HGFC, a higher-eve custering impicity empoys a more goba simiarity measure caused by muti-hop Markov random waks: Proposition 4.2. For a given hierarchica custering structure that starts from a bottom graph G 0 (V 0, E 0 ) to a higher eve G k (V k, E k ), the vertices V at eve 0 < k induces an equivaent adjacency matrix of V 0, which is A t with t = 2 1 as defined in Eq. (8). Therefore the presented hierarchica custering agorithm HGFC appies different sizes of time windows to examine random waks, and derives different scaes of simiarity measures to expose the oca and goba custering structures of data manifods. Fig. 2 iustrates the empoyed simiarities of vertices to a fixed vertex in custering eves = 2 and 6, which corresponds to time periods t = 2 and 32. It can be seen that for a short period t = 2, the simiarity is very oca and heps to uncover ow-eve custers, whie in a onger period t = 32 the simiarity function is rather goba. 5 Empirica study We appy HGFC on USPS handwritten digits and Newsgroup text data. For USPS data we use the images of digits 1, 2, 3 and 4, with respectivey 1269, 929, 824 and 852 images per cass. Each image is represented as a 256-dimension vector. The text data contain totay 3970 documents covering 4 categories, autos, motorcyces, baseba, and hockey. Each document is represented by an 8014-dimension TFIDF feature vector. Our method empoys a 10-nearest-neighbor graph, with the simiarity measure RBF for USPS and cosine for Newsgroup. We perform 4-eve HGFC, and set the custer number, respectivey from bottom to top, to be 100, 20, 10 and 4 for both data sets. We compare HGFC with two popuar aggomerative hierarchica custering agorithms, singe ink and compete ink (e.g., [3]). Both methods merge two cosest custers at each step.

6 Figure 3: Visuaization of HGFC for USPS data set. Left: mean images of the top 3 custering eves, aong with a Hinton graph representing the soft (probabiistic) assignments of randomy chosen 10 digits (shown on the eft) to the top 3rd eve custers; Midde: a Hinton graph showing the soft custer assignments from top 3rd eve to top 2nd eve; Right: a Hinton graph showing the soft assignments from top 2nd eve to top 1st eve. Figure 4: Comparison of custering methods on USPS (eft) and Newsgroup (right), evauated by normaized mutua information (NMI). Higher vaues indicate better quaities. Singe ink defines the custer distance to be the smaest point-wise distance between two custers, whie compete ink uses the argest one. A third compared method is normaized cut [5], which partitions data into two custers. We appy the agorithm recursivey to produce a top-down hierarchy of 2, 4, 8, 16, 32 and 64 custers. We aso compare with the k-means agorithm, k = 4, 10, 20 and 100. Before showing the comparison, we visuaize a part of custering resuts for USPS data in Fig. 3. On top of the eft figure, we show the top three eves of the hierarchy with respectivey 4, 10 and 20 custers, where each custer is represented by its mean image via an average over a the images weighted by their posterior probabiities of beonging to this custer. Then 10 randomy samped digits with soft custer assignments to the top 3rd eve custers are iustrated with a Hinton graph. The midde and right figures in Fig. 3 show the assignments between custers across the hierarchy. The cear diagona bock structure in a the Hinton graphs indicates a very meaningfu custer hierarchy.

7 Normaized cut HGFC K-means Tabe 1: Confusion matrices of custering resuts, 4 custers, USPS data. In each confusion matrix, rows correspond true casses and coumns correspond the found custers. Normaized cut HGFC K-means autos motor baseba hockey Tabe 2: Confusion matrices of custering resuts, 4 custers, Newsgroup data. In each confusion matrix, rows correspond true casses and coumns correspond the found custers. We compare the custering methods by evauating the normaized mutua information (NMI) in Fig. 4. It is defined to be the mutua information between custers and true casses, normaized by the maximum of margina entropies. Moreover, in order to more directy assess the custering quaity, we aso iustrate the confusion matrices in Tabe 1 and Tabe 2, in the case of producing 4 custers. We drop out the confusion matrices of singe ink and compete ink in the tabes, for saving spaces and aso due to their ceary poor performance compared with others. The resuts show that singe ink performs poory, as it greediy merges nearby data and tends to form a big custer with some outiers. Compete ink is more baanced but unsatisfactory either. For the Newsgroup data it even gets stuck at the 3601-th merge because a the simiarities between custers are 0. Top-down hierarchica normaized cut obtains reasonabe resuts, but sometimes cannot spit one big custer (see the tabes). The confusion matrices indicates that k-means does we for digit images but reativey worse for high-dimension textua data. In contrast, Fig. 4 shows that HGFC gives significanty higher NMI vaues than competitors on both tasks. It aso produces confusion matrices with cear diagona structures (see tabes 1 and 2), which indicates a very good custering quaity. 6 Concusion and Future Work In this paper we have proposed a probabiistic graph partition method for custering data objects based on their pairwise simiarities. A nove hierarchica custering agorithm HGFC has been derived, where a higher eve in HGFC corresponds to a statistica mode of random wak transitions in a onger period, giving rise to a more goba custering structure. Experiments show very encouraging resuts. In this paper we have empiricay specified the number of custers in each eve. In the near future we pan to investigate effective methods to automaticay determine it. Another direction is hierarchica custering on directed graphs, as we as its appications in web mining.

8 Appendix Proof of Theorem 2.1. We first notice that p λp = ij wij under constraints i hip = 1. Therefore we can normaize W by ij wij and after convergence mutipy a λp by this quantity to get the soution. Under this assumption we are maximizing L(H, Λ) = ij wij og(hλh ) ij with an extra constraint p λp = 1. We first fix λp and show update Eq. (3) wi not decrease L(H) L(H, Λ). We prove this by constructing an auxiiary function f(h, H ) such that f(h, H ) L(H) and f(h, H) = L(H). Then we know the update H t+1 = arg max H f(h, H t ) wi not decrease L(H) since L(H t+1 ) f(h t+1, H t ) f(h t, H t ) = L(H t ). Define f(h, H ) = ( ) ij wij h ip λph jp p h i λ h og h ipλ ph jp og h ip λph jp j h i λ h. f(h, H) = L(H) can be easiy verified, and f(h, H ) L(H) aso foows if we use concavity of og function. Then it is straightfor- j ward to verify Eq. (3) by setting the derivative of f with respect to h ip to be zero. The normaization is due to the constraints and can be formay derived from this procedure with a Lagrange formaism. Simiary we can define an auxiiary function for Λ with H fixed, and verify Eq. (4). Proof of Proposition 4.1. (i) foows directy from the proof of Theorem 2.1. To prove (ii) we take u p as the missing data and foow the standard way to derive the EM agorithm. In the E-step we estimate the a posteriori probabiity of taking u p for pair (v i, v j) using Bayes rue: ˆp(u p v i, v j) p(v i u p)p(v j u p)p(u p). And then in the M-step we maximize the compete data ikeihood ˆL(G) = ij p wij ˆp(up vi, vj) og p(vi up)p(vj up)p(up) with respect to mode parameters h ip = p(v i u p) and λ p = p(u p), with constraints i hip = 1 and p λp = 1. By setting the corresponding derivatives to zero we obtain h ip j wij ˆp(up vi, vj) and λp ij wij ˆp(up vi, vj). It is easy to check that they are equivaent to updates Eq. (3) and Eq. (4) respectivey. Proof of Proposition 4.2. We give a brief proof. Suppose that at eve the data-custer reationship is described by K (V 0, V, F ) (see Eq. (6)) with adjacency matrix B, degrees D 0 for V 0, and degrees Λ for V. In this case the induced adjacency matrix of V 0 is W = B Λ 1 B, and the adjacency matrix of V is W = B D 1 0 B. Let K (V, V +1, F +1 ) be the bipartite graph connecting V and V +1, with the adjacency B +1 and degrees Λ +1 for V +1. Then the adjacency matrix of V 0 induced by eve + 1 is W +1 = B Λ 1 B +1 Λ 1 +1 B +1Λ 1 B = W D 1 0 W, where reations B +1 Λ 1 +1 B +1 = B D 1 0 B and W = B Λ 1 B are appied. Given the initia condition from the bottom eve W1 = W 0, it is not difficut to obtain W = A t with t = 2 1. References [1] J. Godberger and S. Roweis. Hierarchica custering of a mixture mode. In L.K. Sau, Y. Weiss, and L. Bottou, editors, Neura Information Processing Systems 17 (NIPS*04), pages , [2] K.A. Heer and Z. Ghahramani. Bayesian hierarchica custering. In Proceedings of the 22nd Internationa Conference on Machine Learning, pages , [3] S. D. Kamvar, D. Kein, and C. D. Manning. Interpreting and extending cassica aggomerative custering agorithms using a mode-based approach. In Proceedings of the 19th Internationa Conference on Machine Learning, pages , [4] Danie D. Lee and H. Sebastian Seung. Agorithms for non-negative matrix factorization. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neura Information Processing Systems 13 (NIPS*00), pages , [5] Jianbo Shi and Jitendra Maik. Normaized cuts and image segmentation. IEEE Transactions on Pattern Anaysis and Machine Inteigence, 22(8): , [6] D. Zhou, B. Schökopf, and T. Hofmann. Semi-supervised earning on directed graphs. In L.K. Sau, Y. Weiss, and L. Bottou, editors, Advances in Neura Information Processing Systems 17 (NIPS*04), pages , 2005.