On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering

Transcription

1 On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering Chris Ding, Xiaofeng He, Horst D. Simon Published on SDM 05 Hongchang Gao

2 Outline NMF NMF Kmeans NMF Spectral Clustering NMF Bipartite graph Kmeans

3 Outline NMF NMF Kmeans NMF Spectral Clustering NMF Bipartite graph Kmeans

4 NMF Paatero and apper (1994) Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values Environmetrices Lee and Seung (1999, 2000) Learning the parts of objects by non-negative matrix factorization, Nature Algorithms for non-negative matrix factorization, NIPS

5 NMF Matrix Factorization is widely used in machine learning, such as SVD X = U Σ V mixed nonneg interpretation of basis vectors is difficult due to mixed signs mixed

6 NMF Nonnegative Matrix Factorization where X = F G columns of F are the underlying basis vectors rows of G give the weights associated with each basis vector nonneg nonneg X R, F R, G R d n d k n k

7 Outline NMF NMF Kmeans NMF Spectral NMF Bipartite graph Kmenas

8 Kmeans Kmeans clustering is one of most widely used clustering method.

9 Kmeans Reformulate Kmeans Clustering Cluster membership indicators:

10 Kmeans Objective function Replace W = X X, which is the standard inner-product linear Kernel matrix

11 Kernel Kmeans Map x to higher dimension space: Kernel Kmeans objective:

12 NMF Kmeans Orthogonal symmetric NMF is equivalent to Kernel Kmeans clustering

13 Kernel Kmeans=>Symmetric NMF Factorization is equivalent to Kernel K-means clustering with the strict orthogonality relaxed H = arg max r( H WH ) H H= I, H 0 = arg min 2 r( H WH ) H H= I, H 0 = arg min W 2 r( H WH ) + H H H H= I, H 0 = arg min W HH H H= I, H Relaxing the orthogonality H H = I completes the proof

14 Symmetric NMF=> Kernel Kmeans W = HH factorization retains H orthogonality approxiamately. 2 Proof. min W HH is equivalent to max r( H WH ) H 0 min H H H 0 he first one recover the objective 2

15 Symmetric NMF=> Kernel Kmeans he second one Minimize the first term, we get Minimize the second term We should make sure H cannot be all zero

16 Outline NMF NMF Kmeans NMF Spectral NMF Bipartite graph Kmeans

17 Spectral Clustering Spectral clustering objective functions

18

19 Spectral Clustering Reformulate the objective based on Ncut J Replace hen, 1 cut( V, V V ) 1 h ( D W ) h = = K vol V K h Dh K K l l l l l= 1 ( l) l= 1 l l z l = D h 1/2 l 1/2 D hl K ~ K ~ l l l l l= 1 l= 1 = = J z ( I W ) z z z r( Z W Z)

20 NMF Spectral Clustering he objective of spectral clustering max r ( Z W Z ) Z Z= IZ, 0 his is identical to the Kernel Kmeans clustering Spectral Clustering Kernel Kmeans NMF ~

21 Outline NMF NMF Kmeans NMF Spectral NMF Bipartite graph Kmenas

22 Bipartite graph Kmeans Simultaneous clustering of rows and columns

23 Bipartite graph Kmeans Simultaneously cluster the rows and columns of data matrix B= ( x, x x ) 1 2,..., n Row Clustering max ( r F BB F ) F F= IF, 0 Column Clustering max ( r G B BG ) GG= IG, 0

24 Bipartite graph Kmeans Equivalent problem: Solution hen, Bg = λ f, B f = λ g k k k k k k B Bg = λ g, BB f = λ f 2 2 k k k k k k

25 Bipartite graph Kmeans=>NMF he simultaneous row and column Kmeans clustering is equivalent to the following optimization problem

26 Bipartite graph Kmeans=>NMF Proof. max r( F BG) FG, min r( F BG) FG, B r F BG + r F FG G 2 min 2 ( ) ( ) FG, min B FG FG, herefore, NMF is equivalent to Kmeans clustering with relaxed orthogonality contraints. 2

27 NMF=>Bipartite graph Kmeans In the previous, we assume both F and G are orthogonal. If one of them is orthogonal, we 2 can explicitly write B FG as a Kmeans clustering objective function. NMF with orthogonal G is identical to Kmeans clustering of the columns of B.

28 NMF=>Bipartite graph Kmeans Proof. At first, normalize the row of G, s.t. hen, for the objective function We have

29 NMF=>Bipartite graph Kmeans he orthogonality condition of G implies that in each row of G, only one element is nonzero and g ik = 0,1 Summing over i: J = x f 2 k= 1 i C which is the Kmeans clustering K k i k 2

30 Reference Ding, Chris HQ, Xiaofeng He, and Horst D. Simon. "On the Equivalence of Nonnegative Matrix Factorization and Spectral Clustering." SDM. Vol Li, ao, and Chris Ding. "he relationships among various nonnegative matrix factorization methods for clustering." Data Mining, ICDM'06. Sixth International Conference on. IEEE, Von Luxburg, Ulrike. "A tutorial on spectral clustering." Statistics and computing 17.4 (2007): Shi, Jianbo, and Jitendra Malik. "Normalized cuts and image segmentation." Pattern Analysis and Machine Intelligence, IEEE ransactions on 22.8 (2000):

31 hanks