Machine Learning for Data Science (CS4786) Lecture 11

Transcription

1 Machine Learning for Data Science (CS4786) Lecture 11 Spectral clustering Course Webpage :

2 ANNOUNCEMENT 1 Assignment P1 the Diagnostic assignment 1 will be posted on Tuesday, March 15th. 2 Assignment P1 due on Thursday March 17th. 3 Individual submissions. Do not work in groups. 4 P1 will be simpler than the regular assignments and no coding.

3 SPECTRAL CLUSTERING Input: Similarity matrix A A i,j = A j,i > 0 indicates similarity between elements x i and x j n A n Example: A i,j = exp( d(x i, x j )) A is adjacency matrix of a graph

4 LAPLACIAN MATRIX Let D be diagonal matrix with D i,i = n j=1 A i,j (degree of vertex i) Laplacian matrix defined as L = D A Fact 1: Laplacian for a connected graph has exactly one 0 eigenvalue, corresponding eigenvector is 1 = (1,...,1) n Proof: Sum of each row of L is 0 as D i,i = n j=1 A i,j and L = D A Fact 2: For general graph, number of 0 eigenvalues correspond to number of connected components. Proof: L is block diagonal, one block for each connected component. Use connected graph result on each component.

5 GRAPH CLUSTERING Fact: For general graph, number of 0 eigenvalues correspond to number of connected components. The corresponding eigenvectors are all 1 s on the nodes of connected components Proof: L is block diagonal. Use connected graph result on each component.

6 GRAPH CLUSTERING: CUTS Partition nodes so that as few edges are cut (Mincut) What has this got to do with the Laplacian matrix?

7 CUTS AND LAPLACIAN Consider case when we have/want 2 clusters. Let c j = 1 if x j belongs to cluster 0 and c j = 1 if x j belongs to cluster 1 CUT = (i,j) E 1 ci c j = 1 2 c Lc

8 SPECTRAL CLUSTERING, K = 2 Hence to find the solution we need to solve for Minimize c Lc s.t. i [n], c i = 1 Since i [n], c i = 1, we have c 2 = n and so relaxing (approximating) the optimization: Minimize c Lc s.t. c 2 = n Hence solution c to above is an Eigen vector, first smallest one is the all 1 s vector (for connected graph), second smallest one is our solution To get clustering assignment we simply threshold at 0

9 SPECTRAL CLUSTERING ALGORITHM (UNNORMALIZED) 1 Given matrix A calculate diagonal matrix D s.t. D i,i = n j=1 A i,j 2 Calculate the Laplacian matrix L = D A 3 Find eigen vectors v 1,...,v n of L (ascending order of eigenvalues) 4 Pick the K eigenvectors with smallest eigenvalues to get y 1,...,y n R K 5 Use K-means clustering algorithm on y 1,...,y n

10 EXAMPLE

11 SPECTRAL CLUSTERING (UNNORMALIZED) Min-cut on a graph can be efficiently computed Why bother with the approximate algorithm Is cut even a good measure?

12 NORMALIZED CUT Why cut is perhaps not a good measure?

13 RATIO CUT Why cut is perhaps not a good measure? Fixes?

14 RATIO CUT Why cut is perhaps not a good measure? Fixes? Perhaps Ratio Cut CUT(C 1, C 2 ) 1 C 2 C C 2 C Set c i = 1 if i C 1 C 1 C 2 otherwise Verify that c Lc = n Ratio Cut and c 2 = n (and c 1) Relaxed solution is same as Unnormalized Spectral clustering

15 NORMALIZED CUT Normalized cut: Minimize sum of ratio of number of edges cut per cluster and number of edges within cluster NCUT = j CUT(C j ) Edges(C j ) Example K = 2 CUT(C 1, C 2 ) 1 Edges(C 1 ) + 1 Edges(C 2 ) This is an NP hard problem!

16 NORMALIZED CUT Normalized cut: Minimize sum of ratio of number of edges cut per cluster and number of edges within cluster NCUT = j CUT(C j ) Edges(C j ) Example K = 2 CUT(C 1, C 2 ) 1 Edges(C 1 ) + 1 Edges(C 2 ) This is an NP hard problem!... so relax

17 NORMALIZED CUT First note that Edges(C i ) = k xk C i D k,k Edges(C 2 ) Edges(C Set c i = 1 ) if i C 1 Edges(C 1 ) Edges(C 2 ) otherwise Verify that c Lc = E NCut and c Dc = E (and Dc 1) Hence we relax Minimize NCUT(C) to Minimize c Lc c Dc s.t. Dc 1 Solution: Find second smallest eigenvectors of L = I D 12 AD 12

18 SPECTRAL CLUSTERING ALGORITHM (NORMALIZED) 1 Given matrix A calculate diagonal matrix D s.t. D i,i = n j=1 A i,j 2 Calculate the normalized Laplacian matrix L = I D 12 AD 12 3 Find eigen vectors v 1,...,v n of L (ascending order of eigenvalues) 4 Pick the K eigenvectors with smallest eigenvalues to get y 1,...,y n R K 5 Use K-means clustering algorithm on y 1,...,y n

19 NORMALIZED CUT: ALTERNATE VIEW If we perform random walk on graph, its the partition of graph into group of vertices such that the probability of transiting from one group to another is minimized Transition matrix: D 1 A Largest eigenvalues and eigenvectors of above matrix correspond to smallest eigenvalues and eigenvectors of D 1 L = I D 1 A For K-nearest neighbor graph (K-regular), same as normalized Laplacian