Communities, Spectral Clustering, and Random Walks

Size: px

Start display at page:

Download "Communities, Spectral Clustering, and Random Walks"

Randell Curtis
5 years ago
Views:

1 Communities, Spectral Clustering, and Random Walks David Bindel Department of Computer Science Cornell University 3 Jul 202

2 Spectral clustering recipe Ingredients:. A subspace basis with useful information (eigenvectors of L, L, B, etc; NMF factors) 2. A way to extract clusters from the basis (sign patterns; (scaled) latent coordinates) What works for small communities? With overlap?

3 Classic spectral bisection Goal: Split graph into equal parts, minimizing edges cut. Strategy: turn into a constrained integer QP Identify partition with label vector s {±} n. Objective: s T L s = 4 edges cut Constraint: e T s = 0 This is NP hard, but...

4 Classical spectral bisection Hard: min s T L s s.t. e T s = 0, s {±} n. Easy: min v T Lv s.t. e T v = 0, v R n, v 2 = n. v is the Fiedler vector (second eigenvector of L)

5 Three clusters? Take three eigenvectors as latent coordinates + cluster: V r. r r 2. r 2 r 3. r 3 =... r r 2 r 3 = SR. Equivalent to a matrix factorization!

6 Overlapping communities nz = 3996 A S diag(β)s T, S {0, } n c

7 Spectrum for A with overlap 20 Eigenvalue Index

8 Clustering and overlap Dominant eigenvectors for A: 2 0 Alternate basis for the space: How do we get the latter basis?

9 Recovering indicators minimize s (proxy for sparsity of s) s.t. s = Uy ( s in the right space) s i ( seed constraint) s 0 (componentwise nonnegativity) Given a basis U, want to extract a vector s s.t. s lies close to the span of U s is almost an indicator for a community Some known seeds are in the community

10 Indicators from subspaces: LP edition minimize s (proxy for sparsity of s) s.t. s = Uy ( s in the right space) s i ( seed constraint) s 0 (componentwise nonnegativity) Recovers smallest set containing node i if U = SY exactly. Each set contains at least one element only in that set. (Frequently works if there is not too much overlap.) Got noise? Need thresholding!

11 Alas The method is willing, but the space is weak: Eigenvectors of A may not be ideal what about L, L, transition matrices for random walks? Many small communities = many eigenvectors? So consider: Why eigenvectors? Optimization connection Dynamics connection What might serve better?

12 Optimization and quadratic forms Indicate V V by s {0, } n. Measure subgraph: s T As = E = internal edges s T Ds = edges incident on subgraph s T Ls = edges between V and V s T Bs = surprising internal edges

13 Rayleigh quotients s T As s T = mean internal degree in subgraph s s T Ls s T s = edges cut between V and V relative to V s T As s T = fraction of incident edges internal to V Ds s T Ls s T = fraction of incident edges cut Ds s T Bs s T = mean surprising internal degree in subgraph s s T Bs s T = mean fraction of internal degree that is surprising Ds

14 Rayleigh quotients and eigenvalues Basic connection (M spd): x T Kx x T Mx Easy despite lack of convexity. stationary at x Kx = λmx

15 Rayleigh quotients big and small Decompose: W T MW = I and W T KW = Λ = diag(λ,..., λ n ). For any x 0, x T Mx =, have x = j w j z j x T Kx = j λ j z 2 j So s T Ks s T Ms λ max = s λ j λ max w j z j. So look at invariant subspaces for extreme eigenvalues.

16 The dynamics perspective

17 The random walker Basic idea: extract structure from random walk. Start at seed and walk forward Day : I came up with a funny joke! Day 2: I tell everyone in my family Day 3: My mother tells a friend? Or look at how quickly source is forgotten Day : David came up with a funny joke! Day 2: There s a joke going around Cornell CS. Day 3: I read this bad joke on the web...

18 The random walker Lazy random walk with transition matrix S = (αi + AD )/( + α).. Start at p 0, take k steps. Distribution: p k = S k p 0 ( d/m as k ) 2. End at q 0 after k steps. Conditional distribution on start: q k ( S T ) k q0 ( e/n as k ) Notes: If the graph is undirected, S T = D SD. If the graph is also regular, S T = S.

19 Simon-Ando theory Markov chain with loosely-coupled subchains: Rapid local mixing: after a few steps p k c j= α j,k p (j) where p (j) is a local equilibrium for the jth subchain Slow equilibration: α j,k α j,. Alternately, rapid local mixing looks like: q k c γ j,k s j j= where s j is an indicator for nodes in one subchain.

20 Spectral Simon-Ando picture Exactly decoupled case (c decoupled chains): Eigenvalue one has multiplicity c. Eigenvectors of S are local equilibria. Eigenvectors of S T are indicators for chains. Rapid mixing = large gap to λ c+. Weakly coupled case: Cluster of c eigenvalues near. Eigenvectors of S are combinations of local equilibria. Eigenvectors of S T are combinations of indicators. Large gap between λ c and λ c+.

21 Beyond eigenvectors Connections that suggest using dominant eigenvectors: Optimization of quadratic forms Asymptotic dynamics of random walks Works well for a few big communities what about local ones?

22 Ritz vectors Variational characterization of eigenpairs: x T Kx x T Mx stationary at x Kx = λmx Rayleigh-Ritz approximation: x = Vy (Vy) T K (Vy) (Vy) T M(Vy) stationary at y (V T KV )y = ˆλ(V T MV )y

23 Krylov + Rayleigh-Ritz Usual approach to large-scale symmetric eigenproblems:. Generate a basis for a Krylov subspace K k (A, x 0 ) = span{x 0, Ax 0, A 2 x 0,..., A k x 0 } 2. Use Rayleigh-Ritz on subpace 3. Re-start if subspace gets too large

24 Krylov + Rayleigh-Ritz + dynamics Eigendecomposition picture: p k = N α j λ k j v j j= Krylov + Rayleigh-Ritz: {(µ j, w j )} r j= = Ritz pairs from K r (S, p 0 ) r p k = β j µ k j w j, k < r j=

25 The idea: Ritz subspaces Idea: Use unconverged subspace computations Invariant subspace Long random walks Global information Potentially expensive May need large space Ritz subspace Short random walks Local information Cheap to compute Small space okay

26 Current favorite method. Pick seed nodes j, j 2, Take short random walks (length k) from each seed 3. Extract few Ritz vectors from span{q 0, q,..., q k }. 4. Approx indicators in span of all Ritz vectors. 5. Possibly add more seeds and return to step. 6. Convert raw score vector to a {0, } indicator.

27 Wang test graph 6 7

28 Spectrum for Wang test graph Eigenvalue Index

29 Zachary Karate graph

30 Spectrum for Karate Eigenvalue Index

31 Football graph

32 Spectrum for Football Eigenvalue Index

33 Dolphin graph

34 Spectrum for Dolphin Eigenvalue Index

35 Non-overlapping synthetic benchmark (µ = 0.5) nz = 5746

36 Spectrum for synthetic benchmark Eigenvalue Index

37 Score vector 0.8 Score ,000 Index Score vector for the two-node seed of 492 and 53 in the first LFR benchmark graph. Ten steps, three Ritz vectors.

38 Non-overlapping synthetic benchmark (µ = 0.6) nz = 536

39 Spectrum for synthetic benchmark Eigenvalue Index

40 Score vector Score ,000 Index Score vector for the two-node seed of 492 and 53 in the first LFR benchmark graph. Ten steps, three Ritz vectors.

41 Overlapping synthetic benchmark (µ = 0.3) 000 nodes 47 communities 500 nodes belong to two communities

42 Spectrum for synthetic benchmark Eigenvalue Index

43 Score vector Score ,000 Index Score vector for the two-node seed of 52 and 892. The desired indicator is in red.

44 Score vector.5 Score ,000 Index Score vector for the two-node seed of 52 and twelve reseeds. The desired indicator is in red.

45 Conclusions Classic spectral methods use eigenvectors to cluster, but: We don t need to stop at partitioning! Overlap is okay Key is how we mine the subspace We don t need to stop at eigenvectors! Can also use Ritz vectors Computation is cheap: short random walks Still very much need: Better theoretical grounding Better ideas for thresholding Testing on large-scale examples

Communities, Spectral Clustering, and Random Walks

Communities, Spectral Clustering, and Random Walks David Bindel Department of Computer Science Cornell University 26 Sep 2011 20 21 19 16 22 28 17 18 29 26 27 30 23 1 25 5 8 24 2 4 14 3 9 13 15 11 10 12