Local quality functions for graph clustering

Transcription

1 Loal quality funtions for graph lustering Twan van Laarhoven Institute for Computing and Information Sienes Radoud University Nijmegen, The Netherlands Lorentz workshop on Clustering, Games and Axioms 25th June / 46

2 Outline Graph lustering Loality Case study: Modularity Soft lustering A loal NMF 2 / 46

3 Outline Graph lustering Loality Case study: Modularity Soft lustering A loal NMF 3 / 46

4 Graphs Distane funtions Graphs a d(i, j) E(i, j) a d d 4 / 46

5 Graphs Distane funtions Graphs a d(i, j) E(i, j) a - d d 4 / 46

6 Graphs a d f e g k h i j A symmetri weighted graph (or network) is a pair (V, A) of a finite set V of nodes, and a symmetri matrix A : R V V 0 of edge weights, 5 / 46

7 Graph lustering a d f e g k h i j A lustering C of a graph G = (V, A) is a partition of its nodes (for now). 5 / 46

8 Clustering: formalizations Clustering funtion F : Graph Clustering F a e d = a e d Quality funtion Q : Graph Clustering R Quality relation G Clustering Clustering 6 / 46

9 Clustering: formalizations Clustering funtion F : Graph Clustering Quality funtion Q : Graph Clustering R Q a e d = Quality relation G Clustering Clustering 6 / 46

10 Clustering: formalizations Clustering funtion F : Graph Clustering Quality funtion Q : Graph Clustering R Quality relation G Clustering Clustering a e d a e d 6 / 46

11 (im)possiility results Theorem (Kleinerg, 2002) There is no lustering funtion that is permutation invariant, sale invariant, onsistent and rih. Theorem (Akerman, Ben-David 2008) There is a lustering quality funtion that is permutation invariant, sale invariant, onsistent and rih. Theorem There is a graph lustering funtion that is permutation invariant, sale invariant, onsistent and rih. 7 / 46

12 (im)possiility results Theorem (Kleinerg, 2002) There is no lustering funtion that is permutation invariant, sale invariant, onsistent and rih. Theorem (Akerman, Ben-David 2008) There is a lustering quality funtion that is permutation invariant, sale invariant, onsistent and rih. Theorem There is a graph lustering funtion that is permutation invariant, sale invariant, onsistent and rih. 7 / 46

13 (im)possiility results Theorem (Kleinerg, 2002) There is no lustering funtion that is permutation invariant, sale invariant, onsistent and rih. Theorem (Akerman, Ben-David 2008) There is a lustering quality funtion that is permutation invariant, sale invariant, onsistent and rih. Theorem There is a graph lustering funtion that is permutation invariant, sale invariant, onsistent and rih. 7 / 46

14 Some quality funtions Conneted omponents Q(G, C) = 1[C are the onneted omponents of G]. Total weight of within luster edges Q(G, C) = a, where a uv = C i u Modularity Q(G, C) = C ( a /a VV (a V /a VV ) 2) a ij. j v Many more Q(G, C) = a log(a V /a VV ) C 8 / 46

15 Families of quality funtions Conneted omponents with a threshold Q(G, C) = 1[C are the onneted omponents of a ij τ] Total weight of within luster edges with penalty Q(G, C) = a α C C Generalized modularity Q γ RB (G, C) = ( a /a VV γ(a V /a V ) 2) C Many more Q(G, C) = a log(a V /α) C 9 / 46

16 Outline Graph lustering Loality Case study: Modularity Soft lustering A loal NMF 10 / 46

17 Motivation: Resolution limit 11 / 46

18 Motivation: Resolution limit 11 / 46

19 Motivation: Resolution limit (atually with > 22 nodes) 11 / 46

20 Motivation: Sugraphs In pratie we don t oserve the whole graph Random susampling edges Random susampling nodes Part of the graph a d e f g h k i j l m 12 / 46

21 Motivation: Sugraphs In pratie we don t oserve the whole graph Random susampling edges Random susampling nodes Part of the graph a d e f g h k i j l m 12 / 46

22 Motivation: Sugraphs In pratie we don t oserve the whole graph Random susampling edges Random susampling nodes Part of the graph a d e f g h k i j l m 12 / 46

23 Motivation: Sugraphs In pratie we don t oserve the whole graph Random susampling edges Random susampling nodes Part of the graph a d e f g h k i j l m 12 / 46

24 Motivation: Noise Related to monotoniity/onsisteny. Changes to a part of the graph that don t involve a luster shouldn t affet that luster. Also, noise in one part of the graph should have a minimal effet on the lustering of other parts of the graph. 13 / 46

25 Resolution-limit-free Definition: sugraph A graph G = (V, A) is a sugraph of G = (V, A) if V V and a ij = a ij for all i, j V. Definition: resolution-limit-free (Traag, Van Dooren, 2011) Let C e a Q-optimal lustering of a graph G 1. Then the quality funtion Q is alled resolution-limit-free if for eah sugraph G 2 indued y D C, the lustering D is also Q-optimal. Reformulation A lustering funtion F is resolution-limit-free if for all graphs G and sugraphs G S of G, for all lusterings C of G and D C of G S, it is the ase that if F (G) = C then F (G S ) = D. 14 / 46

26 Resolution-limit-free Definition: sugraph A graph G = (V, A) is a sugraph of G = (V, A) if V V and a ij = a ij for all i, j V. Definition: resolution-limit-free (Traag, Van Dooren, 2011) Let C e a Q-optimal lustering of a graph G 1. Then the quality funtion Q is alled resolution-limit-free if for eah sugraph G 2 indued y D C, the lustering D is also Q-optimal. Reformulation A lustering funtion F is resolution-limit-free if for all graphs G and sugraphs G S of G, for all lusterings C of G and D C of G S, it is the ase that if F (G) = C then F (G S ) = D. 14 / 46

27 Resolution-limit-free Definition: sugraph A graph G = (V, A) is a sugraph of G = (V, A) if V V and a ij = a ij for all i, j V. Definition: resolution-limit-free (Traag, Van Dooren, 2011) Let C e a Q-optimal lustering of a graph G 1. Then the quality funtion Q is alled resolution-limit-free if for eah sugraph G 2 indued y D C, the lustering D is also Q-optimal. Reformulation A lustering funtion F is resolution-limit-free if for all graphs G and sugraphs G S of G, for all lusterings C of G and D C of G S, it is the ase that if F (G) = C then F (G S ) = D. 14 / 46

28 Generalized: loality Definition: Loality A lustering quality funtion Q is loal if for all graphs G 1, G 2, and ommon sugraphs G S of G 1 and G 2, for all sets of lusters C 1, C 2, D, D, suh that C 1 D and C 1 D are lusterings of G 1, C 2 D and C 2 D are lusterings of G 2, and D and D are lusterings of G S, it is the ase that Q(G 1, C 1 D) Q(G 1, C 1 D ) if and only if Q(G 2, C 2 D) Q(G 2, C 2 D ). Intuition Clusterings of a sugraph an e ompared independently of the lustering of the rest of the graph. 15 / 46

29 Additivity Definition: Additivity A lustering quality funtion Q is additive if it an e written as Q(G, C) = q graph (G) + C for some funtions q graph, q lus, q edge. Lemma q lus () + q edge (a ij, i C j). i,j V All additive lustering quality funtions are loal. (the onverse is not true) 16 / 46

30 Loality Intuition: Additivity Q a e d = Q a + Q e d 17 / 46

31 Loality Intuition: Indued sugraphs Q a e d Q a e d Q a Q a 17 / 46

32 Loality Intuition: Union of two graphs and two lusterings Q a Q a and Q e d Q e d Q a e d Q a e d 17 / 46

33 Loality Intuition: Changing other parts of the lustering should have no effet Q a e d Q a e d Q a e d Q a e d 17 / 46

34 Loality Intuition: Adding/removing edges in other lusters should have no effet Q a e d Q a e d Q a e d Q a e d 17 / 46

35 Loality Q a Q a Q a Q a 17 / 46

36 Charaterization Whih quality funtions are loal? Quality of a luster in D an only depend on edges in G S. So only internal onnetivity/ohesiveness an e onsidered. Union of graphs and lusterings doesn t hange optima, so there an t e parameter for fixed numer of lusters. Definition: neighoring sugraph A sugraph G S of G is alled neighored y G N, if a ij = 0 for all i V S and j V \ V N. Intuition: We might are only aout loality wrt. far away nodes, not neighors. 18 / 46

37 Charaterization Whih quality funtions are loal? Quality of a luster in D an only depend on edges in G S. So only internal onnetivity/ohesiveness an e onsidered. Union of graphs and lusterings doesn t hange optima, so there an t e parameter for fixed numer of lusters. Definition: neighoring sugraph A sugraph G S of G is alled neighored y G N, if a ij = 0 for all i V S and j V \ V N. Intuition: We might are only aout loality wrt. far away nodes, not neighors. 18 / 46

38 Fixed-size loality Idea Require loality only for k-lusterings. Definition: Fixed-size loality A lustering quality funtion Q is fixed-size loal if for all k 1, for all graphs G 1, G 2, and ommon sugraphs G S of G 1 and G 2, for all sets of lusters C 1, C 2, D, D, suh that C 1 D and C 1 D are k-lusterings of G 1, C 2 D and C 2 D are k-lusterings of G 2, and D and D are lusterings of G S, it is the ase that Q(G 1, C 1 D) Q(G 1, C 1 D ) if and only if Q(G 2, C 2 D) Q(G 2, C 2 D ). 19 / 46

39 Weak loality Definition: Weak loality A quality funtion Q is weakly loal if for all graphs G 1 = (V 1, A 1 ) and G 2 = (V 2, A 2 ), with ommon sugraph G N, and sugraph G S that is neighored y G N, sets of nodes C 1 P(V 1 ), C 2 P(V 2 ), and D, D P(V S ), suh that C 1 D and C 1 D are lusterings of G 1, suh that C 2 D and C 2 D are lusterings of G 2, it is the ase that if Q(G 1, C 1 D) Q(G 1, C 1 D ) then Q(G 2, C 2 D) Q(G 2, C 2 D ). 20 / 46

40 Outline Graph lustering Loality Case study: Modularity Soft lustering A loal NMF 21 / 46

41 Modularity Intuition: Balane within luster edges against luster volume. Definition: Q modularity (G, C) = C ( a ( av ) ) 2. a VV a VV Loality The term a V = i,j V a ij preludes loality, ut maye weakly loal? 22 / 46

42 Modularity is not weakly loal ( Q modularity ( Q modularity ( Q modularity ( Q modularity a d a d ) ) = 0.3 = a d x y a d x y ) = 0.3 ) = / 46

43 Idea: Fix the sale Pik a onstant M, Q M-fixed (G, C) = C ( a ( M av ) ) 2 M Issues Fails other axioms (monotoniity). 24 / 46

44 Proof: modularity is not monotoni Pik a onstant M, Q M-fixed (G, C) = C ( a ( M av ) ) 2 M Take a V = w + (within + etween) Q M-fixed (G, C) w = 1 M 2w + 2 M 2. This is negative when 2a V > M, so not monotoni. 25 / 46

45 Idea 2: Add some v to the denominator Q M,γ (G, C) = ( w ( v ) ) 2. M + γv M + γv C This quality funtion is permutation invariant, ontinuous and loal. monotoni for all M 0 and γ 2. rih for all M 0 and γ 1. sale invariant for M = / 46

46 Outline Graph lustering Loality Case study: Modularity Soft lustering A loal NMF 27 / 46

47 Non-negative Matrix Fatorization The idea Fatorize a matrix into non-negative fators, A W T H s.t. w ki 0, h kj 0. This is a general reipe, still have to deide what means. Variants Symmetri: W = H. Proailisti or regularized variants. 28 / 46

48 NMF graph lustering The idea G = (V, A), fatorize A. Intuition Eah node i spends a fration w ki of time in luster k. There they meet other nodes in that same luster. Gives soft lusters: A node an e in more than one luster (overlapping lusters). Memerships are fuzzy. 29 / 46

49 Formalizing soft lustering Definition: soft lustering A soft lustering of a graph G = (V, A) is a set C of ojets alled lusters, together with a funtion supp : C P(V ), alled the support. Remarks Clusters an have different elements, depending on the method. For NMF: the support of a luster is the set of nodes for whih w i > / 46

50 Is NMF resolution limit free? What is the indued sugraph if lusters are soft? Common approah: turn soft lustering into hard lustering: Assign eah node to the luster with highest memership. The answer is then: no. 31 / 46

51 Is NMF resolution limit free? What is the indued sugraph if lusters are soft? Common approah: turn soft lustering into hard lustering: Assign eah node to the luster with highest memership. The answer is then: no. 31 / 46

52 Is NMF resolution limit free? (take 2) Take the indued sugraph ased on the support. The answer is still no / 46

53 Is NMF resolution limit free? (take 2) Take the indued sugraph ased on the support. The answer is still no / 46

54 Loal quality funtions Definition A lustering quality funtion Q is loal if for all graphs G 1, G 2, and ommon sugraphs G S of G 1 and G 2, for all lusterings C 1 of G 1, C 2 of G 2, and D, D of G S, suh that supp(c 1 C 2 ) supp(d D ) =, it is the ase that Q(G 1, C 1 D) Q(G 1, C 1 D ) if and only if Q(G 2, C 2 D) Q(G 2, C 2 D ). 33 / 46

55 Loal quality funtions 34 / 46

56 Additive quality funtions Definition A qualify funtion is additive if it an e written as q(g, C) = q graph (G) + q lus () C + ( ) q node { C i supp()} i V + ( q edge aij, { C i, j supp()} ) i V j V for some funtions q graph, q lus, q node, q edge. 35 / 46

57 Outline Graph lustering Loality Case study: Modularity Soft lustering A loal NMF 36 / 46

58 First some notation For symmetri methods: â ij = h i h j. C s.t. i,j supp() For asymmetri methods: â ij = w i h j. C s.t. i,j supp() 37 / 46

59 Some NMF methods SymNMF: q SymNMF (G, C) = 1 2 (a ij â ij ) 2. i,j V Psorakis et al.: q BayNMF = ( a ij log a ) ij + â ij â ij i V j V 1 ( β wi 2 + ) β hi 2 2 V log β 2 i V C i V C( β (a 1) log β ) κ, 38 / 46

60 Proailisti NMF Optimize log likelihood, V C Q(C, G) = log P(C G). Assume latent luster struture C, A P(C V, A) = P(A C, V )P(C V )P(V )/P(V, A). Then, Q(C, G) = log P(A C, V ) + log P(C V ) + κ, where κ is onstant for fixed V, A. 39 / 46

61 Proailisti NMF (ont.) Assume edges independent, P(A C, V ) = P(a ij C, V ). i,j V Gaussian likelihood a ij N (â ij, 1) â ij = h i h j h i HN (0, σ) 40 / 46

62 Proailisti NMF (ont.) Gaussian likelihood a ij N (â ij, 1) â ij = h i h j Gives regularized symmetri NMF: q(c, G) = 1 2 h i HN (0, σ) i,j V (a ij â ij ) 2 1 2σ 2 C i V + V 2 log 2π + C V log πσ 2 /2. h 2 i 40 / 46

63 Proailisti NMF (ont.) Model of Psorakis et al. a ij Poisson(â ij ) â ij = h i w j h i HN (0, 1/ β ) w i HN (0, 1/ β ) β Gamma(a, ) Numer of lusters not speified! 41 / 46

64 A loal prior Make log P(C A, V ) e an additive quality funtion. Terms per node, per edge, and per luster. Idea: lusters depend on support S, P(C V ) = P(C V, S)P(S V ). Assume memership oeffiients are independent given support C = {C s s S} P(C s V, s) = ( P(h i ) C i s i V \s ) δ(h i, 0), 42 / 46

65 A loal prior on supports Use two aspets: 1 how many lusters over eah node (node term) 2 how many nodes are in eah luster (luster term). P(S V ) = 1 f (n i ) g( s ), Z s S i V f (n i ) = 1[n i = 1] gives hard lustering. g( s ) = ( s 1)! gives Chinese Restaurant Proess. Poisson prior on the numer of lusters per node. f (n i ) = λ n i /(n i!)e λ 43 / 46

66 Experiment setup D 1 D 2 C A ring of 10 liques. Eah irle is a lique. Possile lusterings of two liques. Eah irle is a node. 44 / 46

67 Cluster size optimal luster size Psorakis no prior Poisson prior, λ = Numer of liques 45 / 46

68 Loal quality funtions for graph lustering Twan van Laarhoven Institute for Computing and Information Sienes Radoud University Nijmegen, The Netherlands Lorentz workshop on Clustering, Games and Axioms 25th June / 46