Chapter 11. Matrix Algorithms and Graph Partitioning. M. E. J. Newman. June 10, M. E. J. Newman Chapter 11 June 10, / 43

Transcription

1 Chapter 11 Matrix Algorithms and Graph Partitioning M. E. J. Newman June 10, 2016 M. E. J. Newman Chapter 11 June 10, / 43

2 Table of Contents 1 Eigenvalue and Eigenvector Eigenvector Centrality The second largest eigenvalue and eigenvector 2 Dividing Networks into Clusters Partitioning and Community Detection 3 Graph Partitioning Why Partitioning is Hard The Kernighan-Lin Algorithm 4 Community Detection Simple Modularity Maximization Spectral Modularity Maximization Division into More than Two Groups Other Modularity Maximization Methods M. E. J. Newman Chapter 11 June 10, / 43

3 Eigenvalue and Eigenvector Eigenvector Centrality Leading Eigenvectors and Eigenvector Centrality Eigenvector Centrality Ax = κ 1 x, (1) where κ 1 is the largest eigenvalue and x is the corresponding eigenvector. the eigenvector centrality of a node i is defined to be the ith element of the leading eigenvector of the adjacency matrix. There are many ways to calculate the eigenvalue/eigenvector using computer. See for example Numerical Recipes in C. M. E. J. Newman Chapter 11 June 10, / 43

4 Eigenvalue and Eigenvector Eigenvector Centrality Leading Eigenvectors and Eigenvector Centrality Power method One of a simple and fast methods for calculating the eigenvector centrality. 1. Start with any initial vector x(0). 2. Multiply it repeatedly by the adjacency matrix A (see Sec. 7.2). x(t) = A t x(0). (2) As t x(t) will converge to the leading eigenvector of A. M. E. J. Newman Chapter 11 June 10, / 43

5 Eigenvalue and Eigenvector Eigenvector Centrality Leading Eigenvectors and Eigenvector Centrality Caveats of the Power method 1. The method will not work if the initial vector x(0) happens to be orthogonal to the leading eigenvector. Solution Choose the initial vector to have all elements positive. 2. The elements of the vector have a tendency to grow on each iteration. get multiplied approximately a factor of the leading eigenvalue which is larger than 1. Sol periodically renormalize the vector. 3. How long do we need to go on multiplying by A before the result converges to the leading eigenvalue? Sol. perform the calculation in parallel for two different initial vectors. And watch to see when they reach the same value within some prescribed tolerance. Finding the largest eigenvalue κ 1. Once the algorithm find the leading eigenvector, one more multiplication by the adjacency matrix will multiply that vector by exactly a factor κ 1. M. E. J. Newman Chapter 11 June 10, / 43

6 Eigenvalue and Eigenvector Eigenvector Centrality Calculating Other Eigenvalues and Eigenvectors Power methods calculate the largest eigenvalue of a matrix and the corresponding eigenvector. Sometimes we need other eigenvalues and eigenvectors. E.g. algebraic connectivity: the second eigenvalue of the graph Laplacian λ 2. λ 2 is non-zero iff the network is connected. Related to the bisection of the graph. Finding the smallest eigenvalue and eigenvector: Let λ 1 λ 2 λ N be the eigenvalues of matrix L and v 1, v 2,, v N be the corresponding eigenvectors. (λ N I L)v i = (λ N λ i )v i (3) Eq. (3) means that λ N λ i is an eigenvalue of λ N I L with corresponding eigenvector v i. For i = 1, λ N λ i becomes the largest eigenvalue. 1. Find λ N using power method. 2. Find λ 1 from Eq. (3) using power method again. M. E. J. Newman Chapter 11 June 10, / 43

7 Eigenvalue and Eigenvector The second largest eigenvalue and eigenvector Calculating Other Eigenvalues and Eigenvectors Finding the second largest eigenvalue and eigenvector. Use Gram-Schmidt orthogonalization. Let v 1 be the normalized eigenvector corresponding to the largest eigenvalue of a matrix A. Choose an arbitrary vector y which is orthogonal to v 1. choose any starting vector x and define y as y = x (v T 1 x)v 1 (4) { vi T y = vt i x (vt 1 x)(vt i v 1) = vi T 0 if i = 1 x (vt 1 x)δ i1 = vi T x otherwise (5) Therefore, the expansion of y in terms of the eigenvectors of A has no term in v 1 : N y = c i v i, (6) wherec i = v T i y. Use the vector y as the starting vector for repeated multiplication by A. i=2 M. E. J. Newman Chapter 11 June 10, / 43

8 Eigenvalue and Eigenvector The second largest eigenvalue and eigenvector Calculating Other Eigenvalues and Eigenvectors Validity of the Gram-Schmidt orthogonalization with power method: After multiplying y by A a total of t times, we have y(t) = A t y(0) = κ t 2 The ratio κ i /κ 2 < 1 for all i > 2. κ i /κ 2 0 as t. N i=2 ( ) t κi c i v i. (7) κ 2 Finding more eigenvectors and eigenvalues: not trivial, see, for example, Numerical Recipe in C M. E. J. Newman Chapter 11 June 10, / 43

9 Dividing Networks into Clusters Dividing Networks into Clusters Graph Partitioning and Community Detection Divide the nodes of a network into groups, clusters, or communities based on the pattern of edges in the network. Divide the nodes so that the groups formed are tightly knit with many edges inside groups and only a few edges between groups. M. E. J. Newman Chapter 11 June 10, / 43

10 Dividing Networks into Clusters Partitioning and Community Detection Partitioning and Community Detection Graph partitioning The number and size of the groups are fixed by the experimenter. Parallelization of the computation for networks. Minize the interprocessor communication as much as possible. Balance the workload by assigning (roughly) the same number of nodes to each processor. Community Detection The number and size of the groups are unspecified. The number and size of the groups are determined by the network itself. Community detection can provides clues about the nature of the interaction within the community represented. can be used as a tool for understanding the structure of a network, for shedding light on large scale patterns of connections that may not be easily visible in the raw network topology. M. E. J. Newman Chapter 11 June 10, / 43

11 Graph Partitioning Graph Partitioning What will we learn here? Kernighan-Lin algorithm (not based on the matrix methods) Spectral partitioning method (based on the spectral properties of graph Laplacian. M. E. J. Newman Chapter 11 June 10, / 43

12 Graph Partitioning Why Partitioning is Hard Why Partitioning is Hard cut size Graph bisection: Divide a network into two parts. The simplest graph partitioning. The repetition of bisection method is the commonest approach to the partitioning of networks into arbitrary numbers of parts. the number of edges running between vertices in different group minimize the cut size M. E. J. Newman Chapter 11 June 10, / 43

13 Graph Partitioning Why Partitioning is Hard Why Partitioning is Hard For Exhaustive Search Exhaustive search: bisect a network by looking through all possible divisions of the network into two parts of the required sizes and choosing one with the smallest cut size. The number of ways of dividing a network of N vertices into two groups of N 1 and N 2 is N! N 1!N 2!. Using Stirling s Formula and using the relation we obtain N! N 1!N 2! N! 2πN ( ) N N. e N 1 + N 2 = N, 2πN(N/e) N 2πN1 (N 1 /e) N1 2πN2 (N 2 /e) N2 N N+1/2 N N1+1/2 1 N N2+1/2 2 (8) M. E. J. Newman Chapter 11 June 10, / 43

14 Graph Partitioning Why Partitioning is Hard Why Partitioning is Hard For Exhaustive Search If we divide the network into two parts of equal size N/2, then N! N 1!N 2! N N+1/2 2N+1 =. (9) (N/2) N+1 N The amount of time or complexity goes up roughly exponentially with N. The goal of partitioning algorithm is to find a pretty good division not the perfect one. Heuristic algorithm. M. E. J. Newman Chapter 11 June 10, / 43

15 Graph Partitioning The Kernighan-Lin Algorithm The Kernighan-Lin (KL) Algorithm 1. We start by dividing the vertices of network into tow groups of the required sizes in any way we like. E.g.] Randomly divide the vertices into two groups. 2. For each pair (i, j), of vertices such that i is in one of the group and j belongs to the other, calculate the change of the cut size between the groups if we were to interchange i and j. 3. Among all pairs (i, j) we find the pair that reduce the cut size by the largest amount. 4. If no pair reduces the cut size, find the pair that increase it by the smallest amount. 5. Then swap that pair of vertices. 6. Repeat the process. Restriction: each vertex can only be moved once. 7. When there is no more pairs of nodes to be swapped, stop the swapping. M. E. J. Newman Chapter 11 June 10, / 43

16 Graph Partitioning The Kernighan-Lin Algorithm The Kernighan-Lin Algorithm cont d 8. When all swaps have been completed, go back through every state that the network passed through during the swapping procedure and choose among them the state in which the cut size takes its smallest value. 9. Finally, this entire process is performed repeatedly, starting each time with the best division of the network found on the last time. 10. Continue until no improvement in the cut size occurs. Slow algorithm: O(N 3 ) for a sparse network. M. E. J. Newman Chapter 11 June 10, / 43

17 Graph Partitioning Cut size Introduced by Fiedler. Use the matrix properties of the graph Laplacian. R = 1 2 i, j in different groups A ij, (10) Let us define a set of quantities s i for each node i as s i = { +1 if vertex i belongs to group 1, 1 if vertex i belongs to group 2. (11) M. E. J. Newman Chapter 11 June 10, / 43

18 Graph Partitioning Then { 1 1 if i and j are in different groups, 2 (1 s is j ) = 0 if i and j are in the same group Thus Eq. (10) becomes for all i and j. Since s i = ±1, s 2 i = 1 and A ij = ij i. (12) R = 1 A ij (1 s i s j ). (13) 4 ij k i = i k i s 2 i = ij k i δ ij s i s j (14) Eq. (10) becomes R = 1 (k i δ ij A ij ) s i s j = 1 L ij s i s j, (15) 4 4 ij where L ij = k i δ ij A ij is the ijth element of the graph Laplacian. M. E. J. Newman Chapter 11 June 10, / 43 ij

19 Graph Partitioning In the matrix form R = 1 4 st Ls. (16) Goal Here s is the vector with elements s i. Eq. (16) gives us a matrix formulation of the graph partitioning. L specifies the network structure. s defines a division of the network into groups. Find the vector s which minimize Eq. (16). The restriction that s i has the special values s i = ±1 makes the problem be hard to minimize Eq. (16). Use the relaxation method. M. E. J. Newman Chapter 11 June 10, / 43

20 Graph Partitioning Constraints I: 1 s i = ±1 the vector s always points to one of the 2 N corners of an N-dimensional hypercube. 2 The length of s is N: s = N or s 2 i = N. (17) Relax the constraint 1 on the vector s direction. s can point to any direction in its N-dimensional space. s can point to any location on the surface of a hypersphere of radius N. i M. E. J. Newman Chapter 11 June 10, / 43

21 Graph Partitioning Constraint II The numbers of s i s that are equal to +1 and 1 respectively equal the desired sizes of the two groups. Let N 1 and N 2 are the sizes of group 1 and 2. s i = N 1 N 2 (18) or in vector notation where 1 = (1, 1,, 1). i 1 T s = N 1 N 2, (19) Now the problem to solve is the minimization of Eq. (16) under the two constraints Eq. (17) and Eq. (18) M. E. J. Newman Chapter 11 June 10, / 43

22 Graph Partitioning Two constraints two Lagrange multipliers, λ and 2µ. L jk s j s k + λ N s 2 j + 2µ (N 1 N 2 ) s i jk j j s j = 0 (20) Performing the derivatives, we find that L ij s j = λs i + µ, (21) j or in matrix form Ls = λs + µ1. (22) M. E. J. Newman Chapter 11 June 10, / 43

23 Graph Partitioning Finding µ Since 1 is an eigenvector of the Laplacian with eigenvalue 0, i.e., L1 = 0, multiplying Eq. (22) on the left by 1 T and from Eq. (18), λ(n 1 N 2 ) + µn = 0 or µ = N 1 N 2 λ (23) N M. E. J. Newman Chapter 11 June 10, / 43

24 Graph Partitioning Eigenvalue Equation Let us define a new vector x as Lx = L (s + µ ) λ 1 = Ls = λs + µ1 = λx. (24) x is an eigenvector of the Laplacian with eigenvalue λ. Porperties of x 1 T x = 1 T s µ λ 1T 1 = (N 1 N 2 ) N 1 N 2 N = 0. (25) N Thus, x is orthogonal to 1. M. E. J. Newman Chapter 11 June 10, / 43

25 Graph Partitioning The remaining thing is to determine the value of λ which gives the smallest value of R. Choosing λ and the corresponding eigenvector From Eq. (24), R = 1 4 st Ls = 1 4 xt Lx = 1 4 λxt x. (26) Therefore, x T x = s T s + µ λ = N 2 N 1 N 2 N = 4 N 1N 2 N. ( s T T s ) + µ2 λ 2 1T 1 (N 1 N 2 ) + (N 1 N 2 ) 2 N N (27) R = N 1N 2 λ. (28) N M. E. J. Newman Chapter 11 June 10, / 43

26 Graph Partitioning Choosing λ and the corresponding eigenvector R λ. Choose x proportional to the eigenvalue v 2 corresponding to the second lowest eigenvalue λ 2, with its normalization fixed by Eq. (27). Finding optimal s From Eq. (24) or s = x + N 1 N 2 1. (29) N s i = x i + N 1 N 2. (30) N M. E. J. Newman Chapter 11 June 10, / 43

27 Graph Partitioning Finding Real s s i should be ±1. Exactly N 1 of them are +1 and N 2 are 1. Choose s to maximize the product s T ( x + N 1 N 2 N ) 1 = i ( s i x i + N ) 1 N 2. (31) N The maximum is achieved by assigning s i = +1 for the vertices with the largest (positive) value of x i + (N 1 N 2 )/N and s i = 1 for the remainder. the most positive value of x i + (N 1 N 2 )/N the most positive value of x i. 1 Calculate v 2, which has N elements, one for each vertex in the network. 2 Place the N 1 vertices with the most positive elements in group 1. 3 Place the rest in group 2. If N 1 N 2, then it is arbitrary which group we call group 1 and which we call group 2. Two different ways of making the split either the larger or the smaller group correspond to the more positive value. Compute R for both splits and choose the one with the smaller value. M. E. J. Newman Chapter 11 June 10, / 43

28 Graph Partitioning Final Algorithm 1 Calculate the eigenvector v 2 corresponding to the second smallest eigenvalue λ 2 of the graph Laplacian. 2 Sort the elements of the eigenvector in order from largest to smallest. 3 Put the vertices corresponding to the N 1 largest elements in group 1, the rest in group 2, and calculate R. 4 Put the vertices corresponding to the N 1 smallest elements in group 1, the rest in group 2, and calculate R 5 Between these two divisions of the network, choose the one that gives the smaller value of R. Complexity: O(Nm) or O(N 2 ). The second eigenvalue is called the algebraic connectivity, in this reason. Algebraic connectivity measure of how easily a network can be divided. M. E. J. Newman Chapter 11 June 10, / 43

29 Community Detection Community Detection Community: the naturally occuring groups in a network regardless of their size or number The basic goal is similar to that of graph partitioning. separate the network into groups of vertices that have fewer connection between them. However, the number and the size of the groups is not fixed. M. E. J. Newman Chapter 11 June 10, / 43

30 Community Detection Community Detection Start from the simplest example. Divide a network into just two non-overlapping groups without any constraint on the size of the groups, other than that the sum of the sizes should be N. First guess: Can the minimum cut size without any constraint on the sizes of groups give the community? Trivial division: all the nodes belongs in one group and none of them in the other [cut size]=0. Cut size is not a good measure for the community detection. Consider the number of edges within group, not the number of edges between groups. Find a measure that quantifies how many edges lie within groups in our network relative to the number of such edges expected on the basis of random chance. Modularity in Chap. 7. Modularity maximization is a hard problem heuristic algorithms. M. E. J. Newman Chapter 11 June 10, / 43

31 Community Detection Simple Modularity Maximization Simple Modularity Maximization One of the straightforward algorithms: similar to the KL algorithm. Divide networks into two communities. 1 Start from some initial division, for example, random division with equally sized groups. 2 Consider each vertex in the network in turn and calculate how much the modularity would change if that vertex were moved to the other group. 3 Choose a vertex whose movement would most increase, or least decrease, the modularity. 4 Move the chosen vertex. 5 Repeat the process form (2). An important constraint: a vertex once moved cannot be moved again, at least on this round of the algorithm. When all nodes have been moved exactly once, go back over the states through which the network has passed and select the one with the highest modularity. Use that state as the starting condition for another round of the same algorithm. Repeat the whole process until the modularity no longer improved. M. E. J. Newman Chapter 11 June 10, / 43

32 Community Detection Simple Modularity Maximization Simple Modularity Maximization Example: Karate Club complexity: O(N m) M. E. J. Newman Chapter 11 June 10, / 43

33 Community Detection Spectral Modularity Maximization Spectral Modularity Maximization Modularity Q = 1 2m ij ( A ij k ) ik j δ(c i, c j ) = 1 2m 2m c i is the group or community to which vertex i belongs. δ(m, n) is the Kronecker delta. Note that B ij has the property and also B ij = j j B ij δ(c i, c j ) (32) ij B ij = A ij k ik j 2m. (33) A ij k i 2m j k j = k i k i 2m = 0, (34) 2m B ij = 0. (35) i M. E. J. Newman Chapter 11 June 10, / 43

34 Community Detection Spectral Modularity Maximization Spectral Modularity Maximization Let us define a set of quantities s i for each node i as s i = { +1 if vertex i belongs to group 1, 1 if vertex i belongs to group 2. (36) Note that 1 2 (s is j + 1) = 1 if i and j belong to the same group and zero otherwise Substituting Eq. (37) into Eq. (32) and from Eq. (34): Q = 1 4m δ(c i, c j ) = 1 2 (s is j + 1). (37) ij B ij (s i s j + 1) = 1 4m B ij s i s j (38) ij M. E. J. Newman Chapter 11 June 10, / 43

35 Community Detection Spectral Modularity Maximization Spectral Modularity Maximization In matrix form Q = 1 4m st Bs. (39) B is called the modularity matrix. Eq. (39) is similar to Eq. (26). Goal: Find the value of s that maximize Eq. (39). The number of elements with value +1 or 1 is not fixed the sizes of groups are unconstrained. Use the relaxation method as before. s points to one of the corners of an n dimensional hypercube s points any direction with a fixed length: s T s = i s 2 i = N (40) M. E. J. Newman Chapter 11 June 10, / 43

36 Community Detection Spectral Modularity Maximization Spectral Modularity Maximization Using a single Lagrange multiplier β: B jk s j s k + β N s i jk j s 2 j = 0. (41) Then we obtain or in matrix form B ij s j = βs i, (42) j Bs = βs. (43) Eq. (43) implies that s is one of the eigenvector of B. M. E. J. Newman Chapter 11 June 10, / 43

37 Community Detection Spectral Modularity Maximization Spectral Modularity Maximization Substituting Eq. (43) into Eq. (39) and from Eq. (40), we obtain Q = 1 4m βst s = N β. (44) 4m Maximum modularity the largest value of β (eigenvalue). Therefore, we should choose s to be the u 1 corresponding to the largest eigenvalue of B. As before, we cannot typically choose s = u 1. - because the elements of s should be ±1. Maximize the product s T u 1 = i s i [u 1 ] i (45) M. E. J. Newman Chapter 11 June 10, / 43

38 Community Detection Spectral Modularity Maximization Spectral Modularity Maximization The maximum of Eq. (45) is achieved when each term in the sum is non-negative, i.e., when s i = { +1 if [u1 ] i > 0, 1 if [u 1 ] i < 0. (46) If [u 1 ] i = 0 then choose whichever we prefer. Since B is not a sparse matrix: complexity O(N 3 ). O(N 2 ) is possible: see [M. E. J. Newman, PNAS 103, 8577 (2006)] M. E. J. Newman Chapter 11 June 10, / 43

39 Community Detection Division into More than Two Groups Divsision into More than Two Groups Communities are natural grouping of vertices in networks. There is no reason to suppose that networks will have just two groups. The number of communities should be fixed by the structure of the network and not by the experimenter. Maximize modularity over divisions into any number of groups. There are a number of community detection algorithms. The simpler one: extension of the method of previous sections. repeated bisection of a network We start by dividing the network first into two parts. Then we further subdivide those parts in to smaller ones, and so on. caution: Unlike the graph partitioning, the modularity of the complete network does not break up into independent contributions from the separate communities and the individual maximization of the modularity of those communities treated as separate networks will not, in general, produce the maximum modularity for the network as a whole. M. E. J. Newman Chapter 11 June 10, / 43

40 Community Detection Division into More than Two Groups Divsision into More than Two Groups Consider explicitly the change Q in the modularity of the entire network upon further bisecting a community c of size N c. Q = = = = 1 1 B ij (s i s j + 1) 2m 2 i,j c i,j c 1 B ij s i s j B ij 4m i,j c i,j c 1 B ij s i s j B ij s 2 i 4m 1 4m i,j c i,j c B ij [ ] B ij δ ij B ik s i s j (47) i,j c k c M. E. J. Newman Chapter 11 June 10, / 43

41 Community Detection Division into More than Two Groups Divsision into More than Two Groups In matrix form Here B (c) is the N c N c matrix: B (c) ij Q = 1 4m st B (c) s. (48) = B ij δ ij B ik. (49) k c Eq. (48) is the same form as Eq. (39). Maximizing Q is just finding the leading eigenvector and dividing the network according to the sign of its elements. Repeat the procedure until Q > 0 our bisection algorithm will put all vertices in one of its two groups and none in the other: indivisible state! This works well but not perfect! M. E. J. Newman Chapter 11 June 10, / 43

42 Community Detection Division into More than Two Groups Divsision into More than Two Groups Ex.) Failure of the bisection method: M. E. J. Newman Chapter 11 June 10, / 43

43 Community Detection Other Modularity Maximization Methods Other Modularity Maximization Methods General minimization/maximization algorithms. See the References in the textbook for the details. Using the method of simulated annealing minus modularity energy minimization of energy very slow algorithm Genetic Algorithm Assign fitness to each division: the fitness is proportional to the modularity. Over a series of generations one simulates the preferential reproduction of high-modularity divisions, while those of low modularity die out. very slow like the simulated annealing. Greedy Algrothm Start with each vertex in the network in a one-vertex group of its own. Successively amalgamate groups in pairs, choosing at each step the pair whose amalgamation gives the biggest increase in modularity, or the smallest decrease if no choice gives an increase. Finally all vertices merges into a single large community. Go back over the state through witch the network passed during the course of the algorithm and select the one with the highest value of Q. M. E. J. Newman Chapter 11 June 10, / 43