arxiv: v1 [cs.ds] 5 Feb 2016

Transcription

1 Compressive spectral clustering Nicolas Tremblay, Gilles Puy, Rémi Gribonval, and Pierre Vandergheynst arxiv:6.8v [cs.ds] 5 Feb 6 Abstract. Spectral clustering has become a popular technique due to its high performance in many contexts. It comprises three main steps: create a similarity graph between N objects to cluster, compute the first k eigenvectors of its Laplacian matrix to define a feature vector for each object, and run k-means on these features to separate objects into k classes. Each of these three steps becomes computationally intensive for large N and/or k. We propose to speed up the last two steps based on recent results in the emerging field of graph signal processing: graph filtering of random signals, and random sampling of bandlimited graph signals. We prove that our method, with a gain in computation time that can reach several orders of magnitude, is in fact an approximation of spectral clustering, for which we are able to control the error. We test the performance of our method on artificial and real-world network data.. Introduction Spectral clustering (SC) is a fundamental tool in data mining []. Given a set of N data points {x,..., x N }, the goal is to partition this set into k weakly inter-connected clusters. Several spectral clustering algorithms exist, e.g., [ 5], but all follow the same scheme. First, compute weights W ij that model the similarity between pairs of data points (x i, x j ). This gives rise to a graph G with N nodes and adjacency matrix W = (W ij ) i,j N R N N. Second, compute the first k eigenvectors U k := (u,..., u k ) R N k of the Laplacian matrix L R N N associated to G (see Sec. for L s definition). And finally, run k-means using the rows of U k as feature vectors to partition the N data points into k clusters. This k-way scheme is a generalisation of Fiedler s pioneering work [6]. SC is mainly used in two contexts: ) if the N data points show particular structures (e.g., concentric circles) for which naive k-means clustering fails; ) if the input data is directly a graph G modeling a network [7], such as social, neuronal, or transportation networks. SC suffers nevertheless from three main computational bottlenecks for large N and/or k: the creation of the similarity matrix W; the partial eigendecomposition of the graph Laplacian matrix L; and k-means... Related work Circumventing these bottlenecks has raised a significant interest in the past decade. Several authors have proposed ideas to tackle the eigendecomposition bottleneck, e.g., via the power method [8, 9], via a careful optimisation of diagonalisation algorithms in the context of SC [], or via matrix column-subsampling such as in the Nyström method [], the nspec and cspec methods of [], or in [3, 4]. All these methods aim to quickly compute feature vectors, but k-means is still applied on N feature vectors. Other authors have proposed to circumvent k-means in high dimension by subsampling a few data points out of the N available ones, applying SC on its reduced similarity graph, and interpolating the results back on the complete dataset. One can find such methods in [5] and [] s espec proposition, where two different interpolation methods are used. Both methods are heuristic: there is no proof that these methods approach the results of SC. Also, let us mention [6] that circumvents both the eigendecomposition and the k-means bottlenecks: the authors reduce the graph s size by successive aggregation of nodes, apply SC on this small graph, and propagate the results on the complete graph using kernel k-means to control interpolation errors. The kernel is computed so that kernel k-means and SC share the same objective function [7]. Finally, we mention This work was partly funded by the European Research Council, PLEASE project (ERC-StG--7796), and by the Swiss National Science Foundation, grant -5435/ - Towards Signal Processing on Graphs. N. Tremblay, G. Puy, R. Gribonval and P. Vandergheynst are with INRIA Rennes - Bretagne Atlantique, Campus de Beaulieu, FR-354 Rennes Cedex, France. N. Tremblay and P. Vandergheynst are also with the Institute of Electrical Engineering, Ecole Polytechnique Fédérale de Lausanne (EPFL), CH-5 Lausanne, Switzerland.

2 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST works [8, 9] that concentrate on reducing the feature vectors dimension in the k-means problem, but do not sidestep the eigendecomposition nor the large N issues... Contribution: compressive clustering In this work, inspired by recent advances in the emerging field of graph signal processing [, ], we circumvent SC s last two bottlenecks and detail a fast approximate spectral clustering method for large datasets, as well as the supporting theory. We suppose that the Laplacian matrix L R N N of G is given. Our method is made of two ingredients. The first ingredient builds upon recent works [,3] that avoid the costly computation of the eigenvectors of L by filtering O(log(k)) random signals on G that will then serve as feature vectors to perform clustering. We show in this paper how to incorporate the effects of non-ideal, but computationally efficient, graph filters on the quality of the feature vectors used for clustering. The second ingredient uses a recent sampling theory of bandlimited graph-signals [4] to reduce the computational complexity of k-means. Using the fact that the indicator vectors of each cluster are approximately bandlimited on G, we prove that clustering a random subset of O(k log(k)) nodes of G using random features vectors of size O(log(k)) is sufficient to infer rapidly and accurately the cluster label of all N nodes of the graph. Note that the complexity of k-means is reduced to O(k log (k)) instead of O(Nk ) for SC. One readily sees that this method scales easily to large datasets, as will be demonstrated on artifical and real-world datasets containing up to N = 6 nodes. The proposed compressive spectral clustering method can be summarised as follows: generate a feature vector for each node by filtering O(log(k)) random Gaussian signals on G; sample O(k log(k)) nodes from the full set of nodes; cluster the reduced set of nodes; interpolate the cluster indicator vectors back to the complete graph.. Background.. Graph signal processing Let G = (V, E, W) be an undirected weighted graph with V the set of N nodes, E the set of edges, and W the weighted adjacency matrix such that W ij = W ji is the weight of the edge between nodes i and j. The graph Fourier matrix. Consider the graph s normalized Laplacian matrix L = I D / WD / where I is the identity matrix in dimension N, and D is diagonal with D ii = j i W ij the strength of node i. L is real symmetric and positive semi-definite, therefore diagonalizable in an orthogonal basis. Its spectrum is composed of its set of sorted eigenvalues = λ... λ N [5], and of the orthonormal matrix U := (u u... u N ) containing its eigenvectors. We denote by Λ R N N the diagonal matrix containing the eigenvalues of L. By analogy to the continuous Laplacian operator whose eigenfunctions are the classical Fourier modes and eigenvalues their squared frequencies, the columns of U are considered as the graph s Fourier modes, and { λ l } l as its set of associated frequencies []. Other types of graph Fourier matrices have been proposed, e.g., [6], but in order to exhibit the link between graph signal processing and SC (that partially diagonalizes the Laplacian matrix), the Laplacian-based Fourier matrix appears more natural. Graph filtering. The graph Fourier transform ˆx of a signal x defined on the nodes of the graph (called a graph signal) reads: ˆx = U x. Given a continuous filter function h defined on [, ], its associated graph filter operator H R N N is defined as H := h(l) = Uh(Λ)U, where h(λ) := diag(h(λ ), h(λ ),, h(λ N )). The signal x filtered by h is Hx. In the following, we consider ideal low-pass filters, denoted by h λc, that satisfy, for all λ [, ], () h λc (λ) =, if λ λ c, and h λc (λ) =, if not. Denote by H λc the graph filter operator associated to h λc.

3 Compressive spectral clustering 3 Algorithm Spectral Clustering [] Input: The Laplacian matrix L, the number of clusters k Compute U k R N k, L s first k eigenvectors: U k = (u u u k ). Form the matrix Y k R N k from U k by normalizing each of U k s rows to unit length: (Y k ) ij = (U k ) ij / k j= U ij. 3 Treat each node i as a point in R k by defining its feature vector f i R k as the transposed i-th row of Y k : f i := Y k δ i, where δ i (j) = if j = i and otherwise. 4 To obtain k clusters, run k-means with the Euclidean distance: () D ij := f i f j Fast graph filtering. In order to filter a signal by h without diagonalizing L, one may approximate h by a polynomial h of order p satisfying p h(λ) := α l λ l h(λ) for all λ [, ], where α,..., α p R. In matrix form, we have p H := h(l) = α l L l H. l= l= Let us highlight that we never compute the potentially dense matrix H in practice. Indeed, we will only be interested in the result of the filtering operation, i.e., Hx = p l= α ll l x Hx for x R N, that can be obtained with only p successive matrix-vector multiplications with L. The computational complexity of filtering a signal is thus O(p#E), where #E is the number of edges of G... Spectral clustering We choose here Ng et al. s method [] based on the normalized Laplacian as our standard SC method. The input is the adjacency matrix W representing the pairwise similarity of all the N objects to cluster. After computing its Laplacian L, follow Alg. to find k classes. 3. Principles of CSC Compressive spectral clustering (CSC) circumvents two of SC s bottlenecks, the partial diagonalisation of the Laplacian and the high-dimensional k-means, thanks to the following ideas. ) Perform a controlled estimation D ij of the spectral clustering distance D ij (see Eq ()), without partially diagonalizing the Laplacian, by fast filtering a few random signals with the polynomial approximation h λk of the ideal low pass filter h λk (see Eq. ()). A theorem recently published independently by two teams [, 3] shows that this is possible when there is no normalisation step (step in Alg. ) and when the order p of the polynomial approximation tends to infinity, i.e., when h λk = h λk. In Sec. 3., we provide a first extension of this theorem that takes into account normalisation. A complete extension that also takes into account the errors due to a finite-order polynomial approximation will be presented in Sec. 4.. ) Run k-means on n randomly selected feature vectors out of the N available ones - thus clustering the corresponding n nodes into k groups - and interpolate the result back on the full graph. To guarantee robust reconstruction, we take advantage of recent results of [4] on random sampling of In network analysis, the raw data is directly W. In the case where one starts with a set of data points (x,..., x N ), the first step consists in deriving W from the pairwise similarities s(x i, x j ). See [7] for several choices of similarity measure s and several ways to create W from the s(x i, x j ).

4 4 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST k-bandlimited graph signals. In Sec. 3., we explain why these results are applicable to clustering and show that it is sufficient to sample O(k log k) features only! Note that to cluster a dataset into k groups, one needs at least k samples. The above bound is thus optimal up to the extra log k factor. 3.. Ideal filtering of random signals Definition 3. (Local cumulative coherence). Given a graph G, the local cumulative coherence of order k at node i is v k (i) := U k δ k i = j= U ij. Let us define the diagonal matrix: V k (i, i) = /v k (i). Note that we assume that v k (i) >. Indeed, in the pathologic cases where v k (i) = for some nodes i, Step of the standard SC algorithm cannot be run either. Now, consider the matrix R = (r r r d ) R N d consisting of d random signals r i, whose components are independent Bernouilli, Gaussian, or sparse (as in Theorem. of [8]) random variables. To fix ideas in the following, we consider the components as independent random Gaussian variables of mean zero and variance /d. Consider the coherence-normalized filtered version of R, V k H λk R R N d, and define node i s new feature vector f i R d as the transposed i-th line of this filtered matrix: f i := (V k H λk R) δ i. The following theorem shows that, for large enough d, D ij := f i f j = (Vk H λk R) (δ i δ j ) is a good estimation of D ij with high probability. Theorem 3.. Let ɛ ], ] and β > be given. If d is larger than then with probability at least N β, we have for all (i, j) {,..., N}. 4 + β ɛ / ɛ 3 log N, /3 ( ɛ)d ij D ij ( + ɛ)d ij. Proof. The proof is an instance of the Johnson-Lindenstrauss lemma. Note that H λk = U k U k, and that Y k = V k U k. We rewrite f i f j in a form that will let us apply the Johnson-Lindenstrauss lemma of norm conservation: (3) f i f j = R H λ k V k (δ i δ j ) = R U k U k V k (δ i δ j ) = R U k (f i f j ) where the f i are the standard SC feature vectors. Applying Theorem. of [8] (an instance of the Johnson-Lindenstrauss lemma) to R U k (f i f j ), the following holds. If d is larger than: (4) 4 + β ɛ / ɛ 3 log N, /3 then with probability at least N β, we have, (i, j) {,..., N} : ( ɛ) U k (f i f j ) D ij ( + ɛ) U k (f i f j ). As the columns of U k are orthonormal, we end the proof: (i, j) [, N] U k (f i f j ) = f i f j = D ij. In Sec. 4., we generalize this result to the real-world case where the low-pass filter is approximated by a finite order polynomial; we also prove that, as announced in the introduction, one only needs d = O(log k) features when using the downsampling scheme that we now detail. Throughout this paper,. stands for the usual l -norm.

5 Compressive spectral clustering Downsampling and interpolation For j =,..., k, let us denote by c j R N the ground-truth indicator vector of cluster C j, i.e., { if i Cj, (c j ) i := i {,..., N}. otherwise, To estimate c j, one could run k-means on the N feature vectors { f,..., f N }, as done in [, 3]. Yet, this is still inefficient for large N. To reduce the computational cost further, we propose to run k-means on a small subset of n feature vectors only. The goal is then to infer the labels of all N nodes from the labels of the n sampled nodes. To this end, we need ) a low-dimensional model that captures the regularity of the vectors c j, ) to make sure that enough information is preserved after sampling to be able to recover the vectors c j, and 3) an algorithm that rapidly and accurately estimates the vectors c j by exploiting their regularity The low-dimensional model For a simple regular (with nodes of same degree) graph of k disconnected clusters, it is easy to check that {c,..., c k } form a set of orthogonal eigenvectors of L with eigenvalue. All indicator vectors c j therefore live in span(u k ). For general graphs, we assume that the indicator vectors c j live close to span(u k ), i.e., the difference between any c j and its orthogonal projection onto span(u k ) is small. Experiments in Section 5 will confirm that it is a good enough model to recover the cluster indicator vectors. In graph signal processing words, one can say that c j is approximately k-bandlimited, i.e., its k first graph Fourier coefficients bear most of its energy. There has been recently a surge of interest around adapting classical sampling theorems to such bandlimited graph signals [9 3]. We rely here on the random sampling strategy proposed in [4] to select a subset of n nodes Sampling and interpolation The subset of feature vectors is selected by drawing n indices Ω := {ω,..., ω n } uniformly at random from {,..., N} without replacement. Running k-means on the subset of features { f ω,..., f ωn } thus yields a clustering of the n sampled nodes into k clusters. We denote by c r j Rn the resulting low-dimensional indicator vectors. Our goal is now to recover c j from c r j. Consider that k-means is able to correctly identify c,..., c k R N using the original set of features {f,..., f N } with the SC algorithm (otherwise, CSC is doomed to fail from the start). Results in [,3] show that k-means is also able to identify the clusters using the feature vectors { f,..., f N }. This is explained theoretically by the fact that the distance between all pairs of feature vectors is preserved (see Theorem 3.). Then, as choosing a subset { f ω,..., f ωn } of { f,..., f N } does not change the distance between the feature vectors, we can hope that k-means correctly clusters the n sampled nodes, provided that each cluster is sufficiently sampled. Experiments in Sec. 5 will confirm this intuition. In this ideal situation, we have (5) c r j = M c j, where M R n N is the sampling matrix satisfying: { if j = ωi, (6) M ij := otherwise. To recover c j from its n observations c r j, Puy et al. [4] show that the solution to the optimisation problem (7) min Mx c r j + γ x R N x g(l)x, is a faithful 3 estimation of c j, provided that c j is close to span(u k ) and that M satisfies the restricted isometry property (discussed in the next subsection). In (7), γ > is a regularisation parameter 3 precise error bounds are provided in [4].

6 6 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST and g a positive non-decreasing polynomial function (see Section. for the definition of g(l)). This reconstruction scheme is proved to be robust to: ) observation noise, i.e., to imperfect clustering of the n nodes in our context; ) model errors, i.e., the indicator vectors do not need to be exactly in span(u k ) for the method to work. Also, the performance is shown to depend on the ratio g(λ k )/g(λ k+ ). The smaller it is, the better the reconstruction. To decrease this ratio, we decide to approximate the ideal high-pass filter g λk (λ) = h λk (λ) for the reconstruction. Remark that this filter favors the recovery of signals living in span(u k ). The approximation g λk of g λk is obtained using a polynomial (as in Sec..), which permits us to find fast algorithms to solve (7) How many features to sample? We terminate this section by providing the theoretical number of features n one needs to sample in order to make sure that the indicator vectors can be faithfully recovered. This number is driven by the following quantity. Definition 3.3 (Global cumulative coherence). The global cumulative coherence of order k of the graph G is ν k := N max i N {v k (i)}. It is shown in [4] that ν k [k /, N / ]. Theorem 3.4 ( [4]). Let M be a random sampling matrix constructed as in (6). For any δ, ɛ ], [, (8) ( δ) x N n M x ( + δ) x for all x span(u k ) with probability at least ɛ provided that n 6 ( ) k δ ν k log. ɛ The above theorem presents a sufficient condition on n ensuring that M satisfies the restricted isometry property (8). This condition is required to ensure that the solution of (7) is an accurate estimation of c j. The above theorem thus indicates that sampling O(νk log k) features is sufficient to recover the cluster indicator vectors. For a simple regular graph G made of k disconnected clusters, we have seen that U k = (c,..., c k ) up to a normalisation of the vectors. Therefore, ν k = N / / min i {N / i }, where N i is the size of the i th cluster. If the clusters have the same size N i = N/k then ν k = k /, the lower bound on ν k. In this simple optimal scenario, sampling O(νk log k) = O(k log k) features is thus sufficient to recover the cluster indicator vectors. The attentive reader will have noticed that for graphs where νk N, no downsampling is possible. Yet, a simple solution exists in this situation: variable density sampling. Indeed, it is proved in [4] that, whatever the graph G, there always exists an optimal sampling distribution such that n = O(k log k) samples are sufficient to satisfy Eq. (8). This distribution depends on the profile of the local cumulative coherence and can be estimated rapidly (see [4] for more details). In this paper, we only consider uniform sampling to simplify the explanations, but keep in mind that in practice results will always be improved if one uses variable density sampling. Note also that one cannot expect to sample less than k nodes to find k clusters. Up to the extra log(k), our result is optimal. 4. CSC in practice We have detailed the two fundamental theoretical notions supporting our algorithm, presented in Alg. and discussed in Sec. 4.. However, some steps in Alg. still need to be clarified. In particular, Sec. 4. provides an extension of Theorem 3. that takes into account the use of a non-ideal low-pass filter (to handle the practical case where the order of the polynomial approximation is finite). This theorem in fine explains and justifies Step 4 of Alg.. Then, in Sec. 4.3, we discuss important details of the method such as the estimation of λ k (Step ) and the choice of the polynomial approximation (Step ). We finish this section with complexity considerations.

7 Compressive spectral clustering 7 Algorithm Compressive Spectral Clustering Input: The Laplacian matrix L, the number of clusters k; and parameters typically set to n = k log k, d = 4 log n, p = 5 and γ = 3. Estimate L s k-th eigenvalue λ k as in Sec Compute the polynomial approximation h λk of order p of the ideal low-pass filter h λk. 3 Generate d random Gaussian signals of mean and variance /d: R = (r r r d ) R N d. 4 Filter R with H λk = h λk (L) as in Sec.. and define, for each node i, its feature vector f i R d : ) ] / ). f i = [( Hλk R δi ( Hλk R δi 5 Generate a random sampling matrix M R n N as in Eq. (6) and keep only n feature vectors: ( f ω... f ωn ) = M( f... f N ). 6 Run k-means on the reduced dataset with the Euclidean distance: Dr ij = f ωi f ωj to obtain k reduced indicator vectors c r j R n, one for each cluster. 7 Interpolate each reduced indicator vector c r j with the optimisation problem of Eq. (7), to obtain the k indicator vectors c j R N on the full set of nodes. 4.. Algorithm As for SC (see Sec..), the algorithm starts with the adjacency matrix W of a graph G. After computing its Laplacian L, the CSC algorithm is summarized in Alg.. The output c j (i) is not binary and in fact quantifies how much node i belongs to cluster j, useful for fuzzy partitioning. To obtain an exact partition of the nodes, we normalize each indicator vector c j, and assign node i to the cluster j for which c j (i)/ c j is maximal. This is the procedure that we follow in the experiments of Sec Non-ideal filtering of random signals In this section, we improve Theorem 3. by studying how the error of the polynomial approximation h λk of h λk propagates to the spectral distance estimation, and by taking into account the fact that k-means is performed on the reduced set of features ( f ω... f ωn ) = M( f... f N ). We denote by MY k R n k the ideal reduced feature matrix. We have (f ω f ωn ) = M(f f N ) = MY k. The actual distances we want to estimate using random signals are thus, for all (i, j) {,..., n} D r ij := fωi f ωj = Y k M (δ r i δ r j ), where the {δi r } are here Diracs in n dimensions. Consider the random matrix R = (r r r d ) R N d constructed as in Sec. 3.. Its filtered, normalized and reduced version is MV k Hλk R R n d. The new feature vector f ωi R d associated to node ω i is thus f ωi = (MV k Hλk R) δi r. The normalisation of Step 4 of Alg. approximates fastly the action of V k in the above equation. More details and justifications are provided in the Important remark at the end of this section. The Euclidean distance between any two features reads as D ij r := f ωi f R ωj = H λ k V k M (δi r δj r ). We now study how well Dr ij estimates D r ij. Approximation error. Denote e(λ) the approximation error of the ideal low-pass filter: λ [, ], In the form of graph filter operators, one has e(λ) := h λk (λ) h λk (λ). h λk (L) = H λk = H λk + E = h λk (L) + e(l).

8 8 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST We model the error e using two parameters: e (resp. e ) the maximal error for λ λ k (resp. λ > λ k ). We have e := sup λ {λ,...,λ k } e(λ), e := sup λ {λ k+,...,λ N } e(λ). The resolution parameter. In some cases, the ideal reduced spectral distance Dij r may be null. In such cases, approximating Dij r = using a non-ideal filter is not possible. In fact, non-ideal filtering introduces an irreducible error on the estimation of the feature vectors that is not possible to compensate in general. We thus introduce a resolution parameter Dmin r below which the distances Dij r do not need to be approximated exactly, but should remain below Dr min (up to a tolerated error). Theorem 4. (General norm conservation theorem). Let Dmin r ], ] be a chosen resolution parameter. For any δ ], ], β >, if d is larger than then, for all (i, j) {,..., n}, and 6( + β) δ δ 3 log n, /3 ( δ)d r ij D r ij ( + δ)d r ij, if D r ij D r min, D r ij < ( + δ)d r min, if D r ij < D r min, with probability at least n β provided that (9) e e + e Dmin r min i{v k (i)} δ + δ. Proof. The proof follows the general ideas of Theorem 3. s proof, with a more careful use of the Johnson-Lindenstrauss lemma that takes into account error propagation. Recall that: D ij r := f ωi f R ωj = H λ k V k M δij r, where δij r = δr i δr j. Given that H λk = H λk + E and using the triangle inequality in the definition of D ij r, we obtain R H λ k V k M δij r R E V k M δij r Dr ij R E V k M δij r + R H λ k V k M δij r, We continue the proof by bounding R H λ k V k M δij r and R E V k M δij r separately. Let δ ], ]. To bound R H λ k V k M δij r, we set ɛ = δ/ in Theorem 3.. This proves that if d is larger than 6( + β) d = δ δ 3 log n, /3 then with probability at least n β, ( δ ) Dij r ( R H λ k V k M δij r + δ ) D r ij, for all (i, j) {,..., n}. To bound R E V k M δij r, we use Theorem. in [8]. This theorem proves that if d > d, then with probability at least n β, ( R E V k M δij r + δ ) E V k M δij r, for all (i, j) {,..., n}. Using the union bound and (), we deduce that, with probability at least n β, ( δ ) ( Dij r + δ ) ( E V k M δij r Dr ij + δ ) ( E V k M δij r + + δ ) D r ij, for all (i, j) {,..., n} provided that d > d.

9 Compressive spectral clustering 9 Then, as e is bounded by e on the first k eigenvalues of the spectrum and by e on the remaining ones, we have E V k M δij r = Ue(Λ)U V k M δij r = e(λ)u V k M δij r = N l= e e(λ l ) (MVk u l ) δij r k (MVk u l ) δij r + e l= N l=k+ (MVk u l ) δij r = e U ( k V k M δij r + e U V k M δij r U k V k M δij r ) = (e e ) U k V k M δij r + e U V k M δij r = (e e ) (Dij) r + e V (e e ) (D r ij) + The last step follows from the fact that N V = k M δ r ij l= Define, for all (i, j) {,..., n} : v k (l) (M δ r ij)(l) = e ij := k M δ r ij e min i {v k (i) }. v k (ω i ) + v k (ω j ) min i {v k (i)} e e e Dr ij + min i {v k (i)}. Thus, the above inequality may be rewritten as: E V k M δij r eij, for all (i, j) {,..., n}, which combined with () yields ( δ ) ( Dij r + δ ) e ij D ij r ( + δ ) ( e ij + + δ ) D r ij, for all (i, j) {,..., n}, with probability at least n β provided that d > d. Let us now separate two cases. In the case where Dij r Dr min >, we have ( e ij = e ) ij Dij r Dij r = e e + e Dij r min Dij r i{v k (i)} ( ) e e + e Dmin r min Dij r i{v k (i)} δ + δ Dr ij. provided that Eq. (9) holds. Combining the last inequality with () proves the first part of the theorem. In the case where Dij r < Dr min, we have e ij < e e e Dr min + min i {v k (i)} δ + δ Dr min. provided that Eq. (9) holds. Combining the last inequality with () terminates the proof. Consequence of Theorem 4.. All distances smaller (resp. larger) than the chosen resolution parameter Dmin r are estimated smaller than ( + δ)dr min (resp. correctly estimated up to a relative error δ). Moreover, for a fixed distance estimation error δ, the lower we decide to fix Dmin r, the lower

10 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST should also be the errors e and/or e to ensure that Eq. (9) still holds, which implies an increase of the order p of the polynomial approximation of the ideal filter h λk, and ultimately, that means a higher computation time for the filtering operation of the random signals. Important remark. The feature matrix V k H λk R can be easily computed if one knows the cut-off value λ k and the local cumulative coherences v k (i). Unfortunately, this is not the case in practice. We propose a solution to estimate λ k in Sec To estimate v k (i), one can use the results in Sec. 4 of [4] showing that v k (i) = U k δ i (H λk R) δ i. Thus, a practical way to estimate V k Hλk R is to first compute H λk R and then normalize its rows to unit length, as done in Step 4 of Alg Polynomial approximation and estimation of λ k The polynomial approximation. Theorem 4. uses a separate control on e(λ) below λ k (with e ) and above λ k (with e ). To have such a control in practice, one would need to use rational filters (ratio of two polynomials) to approximate h λk. Such filters have been introduced in the graph context [33], but they involve another optimisation step that would burden our main message. We prefer to simplify our analysis by using polynomials for which only the maximal error can be controlled. We write () e m := max(e, e ) = sup e(λ). λ {λ,...,λ N } In this easier case, one can show that Theorem 4. is still valid if Eq. (9) is replaced by em () Dmin r min i{v k (i)} δ + δ. In our experiments, we could follow [34] and use truncated Chebychev polynomials to approximate the ideal filter, as these polynomials are known to require a small degree to ensure a given tolerated maximal error e m. We prefer to follow [35] who suggest to use Jackson-Chebychev polynomials: Chebychev polynomials to which are added damping multipliers to alleviate the unwanted Gibbs oscillations around the cut-off frequency λ k. The polynomial s order p. For a fixed δ, Dmin r, and min i{v k (i)}, one should use the Jackson- Chebychev polynomial of smallest order p ensuring that e m satisfies Eq. (), in order to optimize the computation time while making sure that Theorem 4. applies. Studying p theoretically without computing the Laplacian s complete spectrum (see Eq. ()) is beyond the scope of this paper. Experimentally, p = 5 yields good results (see Fig. c). Estimation of λ k. The fast filtering step is based on the polynomial approximation of h λk, which is itself parametrized by λ k. Unless we compute the first k eigenvectors of L, thereby partly loosing our efficiency edge on other methods, we cannot know the value of λ k with infinite precision. To estimate it efficiently, we use Alg. in [4], where an estimation of λ k is obtained as a by-product. This estimation uses the fact that Tr(R H λ k H λk R) = Tr(R U k U kr) k with high probability. Lemma 5.3 of [36] shows that this eigencount estimator yields the exact result (i.e., k) with probability δ provided that d 4k log(/δ). In practice, experiments in [35] show that fewer random signals are sufficient in some cases. Inspired by these results and as in [4], we choose log N random signals for this estimation task. Starting from λ = and using the above estimator, the algorithm proceeds by dichotomy on λ until the interval [, λ] contains k eigenvalues. For computational efficiency, we perform this eigencount task with Jackson-Chebychev polynomial approximation of the ideal low-pass filters Complexity considerations The complexity of steps, 3 and 5 of Alg. are not detailed as they are insignificant compared to the others. First, note that fast filtering a graph signal costs O(p #E). 4 Therefore, Step costs O(p #E log N) per iteration of the dichotomy, and Step 4 costs O(p #E log n) (as d = O(log n)). Step 7 requires to solve Eq. (7) with the polynomial approximation of g λk (λ) = h λk (λ). When solved, e.g., by conjugate gradient or gradient descent, this step costs a fast filtering operation per 4 Recall that p is the order of the polynomial filter.

11 Compressive spectral clustering iteration of the solver and for each of the k classes. Step 7 thus costs O(p #Ek). Also, the complexity of k-means to cluster Q feature vectors of dimension r into k classes is O(kQr) per iteration. Therefore, Step 6 with Q = n and r = d = O(log(n)) costs O(kn log n). CSC s complexity is thus O (kn log n + p #E (log N + log n + k)). In practice, we are interested in sparse graphs: #E = O(N). Using the fact that n = O(k log k), CSC s complexity simplifies to O ( k log k + pn (log N + k) ). SC s k-means step has a complexity of O(Nk ) per iteration. In many cases 5 this sole task is more expensive than the CSC algorithm. On top of this, SC has the additional complexity of computing the first k eigenvectors of L, for which the cost of ARPACK - a popular eigenvalue solver - is at least O(k 3 ) (see details in Sec. 3. of [37]). In general, the complexity is much larger with additional terms growing linearly with N. This study suggests that CSC is faster than SC for large N and/or k. The above algorithms number of iterations are not taken into account as they are difficult to predict theoretically. Yet, the following experiments confirm the superiority of CSC over SC in terms of computational time. 5. Experiments We first perform well-controlled experiments on the Stochastic Block Model (SBM), a model of random graphs with community structures, that was showed suitable as a benchmark for SC in [38]. We also show performance results on a large real-world network. Implementation was done in Matlab R5a, using the built-in function kmeans with replicates, and the function eigs for SC. Experiments were done on a laptop with a.6 GHz Intel i7 dual-core processor running OS Fedora release with 6 GB of RAM. The fast filtering part of CSC uses the gsp cheby op function of the GSP toolbox [39]. Equation (7) is solved using Matlab s gmres function. All our results may be reproduced with the toolbox provided here [4]. 5.. The Stochastic Block Model What distinguishes the SBM from Erdos-Renyi graphs is that the probability of connection between two nodes i and j is not uniform, but depends on the community label of i and j. More precisely, the probability of connection between nodes i and j equals q if they are in the same community, and q if not. In a first approach, we look at graphs with k communities, all of same size N/k. Furthermore, instead of considering the probabilities, one may fully characterize a SBM by providing their ratio ɛ = q q, as well as the average degree s of the graph. The larger ɛ, the more difficult the community structure s detection. In fact, Decelle et al. [4] show that a critical value ɛ c exists above which community detection is impossible at the large N limit: ɛ c = (s s)/(s + s(k )). 5.. Performance results In Figs. a-d), we compare the recovery performance of CSC versus SC for different parameters. The performance is measured by the Adjusted Rand similarity index [4] between the SBM s ground truth and the obtained partitions. It varies between and. The higher it is, the better is the reconstruction. These figures show that the performance of CSC saturates at the default values of n, d, p and γ (see top of Alg. ). We also perform experiments on a SBM with N = 3, k =, s = 6 and hetereogeneous community sizes. More specifically, the list of community sizes is chosen to be: 5,, 5,, 5, 3, 35, 4, 45, 5, 5, 55, 6, 65, 7, 75, 8, 85, 9 and 95 nodes. In this scenario, there is no theoretical value of ɛ over which it is proven that recovery is impossible in the large N limit. Instead, we vary ɛ between and. and show the recovery performance results with respect to n, d, p and γ in Fig.. Results are similar to the homogeneous case presented in Figs. a-d). We show in Fig. e) the estimation results of λ k for different values of ɛ. We see that it is overestimated in the SBM context. As long as the estimated value stays under λ k+, this overestimation 5 Roughly, all cases for which k > p(log N + k).

12 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST.5 SC n = k log(k) n = k log(k) n = 3 k log(k) n = 4 k log(k).5. ǫ c.5 a) ǫ ǫ c.5 e) ǫ λ k λ k+ λ k est SC.5 d = log(n) d = 3 log(n) d = 4 log(n) d = 5 log(n).5. ǫ c.5 b) ǫ f) # of classes k N= 4, CSC N= 4, PM N= 4, SC N= 5, CSC N= 5, PM N= 5, SC N= 6, CSC N= 6, PM N= 6, SC.5 SC p = p = p = 5 p =.5. ǫ c.5 c) ǫ Computation time (s) g) # of classes k SC.5 γ = -5 γ = -3 γ = - γ =.5. ǫ c.5 d) ǫ h) SC CSC k=5 7h7m,.84 hm,.83 k=5 5h9m,.84 3h34m,.84 7h36m (eigs) k= + at least h h8m,.84 for k-means, unknown Figure. (a-d): recovery performance of CSC on a SBM with N = 3, k =, s = 6 versus ɛ, for different n, d, p, γ. Default is n = k log k, d = 4 log n, p = 5 and γ = 3. All results are averaged over graph realisations. e) Estimation of λ k ( λ k est ) and the true values of λ k and λ k+, versus ɛ on the same SBM. f-g) Performance and time of computation on a SBM with ɛ = ɛ c/4 and different values of N and k; for CSC, PM (Power Method) and SC. For N = 6 and k =, we stopped SC (and PM) after h of computation. Figures f and g are averaged over 3 graph realisations. N.B.: Fig. f is zoomed around high values of the recovery score, and Fig. g is plotted in log-log. h) Time of computation (in hours) and modularity (in bold) of the obtained partitions for SC and CSC on the Amazon graph. For k =, SC s eigendecomposition converges in 7h, and we stopped k-means after hours of computation..5 SC n = k log(k) n = k log(k) n = 3 k log(k) n = 4 k log(k).. a) ǫ.5 SC d = log(n) d = 3 log(n) d = 4 log(n) d = 5 log(n).. b) ǫ SC p = p = p = 5.5 p =.. c) ǫ SC.5 γ = -5 γ = -3 γ = - γ =.. d) ǫ Figure. (a-d): recovery performance of CSC on a SBM with N = 3, k =, s = 6 and hetereogeneous community sizes versus ɛ, for different n, d, p, γ. Default is n = k log k, d = 4 log n, p = 5 and γ = 3. All results are averaged over graph realisations. does not have a strong impact on the method. On the other hand, as ɛ becomes larger than.6, our estimation of λ k is larger than λ k+, which means that our feature vectors start to integrate some unwanted information from eigenvectors outside of span(u k ). Even though the impact of this additional information is application-dependent and in some cases insignificant, further efforts to improve the estimation of λ k would be beneficial to our proposition. In Figs. f-g) we fix ɛ to ɛ c /4, n, d, p and γ to the values given in Alg., and vary N and k. We compare the recovery performance and the time of computation of CSC, SC and Boutsidis power method [8]. The power method (PM), in a nutshell, ) applies the Laplacian matrix to the power r to k random signals, ) computes the left singular vectors of the N k obtained matrix, to extract feature vectors, 3) applies k-means in high-dimension (like SC) with these feature vectors. In our experiments, we use r =. The recovery performances are nearly identical in all situations, even though CSC is only a few percents under SC and PM (Fig. f is zoomed around the high values of the recovery score). For the time of computation, the experiments confirm that all three methods are roughly linear in N and polynomial in k (Fig. g is plotted in log-log), with a lower exponent for CSC than for SC and PM; such that SC and PM are faster for k = but CSC becomes up to an order of magnitude faster as k increases to. Note that the SBM is favorable to SC as Matlab s function eigs converges very fast in this case, e.g., for N = 5, it finds the first k = eigenvectors in less

13 Compressive spectral clustering 3 than minutes! PM sidesteps successfully the cost of eigs, but the cost of k-means in high-dimension is still a strong bottleneck. We finally compare CSC and SC on a real-world dataset: the Amazon co-purchasing network [43]. It is an undirected connected graph comprising N = nodes and #E = edges. The results are presented in Fig. h) for three values of k. As there is no clear ground truth in this case, we use the modularity [44] to measure the algorithm s clustering performance, a well-known cost function that measures how well a given partition separates a network in different communities. Note that the replicates of k-means would not converge for SC with the default maximum number of iterations set to. For a fair comparison with CSC, we used only replicates with a maximum number of iterations set to for SC s k-means step. We see that for the same clustering performance, CSC is much faster than SC, especially as k increases. The PM algorithm on this dataset does not perform well: even though the features are estimated quickly, they apparently do not form clear classes such that its k-means step takes even longer than SC s. For the three values of k, we stopped the PM algorithm after a time of computation exceeding SC s. 6. Conclusion By graph filtering O(log k) random signals with polynomial filters approximating the low-pass ideal filter, we are able to construct feature vectors whose interdistances approach the standard SC feature distances. Then, building upon compressive sensing results, we show that one can sample O(k log k) nodes from the set of N nodes, cluster this reduced set of nodes and interpolate the result back to the whole graph. If the low-dimensional k-means result is correct, i.e., if Eq. (5) is verified, we guarantee that the interpolation is a good approximation of the SC result. To improve the clustering result of the reduced set of nodes, one could build upon the concept of community cores [45]. In fact, as the filtering and the low-dimensional clustering steps are fairly cheap to compute, one could repeat these steps for different random signals, keep the sets of nodes that are always classified together and use only these stable cores for interpolation. Also, note that the SC algorithm based on the combinatorial Laplacian L = D W can also be successfully approximated by CSC. Using this Laplacian, the normalisation step of SC s algorithm (Step of Alg. ) is no longer required [7]. To adapt CSC to this version of SC, one only needs to skip the normalisation step of CSC: simply change step 4 of Alg. to f ) i = ( Hλk R δi. In fact, the Fourier matrix (the eigenvector matrix of L) is still orthogonal in this case and all proofs stay valid. On the other hand, for the version of SC based on the random walk Laplacian L = I D W [3], the Fourier matrix is no longer orthogonal, its inverse is no longer equal to its transpose, and the sampling theorems of [4] need yet to be extended to this case. Nevertheless, even without such potential improvements, our experiments show that CSC proves efficient and accurate in synthetic and real-world datasets; and could be preferred to SC for large N and/or k.. References [] M. Nascimento and A. de Carvalho, Spectral methods for graph clustering a survey, European Journal of Operational Research, vol., no., pp. 3,. [] A. Ng, M. Jordan, and Y. Weiss, On spectral clustering: Analysis and an algorithm, in Advances in Neural Information Processing Systems 4, T. Dietterich, S. Becker, and Z. Ghahramani, Eds. MIT Press,, pp [3] J. Shi and J. Malik, Normalized cuts and image segmentation, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol., no. 8, pp ,. [4] M. Belkin and P. Niyogi, Laplacian eigenmaps for dimensionality reduction and data representation, Neural computation, vol. 5, no. 6, pp , 3. [5] L. Zelnik-Manor and P. Perona, Self-tuning spectral clustering, in Advances in neural information processing systems, 4, pp [6] M. Fiedler, Algebraic connectivity of graphs, Czechoslovak mathematical journal, vol. 3, no., pp , 973. [7] S. White and P. Smyth, A spectral clustering approach to finding communities in graph. in SDM, vol. 5. SIAM, 5, pp

14 4 N. TREMBLAY, G. PUY, R. GRIBONVAL, AND P. VANDERGHEYNST [8] K. P. Boutsidis, C. and A. Gittens, Spectral clustering via the power method - provably. in Proceedings of the 3nd International Conference on Machine Learning (ICML-5), Lille, France, 5, pp [9] F. Lin and W. W. Cohen, Power iteration clustering. in Proceedings of the 7th International Conference on Machine Learning (ICML-), Haifa, Israel,, pp [] T.-Y. Liu, H.-Y. Yang, X. Zheng, T. Qin, and W.-Y. Ma, Fast large-scale spectral clustering by sequential shrinkage optimization, in Advances in Information Retrieval, 7, pp [] C. Fowlkes, S. Belongie, F. Chung, and J. Malik, Spectral grouping using the nystrom method, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 6, no., pp. 4 5, 4. [] L. Wang, C. Leckie, K. Ramamohanarao, and J. Bezdek, Approximate spectral clustering, in Advances in Knowledge Discovery and Data Mining, 9, pp [3] X. Chen and D. Cai, Large scale spectral clustering with landmark-based representation. in Proceedings of the 5th AAAI Conference on Artificial Intelligence,. [4] T. Sakai and A. Imiya, Fast spectral clustering with random projection and sampling, in Machine Learning and Data Mining in Pattern Recognition, 9, pp [5] D. Yan, L. Huang, and M. Jordan, Fast approximate spectral clustering, in Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ser. KDD 9, New York, NY, USA, 9, pp [6] I. Dhillon, Y. Guan, and B. Kulis, Weighted graph cuts without eigenvectors a multilevel approach, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 9, no., pp , 7. [7] M. Filippone, F. Camastra, F. Masulli, and S. Rovetta, A survey of kernel and spectral methods for clustering, Pattern Recognition, vol. 4, no., pp. 76 9, 8. [8] C. Boutsidis, A. Zouzias, M. W. Mahoney, and P. Drineas, Stochastic dimensionality reduction for k-means clustering, arxiv, abs/.897,. [9] M. B. Cohen, S. Elder, C. Musco, C. Musco, and M. Persu, Dimensionality reduction for k-means clustering and low rank approximation, in Proceedings of the 47th Annual ACM on Symposium on Theory of Computing. ACM, 5, pp [] D. Shuman, S. Narang, P. Frossard, A. Ortega, and P. Vandergheynst, The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains, Signal Processing Magazine, IEEE, vol. 3, no. 3, pp , 3. [] A. Sandryhaila and J. Moura, Big data analysis with signal processing on graphs: Representation and processing of massive data sets with irregular structure, Signal Processing Magazine, IEEE, vol. 3, no. 5, pp. 8 9, 4. [] N. Tremblay, G. Puy, P. Borgnat, R. Gribonval, and P. Vandergheynst, Accelerated spectral clustering using graph filtering of random signals, in Acoustics, Speech and Signal Processing (ICASSP), IEEE International Conference on, 6, accepted. [3] D. Ramasamy and U. Madhow, Compressive spectral embedding: sidestepping the SVD, in Advances in Neural Information Processing Systems 8, 5, pp [4] G. Puy, N. Tremblay, R. Gribonval, and P. Vandergheynst, Random sampling of bandlimited signals on graphs, arxiv, vol. abs/5.58, 5. [5] F. Chung, Spectral graph theory. Amer Mathematical Society, 997, no. 9. [6] A. Sandryhaila and J. Moura, Discrete signal processing on graphs, Signal Processing, IEEE Transactions on, vol. 6, no. 7, pp , 3. [7] U. von Luxburg, A tutorial on spectral clustering, Statistics and Computing, vol. 7, no. 4, pp , 7. [8] D. Achlioptas, Database-friendly random projections: Johnson-lindenstrauss with binary coins, Journal of Computer and System Sciences, vol. 66, no. 4, pp , 3. [9] S. Chen, R. Varma, A. Sandryhaila, and J. Kovacevic, Discrete signal processing on graphs: Sampling theory, Signal Processing, IEEE Transactions on, vol. 63, no. 4, pp , 5. [3] A. Anis, A. Gadde, and A. Ortega, Efficient sampling set selection for bandlimited graph signals using graph spectral proxies, arxiv, vol. abs/5.97, 5. [3] M. Tsitsvero, S. Barbarossa, and P. D. Lorenzo, Signals on graphs: Uncertainty principle and sampling, arxiv, vol. abs/57.88, 5. [3] A. Marques, S. Segarra, G. Leus, and A. Ribeiro, Sampling of graph signals with successive local aggregations, Signal Processing, IEEE Transactions on, vol. PP, no. 99, pp., 5. [33] X. Shi, H. Feng, M. Zhai, T. Yang, and B. Hu, Infinite impulse response graph filters in wireless sensor networks, Signal Processing Letters, IEEE, vol., no. 8, pp. 3 7, 5. [34] D. Shuman, P. Vandergheynst, and P. Frossard, Chebyshev polynomial approximation for distributed signal processing, in Distributed Computing in Sensor Systems and Workshops (DCOSS), International Conference on,, pp. 8. [35] E. D. Napoli, E. Polizzi, and Y. Saad, Efficient estimation of eigenvalue counts in an interval, arxiv, vol. abs/38.475, 3. [36] H. Avron and S. Toledo, Randomized algorithms for estimating the trace of an implicit symmetric positive semidefinite matrix, J. ACM, vol. 58, no., pp. 8: 8:34,. [37] W.-Y. Chen, Y. Song, H. Bai, L. C.-J, and E. Chang, Parallel spectral clustering in distributed systems, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 33, no. 3, pp ,.

15 Compressive spectral clustering 5 [38] J. Lei and A. Rinaldo, Consistency of spectral clustering in stochastic block models, Ann. Statist., vol. 43, no., pp. 5 37, 5. [39] N. Perraudin, J. Paratte, D. Shuman, V. Kalofolias, P. Vandergheynst, and D. Hammond, Gspbox: A toolbox for signal processing on graphs, arxiv, vol. abs/48.578, 4. [4] [4] A. Decelle, F. Krzakala, C. Moore, and L. Zdeborová, Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications, Phys. Rev. E, vol. 84, p. 666,. [4] L. Hubert and P. Arabie, Comparing partitions, Journal of classification, vol., no., pp. 93 8, 985. [43] J. Yang and J. Leskovec, Defining and evaluating network communities based on ground-truth, Knowledge and Information Systems, vol. 4, no., pp. 8 3, 5. [44] M. E. J. Newman and M. Girvan, Finding and evaluating community structure in networks, Phys. Rev. E, vol. 69, p. 63, 4. [45] M. Seifi, I. Junier, J.-B. Rouquier, S. Iskrov, and J.-L. Guillaume, Stable community cores in complex networks, in Complex Networks, 3, pp