Sampling, Inference and Clustering for Data on Graphs

Transcription

1 Sampling, Inference and Clustering for Data on Graphs Pierre Vandergheynst École Polytechnique Fédérale de Lausanne (EPFL) School of Engineering & School of Computer and Communication Sciences Joint work with Gilles Puy (INRIA), Nicolas Tremblay (INRIA) and Rémi Gribonval (INRIA) CO Ccc COSMO 21 Chania, Crete, May

2 Motivation Energy Networks Social Networks Transportation Networks Biological Networks Point Clouds 2

3 Goal Given partially observed information at the nodes of a graph? Can we robustly and efficiently infer missing information? What signal model? How many observations? Influence of the structure of the graph? 3

4 Notations G = {V, E, W} weighted, undirected V is the set of n nodes E is the set of edges W 2 R n n is the weighted adjacency matrix L 2 R n n combinatorial graph Laplacian L := D normalised Laplacian L := I D 1/2 WD 1/2 W diagonal degree matrix D has entries d i := P i6=j W ij 4

5 SP on Graphs Cheat Sheet x 2 R n L 2 R n n L = U U Graph Fourier Frequencies a (scalar valued) signal Laplacian Filter and Filtering = F T! 2 F g(l) =Ug( )U g(l)x g? x \g(l)x(k) =g( k )ˆx(k) [g? x(!) =ĝ(!)ˆx(!) 5

6 Notations L is real, symmetric PSD orthonormal eigenvectors U 2 R n n Graph Fourier Matrix non-negative eigenvalues , n L = U U k-bandlimited signals Fourier coefficients x = U k ˆx k x 2 R n ˆx = U x ˆx k 2 R k U k := (u 1,...,u k ) 2 R n k first k eigenvectors only 6

7 Sampling Model nx p 2 R n p i > 0 kpk 1 = P := diag(p) 2 R n n i=1 p i =1 Draw independently m samples (random sampling) P(! j = i) =p i, 8j 2 {1,...,m} and 8i 2 {1,...,n} y j := x!j, 8j 2 {1,...,m} y = Mx 7

8 Sampling Model ku k ik 2 ku ik 2 = ku k ik 2 k i k 2 = ku k ik 2 How much a perfect impulse can be concentrated on first k eigenvectors Carries interesting information about the graph Ideally: p i large wherever ku k ik 2 is large Graph Coherence k p := max 16i6n Rem: k p n p 1/2 i ku k ik 2 o > p k 8

9 Stable Embedding Theorem 1 (Restricted isometry property). Let M be a random subsampling matrix with the sampling distribution p. Forany, 2 (0, 1), with probability at least 1, (1 ) kxk m MP 1/2 x 2 6 (1 + ) kxk 2 2 (1) 2 for all x 2 span(u k ) provided that m > 3 2k 2 ( k p) 2 log. (2) MP 1/2 x = P 1/2 Mx Only need M, re-weighting offline ( k p) 2 > k Need to sample at least k nodes Proof similar to CS in bounded ONB but simpler since model is a subspace (not a union) 9

10 Stable Embedding ( k p) 2 > k Need to sample at least k nodes Can we reduce to optimal amount? Variable Density Sampling p i := ku k ik 2 2 k, i =1,...,n is such that: ( k p) 2 = k and depends on structure of graph Corollary 1. Let M be a random subsampling matrix constructed with the sampling distribution p.forany, 2 (0, 1), with probability at least 1, (1 ) kxk m MP 1/2 x (1 + ) kxk 2 2 for all x 2 span(u k ) provided that m > 3 2 k log 2k. 10

11 Recovery Procedures (Inference) y = Mx + n y 2 R m x 2 span(u k ) stable embedding Standard Decoder min z2span(u k ) P 1/2 (Mz y) 2 need projector re-weighting for RIP 11

12 Recovery Procedures (Inference) y = Mx + n y 2 R m x 2 span(u k ) stable embedding Efficient Decoder: min P 1/2 z2r n (Mz y) 2 + z g(l)z 2 soft constrain on frequencies efficient implementation 12

13 Analysis of Standard Decoder Standard Decoder: min z2span(u k ) P 1/2 (Mz y) 2 Theorem 1. Let be a set of m indices selected independently from {1,...,n} with sampling distribution p 2 R n,andm the associated sampling matrix. Let, 2 (0, 1) and m > 3 2 ( k p) 2 log 2k. With probability at least 1, the following holds for all x 2 span(u k ) and all n 2 R m. i) Let x be the solution of Standard Decoder with y = Mx + n. Then, kx xk 2 6 Exact recovery when noiseless 2 p P 1/2 n m (1 ) 2. (1) ii) There exist particular vectors n 0 2 R m such that the solution x of Standard Decoder with y = Mx + n 0 satisfies kx xk 2 > 1 p m (1 + ) P 1/2 n 0 2. (2) 13

14 Analysis of Efficient Decoder Efficient Decoder: min P 1/2 z2r n (Mz y) 2 + z g(l)z 2 non-negative non-decreasing = penalizes high-frequencies Favours reconstruction of approximately band-limited signals Ideal filter yields Standard Decoder i k (t) := 0 if t 2 [0, k], +1 otherwise, 14

15 Analysis of Efficient Decoder Theorem 1. Let, M, P, m as before and M max > 0 be a constant such that MP 1/2 6 M 2 max. Let, 2 (0, 1). With probability at least 1, the following holds for all x 2 span(u k ),alln2r n,all > 0, and all nonnegative and nondecreasing polynomial functions g such that g( k+1 ) > 0. Let x be the solution of E cient Decoder with y = Mx + n. Then, "! k 1 M xk 2 6 p 2+ p max P 1/2 n m(1 ) g( k+1 ) 2 s g( k ) + M max g( k+1 ) + p! # g( k ) kxk2, (1) and k k p g( k+1 ) P 1/2 n 2 + s g( k ) g( k+1 ) kxk 2, (2) where := U k U k x and := (I U k U k ) x. 15

16 Analysis of Efficient Decoder Noiseless case: x xk 2 6 s 1 g( p k ) M max m(1 ) g( k+1 ) + p! g( k ) kxk 2 + s g( k ) g( k+1 ) kxk 2 g( k )=0 + non-decreasing implies perfect reconstruction Otherwise: choose as close as possible to 0 and seek to minimise the ratio g( k )/g( k+1 ) Noise: kp 1/2 nk 2 / kxk 2 Choose filter to increase spectral gap? Clusters are of course good 16

17 Estimating the Optimal Distribution Need to estimate ku k ik 2 2 Filter random signals with ideal low-pass filter: r b k = U diag( 1,..., k, 0,...,0) U r = U k U k r E (r b k ) 2 i = i U ku k E(rr ) U k U k i = ku k ik 2 2 In practice, one may use a polynomial approximation of the ideal filter and: p i := P n i=1 P L l=1 (rl c k ) 2 i P L l=1 (rl c k ) 2 i L > C 2 log 2n 17

18 Experiments unbalanced clusters 18

19 Experiments 19

20 Experiments 20

21 Experiments 7% 21

22 Compressive Spectral Clustering Clustering equivalent to recovery of cluster assignment functions Well-defined clusters -> band-limited assignment functions! Generate features by filtering random signals by Johnson-Lindenstrauss = /2 3 /3 log n 22

23 Compressive Spectral Clustering Clustering equivalent to recovery of cluster assignment functions Well-defined clusters -> band-limited assignment functions! Generate features by filtering random signals by Johnson-Lindenstrauss = /2 3 /3 log n Each feature map is smooth, therefore keep m > 6 k 2 2 k log 0 Use k-means on compressed data and feed into Efficient Decoder 23

24 Compressive Spectral Clustering log k k log k 24

25 Conclusion Stable, robust and universal random sampling of smoothly varying information on graphs. Tractable decoder with guarantees Optimal sampling distribution depends on graph structure Can be used for inference, (SVD less) compressive clustering 25

26 Thank you! 26