Graphs in Machine Learning

Transcription

1 Graphs in Machine Learning Michal Valko Inria Lille - Nord Europe, France TA: Pierre Perrault Partially based on material by: Ulrike von Luxburg, Gary Miller, Mikhail Belkin October 16th, 2017 MVA 2017/2018

2 Previous Lecture similarity graphs different types construction sources of graphs practical considerations spectral graph theory Laplacians and their properties symmetric and asymmetric normalization random walks recommendation on a bipartite graph resistive networks recommendation score as a resistance? Laplacian and resistive networks resistance distance and random walks Michal Valko Graphs in Machine Learning SequeL - 2/54

3 This Lecture geometry of the data and the connectivity spectral clustering manifold learning with Laplacians eigenmaps Gaussian random fields and harmonic solution graph-based semi-supervised learning and manifold regularization transductive learning inductive and transductive semi-supervised learning Michal Valko Graphs in Machine Learning SequeL - 3/54

4 Next Class: Lab Session by Pierre Perrault cca. 10h00-10h30 optional help with setup, 10h30-12h30: TD Salle Condorcet Download the image and set it up BEFORE the class Matlab/Octave Short written report (graded) All homeworks together account for 40% of the final grade Content Graph Construction Test sensitivity to parameters: σ, k, ε Spectral Clustering Spectral Clustering vs. k-means Image Segmentation Michal Valko Graphs in Machine Learning SequeL - 4/54

5 PhD student position on the topic of sequential learning (mathematical statistics and machine learning) at Uni Magdeburg. PhD candidate will focus on developing sequential learning algorithms with mathematical guarantees for learning on given non-stationary processes that are relevant in the context of recommendation systems, and on implementation of the algorithms that will be developed. S/He will also work on the eye tracker based application of the project. A degree in machine learning or in mathematics with an interest in theoretical computer science will be preferred. Uni Magdeburg (90 minutes away from Berlin, Germany, by public transports) and universities in nearby Berlin offer a highly motivating and rich research environment. The PhD candidates will be advised by Dr. Alexandra Carpentier (contact Ausschreibung_SFB1294_A03_2ndround-4-2.pdf for more info). Michal Valko Graphs in Machine Learning SequeL - 5/54

6 Use of Laplacians: Movie recommendation How to do movie recommendation on a bipartite graph? Adam Barbara Céline viewer 1 viewer 2 viewer 3 ranking ranking ranking ranking movie A movie B movie C Blade Runner 2049 Cars 3 Capitaine Superslip Question: Do we recommend Capitaine Superslip to Adam? Let s compute some score(v, m)! Michal Valko Graphs in Machine Learning SequeL - 6/54

7 Use of Laplacians: Movie recommendation How to compute the score(v, m)? Using some graph distance! Idea 1 : maximally weighted path score(v, m) = max vpm weight(p) = max vpm e P ranking(e) Idea 2 : change the path weight score 2 (v, m) = max vpm weight 2 (P) = max vpm min e P ranking(e) Idea 3 : consider everything score 3 (v, m) = max flow from m to v Michal Valko Graphs in Machine Learning SequeL - 7/54

8 Laplacians and Resistive Networks How to compute the score(v, m)? Idea 4 : view edges as conductors score 4 (v, m) = effective resistance between m and v v i + C C conductance R resistance i current V voltage C = 1 R i = CV = V R Michal Valko Graphs in Machine Learning SequeL - 8/54

9 Resistive Networks: Some high-school physics Michal Valko Graphs in Machine Learning SequeL - 9/54

10 Resistive Networks resistors in series 1 R = R R n C = 1 C C N i = V R conductors in parallel C = C C N i = VC Effective Resistance on a graph Take two nodes: a b. Let V ab be the voltage between them and i ab the current between them. Define R ab = V ab i ab and C ab = 1 R ab. We treat the entire graph as a resistor! Michal Valko Graphs in Machine Learning SequeL - 10/54

11 Resistive Networks: Optional Homework (ungraded) Show that R ab is a metric space. 1. R ab 0 2. R ab = 0 iff a = b 3. R ab = R ba 4. R ac R ab + R bc The effective resistance is a distance! Michal Valko Graphs in Machine Learning SequeL - 11/54

12 How to compute effective resistance? Kirchhoff s Law flow in = flow out V 1 V 3 C 3 V C 1 C 2 V 2 V = C 1 C V 1 + C 2 C V 2 + C 3 C V 3 (convex combination) residual current = CV C 1 V 1 C 2 V 2 C 3 V 3 Kirchhoff says: This is zero! There is no residual current! Michal Valko Graphs in Machine Learning SequeL - 12/54

13 Resistors: Where is the link with the Laplacian? General case of the previous! d i = j c ij = sum of conductances d i if i = j, L ij = c ij if (i, j) E, 0 otherwise. v = voltage setting of the nodes on graph. (Lv) i = residual current at v i as we derived Use: setting voltages and getting the current Inverting injecting current and getting the voltages The net injected has to be zero Kirchhoff s Law. Michal Valko Graphs in Machine Learning SequeL - 13/54

14 Resistors and the Laplacian: Finding R ab Let s calculate R 1N to get the movie recommendation score! L 0 v 2. v n 1 1 = i 0. 0 i i = V R V = 1 R = 1 i Return R 1N = 1 i Doyle and Snell: Random Walks and Electric Networks doyle/docs/walks/walks.pdf Michal Valko Graphs in Machine Learning SequeL - 14/54

15 Resistors and the Laplacian: Finding R 1N Lv = (i, 0,..., i) T boundary valued problem For R 1N V 1 and V N are the boundary (v 1, v 2,..., v N ) is harmonic: V i interior (not boundary) V i is a convex combination of its neighbors Michal Valko Graphs in Machine Learning SequeL - 15/54

16 Resistors and the Laplacian: Finding R 1n From the properties of electric networks (cf. Doyle and Snell) we inherit the useful properties of the Laplacians! Example: Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions (later in the course) Maximum Principle If f = v is harmonic then min and max are on the boundary. Uniqueness Principle If f and g are harmonic with the same boundary then f = g Michal Valko Graphs in Machine Learning SequeL - 16/54

17 Resistors and the Laplacian: Finding R 1N Alternative method to calculate R 1N : 1 0 def Lv =. = i ext Return R 1N = v 1 v N Why? 0 1 Question: Does v exist? L does not have an inverse :(. Not unique: 1 in the nullspace of L : L(v + c1) = Lv + cl1 = Lv Moore-Penrose pseudo-inverse solves LS Solution: Instead of v = L 1 i ext we take v = L + i ext We get: R 1N = v 1 v N = i T extv = i T extl + i ext. Notice: We can reuse L + to get resistances for any pair of nodes! Michal Valko Graphs in Machine Learning SequeL - 17/54

18 What? A pseudo-inverse? Eigendecomposition of the Laplacian: L = QΛQ T = N λ i q i q T i = i=1 N λ i q i q T i i=2 Pseudo-inverse of the Laplacian: L + = QΛ + Q T = N i=2 1 λ i q i q T i Moore-Penrose pseudo-inverse solves a least squares problem: v = arg min Lx i ext 2 = L + i ext x Michal Valko Graphs in Machine Learning SequeL - 18/54

19 How to rule the world? Let s make France great again! Michal Valko Graphs in Machine Learning SequeL - 19/54

20 How to rule the world? Michal Valko Graphs in Machine Learning SequeL - 20/54

21 How to rule the world: AI is here 1d71fe2e-d391-11e2-b05f-3ea3f0e7bb5a_story.html Talk of Rayid Ghaniy: Michal Valko Graphs in Machine Learning SequeL - 21/54

22 Application of Graphs for ML: Clustering Michal Valko Graphs in Machine Learning SequeL - 22/54

23 Application: Clustering - Recap What do we know about the clustering in general? ill defined problem (different tasks different paradigms) I know it when I see it inconsistent (wrt. Kleinberg s axioms) number of clusters k need often be known difficult to evaluate What do we know about k-means? hard version of EM clustering sensitive to initialization optimizes for compactness yet: algorithm-to-go Michal Valko Graphs in Machine Learning SequeL - 23/54

24 Spectral Clustering: Cuts on graphs A B C F E G D H I J K Michal Valko Graphs in Machine Learning SequeL - 24/54

25 Spectral Clustering: Cuts on graphs A B C F E G D H I J K Defining the cut objective we get the clustering! Michal Valko Graphs in Machine Learning SequeL - 24/54

26 Spectral Clustering: Cuts on graphs A B C F E G D H I J K MinCut: cut(a, B) = i A,j B w ij Are we done? Can be solved efficiently, but maybe not what we want.... Michal Valko Graphs in Machine Learning SequeL - 24/54

27 Spectral Clustering: Balanced Cuts Let s balance the cuts! MinCut RatioCut cut(a, B) = RatioCut(A, B) = i A,j B i A,j B w ij ( 1 w ij A + 1 ) B Normalized Cut NCut(A, B) = i A,j B ( 1 w ij vol(a) + 1 ) vol(b) Michal Valko Graphs in Machine Learning SequeL - 25/54

28 Spectral Clustering: Balanced Cuts ( 1 RatioCut(A, B) = cut(a, B) A + 1 B ( 1 NCut(A, B) = cut(a, B) vol(a) + 1 vol(b) ) ) Can we compute this? RatioCut and NCut are NP hard :( Approximate! Michal Valko Graphs in Machine Learning SequeL - 26/54

29 Spectral Clustering: Relaxing Balanced Cuts Relaxation for (simple) balanced cuts for 2 sets min cut(a, B) s.t. A = B A,B Graph function f for cluster membership: f i = { 1 if V i A, 1 if V i B. What it is the cut value with this definition? cut(a, B) = w i,j = 1 4 w i,j (f i f j ) 2 = 1 2 f T Lf i A,j B What is the relationship with the smoothness of a graph function? i,j Michal Valko Graphs in Machine Learning SequeL - 27/54

30 Spectral Clustering: Relaxing Balanced Cuts cut(a, B) = w i,j = 1 4 w i,j (f i f j ) 2 = 1 2 f T Lf i A,j B i,j A = B = i f i = 0 = f 1 N f = N objective function of spectral clustering min f f T Lf s.t. f i = ±1, f 1 N, f = N Still NP hard :( Relax even further! f i = ±1 f i R Michal Valko Graphs in Machine Learning SequeL - 28/54

31 Spectral Clustering: Relaxing Balanced Cuts objective function of spectral clustering min f f T Lf s.t. f i R, f 1 N, f = N Rayleigh-Ritz theorem If λ 1 λ N are the eigenvectors of real symmetric L then x T Lx x T x x T Lx λ 1 = min x 0 x T x = min x T Lx x T x=1 λ N = max x 0 Rayleigh quotient x T Lx x T x = max x T Lx x T x=1 How can we use it? Michal Valko Graphs in Machine Learning SequeL - 29/54

32 Spectral Clustering: Relaxing Balanced Cuts objective function of spectral clustering min f f T Lf s.t. f i R, f 1 N, f = N Generalized Rayleigh-Ritz theorem (Courant-Fischer-Weyl) If λ 1 λ N are the eigenvectors of real symmetric L and v 1,..., v N the corresponding orthogonal eigenvalues, then for k = 1 : N 1 λ k+1 = λ N k = x T Lx min x 0,x v 1,...v k x T x = min x T Lx x T x=1,x v 1,...v k x T Lx max x 0,x v n,...v N k+1 x T x = max x T Lx x T x=1,x v N,...v N k+1 Michal Valko Graphs in Machine Learning SequeL - 29/54

33 Rayleigh-Ritz theorem: Quick and dirty proof When we reach the extreme points? x ( ) xt Lx = x T x x By matrix calculus (or just calculus): ( ) f (x) = 0 f (x)g(x) = f (x)g (x) g(x) x T Lx x = 2Lx and x T x x = 2x When f (x)g(x) = f (x)g (x)? Lx (x T x) = (x T Lx) x Lx = xt Lx x Lx = λx x T x Conclusion: Extremes are the eigenvectors with their eigenvalues Michal Valko Graphs in Machine Learning SequeL - 30/54

34 Spectral Clustering: Relaxing Balanced Cuts objective function of spectral clustering min f f T Lf s.t. f i R, f 1 N, f = N Solution: second eigenvector The solution may not be integral. cluster i = How do we get the clustering? What to do? { 1 if f i 0, 1 if f i < 0. Works but this heuristics is often too simple. In practice, cluster f using k-means to get {C i } i and assign: { 1 if i C 1, cluster i = 1 if i C 1. Michal Valko Graphs in Machine Learning SequeL - 31/54

35 Spectral Clustering: Approximating RatioCut Wait, but we did not care about approximating mincut! RatioCut RatioCut(A, B) = i A,j B ( 1 w ij A + 1 ) B Define graph function f for cluster membership of RatioCut: B A if V i A, f i = A B if V i B. f T Lf = 1 2 w i,j (f i f j ) 2 = ( A + B )RatioCut(A, B) i,j Michal Valko Graphs in Machine Learning SequeL - 32/54

36 Spectral Clustering: Approximating RatioCut Define graph function f for cluster membership of RatioCut: B A if V i A, f i = A B if V i B. f i = 0 i i f 2 i = N objective function of spectral clustering (same - it s magic!) min f f T Lf s.t. f i R, f 1 N, f = N Michal Valko Graphs in Machine Learning SequeL - 33/54

37 Spectral Clustering: Approximating NCut Normalized Cut NCut(A, B) = i A,j B ( 1 w ij vol(a) + 1 ) vol(b) Define graph function f for cluster membership of NCut: vol(a) vol(b) if V i A, f i = vol(b) vol(a) if V i B. (Df) T 1 n = 0 f T Df = vol(v) f T Lf = vol(v)ncut(a, B) objective function of spectral clustering (NCut) min f f T Lf s.t. f i R, Df 1 N, f T Df = vol(v) Michal Valko Graphs in Machine Learning SequeL - 34/54

38 Spectral Clustering: Approximating NCut objective function of spectral clustering (NCut) min f f T Lf s.t. f i R, Df 1 N, f T Df = vol(v) Can we apply Rayleigh-Ritz now? Define w = D 1/2 f objective function of spectral clustering (NCut) min w wt D 1/2 LD 1/2 w s.t. w i R, w D 1/2 1 N, w 2 = vol(v) objective function of spectral clustering (NCut) min w wt L sym w s.t. w i R, w v 1,Lsym, w 2 = vol(v) Michal Valko Graphs in Machine Learning SequeL - 35/54

39 Spectral Clustering: Approximating NCut objective function of spectral clustering (NCut) min w wt L sym w s.t. w i R, w v 1,Lsym, w = vol(v) Solution by Rayleigh-Ritz? w = v 2,Lsym f = D 1/2 w f is a the second eigenvector of L rw! tl;dr: Get the second eigenvector of L/L rw for RatioCut/NCut. Michal Valko Graphs in Machine Learning SequeL - 36/54

40 Spectral Clustering: Approximation These are all approximations. How bad can they be? Example: cockroach graphs No efficient approximation exist. Other relaxations possible. glmiller/publications/papers/gumi95.pdf Michal Valko Graphs in Machine Learning SequeL - 37/54

41 Spectral Clustering: 1D Example Elbow rule/eigengap heuristic for number of clusters Michal Valko Graphs in Machine Learning SequeL - 38/54

42 Spectral Clustering: Understanding Compactness vs. Connectivity For which kind of data we can use one vs. the other? Any disadvantages of spectral clustering? Michal Valko Graphs in Machine Learning SequeL - 39/54

43 Spectral Clustering: 1D Example - Histogram publications/luxburg07_tutorial.pdf Michal Valko Graphs in Machine Learning SequeL - 40/54

44 Spectral Clustering: 1D Example - Eigenvectors Michal Valko Graphs in Machine Learning SequeL - 41/54

45 Spectral Clustering: Bibliography M. Meila et al. A random walks view of spectral segmentation. In: International Conference on Artificial Intelligence and Statistics (2001) L sym Andrew Y Ng, Michael I Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. In: Neural Information Processing Systems L rm J Shi and J Malik. Normalized Cuts and Image Segmentation. In: IEEE Transactions on Pattern Analysis and Machine Intelligence 22 (2000), pp Things can go wrong with the relaxation: Daniel A. Spielman and Shang H. Teng. Spectral partitioning works: Planar graphs and finite element meshes. In: Linear Algebra and Its Applications 421 (2007), pp Michal Valko Graphs in Machine Learning SequeL - 42/54

46 Michal Valko ENS Paris-Saclay, MVA 2017/2018 SequeL team, Inria Lille Nord Europe