Graphs in Machine Learning

Size: px

Start display at page:

Download "Graphs in Machine Learning"

Elizabeth Grant
5 years ago
Views:

1 Graphs in Machine Learning Michal Valko Inria Lille - Nord Europe, France TA: Pierre Perrault Partially based on material by: Ulrike von Luxburg, Gary Miller, Doyle & Schnell, Daniel Spielman October 10th, 2018 MVA 2018/2019

2 Previous lecture where do the graphs come from? social, information, utility, and biological networks we create them from the flat data random graph models specific applications and concepts maximizing influence on a graph gossip propagation, submodularity, proof of the approximation guarantee Google pagerank random surfer process, steady state vector, sparsity online semi-supervised learning label propagation, backbone graph, online learning, combinatorial sparsification, stability analysis Erdős number project, real-world graphs, heavy tails, small world when did this happen? similarity graphs different types construction practical considerations Michal Valko Graphs in Machine Learning SequeL - 2/43

3 This lecture Laplacians and their properties spectral graph theory random walks recommendation on a bipartite graph resistive networks recommendation score as a resistance? Laplacian and resistive networks resistance distance and random walks PS: some students have started working on their projects already Michal Valko Graphs in Machine Learning SequeL - 3/43

4 Similarity Graphs: ε or k-nn? DEMO IN CLASS tutorial.pdf Michal Valko Graphs in Machine Learning SequeL - 4/43

5 Generic Similarity Functions Gaussian similarity function/heat function/rbf: Cosine similarity function: Typical Kernels ( xi x j 2 ) s ij = exp 2σ 2 s ij = cos(θ) = ( x T i x j ) x i x j Michal Valko Graphs in Machine Learning SequeL - 5/43

6 Similarity Graphs G = (V, E) - with a set of nodes V and a set of edges E Michal Valko Graphs in Machine Learning SequeL - 6/43

7 Sources of Real Networks default.htm Michal Valko Graphs in Machine Learning SequeL - 7/43

8 L = D W graph Laplacian the only matrix that matters

9 2 Graph Laplacian L = G = (V, E) - with a set of nodes V and a set of edges E A W D L = D W adjacency matrix weight matrix (diagonal) degree matrix graph Laplacian matrix L is SDD! Michal Valko Graphs in Machine Learning SequeL - 9/43

10 Properties of Graph Laplacian Graph function: a vector f R N assigning values to nodes: f : V(G) R. f T Lf = 1 w i,j (f i f j ) 2 = S G (f) 2 i,j N Michal Valko Graphs in Machine Learning SequeL - 10/43

11 Recap: Eigenwerte und Eigenvektoren A vector v is an eigenvector of matrix M of eigenvalue λ Mv = λv. If (λ 1, v 1 ) are (λ 2, v 2 ) eigenpairs for symmetric M with λ 1 λ 2 then v 1 v 2, i.e., v T 1 v 2 = 0. If (λ, v 1 ), (λ, v 2 ) are eigenpairs for M then (λ, v 1 + v 2 ) is as well. For symmetric M, the multiplicity of λ is the dimension of the space of eigenvectors corresponding to λ. N N symmetric matrix has N eigenvalues (w/ multiplicities). Michal Valko Graphs in Machine Learning SequeL - 11/43

12 Eigenvalues, Eigenvectors, and Eigendecomposition A vector v is an eigenvector of matrix M of eigenvalue λ Mv = λv. Vectors {v i } i form an orthonormal basis with λ 1 λ 2... λ N. i Mv i = λ i v i MQ = QΛ Q has eigenvectors in columns and Λ has eigenvalues on its diagonal. Right-multiplying MQ = QΛ by Q T we get the eigendecomposition of M: M = MQQ T = QΛQ T = i λ iv i v T i Michal Valko Graphs in Machine Learning SequeL - 12/43

13 M = L: Properties of Graph Laplacian We can assume non-negative weights: w ij 0. L is symmetric L positive semi-definite f T Lf = 1 2 i,j N w i,j(f i f j ) 2 Recall: If Lf = λf then λ is an eigenvalue (of the Laplacian). The smallest eigenvalue of L is 0. Corresponding eigenvector: 1 N. All eigenvalues are non-negative reals 0 = λ 1 λ 2 λ N. Self-edges do not change the value of L. Michal Valko Graphs in Machine Learning SequeL - 13/43

14 Properties of Graph Laplacian The multiplicity of eigenvalue 0 of L equals to the number of connected components. The eigenspace of 0 is spanned by the components indicators. Proof: If (0, f) is an eigenpair then 0 = 1 2 i,j N w i,j(f i f j ) 2. Therefore, f is constant on each connected component. If there are k components, then L is k-block-diagonal: L 1 L 2 L =... Lk For block-diagonal matrices: the spectrum is the union of the spectra of L i (eigenvectors of L i padded with zeros elsewhere). For L i (0, 1 Vi ) is an eigenpair, hence the claim. Michal Valko Graphs in Machine Learning SequeL - 14/43

15 Smoothness of the Function and Laplacian f = (f 1,..., f N ) T : graph function Let L = QΛQ T be the eigendecomposition of the Laplacian. Diagonal matrix Λ whose diagonal entries are eigenvalues of L. Columns of Q are eigenvectors of L. Columns of Q form a basis. α: Unique vector such that Qα = f Note: Q T f = α Smoothness of a graph function S G (f) S G (f) = f T Lf = f T QΛQ T f = α T Λα = α 2 Λ = N λ i αi 2 i=1 Smoothness and regularization: Small value of (a) S G (f) (b) Λ norm of α (c) α i for large λ i Michal Valko Graphs in Machine Learning SequeL - 15/43

16 Smoothness of the Function and Laplacian N S G (f) = f T Lf = f T QΛQ T f = α T Λα = α 2 Λ = λ i αi 2 Eigenvectors are graph functions too! What is the smoothness of an eigenvector? Spectral coordinates of eigenvector v k : Q T v k = e k i=1 N S G (v k )=v T k Lv k =v T k QΛQT v k = e T k Λe k = e k 2 Λ = λ i (e k ) 2 i = λ k i=1 The smoothness of k-th eigenvector is the k-th eigenvalue. Michal Valko Graphs in Machine Learning SequeL - 16/43

17 Laplacian of the Complete Graph K N What is the eigenspectrum of L KN? L KN = From before: we know that (0, 1 N ) is an eigenpair. N N N N N 1 If v 0 N and v 1 N = i v i = 0. To get the other eigenvalues, we compute (L KN v) 1 and divide by v 1 (wlog v 1 0). N (L KN v) 1 = (N 1)v 1 v i = Nv 1. i=2 What are the remaining eigenvalues/vectors? Michal Valko Graphs in Machine Learning SequeL - 17/43

18 Normalized Laplacians L un = D W L sym = D 1/2 LD 1/2 = I D 1/2 WD 1/2 L rw = D 1 L = I D 1 W f T L sym f = 1 2 ( ) 2 f i w i,j f j di dj i,j N (λ, u) is an eigenpair for L rw iff (λ, D 1/2 u) is an eigenpair for L sym Michal Valko Graphs in Machine Learning SequeL - 18/43

19 Normalized Laplacians L sym and L rw are PSD with non-negative real eigenvalues 0 = λ 1 λ 2 λ 3 λ N. (λ, u) is an eigenpair for L rw iff (λ, u) solve the generalized eigenproblem Lu = λdu. (0, 1 N ) is an eigenpair for L rw. (0, D 1/2 1 N ) is an eigenpair for L sym. Multiplicity of eigenvalue 0 of L rw or L sym equals to the number of connected components. Proof: As for L. Michal Valko Graphs in Machine Learning SequeL - 19/43

20 Laplacian and Random Walks on Undirected Graphs stochastic process: vertex-to-vertex jumping transition probability v i v j is p ij = w ij /d i def d i = j w ij transition matrix P = (p ij ) ij = D 1 W (notice L rw = I P) if G is connected and non-bipartite unique stationary distribution π = (π 1, π 2, π 3,..., π N ) where π i = d i /vol(v ) vol(g) = vol(v ) = vol(w) def = i d i = i,j w ij π = 1T W vol(w) verifies πp = π as: πp = 1T WP vol(w) = 1T DP vol(w) = 1T DD 1 W = 1T W vol(w) vol(w) = π What s the difference from the PageRank TM? Michal Valko Graphs in Machine Learning SequeL - 20/43

21 score(v, m) recommendation on a bipartite graph with the graph Laplacian!

22 Use of Laplacians: Movie recommendation How to do movie recommendation on a bipartite graph? Adam Barbara Céline viewer 1 viewer 2 viewer 3 ranking ranking ranking ranking movie A movie B movie C Blade Runner 2049 Cars 3 Capitaine Superslip Question: Do we recommend Capitaine Superslip to Adam? Let s compute some score(v, m)! Michal Valko Graphs in Machine Learning SequeL - 22/43

23 Use of Laplacians: Movie recommendation How to compute the score(v, m)? Using some graph distance! Idea 1 : maximally weighted path score(v, m) = max vpm weight(p) = max vpm e P ranking(e) Idea 2 : change the path weight score 2 (v, m) = max vpm weight 2 (P) = max vpm min e P ranking(e) Idea 3 : consider everything score 3 (v, m) = max flow from m to v Michal Valko Graphs in Machine Learning SequeL - 23/43

24 Laplacians and Resistive Networks How to compute the score(v, m)? Idea 4 : view edges as conductors score 4 (v, m) = effective resistance between m and v v i + C C conductance R resistance i current V voltage C = 1 R i = CV = V R Michal Valko Graphs in Machine Learning SequeL - 24/43

25 Resistive Networks: Some high-school physics Michal Valko Graphs in Machine Learning SequeL - 25/43

26 Resistive Networks resistors in series 1 R = R R n C = 1 C C N i = V R conductors in parallel C = C C N i = VC Effective Resistance on a graph Take two nodes: a b. Let V ab be the voltage between them and i ab the current between them. Define R ab = V ab i ab and C ab = 1 R ab. We treat the entire graph as a resistor! Michal Valko Graphs in Machine Learning SequeL - 26/43

27 Resistive Networks: Optional Homework (ungraded) Show that R ab is a metric space. 1. R ab 0 2. R ab = 0 iff a = b 3. R ab = R ba 4. R ac R ab + R bc The effective resistance is a distance! Michal Valko Graphs in Machine Learning SequeL - 27/43

28 How to compute effective resistance? Kirchhoff s Law flow in = flow out V 1 V 3 C 3 V C 1 C 2 V 2 V = C 1 C V 1 + C 2 C V 2 + C 3 C V 3 (convex combination) residual current = CV C 1 V 1 C 2 V 2 C 3 V 3 Kirchhoff says: This is zero! There is no residual current! Michal Valko Graphs in Machine Learning SequeL - 28/43

29 Resistors: Where is the link with the Laplacian? General case of the previous! d i = j c ij = sum of conductances d i if i = j, L ij = c ij if (i, j) E, 0 otherwise. v = voltage setting of the nodes on graph. (Lv) i = residual current at v i as we derived Use: setting voltages and getting the current Inverting injecting current and getting the voltages The net injected has to be zero Kirchhoff s Law. Michal Valko Graphs in Machine Learning SequeL - 29/43

30 Resistors and the Laplacian: Finding R ab Let s calculate R 1N to get the movie recommendation score! L 0 v 2. v n 1 1 = i 0. 0 i i = V R V = 1 R = 1 i Return R 1N = 1 i Doyle and Snell: Random Walks and Electric Networks Michal Valko Graphs in Machine Learning SequeL - 30/43

31 Resistors and the Laplacian: Finding R 1N Lv = (i, 0,..., i) T boundary valued problem For R 1N V 1 and V N are the boundary (v 1, v 2,..., v N ) is harmonic: V i interior (not boundary) V i is a convex combination of its neighbors Michal Valko Graphs in Machine Learning SequeL - 31/43

32 Resistors and the Laplacian: Finding R 1n From the properties of electric networks (cf. Doyle and Snell) we inherit the useful properties of the Laplacians! Example: Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions (later in the course) Maximum Principle If f = v is harmonic then min and max are on the boundary. Uniqueness Principle If f and g are harmonic with the same boundary then f = g Michal Valko Graphs in Machine Learning SequeL - 32/43

33 Resistors and the Laplacian: Finding R 1N Alternative method to calculate R 1N : 1 0 def Lv =. = i ext Return R 1N = v 1 v N Why? 0 1 Question: Does v exist? L does not have an inverse :(. Not unique: 1 in the nullspace of L : L(v + c1) = Lv + cl1 = Lv Moore-Penrose pseudo-inverse solves LS Solution: Instead of v = L 1 i ext we take v = L + i ext We get: R 1N = v 1 v N = i T extv = i T extl + i ext. Notice: We can reuse L + to get resistances for any pair of nodes! Michal Valko Graphs in Machine Learning SequeL - 33/43

34 What? A pseudo-inverse? Eigendecomposition of the Laplacian: L = QΛQ T = N λ i q i q T i = i=1 N λ i q i q T i i=2 Pseudo-inverse of the Laplacian: L + = QΛ + Q T = N i=2 1 λ i q i q T i Moore-Penrose pseudo-inverse solves a least squares problem: v = arg min Lx i ext 2 = L + i ext x Michal Valko Graphs in Machine Learning SequeL - 34/43

35 cut(a, B) = 1 2 ft Lf spectral clustering with connectivity beyond compactness

36 How to rule the world? Let s make France great again! Michal Valko Graphs in Machine Learning SequeL - 36/43

37 How to rule the world? Michal Valko Graphs in Machine Learning SequeL - 37/43

How to rule the world: AI is here https://www.washingtonpost.com/opinions/obama-the-big-data-president/2013/06/14/ 1d71fe2e-d391-11e2-b05f-3ea3f0e7bb5a_story.html https://www.

38 How to rule the world: AI is here 1d71fe2e-d391-11e2-b05f-3ea3f0e7bb5a_story.html Talk of Rayid Ghaniy: Michal Valko Graphs in Machine Learning SequeL - 38/43

39 Application of Graphs for ML: Clustering Michal Valko Graphs in Machine Learning SequeL - 39/43

40 Application: Clustering - Recap What do we know about the clustering in general? ill defined problem (different tasks different paradigms) I know it when I see it inconsistent (wrt. Kleinberg s axioms) number of clusters k need often be known difficult to evaluate What do we know about k-means? hard version of EM clustering sensitive to initialization optimizes for compactness yet: algorithm-to-go Michal Valko Graphs in Machine Learning SequeL - 40/43

41 Spectral Clustering: Cuts on graphs A B C F E G D H I J K Michal Valko Graphs in Machine Learning SequeL - 41/43

42 Spectral Clustering: Cuts on graphs A B C F E G D H I J K Defining the cut objective we get the clustering! Michal Valko Graphs in Machine Learning SequeL - 41/43

43 Spectral Clustering: Cuts on graphs A B C F E G D H I J K MinCut: cut(a, B) = i A,j B w ij Are we done? Can be solved efficiently, but maybe not what we want.... Michal Valko Graphs in Machine Learning SequeL - 41/43

44 Next class on Wednesday, October 17th at 14:00! S. des Conférences S. Visio DSI S. Renaudeau C518 Bretécher Amphi Tocqueville Uderzo Amphi Marie Curie S. des Comm. Amphi 121 Condorcet FCD Fonteneau 131 bis Amphi 109 Amphi e-media Michal Valko Graphs in Machine Learning SequeL - 42/43

45 Michal Valko ENS Paris-Saclay, MVA 2018/2019 SequeL team, Inria Lille Nord Europe

Graphs in Machine Learning

Graphs in Machine Learning Michal Valko Inria Lille - Nord Europe, France TA: Pierre Perrault Partially based on material by: Ulrike von Luxburg, Gary Miller, Mikhail Belkin October 16th, 2017 MVA 2017/2018