Data Mining and Analysis: Fundamental Concepts and Algorithms
|
|
- Cynthia Butler
- 5 years ago
- Views:
Transcription
1 : Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA 2 Department of Computer Science Universidade Federal de Minas Gerais, Belo Horizonte, Brazil Chapter 16: Spectral & Graph Clustering Chapter 16: Spectral & Graph Clustering 1 /
2 Graphs and Matrices: Adjacency Matrix Given a dataset D = {x i } n i=1 consisting of n points in Rd, let A denote the n n symmetric similarity matrix between the points, given as a 11 a 12 a 1n a 21 a 22 a 2n A = a n1 a n2 a nn where A(i, j) = a ij denotes the similarity or affinity between points x i and x j. We require the similarity to be symmetric and non-negative, that is, a ij = a ji and a ij 0, respectively. The matrix A is the weighted adjacency matrix for the data graph. If all affinities are 0 or 1, then A represents the regular adjacency relationship between the vertices. Chapter 16: Spectral & Graph Clustering 2 /
3 Iris Similarity Graph: Mutual Nearest Neighbors V = n = 150, E = m = 1730 Chapter 16: Spectral & Graph Clustering 3 /
4 Graphs and Matrices: Degree Matrix For a vertex x i, let d i denote the degree of the vertex, defined as d i = n j=1 a ij We define the degree matrix of graph G as the n n diagonal matrix: n d j=1 a 1j d 2 0 = = 0 n j=1 a 2j d n n 0 0 j=1 a nj can be compactly written as (i, i) = d i for all 1 i n. Chapter 16: Spectral & Graph Clustering 4 /
5 Graphs and Matrices: Normalized Adjacency Matrix The normalized adjacency matrix is obtained by dividing each row of the adjacency matrix by the degree of the corresponding node. Given the weighted adjacency matrix A for a graph G, its normalized adjacency matrix is defined as a 11 a 12 a d 1 d 1 1n M = 1 A =. a 21 d 2 a 22 d 2 a n1 d n d 1 a 2n d a n2 d n Because A is assumed to have non-negative elements, this implies that each element of M, namely m ij is also non-negative, as m ij = a ij d i 0. Each row in M sums to 1, which implies that 1 is an eigenvalue of M. In fact, λ 1 = 1 is the largest eigenvalue of M, and the other eigenvalues satisfy the property that λ i 1. Because M is not symmetric, its eigenvectors are not necessarily orthogonal. a nn d n Chapter 16: Spectral & Graph Clustering 5 /
6 Example Graph: Adjacency and Degree Matrices Its adjacency and degree matrices are given as A = = Chapter 16: Spectral & Graph Clustering 6 /
7 Example Graph: Normalized Adjacency Matrix The normalized adjacency matrix is as follows: M = 1 A = The eigenvalues of M are: λ 1 = 1, λ 2 = 0.483, λ 3 = 0.206, λ 4 = 0.045, λ 5 = 0.405, λ 6 = 0.539, λ 7 = 0.7 Chapter 16: Spectral & Graph Clustering 7 /
8 Graph Laplacian Matrix The Laplacian matrix of a graph is defined as L = A n j=1 a 1j 0 0 a 11 a 12 a 1n n 0 j=1 = a 2j a 21 a 22 a 2n n 0 0 j=1 a nj a n1 a n2 a nn j 1 a 1j a 12 a 1n a 21 j 2 = a 2j a 2n a n1 a n2 j n a nj L is a symmetric, positive semidefinite matrix. This means that L has n real, non-negative eigenvalues, which can be arranged in decreasing order as follows: λ 1 λ 2 λ n 0. Because L is symmetric, its eigenvectors are orthonormal. The rank of L is at most n 1, and the smallest eigenvalue is λ n = 0. Chapter 16: Spectral & Graph Clustering 8 /
9 Example Graph: Laplacian Matrix The graph Laplacian is given as L = A = The eigenvalues of L are as follows: λ 1 = 5.618, λ 2 = 4.618, λ 3 = 4.414, λ 4 = 3.382, λ 5 = 2.382, λ 6 = 1.586, λ 7 = 0 Chapter 16: Spectral & Graph Clustering 9 /
10 Normalized Laplacian Matrices The normalized symmetric Laplacian matrix of a graph is defined as j 1 a 1j a 12 a 1n d1 d 1 d1 d 2 a 21 j 2 a 2j a 2n L s = 1/2 L 1/2 d2 d = 1 d2 d a n2 dnd2 a n1 dnd1 d1 d n d2 d n j n a nj dnd n L s is a symmetric, positive semidefinite matrix, with rank at most n 1. The smallest eigenvalueλ n = 0. The normalized asymmetric Laplacian matrix is defined as L a = 1 L = j 1 a 1j d 1 a 12 a 21 d 2. a n1 d n d 1 a 1n d 1 j 2 a 2j a 2n d 2 d a n2 d n j n a nj d n L a is also a positive semi-definite matrix with n real eigenvalues λ 1 λ 2 λ n = 0. Chapter 16: Spectral & Graph Clustering 10 /
11 Example Graph: Normalized Symmetric Laplacian Matrix The normalized symmetric Laplacian is given as L s = The eigenvalues of L s are as follows: λ 1 = 1.7, λ 2 = 1.539, λ 3 = 1.405, λ 4 = 1.045, λ 5 = Zaki 0.794, & Meira Jr. λ(rpi 6 = and0.517, UFMG) λ 7 = 0 Chapter 16: Spectral & Graph Clustering 11 /
12 Example Graph: Normalized Asymmetric Laplacian Matrix The normalized asymmetric Laplacian matrix is given as L a = 1 L = The eigenvalues of L a are identical to those for L s, namelyλ 1 = 1.7, λ 2 = 1.539, λ 3 = Zaki 1.405, & Meira Jr. λ(rpi 4 = and1.045, UFMG) λ 5 = 0.794, Data λ 6 Mining = 0.517, and Analysis λ 7 = 0 Chapter 16: Spectral & Graph Clustering 12 /
13 Clustering as Graph Cuts A k-way cut in a graph is a partitioning or clustering of the vertex set, given as C = {C 1,...,C k }. We require C to optimize some objective function that captures the intuition that nodes within a cluster should have high similarity, and nodes from different clusters should have low similarity. Given a weighted graph G defined by its similarity matrix A, let S, T V be any two subsets of the vertices. We denote by W(S, T) the sum of the weights on all edges with one vertex in S and the other in T, given as W(S, T) = a ij v i S v j T Given S V, we denote by S the complementary set of vertices, that is, S = V S. A (vertex) cut in a graph is defined as a partitioning of V into S V and S. The weight of the cut or cut weight is defined as the sum of all the weights on edges between vertices in S and S, given as W(S, S). Chapter 16: Spectral & Graph Clustering 13 /
14 Cuts and Matrix Operations Given a clustering C = {C 1,...,C k } comprising k clusters. Let c i {0, 1} n be the cluster indicator vector that records the cluster membership for cluster C i, defined as { 1 if v j C i c ij = 0 if v j C i The cluster size can be written as C i = c T i c i = c i 2 The volume of a cluster C i is defined as the sum of all the weights on edges with one end in cluster C i : vol(c i ) = W(C i, V) = n n d r = c ir d rc ir = c ir rsc is = c T i c i v r C i v r C i r=1 s=1 The sum of weights of all internal edges is: W(C i, C i ) = v r C i v s C i a rs = n r=1 s=1 n c ir a rsc is = c T i Ac i We can get the sum of weights for all the external edges as follows: W(C i, C i ) = v r C i v s V C i a rs = W(C i, V) W(C i, C i ) = c i ( A)c i = c T i Lc i Chapter 16: Spectral & Graph Clustering 14 /
15 Clustering Objective Functions: Ratio Cut The clustering objective function can be formulated as an optimization problem over the k-way cut C = {C 1,...,C k }. The ratio cut objective is defined over a k-way cut as follows: min C J rc (C) = k i=1 W(C i, C i ) C i = k i=1 c T i Lc i c T i c i = k i=1 c T i Lc i c i 2 Ratio cut tries to minimize the sum of the similarities from a cluster C i to other points not in the cluster C i, taking into account the size of each cluster. Unfortunately, for binary cluster indicator vectors c i, the ratio cut objective is NP-hard. An obvious relaxation is to allow c i to take on any real value. In this case, we can rewrite the objective as min C J rc (C) = k i=1 c T i Lc i c i 2 = k i=1 ( ) T ( ) ci ci L = c i c i k u T i Lu i The optimal solution comprises the eigenvectors corresponding to the k smallest eigenvalues of L, i.e., the eigenvectors u n, u n 1,...,u n k+1 represent the relaxed cluster indicator vectors. i=1 Chapter 16: Spectral & Graph Clustering 15 /
16 Clustering Objective Functions: Normalized Cut Normalized cut is similar to ratio cut, except that it divides the cut weight of each cluster by the volume of a cluster instead of its size. The objective function is given as min C J nc (C) = k i=1 W(C i, C i ) vol(c i ) = k c T i Lc i c T i c i i=1 We can obtain an optimal solution by allowing c i to be an arbitrary real vector. The optimal solution comprise the eigenvectors corresponding to the k smallest eigenvalues of either the normalized symmetric or asymmetric Laplacian matrices, L s and L a. Chapter 16: Spectral & Graph Clustering 16 /
17 Spectral Clustering Algorithm The spectral clustering algorithm takes a dataset D as input and computes the similarity matrix A. For normalized cut we chose either L s or L a, whereas for ratio cut we choose L. Next, we compute the k smallest eigenvalues and corresponding eigenvectors of the chosen matrix. The main problem is that the eigenvectors u i are not binary, and thus it is not immediately clear how we can assign points to clusters. One solution to this problem is to treat the n k matrix of eigenvectors as a new data matrix: U = y T 1 u n u n 1 u n k+1 y T 2 normalize rows.. = Y y T n We then cluster the new points in Y into k clusters via the K-means algorithm or any other fast clustering method to obtain binary cluster indicator vectors c i. Chapter 16: Spectral & Graph Clustering 17 /
18 Spectral Clustering Algorithm SPECTRAL CLUSTERING (D, k): Compute the similarity matrix A R n n if ratio cut then B L else if normalized cut then B L s or L a Solve Bu i = λ i u i for i = n,...,n k + 1, where λ n λ n 1 λ n k+1 U ( ) u n u n 1 u n k+1 Y normalize rows of U C {C 1,...,C k } via K-means on Y Chapter 16: Spectral & Graph Clustering 18 /
19 Spectral Clustering on Example Graph k = 2, normalized cut (normalized asymmetric Laplacian) u 2 5 6, , u 1 Chapter 16: Spectral & Graph Clustering 19 /
20 Normalized Cut on Iris Graph k = 3, normalized asymmetric Laplacian setosa virginica versicolor C 1 (triangle) C 2 (square) C 3 (circle) Chapter 16: Spectral & Graph Clustering 20 /
21 Maximization Objectives: Average Cut The average weight objective is defined as max C J aw (C) = k i=1 W(C i, C i ) C i = k i=1 c T i Ac i c T i c i = k u T i Au i i=1 where u i is an arbitrary real vector, which is a relaxation of the binary cluster indicator vectors c i. We can maximize the objective by selecting the k largest eigenvalues of A, and the corresponding eigenvectors. max C J aw (C) = u T 1 Au 1 + +u T k Au k = λ 1 + +λ k where λ 1 λ 2 λ n. In general, while A is symmetric, it may not be positive semidefinite. This means that A can have negative eigenvalues, and to maximize the objective we must consider only the positive eigenvalues and the corresponding eigenvectors. Chapter 16: Spectral & Graph Clustering 21 /
22 Maximization Objectives: Modularity Given A, the weighted adjacency matrix, the modularity of a clustering is the difference between the observed and expected fraction of weights on edges within the clusters. The clustering objective is given as max C J Q (C) = k i=1 ( c T i Ac i tr( ) (dt i c i ) 2 ) tr( ) 2 = k c T i Qc i i=1 where Q is the modularity matrix: Q = 1 ) (A d dt tr( ) tr( ) The optimal solution comprises the eigenvectors corresponding to the k largest eigenvalues of Q. Since Q is symmetric, but not positive semidefinite, we use only the positive eigenvalues. Chapter 16: Spectral & Graph Clustering 22 /
23 Markov Chain Clustering A Markov chain is a discrete-time stochastic process over a set of states, in our case the set of vertices V. The Markov chain makes a transition from one node to another at discrete timesteps t = 1, 2,..., with the probability of making a transition from node i to node j given as m ij. Let the random variable X t denote the state at time t. The Markov property means that the probability distribution of X t over the states at time t depends only on the probability distribution of X t 1, that is, P(X t = i X 0, X 1,...,X t 1 ) = P(X t = i X t 1 ) Further, we assume that the Markov chain is homogeneous, that is, the transition probability is independent of the time step t. P(X t = j X t 1 = i) = m ij Chapter 16: Spectral & Graph Clustering 23 /
24 Markov Chain Clustering: Markov Matrix The normalized adjacency matrix M = 1 A can be interpreted as the n n transition matrix where the entry m ij = a ij d i is the probability of transitioning or jumping from node i to node j in the graph G. The matrix M is thus the transition matrix for a Markov chain or a Markov random walk on graph G. That is, Given node i the transition matrix M specifies the probabilities of reaching any other node j in one time step. In general, the transition probability matrix for t time steps is given as M t 1 M = M t Chapter 16: Spectral & Graph Clustering 24 /
25 Markov Chain Clustering: Random Walk A random walk on G thus corresponds to taking successive powers of the transition matrix M. Let π 0 specify the initial state probability vector at time t = 0. The state probability vector after t steps is π T t = π T t 1M = π T t 2M 2 = = π T 0 M t Equivalently, taking transpose on both sides, we get π t = (M t ) T π 0 = (M T ) t π 0 The state probability vector thus converges to the dominant eigenvector of M T. Chapter 16: Spectral & Graph Clustering 25 /
26 Markov Clustering Algorithm Consider a variation of the random walk, where the probability of transitioning from node i to j is inflated by taking each element m ij to the power r 1. Given a transition matrix M, define the inflation operator Υ as follows: Υ(M, r) = { (mij ) r n a=1 (m ia) r } n i,j=1 The net effect of the inflation operator is to increase the higher probability transitions and decrease the lower probability transitions. The Markov clustering algorithm (MCL) is an iterative method that interleaves matrix expansion and inflation steps. Matrix expansion corresponds to taking successive powers of the transition matrix, leading to random walks of longer lengths. On the other hand, matrix inflation makes the higher probability transitions even more likely and reduces the lower probability transitions. MCL takes as input the inflation parameter r 1. Higher values lead to more, smaller clusters, whereas smaller values lead to fewer, but larger clusters. Chapter 16: Spectral & Graph Clustering 26 /
27 Markov Clustering Algorithm: MCL The final clusters are found by enumerating the weakly connected components in the directed graph induced by the converged transition matrix M t, where the edges are defined as: E = { (i, j) M t (i, j) > 0 } A directed edge (i, j) exists only if node i can transition to node j within t steps of the expansion and inflation process. A node j is called an attractor if M t (j, j) > 0, and we say that node i is attracted to attractor j if M t (i, j) > 0. The MCL process yields a set of attractor nodes, V a V, such that other nodes are attracted to at least one attractor in V a. To extract the clusters from G t, MCL first finds the strongly connected components S 1, S 2,...,S q over the set of attractors V a. Next, for each strongly connected set of attractors S j, MCL finds the weakly connected components consisting of all nodes i V t V a attracted to an attractor in S j. If a node i is attracted to multiple strongly connected components, it is added to each such cluster, resulting in possibly overlapping clusters. Chapter 16: Spectral & Graph Clustering 27 /
28 Algorithm MARKOV CLUSTERING MARKOV CLUSTERING (A, r,ǫ): t 0 Add self-edges to A if they do not exist M t 1 A repeat t t + 1 M t M t 1 M t 1 M t Υ(M t, r) until M t M t 1 F ǫ G t directed graph induced by M t C {weakly connected components in G t } Chapter 16: Spectral & Graph Clustering 28 /
29 MCL Attractors and Clusters r = Chapter 16: Spectral & Graph Clustering 29 /
30 MCL on Iris Graph (a) r = 1.3 (b) r = 2 Chapter 16: Spectral & Graph Clustering 30 /
31 Contingency Table: MCL Clusters versus Iris Types r = 1.3 iris-setosa iris-virginica iris-versicolor C 1 (triangle) C 2 (square) C 3 (circle) Chapter 16: Spectral & Graph Clustering /
Data Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA Department
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationMachine Learning for Data Science (CS4786) Lecture 11
Machine Learning for Data Science (CS4786) Lecture 11 Spectral clustering Course Webpage : http://www.cs.cornell.edu/courses/cs4786/2016sp/ ANNOUNCEMENT 1 Assignment P1 the Diagnostic assignment 1 will
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki Wagner Meira Jr. Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA Department
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More information1 Matrix notation and preliminaries from spectral graph theory
Graph clustering (or community detection or graph partitioning) is one of the most studied problems in network analysis. One reason for this is that there are a variety of ways to define a cluster or community.
More informationSpectral Clustering. Spectral Clustering? Two Moons Data. Spectral Clustering Algorithm: Bipartioning. Spectral methods
Spectral Clustering Seungjin Choi Department of Computer Science POSTECH, Korea seungjin@postech.ac.kr 1 Spectral methods Spectral Clustering? Methods using eigenvectors of some matrices Involve eigen-decomposition
More information1 Matrix notation and preliminaries from spectral graph theory
Graph clustering (or community detection or graph partitioning) is one of the most studied problems in network analysis. One reason for this is that there are a variety of ways to define a cluster or community.
More informationSpectral Clustering. Zitao Liu
Spectral Clustering Zitao Liu Agenda Brief Clustering Review Similarity Graph Graph Laplacian Spectral Clustering Algorithm Graph Cut Point of View Random Walk Point of View Perturbation Theory Point of
More informationFinding normalized and modularity cuts by spectral clustering. Ljubjana 2010, October
Finding normalized and modularity cuts by spectral clustering Marianna Bolla Institute of Mathematics Budapest University of Technology and Economics marib@math.bme.hu Ljubjana 2010, October Outline Find
More informationSpectral Clustering. Guokun Lai 2016/10
Spectral Clustering Guokun Lai 2016/10 1 / 37 Organization Graph Cut Fundamental Limitations of Spectral Clustering Ng 2002 paper (if we have time) 2 / 37 Notation We define a undirected weighted graph
More informationSummary: A Random Walks View of Spectral Segmentation, by Marina Meila (University of Washington) and Jianbo Shi (Carnegie Mellon University)
Summary: A Random Walks View of Spectral Segmentation, by Marina Meila (University of Washington) and Jianbo Shi (Carnegie Mellon University) The authors explain how the NCut algorithm for graph bisection
More informationData Analysis and Manifold Learning Lecture 7: Spectral Clustering
Data Analysis and Manifold Learning Lecture 7: Spectral Clustering Radu Horaud INRIA Grenoble Rhone-Alpes, France Radu.Horaud@inrialpes.fr http://perception.inrialpes.fr/ Outline of Lecture 7 What is spectral
More informationSpectra of Adjacency and Laplacian Matrices
Spectra of Adjacency and Laplacian Matrices Definition: University of Alicante (Spain) Matrix Computing (subject 3168 Degree in Maths) 30 hours (theory)) + 15 hours (practical assignment) Contents 1. Spectra
More informationData Analysis and Manifold Learning Lecture 3: Graphs, Graph Matrices, and Graph Embeddings
Data Analysis and Manifold Learning Lecture 3: Graphs, Graph Matrices, and Graph Embeddings Radu Horaud INRIA Grenoble Rhone-Alpes, France Radu.Horaud@inrialpes.fr http://perception.inrialpes.fr/ Outline
More informationCOMPSCI 514: Algorithms for Data Science
COMPSCI 514: Algorithms for Data Science Arya Mazumdar University of Massachusetts at Amherst Fall 2018 Lecture 8 Spectral Clustering Spectral clustering Curse of dimensionality Dimensionality Reduction
More informationAlgebraic Representation of Networks
Algebraic Representation of Networks 0 1 2 1 1 0 0 1 2 0 0 1 1 1 1 1 Hiroki Sayama sayama@binghamton.edu Describing networks with matrices (1) Adjacency matrix A matrix with rows and columns labeled by
More informationStochastic processes. MAS275 Probability Modelling. Introduction and Markov chains. Continuous time. Markov property
Chapter 1: and Markov chains Stochastic processes We study stochastic processes, which are families of random variables describing the evolution of a quantity with time. In some situations, we can treat
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationLecture 13: Spectral Graph Theory
CSE 521: Design and Analysis of Algorithms I Winter 2017 Lecture 13: Spectral Graph Theory Lecturer: Shayan Oveis Gharan 11/14/18 Disclaimer: These notes have not been subjected to the usual scrutiny reserved
More informationLecture 1 and 2: Introduction and Graph theory basics. Spring EE 194, Networked estimation and control (Prof. Khan) January 23, 2012
Lecture 1 and 2: Introduction and Graph theory basics Spring 2012 - EE 194, Networked estimation and control (Prof. Khan) January 23, 2012 Spring 2012: EE-194-02 Networked estimation and control Schedule
More informationData Mining and Analysis: Fundamental Concepts and Algorithms
Data Mining and Analysis: Fundamental Concepts and Algorithms dataminingbook.info Mohammed J. Zaki 1 Wagner Meira Jr. 2 1 Department of Computer Science Rensselaer Polytechnic Institute, Troy, NY, USA
More informationMarkov Chains and Spectral Clustering
Markov Chains and Spectral Clustering Ning Liu 1,2 and William J. Stewart 1,3 1 Department of Computer Science North Carolina State University, Raleigh, NC 27695-8206, USA. 2 nliu@ncsu.edu, 3 billy@ncsu.edu
More informationSemidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5
Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5 Instructor: Farid Alizadeh Scribe: Anton Riabov 10/08/2001 1 Overview We continue studying the maximum eigenvalue SDP, and generalize
More informationMarkov Chains, Random Walks on Graphs, and the Laplacian
Markov Chains, Random Walks on Graphs, and the Laplacian CMPSCI 791BB: Advanced ML Sridhar Mahadevan Random Walks! There is significant interest in the problem of random walks! Markov chain analysis! Computer
More informationCommunities Via Laplacian Matrices. Degree, Adjacency, and Laplacian Matrices Eigenvectors of Laplacian Matrices
Communities Via Laplacian Matrices Degree, Adjacency, and Laplacian Matrices Eigenvectors of Laplacian Matrices The Laplacian Approach As with betweenness approach, we want to divide a social graph into
More informationCertifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering
Certifying the Global Optimality of Graph Cuts via Semidefinite Programming: A Theoretic Guarantee for Spectral Clustering Shuyang Ling Courant Institute of Mathematical Sciences, NYU Aug 13, 2018 Joint
More informationKnowledge Discovery and Data Mining 1 (VO) ( )
Knowledge Discovery and Data Mining 1 (VO) (707.003) Review of Linear Algebra Denis Helic KTI, TU Graz Oct 9, 2014 Denis Helic (KTI, TU Graz) KDDM1 Oct 9, 2014 1 / 74 Big picture: KDDM Probability Theory
More informationSpectral clustering. Two ideal clusters, with two points each. Spectral clustering algorithms
A simple example Two ideal clusters, with two points each Spectral clustering Lecture 2 Spectral clustering algorithms 4 2 3 A = Ideally permuted Ideal affinities 2 Indicator vectors Each cluster has an
More informationA lower bound for the Laplacian eigenvalues of a graph proof of a conjecture by Guo
A lower bound for the Laplacian eigenvalues of a graph proof of a conjecture by Guo A. E. Brouwer & W. H. Haemers 2008-02-28 Abstract We show that if µ j is the j-th largest Laplacian eigenvalue, and d
More informationSTA141C: Big Data & High Performance Statistical Computing
STA141C: Big Data & High Performance Statistical Computing Lecture 12: Graph Clustering Cho-Jui Hsieh UC Davis May 29, 2018 Graph Clustering Given a graph G = (V, E, W ) V : nodes {v 1,, v n } E: edges
More informationChapter 6. Eigenvalues. Josef Leydold Mathematical Methods WS 2018/19 6 Eigenvalues 1 / 45
Chapter 6 Eigenvalues Josef Leydold Mathematical Methods WS 2018/19 6 Eigenvalues 1 / 45 Closed Leontief Model In a closed Leontief input-output-model consumption and production coincide, i.e. V x = x
More informationLab 8: Measuring Graph Centrality - PageRank. Monday, November 5 CompSci 531, Fall 2018
Lab 8: Measuring Graph Centrality - PageRank Monday, November 5 CompSci 531, Fall 2018 Outline Measuring Graph Centrality: Motivation Random Walks, Markov Chains, and Stationarity Distributions Google
More informationLecture 12 : Graph Laplacians and Cheeger s Inequality
CPS290: Algorithmic Foundations of Data Science March 7, 2017 Lecture 12 : Graph Laplacians and Cheeger s Inequality Lecturer: Kamesh Munagala Scribe: Kamesh Munagala Graph Laplacian Maybe the most beautiful
More informationHYPERGRAPH BASED SEMI-SUPERVISED LEARNING ALGORITHMS APPLIED TO SPEECH RECOGNITION PROBLEM: A NOVEL APPROACH
HYPERGRAPH BASED SEMI-SUPERVISED LEARNING ALGORITHMS APPLIED TO SPEECH RECOGNITION PROBLEM: A NOVEL APPROACH Hoang Trang 1, Tran Hoang Loc 1 1 Ho Chi Minh City University of Technology-VNU HCM, Ho Chi
More informationStatistical Learning Notes III-Section 2.4.3
Statistical Learning Notes III-Section 2.4.3 "Graphical" Spectral Features Stephen Vardeman Analytics Iowa LLC January 2019 Stephen Vardeman (Analytics Iowa LLC) Statistical Learning Notes III-Section
More information7. Symmetric Matrices and Quadratic Forms
Linear Algebra 7. Symmetric Matrices and Quadratic Forms CSIE NCU 1 7. Symmetric Matrices and Quadratic Forms 7.1 Diagonalization of symmetric matrices 2 7.2 Quadratic forms.. 9 7.4 The singular value
More information17.1 Directed Graphs, Undirected Graphs, Incidence Matrices, Adjacency Matrices, Weighted Graphs
Chapter 17 Graphs and Graph Laplacians 17.1 Directed Graphs, Undirected Graphs, Incidence Matrices, Adjacency Matrices, Weighted Graphs Definition 17.1. A directed graph isa pairg =(V,E), where V = {v
More informationSVD, Power method, and Planted Graph problems (+ eigenvalues of random matrices)
Chapter 14 SVD, Power method, and Planted Graph problems (+ eigenvalues of random matrices) Today we continue the topic of low-dimensional approximation to datasets and matrices. Last time we saw the singular
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationMath 1553, Introduction to Linear Algebra
Learning goals articulate what students are expected to be able to do in a course that can be measured. This course has course-level learning goals that pertain to the entire course, and section-level
More informationLecture 14: Random Walks, Local Graph Clustering, Linear Programming
CSE 521: Design and Analysis of Algorithms I Winter 2017 Lecture 14: Random Walks, Local Graph Clustering, Linear Programming Lecturer: Shayan Oveis Gharan 3/01/17 Scribe: Laura Vonessen Disclaimer: These
More informationAPPENDIX A. Background Mathematics. A.1 Linear Algebra. Vector algebra. Let x denote the n-dimensional column vector with components x 1 x 2.
APPENDIX A Background Mathematics A. Linear Algebra A.. Vector algebra Let x denote the n-dimensional column vector with components 0 x x 2 B C @. A x n Definition 6 (scalar product). The scalar product
More informationGraph fundamentals. Matrices associated with a graph
Graph fundamentals Matrices associated with a graph Drawing a picture of a graph is one way to represent it. Another type of representation is via a matrix. Let G be a graph with V (G) ={v 1,v,...,v n
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector
More informationMining of Massive Datasets Jure Leskovec, AnandRajaraman, Jeff Ullman Stanford University
Note to other teachers and users of these slides: We would be delighted if you found this our material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify them to fit
More informationSpectral Theory of Unsigned and Signed Graphs Applications to Graph Clustering. Some Slides
Spectral Theory of Unsigned and Signed Graphs Applications to Graph Clustering Some Slides Jean Gallier Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104,
More informationStrongly Regular Graphs, part 1
Spectral Graph Theory Lecture 23 Strongly Regular Graphs, part 1 Daniel A. Spielman November 18, 2009 23.1 Introduction In this and the next lecture, I will discuss strongly regular graphs. Strongly regular
More informationPhysical Metaphors for Graphs
Graphs and Networks Lecture 3 Physical Metaphors for Graphs Daniel A. Spielman October 9, 203 3. Overview We will examine physical metaphors for graphs. We begin with a spring model and then discuss resistor
More informationMATH 829: Introduction to Data Mining and Analysis Clustering II
his lecture is based on U. von Luxburg, A Tutorial on Spectral Clustering, Statistics and Computing, 17 (4), 2007. MATH 829: Introduction to Data Mining and Analysis Clustering II Dominique Guillot Departments
More informationGraph Partitioning Using Random Walks
Graph Partitioning Using Random Walks A Convex Optimization Perspective Lorenzo Orecchia Computer Science Why Spectral Algorithms for Graph Problems in practice? Simple to implement Can exploit very efficient
More informationMa/CS 6b Class 23: Eigenvalues in Regular Graphs
Ma/CS 6b Class 3: Eigenvalues in Regular Graphs By Adam Sheffer Recall: The Spectrum of a Graph Consider a graph G = V, E and let A be the adjacency matrix of G. The eigenvalues of G are the eigenvalues
More informationSpectral Graph Theory and its Applications. Daniel A. Spielman Dept. of Computer Science Program in Applied Mathematics Yale Unviersity
Spectral Graph Theory and its Applications Daniel A. Spielman Dept. of Computer Science Program in Applied Mathematics Yale Unviersity Outline Adjacency matrix and Laplacian Intuition, spectral graph drawing
More informationSpectral Graph Theory and You: Matrix Tree Theorem and Centrality Metrics
Spectral Graph Theory and You: and Centrality Metrics Jonathan Gootenberg March 11, 2013 1 / 19 Outline of Topics 1 Motivation Basics of Spectral Graph Theory Understanding the characteristic polynomial
More informationClustering compiled by Alvin Wan from Professor Benjamin Recht s lecture, Samaneh s discussion
Clustering compiled by Alvin Wan from Professor Benjamin Recht s lecture, Samaneh s discussion 1 Overview With clustering, we have several key motivations: archetypes (factor analysis) segmentation hierarchy
More informationLecture 1: Graphs, Adjacency Matrices, Graph Laplacian
Lecture 1: Graphs, Adjacency Matrices, Graph Laplacian Radu Balan January 31, 2017 G = (V, E) An undirected graph G is given by two pieces of information: a set of vertices V and a set of edges E, G =
More informationThe maximal stable set problem : Copositive programming and Semidefinite Relaxations
The maximal stable set problem : Copositive programming and Semidefinite Relaxations Kartik Krishnan Department of Mathematical Sciences Rensselaer Polytechnic Institute Troy, NY 12180 USA kartis@rpi.edu
More informationCS168: The Modern Algorithmic Toolbox Lectures #11 and #12: Spectral Graph Theory
CS168: The Modern Algorithmic Toolbox Lectures #11 and #12: Spectral Graph Theory Tim Roughgarden & Gregory Valiant May 2, 2016 Spectral graph theory is the powerful and beautiful theory that arises from
More informationMATH6142 Complex Networks Exam 2016
MATH642 Complex Networks Exam 26 Solution Problem a) The network is directed because the adjacency matrix is not symmetric. b) The network is shown in Figure. (4 marks)[unseen] 4 3 Figure : The directed
More informationAn indicator for the number of clusters using a linear map to simplex structure
An indicator for the number of clusters using a linear map to simplex structure Marcus Weber, Wasinee Rungsarityotin, and Alexander Schliep Zuse Institute Berlin ZIB Takustraße 7, D-495 Berlin, Germany
More information22m:033 Notes: 7.1 Diagonalization of Symmetric Matrices
m:33 Notes: 7. Diagonalization of Symmetric Matrices Dennis Roseman University of Iowa Iowa City, IA http://www.math.uiowa.edu/ roseman May 3, Symmetric matrices Definition. A symmetric matrix is a matrix
More information8.1 Concentration inequality for Gaussian random matrix (cont d)
MGMT 69: Topics in High-dimensional Data Analysis Falll 26 Lecture 8: Spectral clustering and Laplacian matrices Lecturer: Jiaming Xu Scribe: Hyun-Ju Oh and Taotao He, October 4, 26 Outline Concentration
More informationBasic graph theory 18.S995 - L26.
Basic graph theory 18.S995 - L26 dunkel@math.mit.edu http://java.dzone.com/articles/algorithm-week-graphs-and no cycles Isomorphic graphs f(a) = 1 f(b) = 6 f(c) = 8 f(d) = 3 f(g) = 5 f(h) = 2 f(i) = 4
More informationMath 102, Winter Final Exam Review. Chapter 1. Matrices and Gaussian Elimination
Math 0, Winter 07 Final Exam Review Chapter. Matrices and Gaussian Elimination { x + x =,. Different forms of a system of linear equations. Example: The x + 4x = 4. [ ] [ ] [ ] vector form (or the column
More informationFundamentals of Engineering Analysis (650163)
Philadelphia University Faculty of Engineering Communications and Electronics Engineering Fundamentals of Engineering Analysis (6563) Part Dr. Omar R Daoud Matrices: Introduction DEFINITION A matrix is
More information3 Best-Fit Subspaces and Singular Value Decomposition
3 Best-Fit Subspaces and Singular Value Decomposition (SVD) Think of the rows of an n d matrix A as n data points in a d-dimensional space and consider the problem of finding the best k-dimensional subspace
More informationLecture 5: Random Walks and Markov Chain
Spectral Graph Theory and Applications WS 20/202 Lecture 5: Random Walks and Markov Chain Lecturer: Thomas Sauerwald & He Sun Introduction to Markov Chains Definition 5.. A sequence of random variables
More informationRelations Between Adjacency And Modularity Graph Partitioning: Principal Component Analysis vs. Modularity Component Analysis
Relations Between Adjacency And Modularity Graph Partitioning: Principal Component Analysis vs. Modularity Component Analysis Hansi Jiang Carl Meyer North Carolina State University October 27, 2015 1 /
More informationDiffusion and random walks on graphs
Diffusion and random walks on graphs Leonid E. Zhukov School of Data Analysis and Artificial Intelligence Department of Computer Science National Research University Higher School of Economics Structural
More informationWeb Structure Mining Nodes, Links and Influence
Web Structure Mining Nodes, Links and Influence 1 Outline 1. Importance of nodes 1. Centrality 2. Prestige 3. Page Rank 4. Hubs and Authority 5. Metrics comparison 2. Link analysis 3. Influence model 1.
More informationLECTURE NOTE #11 PROF. ALAN YUILLE
LECTURE NOTE #11 PROF. ALAN YUILLE 1. NonLinear Dimension Reduction Spectral Methods. The basic idea is to assume that the data lies on a manifold/surface in D-dimensional space, see figure (1) Perform
More informationChapter 4. Signed Graphs. Intuitively, in a weighted graph, an edge with a positive weight denotes similarity or proximity of its endpoints.
Chapter 4 Signed Graphs 4.1 Signed Graphs and Signed Laplacians Intuitively, in a weighted graph, an edge with a positive weight denotes similarity or proximity of its endpoints. For many reasons, it is
More informationCommunities, Spectral Clustering, and Random Walks
Communities, Spectral Clustering, and Random Walks David Bindel Department of Computer Science Cornell University 26 Sep 2011 20 21 19 16 22 28 17 18 29 26 27 30 23 1 25 5 8 24 2 4 14 3 9 13 15 11 10 12
More informationGAUSSIAN PROCESS TRANSFORMS
GAUSSIAN PROCESS TRANSFORMS Philip A. Chou Ricardo L. de Queiroz Microsoft Research, Redmond, WA, USA pachou@microsoft.com) Computer Science Department, Universidade de Brasilia, Brasilia, Brazil queiroz@ieee.org)
More informationLinear Algebra. Matrices Operations. Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0.
Matrices Operations Linear Algebra Consider, for example, a system of equations such as x + 2y z + 4w = 0, 3x 4y + 2z 6w = 0, x 3y 2z + w = 0 The rectangular array 1 2 1 4 3 4 2 6 1 3 2 1 in which the
More informationBoolean Inner-Product Spaces and Boolean Matrices
Boolean Inner-Product Spaces and Boolean Matrices Stan Gudder Department of Mathematics, University of Denver, Denver CO 80208 Frédéric Latrémolière Department of Mathematics, University of Denver, Denver
More informationData-dependent representations: Laplacian Eigenmaps
Data-dependent representations: Laplacian Eigenmaps November 4, 2015 Data Organization and Manifold Learning There are many techniques for Data Organization and Manifold Learning, e.g., Principal Component
More information(a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? Solution: dim N(A) 1, since rank(a) 3. Ax =
. (5 points) (a) If A is a 3 by 4 matrix, what does this tell us about its nullspace? dim N(A), since rank(a) 3. (b) If we also know that Ax = has no solution, what do we know about the rank of A? C(A)
More informationLinear Algebra review Powers of a diagonalizable matrix Spectral decomposition
Linear Algebra review Powers of a diagonalizable matrix Spectral decomposition Prof. Tesler Math 283 Fall 2018 Also see the separate version of this with Matlab and R commands. Prof. Tesler Diagonalizing
More informationSpectral Graph Theory Lecture 2. The Laplacian. Daniel A. Spielman September 4, x T M x. ψ i = arg min
Spectral Graph Theory Lecture 2 The Laplacian Daniel A. Spielman September 4, 2015 Disclaimer These notes are not necessarily an accurate representation of what happened in class. The notes written before
More informationMATH 567: Mathematical Techniques in Data Science Clustering II
This lecture is based on U. von Luxburg, A Tutorial on Spectral Clustering, Statistics and Computing, 17 (4), 2007. MATH 567: Mathematical Techniques in Data Science Clustering II Dominique Guillot Departments
More informationQuick Tour of Linear Algebra and Graph Theory
Quick Tour of Linear Algebra and Graph Theory CS224W: Social and Information Network Analysis Fall 2014 David Hallac Based on Peter Lofgren, Yu Wayne Wu, and Borja Pelato s previous versions Matrices and
More informationTopics. Vectors (column matrices): Vector addition and scalar multiplication The matrix of a linear function y Ax The elements of a matrix A : A ij
Topics Vectors (column matrices): Vector addition and scalar multiplication The matrix of a linear function y Ax The elements of a matrix A : A ij or a ij lives in row i and column j Definition of a matrix
More informationThe University of Texas at Austin Department of Electrical and Computer Engineering. EE381V: Large Scale Learning Spring 2013.
The University of Texas at Austin Department of Electrical and Computer Engineering EE381V: Large Scale Learning Spring 2013 Assignment Two Caramanis/Sanghavi Due: Tuesday, Feb. 19, 2013. Computational
More information1 Last time: least-squares problems
MATH Linear algebra (Fall 07) Lecture Last time: least-squares problems Definition. If A is an m n matrix and b R m, then a least-squares solution to the linear system Ax = b is a vector x R n such that
More information, some directions, namely, the directions of the 1. and
11. Eigenvalues and eigenvectors We have seen in the last chapter: for the centroaffine mapping, some directions, namely, the directions of the 1 coordinate axes: and 0 0, are distinguished 1 among all
More informationDATA MINING LECTURE 8. Dimensionality Reduction PCA -- SVD
DATA MINING LECTURE 8 Dimensionality Reduction PCA -- SVD The curse of dimensionality Real data usually have thousands, or millions of dimensions E.g., web documents, where the dimensionality is the vocabulary
More informationRelations Graphical View
Introduction Relations Computer Science & Engineering 235: Discrete Mathematics Christopher M. Bourke cbourke@cse.unl.edu Recall that a relation between elements of two sets is a subset of their Cartesian
More informationLecture 13 Spectral Graph Algorithms
COMS 995-3: Advanced Algorithms March 6, 7 Lecture 3 Spectral Graph Algorithms Instructor: Alex Andoni Scribe: Srikar Varadaraj Introduction Today s topics: Finish proof from last lecture Example of random
More informationCS 246 Review of Linear Algebra 01/17/19
1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector
More informationCSC Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming
CSC2411 - Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming Notes taken by Mike Jamieson March 28, 2005 Summary: In this lecture, we introduce semidefinite programming
More information1 9/5 Matrices, vectors, and their applications
1 9/5 Matrices, vectors, and their applications Algebra: study of objects and operations on them. Linear algebra: object: matrices and vectors. operations: addition, multiplication etc. Algorithms/Geometric
More informationNetworks and Their Spectra
Networks and Their Spectra Victor Amelkin University of California, Santa Barbara Department of Computer Science victor@cs.ucsb.edu December 4, 2017 1 / 18 Introduction Networks (= graphs) are everywhere.
More information