Kernels March 13, 2008 Paper: R.R. Coifman, S. Lafon, maps ([Coifman06]) Seminar: Learning with Graphs, Prof. Hein, Saarland University
Kernels Figure: Example Application from [LafonWWW] meaningful geometric descriptions of data sets data parametrization (demo) dimensionality reduction
Kernels 1 Table of Contents 2 Application of to 3 Limit of the Graph Kernels 4 5 and Appendix
Kernels 1 2 Application of to 3 Limit of the Graph Kernels 4 5 and Appendix
Process on a Graph Kernels : Kernel k : X X R on data set X Symmetric: k(x, y) = k(y, x) positivity preserving:k(x, y) 0 represents similarity between points in X defines edge weight matrix W in weighted graph (X, k) Normalized Graph L (rw) = P d(x) = X k(x, y)dµ(y) (degree, discrete: d i = n j=1 w ij) p(x, y) = k(x,y) d(x) ( p(x, y)dµ(y) = 1) X p(x, y) = transition probability in one time step Matrix P defines Markov chain :P t = probability of transition from x to y in t steps
Kernels Process on a Graph : a cluster is a region in which the probability of escaping this region is low. Figure: at time t = 8, right: P 8, left: color from one row of P 8
Kernels Process on a Graph Block structure of P reveals clusters, after 64 steps the closest two clusters have merged Figure: at time t = 64
Process on a Graph All clusters have merged after 1024 time steps Kernels Figure: at time t = 1024
Kernels Distance Goal: relate spectral properties of Markov chain to geometry of the data : distances {D t } t N D t (x, y) 2 p t (x, ) p t (y, ) L 2 (X,dµ/π) D t (x, y) will be small, if there is a large number of short paths between x and y Figure: Example paths for diffusion distance [Maggioni06]
Kernels distances can be computed using eigenvectors ψ l and eigenvalues λ l of P ( ) 1 D t (x, y) = l 1 λ2t l (ψ l (x) ψ l (y)) 2 2 The proof uses the spectral theorem in the Hilbert space (more later) and the fact that the eigenfunctions of P are orthonormal. Using 1 = λ 0 > λ 1 λ 2... the distance may be approximated with the first s eigenvalues.
Kernels : Map Ψ t (x) : X R s Ψ t (x) λ t 1 ψ 1(x) λ t 2 ψ 2(x). λ t sψ s (x) Proposition: The diffusion map Ψ t embeds the data into the Euclidean space R s so that in this space, the Euclidean distance is equal to the diffusion distance (up to relative accuracy), or equivalently Ψ t (x) Ψ t (y) = D t (x, y) Remark: Unlike Eigenmaps, each dimension is weighted by the decreasing eigenvalues.
Example of Eigenfunctions ψ Kernels Figure: First 4 eigenfunctions of a dumb-bell shaped manifold. [Maggioni06]
Kernels is one possible application of diffusion maps. 1 Construct similarity graph 2 Compute normalized 3 Solve generalized eigenvector problem Lu = λdu 4 Define the embedding into k-dimensional Euclidean space via diffusion maps 5 Cluster points y i R k with k-means
Kernels 1 2 Application of to 3 Limit of the Graph Kernels 4 5 and Appendix
New Scenario Kernels Points are sampled from a probability density on a submanifold of R n Sampling often not related to geometry of manifold. Biased data: e.g. more faces from one pose Goal: Recover manifold structure regardless of the distribution of the data Additional concepts needed for this continuous setting Figure: Manifold with density [Learned07]
The Continuous Setting Kernels : Manifold: A space in which every point has a neighborhood which resembles Euclidean space, but in which the global structure may be more complicated, e.g. sphere or 2-d surface : Hilbert Space: An inner product space X on a space S that is complete under the norm f = f, f defined by the inner product,. For example the L 2 norm: f, g = S f (x)g(x)dx Ability to define functions f using a function basis (Φ i ): f = α i Φ i (x) Orthonormal basis similar to vector space: Φ i = 1 i and Φ i, Φ j = 0 i j
The Continuous Setting Kernels : L : X X A function of functions Linear operator: L(λf ) = λlf Eigenfunction of an operator:lf = λf An operator is Hermitian (symmetric) if Lf, g = f, Lg Eigenfunctions of Hermitian operators form an orthonormal basis on their Hilbert space X. Example:Laplace
Kernels : The Laplace Interesting properties: f = n 2 f x 2 i=1 i First eigenfunction is constant ( 2 c x 2 i = 0), Second eigenfunction has to change signs (orthogonal to first) and needs to be only scaled by operator:...
Kernels : The Laplace Interesting properties: f = n 2 f x 2 i=1 i First eigenfunction is constant ( 2 c = 0), xi 2 Second eigenfunction has to change signs (orthogonal to first) and needs to be only scaled by operator:... sine and cosine, since (sin(ωx)) = ω 2 sin(ωx) Hence, the eigenfunctions of form a nice orthonormal basis in X. The is extension of normal to manifolds. Problem: We only have a finite sample from a probability measure p on an m-dimensional submanifold M in R d.
Kernels Limit of the Graph Theorem: Let M be a m-dimensional submanifold in R d, {X i } n i=1 a sample from a probability measure P on M with density p. Then under several conditions on M, p and the kernel k, we have: If neighborhood h 0, number of points in it n and nh m+2 / log n, then the random walk converges to the operator lim n (L(rw) n f )(x) ( s f )(x) Where the weighted operator s = M + s p, f p induces an anisotropic diffusion towards or away from increasing density depending on s.
Kernels Now, we have established ourselves in the awesome world of operators in Hilbert Spaces on submanifolds. Initial motivation was to analyze geometry regardless of sampling distribution. What is the influence of the geometry and the density over the eigenfunctions and the spectrum of the diffusion?
Kernels We introduced a family of weighted operators that allow a scaling of the influence of the density via one parameter s: s = M + s p p, f The smoothness functional induced by s is: S(f ) = f 2 p s dv M and is to be minimized. ( to graph : i,j w ij(fi fj) 2 ) Hence, this functional prefers functions that are smooth in high density regions (see board).
Kernels Construction of the Family of s Now, we have the tools to define a new kernel for the weights of normalized graph s. But what exactly has changed in the construction of diffusion kernels?
Kernels Standard Normalized Graph Fix kernel k(x, y) - - - - d(x) = k(x, y)dµ(y) X p(x, y) = k(x,y) d(x) Kernels Kernel Normalized Graph Fix kernel k(x, y) Renormalize weight into new anisotropic kernel: q(x) = X k(x, y)q(y)dy k (α) = k(x,y) q α (x)q α (y) d (α) (x) = X k(α) (x, y)q(y)dy p (α) = k(α) (x,y) d (α) (x) α = 0: Construct normalized weights for graph α = 1: approximation, the normalization removes the influence of the density and recovers the geometry of the data. s = 2(1 α)
Kernels is one possible application of diffusion maps. 1 Construct similarity graph Apply normalization 2 Compute normalized 3 Solve generalized eigenvector problem Lu = λdu 4 Define the embedding into k-dimensional Euclidean space via diffusion maps 5 Cluster points y i R k with k-means
Kernels Embeddings via Approximation Figure: From left to right: original curves, the densities of points, the embeddings via the graph (α = 0) and the embeddings via the Laplace Beltrami approximation (α = 1). In the latter case, the curve is embedded as a perfect circle and the arclength parametrization is recovered.
Parametrization of Curves Kernels Figure: Live Example from [LafonWWW]
Kernels 1. Map: Ψ t (x) : X R s Ψ t (x) λ t 1 ψ 1(x) λ t 2 ψ 2(x). λ t sψ s (x) data into the Euclidean space so that: Ψ t (x) Ψ t (y) = D t (x, y) 2. Approximation: k (α) = k(x,y) q α (x)q α (y) normalization parameter α steers the influence of the density allows the complete separation of the distribution of the data from the geometry of the underlying manifold
Kernels Coifman06: R.R. Coifman, S. Lafon, maps, Appl. Comput. Harmon. Anal. 21 (1) (2006) 631. LafonWWW: Stephane Lafon s website: http: // www. math. yale. edu/ ~ sl349/ Luxburg07: von Luxburg, U.: A Tutorial on. Statistics and Computing 17(4), 395-416 (12 2007) Hein07: M. Hein, J.-Y. Audibert, U. von Luxburg. Convergence of graph s on random neighborhood graphs, Journal of Machine Learning Research 8, 1325-1370, 2007. Learned07: Manifold Picture from http: // www. cs. umass. edu/ ~ elm/ papers_ by_ research. html Maggioni06: and wavelet bases for value function approximation and their connection to kernel methods, Mauro Maggioni, Yale University, ICML Workshop, June 29th, 2006
Thank you. Kernels