Manifold Learning and it s application

Transcription

1 Manifold Learning and it s application Nandan Dubey SE367

2 Outline 1 Introduction Manifold Examples image as vector Importance Dimension Reduction Techniques 2 Linear Methods PCA Example MDS Perception 3 Non-Linear Techniques 4 LLE 5 Isomap 6 References Further Reading

3 Manifold What is Manifold? Definition Manifold is a mathematical space that on a small enough scale resembles the Euclidean space of a specific dimension. Hence a line and a circle are one-dimensional manifolds, a plane and sphere (the surface of a ball) are two-dimensional manifolds.

4 Examples Introduction contd. S -shape Swiss roll

5 Examples Introduction contd. S -shape Swiss roll Both are two-dim. data embedded in 3D

6 image as vector Images as Vectors

7 Importance Why reduce Dimension? Curse of Dimensionality. Image data(each pixel), spectral coefficients, Text categorization(frequencies of phrases in a document), genes, many more. Practical data lie in subspace which is low dimensional.

8 Dimension Reduction Techniques Dimension reduction techniques Linear methods Principal Component Analysis(PCA) Multidimensional Scaling(MDS) Non-linear Methods Locally Linear Embedding (Roweis and Saul) IsoMap (Tenenbaum, de Silva, and Langford)

9 PCA PCA-Principal Component Analysis PCA Transforms possibly correlated variables into a smaller dimensional uncorrelated variables called principal components. Find linear subspace projection P which preserves the data locations (under quadratic error) Algorithm Evaluate Covariance matrix C = (x µ x )(x µ x ) T Simple eigenvector(v q ) solution Top q eigenvectors of XCC T X T V q =[v 1,... v q ] is a basis for the q-dim subspace Locations given by V T q XC

10 Example

11 Example PCA - Algorithm n is number of data points, D is dimension of input, d is output dimension. Data = {X 1, X 2,..., X n }, X i ɛr D, i = 1,... n goal is y i ɛr d, i =1,..., n d << D minimize reconstruction error min PɛR mxm n i=1 X i PX i 2 where P = U U T UɛR Dxd maximize the variance max y i y j 2 Then y i = U T X i

12 MDS Multidimensional Scaling Given pre-distances D Find Euclidean q-dim space which preserves these distances Solution is given by the eigenstructure of Top q eigenvectors This is exactly the same solution as PCA

13 MDS Deficiencies of Linear Methods Data may not be best summarized by linear combination of features Example: PCA cannot discover 1D structure of a helix

14 Perception How brain store these? Every pixel? Or perceptually meaningful structure?

15 Perception How brain store these? Every pixel? Or perceptually meaningful structure? : Up-down pose Left-right pose Lighting direction So, our brain successfully reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!

16 LLE and Isomap Local relationships: Two solutions which preserve local structure: LLE Locally Linear Embedding (LLE) : Change to a local representation (at each point) Base the local rep. on position of neighboring points Isomap IsoMap: Estimate actual (geodesic) distances in p-dim. space Find q-dim representation preserving those distances

17 Both rely on the locally flat nature of the manifold How do we find a locality in which this is true?

18 Both rely on the locally flat nature of the manifold How do we find a locality in which this is true? k-nearest-neighbors

19 Both rely on the locally flat nature of the manifold How do we find a locality in which this is true? k-nearest-neighbors epsilon-ball

20 Local Linear Embedding Select a local neighborhood Change each point into a coordinate system based on its neighbors Find new (q-dim) coordinates which reproduce these local relationships

21 LLE by image

22 Locally Linear Embedding Find new (q-dim) coordinates which reproduce these local coordinates. This can be solved using the eigenstructure as well: We want the min. variance

23 Locally Linear Embedding minimize n i=1 X i k j=1 W ijx j 2 W is n x n sparse matrix minimize n i=1 y i k j=1 W ijy j 2 or n n i=1 j=1 M ijyi T y i M = n x n matrix M = (I - W) T (I W )

24 Isomap Distance Matrix Build a graph with K-nearest neighbor or ɛ nearest neighbor. Geodesic Distance Infer other interpoint distances by finding shortest paths on the graph (Dijkstra s algorithm) MDS Apply MDS to embed the nearest neighbors.

25 an Application

26 an Application Linear Interpolation

27 an Application Interpolation By Manifold

28 another Application(Isomap)

29 other Applications Example Pose estimation image denoising missing data interpolation

30 Image References Learning-based Methods in Vision Alexei Efros, CMU, Spring 2007 Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul

31 Further Reading Further Reading MDS : Encyclopedia of Cognitive Science Multidimensional Scaling Mark Steyvers PCA: Jonathon Shlens, A Tutorial on Principal Component Analysis or refer Wikipedia ISOMAP: A Global Geometric Framework for Nonlinear Dimensionality Reduction Joshua B. Tenenbaum, Vin de Silva, John C. Langford LLE: Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul

32 Further Reading Further Reading Thank you! ndubey/thesis