Is Manifold Learning for Toy Data only?

Transcription

1 s Manifold Learning for Toy Data only? Marina Meilă University of Washington MMDS Workshop 2016

2 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

3 When to do (non-linear) dimension reduction high-dimensional data p 2 R D, D =64 64 can be described by a small number d of continuous parameters Usually, large sample size n

4 When to do (non-linear) dimension reduction Why? To save space and computation n D data matrix! n s, s D To understand the data better preserve large scale features, suppress fine scale features To use it afterwards in (prediction) tasks

5 When to do (non-linear) dimension reduction Why? To save space and computation n D data matrix! n s, s D To understand the data better preserve large scale features, suppress fine scale features To use it afterwards in (prediction) tasks

6 When to do (non-linear) dimension reduction Richard Powell - The Hertzsprung Russell Diagram, CC BY-SA 2.5, Why? To save space and computation n D data matrix! n s, s D To understand the data better preserve large scale features, suppress fine scale features To use it afterwards in (prediction) tasks

7 How? Brief intro to manifold learning algorithms nput Data p 1,...p n,embeddingdimensionm, neighborhoodscale parameter p 1,...p n R D

8 How? Brief intro to manifold learning algorithms nput Data p 1,...p n,embeddingdimensionm, neighborhoodscale parameter Construct neighborhood graph p, p 0 neighbors i p p 0 2 apple p 1,...p n R D

9 How? Brief intro to manifold learning algorithms nput Data p 1,...p n,embeddingdimensionm, neighborhoodscale parameter Construct neighborhood graph p, p 0 neighbors i p p 0 2 apple Construct a n n matrix its leading eigenvectors are the coordinates (p 1:n) p 1,...p n R D

10 How? Brief intro to manifold learning algorithms nput Data p 1,...p n,embeddingdimensionm, neighborhoodscale parameter Construct neighborhood graph p, p 0 neighbors i p p 0 2 apple Construct a n n matrix its leading eigenvectors are the coordinates (p 1:n) Laplacian Eigenmaps [Belkin & Niyogi 02] Construct similarity matrix S =[S pp 0] p,p 0 2D withs pp 0 = e 1 p p0 2 i p, p 0 neighbors Construct Laplacian matrix L = T 1 S with T = diag(s1) Calculate 1...m = eigenvectors of L (smallest eigenvalues) coordinates of p 2Dare ( 1 (p),... m (p))

11 How? Brief intro to manifold learning algorithms nput Data p 1,...p n,embeddingdimensionm, neighborhoodscale parameter Construct neighborhood graph p, p 0 neighbors i p p 0 2 apple Construct a n n matrix its leading eigenvectors are the coordinates (p 1:n) somap [[Tennenbaum, desilva & Langford 00]] Find all shortest paths in neighborhood graph, construct matrix of distances M =[distance 2 pp 0] use M and Multi-Dimensional Scaling (MDS) to obtain m dimensional coordinates for p 2D

12 Atoyexample(the SwissRoll withahole) points in D 3dimensions samepointsreparametrizedin2d

13 Atoyexample(the SwissRoll withahole) points in D 3dimensions samepointsreparametrizedin2d nput Desired output

14 Embedding in 2 dimensions by di erent manifold learning algorithms nput

15 How to evaluate the results objectively? which of these embedding are correct? if several correct, how do we reconcile them? if not correct, what failed? Algorithms Multidimensional Scaling (MDS), Principal Components (PCA), somap, Locally Linear Embedding (LLE), Hessian Eigenmaps (HE), Laplacian Eigenmaps (LE), Di usion Maps (DM)

16 How to evaluate the results objectively? which of these embedding are correct? if several correct, how do we reconcile them? if not correct, what failed? what if have real data? Spectrum of a galaxy. Source SDSS, Jake VanderPlas

17 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

18 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

19 Preserving topology vs. preserving (intrinsic) geometry Algorithm maps data p 2 R D! (p) =x 2 R m Mapping M! (M) isdi eomorphism preserves topology often satisfied by embedding algorithms Mapping preserves distances along curves in M angles between curves in M areas, volumes

20 Preserving topology vs. preserving (intrinsic) geometry Algorithm maps data p 2 R D! (p) =x 2 R m Mapping M! (M) isdi eomorphism preserves topology often satisfied by embedding algorithms Mapping preserves distances along curves in M angles between curves in M areas, volumes...i.e. is isometry For most algorithms, in most cases, is not isometry Preserves topology Preserves topology + intrinsic geometry

21 Previous known results in geometric recovery Positive results Nash s Theorem: sometric embedding is possible. algorithm based on Nash s theorem (isometric embedding for very low d) [Verma 11] somap recovers (only) flat manifolds isometrically Consistency results for Laplacian and eigenvectors [[Hein & al 07,Coifman & Lafon 06, Ting & al 10, Gine & Koltchinskii 06]] imply isometric recovery for LE, DM in special situations Negative results obvious negative examples No a ne recovery for normalized Laplacian algorithms [Goldberg&al 08] Sampling density distorts the geometry for LE [Coifman& Lafon 06]

22 Our approach: Metric Manifold Learning [Perrault-Joncas,M 10] Given mapping that preserves topology true in many cases Objective augment with geometric information g so that (, g) preserves the geometry Dominique Perrault-Joncas

23 Our approach: Metric Manifold Learning [Perrault-Joncas,M 10] Given mapping that preserves topology true in many cases Objective augment with geometric information g so that (, g) preserves the geometry g is the Riemannian metric. Dominique Perrault-Joncas

24 The Riemannian metric g Mathematically M = (smooth) manifold p point on M T pm = tangent subspace at p g =Riemannianmetricon M g defines inner product on T pm < v, w > = v T g pw for v, w 2 T pm and for p 2M g is symmetric and positive definite tensor field g also called first di erential form (M, g) is a Riemannian manifold Computationally at each point p 2M, g p is a positive definite matrix of rank d

25 All geometric quantities on M involve g Volume element on manifold Z p Vol(W )= det(g)dx 1...dx d. W

26 All geometric quantities on M involve g Volume element on manifold Z p Vol(W )= det(g)dx 1...dx d. W Length of curve c l(c) = Z v b u t X a ij g ij dx i dt dx j dt dt,

27 All geometric quantities on M involve g Volume element on manifold Z p Vol(W )= det(g)dx 1...dx d. W Length of curve c l(c) = Z v b u t X a ij g ij dx i dt dx j dt dt, Under a change of parametrization, g changes in a way that leaves geometric quantities invariant

28 All geometric quantities on M involve g Volume element on manifold Z p Vol(W )= det(g)dx 1...dx d. W Length of curve c l(c) = Z v b u t X a ij g ij dx i dt dx j dt dt, Under a change of parametrization, g changes in a way that leaves geometric quantities invariant Current algorithms: estimate M This talk: estimate g along with M (and in the same coordinates)

29 Problem formulation Given: data set D = {p 1,... p n} sampled from manifold M R D embedding { x i = (p i ), p i 2D} by e.g LLE, somap, LE,... Estimate G i 2 R m m the (pushforward) Riemannian metric for p i 2D in the embedding coordinates The embedding {x 1:n, G 1:n} will preserve the geometry of the original data

30 g for Sculpture Faces n =698grayimagesoffacesinD =64 64 dimensions head moves up/down and right/left LTSA Algoritm

31 somap LTSA Laplacian Eigenmaps

32 Calculating distances in the manifold M Geodesic distance = shortest path on M should be invariant to coordinate changes Original somap Laplacian Eigenmaps

33 Calculating distances in the manifold M true distance d =1.57 Shortest Metric Rel. Embedding f (p) f (p 0 ) Path d G ˆd error Original data % somap s = % LTSA s = % LE s = %

34 Relation between g and =Laplace-BeltramioperatoronM Proposition 1 (Di erential geometric fact) f = p det(h) X 1 X l det(h) k f, k where h = g 1 (matrix inverse)

35 Relation between g and =Laplace-BeltramioperatoronM =div grad on C 2 (R d ), f = P 2 2 j on weighted graph with similarity matrix S, andt p = P pp 0 S pp 0, =diag { t p} S Proposition 1 (Di erential geometric fact) f = p det(h) X 1 X l det(h) k f, k where h = g 1 (matrix inverse)

36 Estimation of g Proposition 2 (Main Result 1) Let be the Laplace-Beltrami operator on M. Then h ij (p) = 1 2 ( i i(p)) ( j j (p)) i (p), j (p) where h = g 1 (matrix inverse) and i, j =1, 2,...m are embedding dimensions

37 Algorithm to Estimate Riemann metric g (Main Result 2) Given dataset D 1. Preprocessing (construct neighborhood graph,...) 2. Find an embedding of D into R m 3. Estimate discretized Laplace-Beltrami operator L 2 R n n 4. Estimate H p = Gp 1 and G p = H p for all p 2D Output ( p, G p) for all p

38 Algorithm to Estimate Riemann metric g (Main Result 2) Given dataset D 1. Preprocessing (construct neighborhood graph,...) 2. Find an embedding of D into R m 3. Estimate discretized Laplace-Beltrami operator L 4. Estimate H p = Gp 1 and G p = H p for all p 4.1 For i, j =1:m, H ij = 1 2 L( i j ) i (L j ) j (L i ) where X Y denotes elementwise product of two vectors X, Y 2 R N 4.2 For p 2D, H p =[Hp ij ] ij and G p = H p Output ( p, G p) for all p

39 Metric Manifold Learning summary Metric Manifold Learning = estimating (pushforward) Riemannian metric G i along with embedding coordinates x i Why useful Measures local distortion induced by any embedding algorithm G i = d when no distortion at p i Corrects distortion ntegrating with the local volume/length units based on G i Riemannian Relaxation (coming next) Algorithm independent geometry preserving method Outputs of di erent algorithms on the same data are comparable

40 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

41 Riemannian Relaxation Sometimes we can dispense with g dea f embedding is isometric, then push-forward metric is identity matrix d

42 Riemannian Relaxation Sometimes we can dispense with g dea f embedding is isometric, then push-forward metric is identity matrix d dea, formalized Measure distortion by loss = P n i=1 G i d 2 where G i is R. metric estimate at point i d is identity matrix teratively change embedding x 1:n to minimize loss

43 Riemannian Relaxation Sometimes we can dispense with g dea f embedding is isometric, then push-forward metric is identity matrix d dea, formalized Measure distortion by loss = P n i=1 G i d 2 where G i is R. metric estimate at point i d is identity matrix teratively change embedding x 1:n to minimize loss More details loss is non-convex is derived from operator norm Extends to s > d embeddings loss = P n i=1 G i U i Ui T 2 Extensions to principal curves and surfaces [Ozertem, Erdogmus 11], subsampling, non-uniform sampling densities

44 Riemannian Relaxation Sometimes we can dispense with g dea f embedding is isometric, then push-forward metric is identity matrix d dea, formalized Measure distortion by loss = P n i=1 G i d 2 where G i is R. metric estimate at point i d is identity matrix teratively change embedding x 1:n to minimize loss More details loss is non-convex is derived from operator norm Extends to s > d embeddings loss = P n i=1 G i U i Ui T 2 Extensions to principal curves and surfaces [Ozertem, Erdogmus 11], subsampling, non-uniform sampling densities mplementation nitialization with e.g Laplacian Eigenmaps Projected gradient descent to (local) optimum

45 Riemannian Relaxation of a deformed sphere target initialization algorithm output

46 Riemannian Relaxation of a deformed sphere target initialization algorithm output Mean-squared error and loss vs. noise amplitude

47 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

48 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

49 Scaling: Statistical viewpoint Rates of convergence as n! 1 Assume data sampled from manifold M with intrinsic dimension d, M, samplingdistributionare wellbehaved kernel bandwidth decreases slowly with n rate of Laplacian n [Wang 15] minimax rate of manifold learning n 1 2 d+6 [Singer 06], andofitseigenvectorsn (5d+6)(d+6) 2 d+2 [Genovese et al. 12]

50 Scaling: Statistical viewpoint Rates of convergence as n! 1 Assume data sampled from manifold M with intrinsic dimension d, M, samplingdistributionare wellbehaved kernel bandwidth decreases slowly with n rate of Laplacian n [Wang 15] minimax rate of manifold learning n 1 2 d+6 [Singer 06], andofitseigenvectorsn (5d+6)(d+6) 2 d+2 [Genovese et al. 12] Estimating M and accurately requires big data

51 Scaling: Computational viewpoint Laplacian Eigenmaps revisited 1. Construct similarity matrix S =[S pp 0] p,p 0 2D withs pp 0 = e 1 p p0 2 i p, p 0 neighbors 2. Construct Laplacian matrix L = T 1 S with T = diag(s1) 3. Calculate 1...m = eigenvectors of L (smallest eigenvalues) 4. coordinates of p 2Dare ( 1 (p),... m (p))

52 Scaling: Computational viewpoint Laplacian Eigenmaps revisited 1. Construct similarity matrix S =[S pp 0] p,p 0 2D withs pp 0 = e 1 p p0 2 i p, p 0 neighbors 2. Construct Laplacian matrix L = T 1 S with T = diag(s1) 3. Calculate 1...m = eigenvectors of L (smallest eigenvalues) 4. coordinates of p 2Dare ( 1 (p),... m (p)) Nearest neighbor search in high dimensions

53 Scaling: Computational viewpoint Laplacian Eigenmaps revisited 1. Construct similarity matrix S =[S pp 0] p,p 0 2D with S pp 0 = e 1 p p0 2 i p, p 0 neighbors 2. Construct Laplacian matrix L = T 1 S with T = diag(s1) 3. Calculate 1...m = eigenvectors of L (smallest eigenvalues) 4. coordinates of p 2Dare ( 1 (p),... m (p)) Nearest neighbor search in high dimensions Sparse Matrix Vector multiplication

54 Scaling: Computational viewpoint Laplacian Eigenmaps revisited 1. Construct similarity matrix S =[S pp 0] p,p 0 2D with S pp 0 = e 1 p p0 2 i p, p 0 neighbors 2. Construct Laplacian matrix L = T 1 S with T = diag(s1) 3. Calculate 1...m = eigenvectors of L (smallest eigenvalues) 4. coordinates of p 2Dare ( 1 (p),... m (p)) Nearest neighbor search in high dimensions Sparse Matrix Vector multiplication Principal eigenvectors of sparse, symmetric, (well conditioned) matrix

55 Manifold Learning with millions of points James McQueen Jake VanderPlas Jerry Zhang Grace Telford mplemented in python, compatible with scikit-learn Designed for performance sparse representation as default incorporates state of the art FLANN package 1 uses amp, lobpcg fast sparse eigensolver for SDP matrices exposes/caches intermediate states (e.g. data set index, distances, Laplacian, eigenvectors) 1 Fast Approximate Nearest Neighbor search

56 James McQueen Jake VanderPlas Jerry Zhang Grace Telford

57 Scalable Manifold Learning in python with megaman English words and phrases taken from Google news (3,000,000 phrases originally represented in 300 dimensions by the Deep Neural Network word2vec [Mikolov et al]) Main sample of galaxy spectra from the Sloan Digital Sky Survey (675,000 spectra originally in 3750 dimensions). preprocessed by Jake VanderPlas, figure by Grace Telford

58 Scalable Manifold Learning in python with megaman English words and phrases taken from Google news (3,000,000 phrases originally represented in 300 dimensions by the Deep Neural Network word2vec [Mikolov et al]) Main sample of galaxy spectra from the Sloan Digital Sky Survey (675,000 spectra originally in 3750 dimensions). preprocessed by Jake VanderPlas, figure by Grace Telford

59 Currently: on single core, embeds all data, all data in memory Near future: Nyström extension, lazy evaluations, multiple charts Next gigaman? scalable geometric/statistical tasks (search for optimal, Riemannian Relaxation, semi-supervised learning, clustering)

60 Outline What is non-linear dimension reduction? Metric Manifold Learning Estimating the Riemannian metric Riemannian Relaxation Scalable manifold learning megaman An application to scientific data

61 Manifold learning for SDSS Spectra of Galaxies (more in next talk!) Main sample of galaxy spectra from the Sloan Digital Sky Survey (675,000 spectra originally in 3750 dimensions). data curated by Grace Telford, noise removal by Jake VanderPlas

62 Chosing the embedding dimension

63 Embedding into 3 dimensions

64 Same embedding... only high density regions another viewpoint how distorted is this embedding?

65 How distorted is this embedding? (ellipses represent G 1 ) i

66 Riemannian Relaxation along principal curves Find principal curves

67 Riemannian Relaxation along principal curves Points near principal curves, colored by log 10 (G i ) (0 means no distortion)

68 Riemannian Relaxation along principal curves Points near principal curves, colored by log 10 (G i ), after Riemannian Relaxation (0 means no distortion)

69 Riemannian Relaxation along principal curves All data after Riemannian Relaxation

70 Manifold learning for sciences and engineering

71 Manifold learning is for toy data and toy problems

72 Manifold learning is for toy data and toy problems Manifold learning should be like PCA tractable automatic first step in data processing pipe-line

73 Manifold learning is for toy data and toy problems Manifold learning should be like PCA tractable automatic first step in data processing pipe-line Metric Manifold learning megaman Use any ML algorithm, estimate distortion by g and correct it (on demand) tractable for millions of data (in progress) implementing quantitative validation procedure (topology preservation, choice of ) future: port classification, regression, clustering to the manifold setting

74 Manifold Learning for engineering and the sciences scientific discovery by quantitative/statistical data analysis manifold learning as preprocessing for other tasks

75 Manifold Learning for engineering and the sciences scientific discovery by quantitative/statistical data analysis manifold learning as preprocessing for other tasks

76 Thank you