Convergence of Laplacian Eigenmaps

Transcription

1 Convergence of Laplacian Eigenmaps ikhail Belkin Deparmen of Compuer Science Ohio Sae Universiy Columbus, OH 4320 Parha Niyogi Deparmen of Compuer Science The Universiy of Chicago Hyde Park, Chicago, IL Absrac Geomerically based mehods for various asks of machine learning have araced considerable aenion over he las few years. In his paper we show convergence of eigenvecors of he poin cloud Laplacian o he eigenfuncions of he Laplace-Belrami operaor on he underlying manifold, hus esablishing he firs convergence resuls for a specral dimensionaliy reducion algorihm in he manifold seing. Inroducion The las several years have seen significan aciviy in geomerically moivaed approaches o daa analysis and machine learning. The unifying premise behind hese mehods is he assumpion ha many ypes of high-dimensional naural daa lie on or near a lowdimensional manifold. Collecively his class of learning algorihms is ofen referred o as manifold learning algorihms. Some recen manifold algorihms include Isomap [4] and Locally Linear Embedding (LLE) [3]. In his paper we provide a heoreical analysis for he Laplacian Eigenmaps inroduced in [2], a framework based on eigenvecors of he graph Laplacian associaed o he poin-cloud daa. ore specifically, we prove ha under cerain condiions, eigenvecors of he graph Laplacian converge o eigenfuncion of he Laplace-Belrami operaor on he underlying manifold. We noe ha in mahemaics he manifold Laplacian is a classical objec of differenial geomery wih a rich radiion of inquiry. I is one of he key objecs associaed o a general differeniable Riemannian manifold. Indeed, several recen manifold learning algorihms are closely relaed o he Laplacian. The eigenfuncion of he Laplacian are also eigenfuncions of hea diffusions, which is he poin of view explored by Coifman and colleagues a Yale Universiy in a series of recen papers on daa analysis (e.g., [6]). Hessian Eigenmaps approach which uses eigenfuncions of he Hessian operaor for daa represenaion was proposed by Donoho and Grimes in [7]. Laplacian is he race of he Hessian. Finally, as observed in [2], he cos funcion ha is minimized o obain he embedding of LLE is an approximaion o he squared Laplacian. In he manifold learning seing, he underlying manifold is usually unknown. Therefore funcional maps from he manifold need o be esimaed using poin cloud daa. The common approximaion sraegy in hese mehods is o consruc an adjacency graph associaed o a poin cloud. The underlying inuiion is ha since he graph is a proxy for he manifold, inference based on he srucure of he graph corresponds o he desired inference based on he geomeric srucure of he manifold. Theoreical resuls o jusify his inuiion have been developed over he las few years. Building on recen resuls on funcional convergence of approximaion for he Laplace-Belrami operaor using hea kernels and resuls on consisency of eigenfuncions for empirical approximaions of such operaors, we show convergence of he Laplacian Eigenmaps algorihm. We noe ha in order o prove convergence of a

2 specral mehod, one needs o demonsrae convergence of he empirical eigenvalues and eigenfuncions. To our knowledge his is he firs complee convergence proof for a specral manifold learning mehod.. Prior and Relaed Work This paper relies on resuls obained in [3, ] for funcional convergence of operaors. I urns ou, however, ha considerably more careful analysis is required o ensure specral convergence, which is necessary o guaranee convergence of he corresponding algorihms. To he bes of our knowledge previous resuls are no sufficien o guaranee convergence for any specral mehod in he manifold seing. Lafon in [0] generalized poinwise convergence resuls from [] o he imporan case of an arbirary probabiliy disribuion on he manifold. We also noe [4], where a similar resul is shown for he case of a domain in R n. Those resuls were furher generalized and presened wih an empirical poinwise convergence heorem for he manifold case in [9]. We observe ha he argumens in his paper are likely o allow one o use hese resuls o show convergence of eigenfuncions for a wide class of probabiliy disribuions on he manifold. Empirical convergence of specral clusering for a fixed kernel parameer was analyzed in [] and is used in his paper. However he geomeric case requires 0. The resuls in his paper as well as in [3, ] are for he case of a uniform probabiliy disribuion on he manifold. Recenly [8] provided deeper probabilisic analysis in ha case. Finally we poin ou ha while he analogies beween he geomery of manifolds and he geomery of graphs are well-known in specral graph heory and in cerain areas of differenial geomery (see, e.g., [5]) he exac naure of ha parallel is usually no made precise. 2 ain Resul The main resul of his paper is o show convergence of eigenvecors of graph Laplacian associaed o a poin cloud daase o eigenfuncions of he Laplace-Belrami operaor when he daa is sampled from a uniform probabiliy disribuion on an embedded manifold. In wha follows we will assume ha he manifold is a compac infiniely differeniable Riemannian submanifold of R N wihou boundary. Recall now ha he Laplace-Belrami operaor on is a differenial operaor : C 2 L 2 defined as f = div ( f) where f is he gradien vecor field and div denoes divergence. is a posiive semi-definie self-adjoin operaor and has a discree specrum on a compac manifold. We will generally denoe is ih smalles eigenvalue by λ i and he corresponding eigenfuncion by e i. See [2] for a horough inroducion o he subjec. We define he operaor L : L 2 () L 2 () as follows (µ is he sandard measure): ( ) L (f)(p) = (4π) k+2 2 e p q 2 4 f(p) dµ q e p q 2 4 f(q) dµ q If x i are he daa poins, he corresponding empirical version is given by ( k+2 ˆL (4π) 2 n(f)(p) = e p x i 2 4 f(p) ) e p x i 2 4 f(x i ) n i i The operaor ˆL n is (he exension of) he poin cloud Laplacian ha forms he basis of he Laplacian Eigenmaps algorihm for manifold learning. I is easy o see ha i acs by marix muliplicaion on funcions resriced o he poin cloud, wih he marix being he corresponding graph Laplacian. We will assume ha x i are randomly i.i.d. sampled from according o he uniform disribuion. Our main heorem shows ha ha here is a way o choose a sequence n, such ha he n eigenfuncions of he empirical operaors ˆL n converge o he eigenfuncions of he Laplace- Belrami operaor in probabiliy.

3 Theorem 2. Le λ n,i be he ih eigenvalue of ˆL n and e n,i be he corresponding eigenfuncion (which, for each fixed i, will be shown o exis for sufficienly small). Le λ i and e i be he corresponding eigenvalue and eigenfuncion of respecively. Then here exiss a sequence n 0, such ha n λn n,i = λ i n en n,i (x) e i(x) 2 = 0 where he is are in probabiliy. 3 Overview of he proof The proof of he main heorem consiss of wo main pars. One is specral convergence of he funcional approximaion L o as 0 and he oher is specral convergence of he empirical approximaion ˆL n o L as he number of daa poins n ends o infiniy. These wo ypes of convergence are hen pu ogeher o obain he main Theorem 2.. Par. The more difficul par of he proof is o show convergence of eigenvalues and eigenfuncions of he funcional approximaion L o hose of as 0. To demonsrae convergence we will ake a differen funcional approximaion H of, where H is he hea operaor. While H does no converge uniformly o hey share an eigenbasis and for each fixed i he ih eigenvalue of H converges o he ih eigenvalue of. We will hen consider he operaor R = H L. A careful analysis of his operaor, which consiues he bulk of he proof paper, shows ha R is a small relaively bounded perurbaion of H in he sense ha for any funcion f we have convergence and lead o he following R f 2 H, f 2 as 0. This will imply specral Theorem 3. Le λ i,λ i,e i,e i be he ih smalles eigenvalues and he corresponding eigenfuncions of and L respecively. Then λ i λ i = 0 0 e i e i 2 = 0 0 Par 2. The second par is o show ha he eigenfuncions of he empirical operaor ˆL n converge o eigenfuncions of L as n in probabiliy. Tha resul follows readily from he previous work in [] ogeher wih he analysis of he essenial specrum of L. The following heorem is obained: Theorem 3.2 For a fixed sufficienly small, le λ n,i and λ i be he ih eigenvalue of ˆL n and L respecively. Le e n,i and e i be he corresponding eigenfuncions. Then n λ n,i = λ i n e n,i(x) e i(x) 2 = 0 assuming ha λ i 2. The convergence is almos sure. Observe ha his implies convergence for any fixed i as soon as is sufficienly small. Symbolically hese wo heorems can be represened by op line of he following diagram: Eig ˆL n n probabilisic.. Eig L 0 deerminisic.. Eig.. n n 0

4 Afer demonsraing wo ypes of convergence resuls in he op line of he diagram a simple argumen shows ha a sequence n can be chosen o guaranee convergence as in he final Theorem 2. and provides he boom arrow. 4 Specral Convergence of Funcional Approximaions. 4. ain Objecs and he Ouline of he Proof Le be a compac smooh smoohly embedded k-dimensional manifold in R N wih he induced Riemannian srucure and he corresponding induced measure µ. As above, we define he operaor L : L 2 () L 2 () as follows: ( ) L (f)(x) = (4π) k+2 2 e x y 2 4 f(x) dµ y e x y 2 4 f(y) dµ y As shown in previous work, his operaor serves as a funcional approximaion o he Laplace-Belrami operaor on. The purpose of his paper is o exend he previous resuls o he eigenvalues and eigenfuncions, which urn ou o need some careful esimaes. We sar by reviewing cerain properies of he Laplace-Belrami operaor and is connecion o he hea equaion. Recall ha he hea equaion on he manifold is given by h(x,) = h(x,) where h(x,) is he hea a ime a poin x. Le f(x) = h(x,0) be he iniial hea disribuion. We observe ha from he definiion of he derivaive f = (h(x,) f(x)) 0 I is well-known (e.g., [2]) ha he soluion o he hea equaion a ime can be wrien as H f(x) := h(x,) = H (x,y)f(y)dµ y Here H is he hea operaor and H (x,y) is he hea kernel of. I is also well-known ha he hea operaor H can be wrien as H = e. We immediaely see ha = H 0 and ha eigenfuncions of H and hence eigenfuncion of H coincide wih eigenfuncions of he Laplace operaor. The ih eigenvalue of H is equal o e λ i, where λ i as usual is he ih eigenvalue of. I is easy o observe ha once he hea kernel H (x,y) is known, finding he Laplace operaor poses no difficuly: ( ) ( ) H f = f(x) H (x,y)f(y)dµ y = f () 0 0 Reconsrucing he Laplacian from a poin cloud is possible because of he fundamenal fac ha he manifold hea kernel H (x,y) can be approximaed by he ambien space Gaussian and hence L is an approximaion o H and can be shown o converge for a fixed f o. This poinwise operaor convergence is discussed in [0, 3, ]. To obain convergence of eigenfuncions, however, one ypically needs he sronger uniform convergence. If A n is a sequence of operaors, we say ha A n A uniformly in L 2 if sup f 2= A n f Af 2 0. This is sufficien for convergence of eigenfuncions and oher specral properies. I urns ou ha his ype of convergence does no hold for funcional approximaion L as 0, which presens a serious echnical obsrucion o proving convergence of specral properies. To observe ha L does no converge uniformly o, observe ha while H

5 converges o for each fixed funcion f, even his convergence is no uniform. Indeed, for a small, we can always choose a sufficienly large λ i / and he corresponding eigenfuncion e i of, s.. ( ) H 2 e i = ( ) λ e λi i λ i Since L is an approximaion o H, uniform convergence canno be expeced and he sandard perurbaion heory echniques do no apply. To overcome his obsacle we need he wo following key ingrediens: Observaion. Eigenfuncions of H coincide wih eigenfuncions of. Observaion 2. L is a small relaively bounded perurbaion of H. While he firs of hese observaions is immediae, he second is he echnical core of his work. The relaive boundedness of he perurbaion will imply convergence of eigenfuncions of L o hose of H and hence, by he Observaion, o eigenfuncions of. We now define he perurbaion operaor R = H The relaive boundedness of he self-adjoin perurbaion operaor R is formalized as follows: Theorem 4. For any 0 < ǫ < 2 k+2 here exiss a consan C, such ha for all sufficienly small R f,f ( H f,f C max 2 k+2 ǫ, k+2 ǫ) 2 In paricular R f,f sup 0 f 2= H f,f = 0 L and hence R is dominaed by H on L 2 as ends o 0. This resul implies ha for small values of, boom eigenvalues and eigenfuncion of L are close o hose of H, which in urn implies convergence. To esablish his resul, we will need wo key esimaes on he size of he perurbaion R in wo differen norms. Proposiion 4.2 Le f L 2. There exiss C R, such ha for all sufficienly small values of R f 2 C f 2 Proposiion 4.3 Le f H k 2 +, where H k 2 + is a Sobolev space. Then here is C R, such ha for all sufficienly small values of R f 2 C f H k 2 + In wha follows we give he proof of he Theorem 4. assuming he wo Proposiions above. The proof of he Proposiions requires echnical esimaes of he hea kernel and can be found he longer version of he paper enclosed. 4.2 Proof of Theorem 4.. Lemma 4.4 Le e be an eigenvecor of wih he eigenvalue λ. Then for some universal consan C e k H + 2 Cλ k+2 4 (2)

6 The deails can be found in he long version. Now we can proceed wih he Proof: [Theorem 4.] Le e i (x) be he ih eigenfuncion of and le λ i be he corresponding eigenvalue. Recall ha e i form an orhonormal basis of L 2 (). Thus any funcion f L 2 () can be wrien uniquely as f(x) = i=0 a ie i (x) where a 2 i <. For echnical resons we will assume ha all our funcions are perpendicular o he consan and he lowes eigenvalue is nonzero. Recall also ha H f = exp( )f, H e i = exp( λ i )e i, H e i = e λi e i (3) Now le us fix and consider he funcion φ(x) = e x ha φ is a concave and increasing funcion of x. Pu x 0 = /. We have: for posiive x. I is easy o check φ(0) = 0 φ(x 0 ) = e φ(x 0 ) x 0 = e Spliing he posiive real line in wo inervals [0,x 0 ], [x 0, ) and using concaviy and monooniciy we observe ha ( e φ(x) min x, ) e e Noe ha 0 =. Therefore for sufficienly small ( ) φ(x) min 2 x, 2 Thus H e i,e i = e λi 2 min (λ i, ) (4) Now ake f L 2, f(x) = a ie i (x). Wihou a loss of generaliy we can assume ha f 2 =. Taking α > 0, we spli f as a sum of f and f 2 as following: f = a i e i, f 2 = a i e i λ i α I is clear ha f = f + f 2 and, since f and f 2 are orhogonal, f 2 2 = f f We will now deal separaely wih f and wih f 2. From he inequaliy (4) above, we observe ha H f,f 2 λ On he oher hand, from he inequaliy (2), we see ha if e i is a basis elemen presen in he basis expansion of f, e i k 2 + H Cα k+2 4 λ i>α Since acs by rescaling basis elemens, we have f H k 2 + Cα k+2 4. Therefore by Proposiion 4.3 for sufficienly small and some consan C R f 2 C α k+2 4 (5)

7 Hence we see ha R f 2 H f,f 2C k+2 α 4 (6) λ Consider now he second summand f 2. Recalling ha f 2 only has basis componens wih eigenvalues greaer han α and using he inequaliy (4) we see ha H H f,f f 2,f 2 ) (α, 2 min f (7) On he oher hand, by Proposiion 4.2 R f 2 2 C f (8) Thus R ( f 2,f 2 H f,f R f 2 2 H f 2,f 2 C max α, ) Finally, collecing inequaliies 6 and 9 we see: R f,f f,f Rf + Rf 2 C H f,f H where C is a consan independen of and α. Choosing α = 2 k+2 +ǫ where 0 < ǫ < 2 k+2 ( ( max α, ) + ) α k+2 4 (9) (0) yields he desired resul. 5 Specral Convergence of Empirical Approximaion Proposiion 5. For sufficienly small ( ) SpecEss (L ) 2, where SpecEss denoes he essenial specrum of he operaor. Proof: As noed before L f is a difference of a muliplicaion operaor and a compac operaor L f(p) = g(p)f(p) Kf () where g(p) = (4π) k+2 2 e p q 2 4 and Kf is a convoluion wih a Gaussian. As noed in [], i is a fac in basic perurbaion heory SpecEss (L ) = rgg where rg g is he range of he funcion g : R. To esimae rg g observe firs ha (4π) k 2 e p q 2 4 dµ q = We hus see ha for sufficienly small (4π) k 2 and hence g() > 2. e p y 2 4 dµ y > 2 Lemma 5.2 Le e be an eigenfuncion of L, L e = λ e, λ < 2. Then e C. We see ha Theorem 3.2 follows easily: Proof: [Theorem 3.2] By he Proposiion 5. we see ha he par of he specrum of L beween 0 and 2 is discree. I is a sandard fac of funcional analysis ha such poins are eigenvalues and here are corresponding eigenspaces of finie dimension. Consider now λ i [0, 2 ] and he corresponding eigenfuncion e i. The Theorem 4 hen follows from Theorem 23 and Proposiion 25 in [], which show convergence of specral properies for he empirical operaors. dµ q

8 6 ain Theorem We are finally in posiion o prove he main Theorem 4.: Proof: [Theorem 4.] From Theorems 3.2 and 3. we obain he following convergence resuls: Eig ˆL n n.. Eig L 0. Eig where he firs convergence is almos surely for λ i 2. Given any i N and any ǫ > 0, we can choose < 2λ i, s.. for all < we have e i e i 2 < ǫ 2. On he oher hand, by using he firs arrow, we see ha { e P n,i e i 2 ǫ } = 0 n 2 Thus for any p > 0 and for each here exiss an N, s.. P { e n,i e i 2 > ǫ} < p Invering his relaionship, we see ha for any N and for any probabiliy p(n) here exiss a N, s.. n>n P { e N n,i e i 2 > ǫ} < p(n) aking p(n) end o zero, we obain convergence in probabiliy. References []. Belkin, Problems of Learning on anifolds, Univ. of Chicago, Ph.D. Diss., [2]. Belkin, P. Niyogi, Laplacian Eigenmaps and Specral Techniques for Embedding and Clusering, NIPS 200. [3]. Belkin, P. Niyogi, Towards a Theoreical Foundaion for Laplacian-Based anifold ehods, COLT [4] O. Bousque, O. Chapelle,. Hein, easure Based Regularizaion, NIPS [5] F. R. K. Chung. (997). Specral Graph Theory. Regional Conference Series in ahemaics, number 92. [6] R.R.Coifman, S. Lafon, A. Lee,. aggioni, B. Nadler, F. Warner and S. Zucker, Geomeric diffusions as a ool for harmonic analysis and srucure definiion of daa, submied o he Proceedings of he Naional Academy of Sciences (2004). [7] D. L. Donoho, C. E. Grimes, Hessian Eigenmaps: new locally linear embedding echniques for high-dimensional daa, PNAS, vol. 00 pp [8] E. Gine, V. Kolchinski, Empirical Graph Laplacian Approximaion of Laplace-Belrami Operaors: Large Sample Resuls, preprin. [9]. Hein, J.-Y. Audiber, U. von Luxburg, From Graphs o anifolds Weak and Srong Poinwise Consisency of Graph Laplacians, COLT [0] S. Lafon, Diffusion aps and Geodesic Harmonics, Ph.D.Thesis, Yale Universiy, [] U. von Luxburg,. Belkin, O. Bousque, Consisency of Specral Clusering, ax Planck Insiue for Biological Cyberneics Technical Repor TR 34, [2] S. Rosenberg, The Laplacian on a Riemannian anifold, Cambridge Univ. Press, 997. [3] Sam T. Roweis, Lawrence K. Saul. (2000). Nonlinear Dimensionaliy Reducion by Locally Linear Embedding, Science, vol 290. [4] J.B.Tenenbaum, V. de Silva, J. C. Langford. (2000). A Global Geomeric Framework for Nonlinear Dimensionaliy Reducion, Science, Vol 290.