arxiv: v2 [physics.data-an] 5 Jul 2012

Transcription

1 Submitte to the Annals of Statistics OPTIMAL TAGET PLAE RECOVERY ROM OISY MAIOLD SAMPLES arxiv:.460v physics.ata-an] 5 Jul 0 By Daniel. Kaslovsky an rançois G. Meyer University of Colorao, Bouler Constructing an efficient parametrization of a large, noisy ata set of points lying close to a smooth manifol in high imension remains a funamental problem. One approach consists in recovering a local parametrization using the local tangent plane. Principal component analysis PCA is often the tool of choice, as it returns an optimal basis in the case of noise-free samples from a linear subspace. To process noisy ata, PCA must be applie locally, at a scale small enough such that the manifol is approximately linear, but at a scale large enough such that structure may be iscerne from noise. Using eigenspace perturbation theory, we stuy the stability of the subspace estimate by PCA as a function of scale, an boun with high probability the angle it forms with the true tangent space. By aaptively selecting the scale that minimizes this boun, our analysis reveals the optimal scale for local tangent plane recovery.. Local Tangent Plane Recovery... Introuction. Large ata sets of points in high-imension often lie close to a smooth low-imensional manifol. A funamental problem in processing such ata sets is the construction of an efficient parameterization that allows for the ata to be well represente in fewer imensions. Such a parameterization may be realize by exploiting the inherent manifol structure of the ata. However, iscovering the geometry of an unerlying manifol from only noisy samples remains an open topic of research. The case of ata sample from a linear subspace is well stuie see 4, 6, 4], for example. The optimal parameterization is given by principal component analysis PCA, as the singular value ecomposition SVD prouces the best low-rank approximation for such ata. However, most interesting manifol-value ata organize on or near a nonlinear manifol. PCA, by projecting ata points onto the linear subspace of best fit, is not Supporte by S Project DMS an S-IGERT in Computational Optical Sensing an Imaging. Supporte by S Project DMS an DOE Awar DE-SC AMS 000 subject classifications: Primary 6H5; seconary 5A4. Keywors an phrases: Manifol-value ata, tangent plane, principal component analysis, local linear moels, curvature, noise.

2 D.. KASLOVSKY AD. G. MEYER optimal in this case, as curvature may only be accommoate by choosing a subspace of imension higher than that of the manifol. Algorithms esigne to process nonlinear ata sets typically procee in one of two irections. One approach is to consier the ata globally an prouce a nonlinear embeing. Alternatively, the ata may be consiere in a piecewise-linear fashion an linear methos such as PCA may be applie locally. The latter is the subject of this work. There have been several versions of localize PCA for tangent plane recovery propose in the literature. While the nee for locality has been acknowlege, a precise treatment of the size of the local neighborhoo is often not aresse. The appropriate neighborhoo size must be a function of intrinsic manifol imensionality, curvature, an noise level. Despite the fact that these properties may change as ifferent regions of the manifol are explore, locality is often efine via an a priori fixe number of neighbors or as the output of an algorithm e.g., 6, 35, 7, 33]. Other methos, 5, 9] aaptively estimate local neighborhoo size but are not equippe with optimality guarantees. The selection of the optimal scale, or neighborhoo size, for local tangent plane recovery is the key contribution of this paper. What is novel about our approach is that we use the geometry of the ata to guie our efinition of locality. On the one han, a neighborhoo must be small enough so that it is approximately linear an avois curvature. On the other han, a neighborhoo must be be large enough to overcome the effects of noise. We use eigenspace perturbation theory to stuy the stability of the tangent plane as the size of the neighborhoo varies. We boun, with high probability, the angle between the recovere linear subspace an the true tangent plane. In oing so, we are able to aaptively select the neighborhoo that minimizes this boun, yieling the best approximate tangent plane. urther, the behavior of this boun emonstrates the non-trivial existence of such an optimal scale. We are also able to accurately an efficiently estimate the curvature of the local neighborhoo. inally, we introuce a geometric uncertainty principle quantifying the limits of noise-curvature perturbation for tangent plane recovery. Our approach is similar to the analysis presente by aler in 4], who stuies the finite-sample properties of the PCA spectrum. Through matrix perturbation theory, aler examines the angle between the leaing finitesample-pca eigenvector an that of the leaing population-pca eigenvector. As a linear moel is assume, perturbation results from noise only. Despite this key ifference, the two analyses utilize similar techniques to boun the effects of perturbation on the PCA subspace an our results re-

3 OPTIMAL TAGET PLAE RECOVERY 3 cover those of aler in the curvature-free setting. aler also reports that sample-pca suffers from a suen loss of tracking of the true ominant eigenvector ue to a crossover between signal an noise eigenvalues. We emonstrate a similar phenomenon, owing to geometry rather than noise. The present work therefore generalizes the stuy of aler to noisy samples from a nonlinear manifol moel. Other recent relate works inclue that of Singer an Wu 9], who use local PCA to buil a tangent plane basis an give an analysis for the neighborhoo size to be use in the absence of noise. Using the hybri linear moel, Zhang, et al. 34] assume ata are samples from a collection of flats affine subspaces an choose an optimal neighborhoo size from which to recover each flat by stuying the least squares approximation error in the form of Jones β-number see 5] an also 8] in which this iea is use for curve enoising. An analysis of noise an curvature for normal estimation of smooth curves an surfaces in R an R 3 is presente by Mitra, et al. ] with application to computer graphics. We also note the work of Maggioni an coauthors 4], in which multiscale PCA is use to iscover the intrinsic imensionality of a ata set. The paper is organize as follows. The remainer of this section provies the intuition an assumptions of our approach an introuces the geometric moel that is use throughout this work. We frame the problem as one of subspace perturbation in Section an stuy the size of the perturbation as a function of scale in Section 3. The selection of the optimal scale is our main result an is presente in Section 4, along with the necessary geometric conitions for tangent plane recovery. umerical results are given in Section 5. We conclue with algorithmic consierations an a iscussion of future irections in Section 6... Problem Setup. Our goal is to recover the best approximation to a local tangent space of a nonlinear -imensional Riemannian manifol M from noisy samples presente in imension D >. Working about a reference point x 0, an approximation to the linear tangent space of M at x 0 is given by the span of the top singular vectors of the centere ata matrix where top refers to the singular vectors associate with the largest singular values. The question becomes: how many neighbors of x 0 shoul be use or in how large of a raius about x 0 shoul we work to recover the best approximation? To answer this question, we examine the noise-curvature trae-off. Given noisy samples of a linear subspace, the quality of PCA approximation improves as more points are inclue. However, the curvature of M prevents

4 4 D.. KASLOVSKY AD. G. MEYER the inclusion of a large number of points. Similarly, there exists a local scale about x 0 such that the effects of curvature are small, as M locally resembles Eucliean space. This suggests allowing only a very small raius about x 0, yet at small scales, the sample points are inistinguishable from noise. We therefore seek a balance an assume there exists a scale large enough to be above the noise level, but still small enough to avoi curvature. This scale reveals a linear structure that is sufficiently ecouple from both the noise an the curvature to be well approximate by a tangent plane. We note that the concept of noise-curvature trae-off has been a subject of interest for ecaes in ynamical systems theory 9]..3. Geometric Data Moel. A -imensional manifol of coimension may be escribe locally by the surface y = fl,..., l, where l i is a coorinate in the tangent plane. After translating the origin, a rotation of the coorinate system can align the coorinate axes with the principal irections associate with the principal curvatures at the given reference point x 0. Aligning the coorinate axes with the plane tangent to M at x 0 gives a local quaratic approximation to the manifol. Using this choice of coorinates, the manifol may be escribe locally ] by the Taylor series of f at the origin x 0 :. y = fl,..., l = κ l + + κ l + o l + + l, where κ,..., κ are the principal curvatures of M at x 0. In this coorinate system, x 0 has the form x 0 = l l l fl,..., l ] T an points in a local neighborhoo of x 0 have similar coorinates. Generalizing to a -imensional manifol of arbitrary coimension in R D, there exist D functions f i l = κi l + + κ i l + o l + + l for i = +,..., D, with κ i,..., κi representing the principal curvatures in coimension i at x 0. Then, given the coorinate system aligne with the principal irections, a point in a neighborhoo of x 0 has coorinates l,..., l, f +,..., f D ]. We truncate this Taylor expansion an use the quaratic approximation. f i l = κi l + + κ i l,

5 OPTIMAL TAGET PLAE RECOVERY 5 i = +,..., D, as the local moel for our analysis. Consier now iscrete samples from M that are contaminate with an aitive Gaussian noise vector e rawn from the 0, σ I D istribution. Each sample x is a D-imensional vector an such samples may be store as columns of a matrix X R D. The coorinate system above allows the ecomposition of x into its linear tangent plane component l, its quaratic curvature component c, an noise e, three D-imensional vectors l = l l l 0 0] T c = 0 0 c + c D ] T e = e e e D ] T such that the last D entries of c are of the form.6 c i = κi l κ i l. We may store the samples of l, c, an e as columns of matrices L, C, E, respectively, such that our ata matrix is ecompose as.7 X = L + C + E. Remark. Of course it is unrealistic for the ata to be observe in the escribe coorinate system. As note, we may use a rotation to align the coorinate axes with the principal irections associate with the principal curvatures. Doing so allows us to write. as well as.7. Because we will ultimately quantify the norm of each matrix using the unitarily-invariant robenius norm, this rotation will not affect our analysis. We therefore procee by assuming that the coorinate axes align with the principal irections. The true tangent plane we wish to recover is given by the PCA of L. Because we o not have irect access to L, we work with X as a proxy, an instea recover a subspace spanne by the corresponing eigenvectors of XX T. We will stuy how close this recovere invariant subspace of XX T is to the corresponing invariant subspace of LL T as a function of scale. Throughout this work, scale refers to the number of points in the local neighborhoo within which we perform PCA. Given a fixe ensity of points, scale may be equivalently quantifie as the raius r about the reference point x 0 efining the local neighborhoo.. Perturbation of Invariant Subspaces. Given the ecomposition of the ata.7, we have. XX T = LL T +CC T +EE T +LC T +CL T +LE T +EL T +CE T +EC T.

6 6 D.. KASLOVSKY AD. G. MEYER To account for the centering require by PCA, efine the sample mean of realizations of ranom variable Y as. ÊY ] = Y i, i= where Y i enotes the ith realization. Let the mean of a matrix M be the matrix ÊM] such that each entry of row i is the sample mean of the ith row of M. Let M enote the centere version of M:.3 M = M ÊM]. Thus we have.4 X X T = L L T + C C T +ẼẼT + L C T + C L T + LẼT +Ẽ L T + CẼT +Ẽ C T. The problem may be pose as a perturbation analysis of invariant subspaces. Rewrite. as.5 where X X T = L L T +,.6 = C C T + ẼẼT + L C T + C L T + LẼT + Ẽ L T + CẼT + Ẽ C T is the perturbation that prevents us from working irectly with L L T. The ominant eigenspace of X X T is therefore a perturbe version of the ominant eigenspace of L L T. Seeking to minimize the effect of this perturbation, we look for the scale at which the ominant eigenspace of X X T is closest to that of L L T. Before proceeing, we review material on the perturbation of eigenspaces relevant to our analysis. The reaer familiar with this topic is invite to skip irectly to Theorem.. The istance between two subspaces of R D can be efine as the spectral norm of the ifference between their respective orthogonal projectors ]. As we will always be consiering two equiimensional subspaces, this istance is equal to the sine of the largest principal angle between the subspaces. We state our results in terms of the robenius norm as it will provie a simplification of Theorem.. Then, by the equivalence of norms, we may efine the optimal scale as.7 = arg min P P,

7 OPTIMAL TAGET PLAE RECOVERY 7 where P an P are the orthogonal projectors onto the subspaces compute from L an X, respectively. The solution to.7 is the main goal of this work. The istance P P may be boune by the classic sin Θ theorem of Davis an Kahan 6]. We will use a version of this theorem presente by Stewart Theorem V..7 of 30], moifie for our specific purpose. irst, we establish some notation, following closely that foun in 30]. Consier the eigenecompositions.8.9 L L T = UΛU T = U U ] Λ U U ] T, X X T = Û ΛÛ T = Û Û] Λ Û Û] T, such that the columns of U are the eigenvectors of L L T an the columns of Û are the eigenvectors of X X T. The columns of U are those eigenvectors associate with the largest eigenvalues in Λ arrange in escening orer. The columns of U are then those eigenvectors associate with the smallest D eigenvalues, an Û is similarly partitione. The subspace we recover is spanne by the columns of Û an we wish to have this subspace as close as possible to the tangent space spanne by the columns of U. The orthogonal projectors onto the tangent an compute subspaces, P an P respectively, are given by P = U U T an P = Û Û T. Define λ to be the th largest eigenvalue of L L T, or the last entry on the iagonal of Λ. ote that λ correspons to variance in a tangent plane irection. We are now in position to state the theorem. ote that we have mae use of the fact that the columns of U are the eigenvectors of L L T, that Λ, Λ are Hermitian iagonal matrices, an that the robenius norm is use to measure istances. The reaer is referre to 30] for the theorem in its original form. Theorem.. Davis & Kahan 6], Stewart 30] Let δ = λ U T U U T U an consier Conition δ > 0 Conition U T U U T U < 4 δ.

8 8 D.. KASLOVSKY AD. G. MEYER Then, provie that conitions an hol,.0 P P U T U δ The two conitions of the theorem have important geometric interpretations. Informally, conition requires that the linear structure we seek to recover be sufficiently ecouple from both the noise an curvature this is consistent with our assumption of the existence of a scale yieling sufficient ecoupling. We may consier δ to be the conition number for subspace recovery. When δ approaches zero, the conition number becomes large, an boun.0 loses meaning as we cannot recover an approximating subspace. In Section 4 we will see that conition naturally gives rise to an uncertainty principle that quantifies the limits of noise-curvature perturbation for tangent plane recovery. We will also see that the secon conition naturally implies that the manifol be sufficiently sample. The solution to.7 is impractical to compute. However,.0 is a tight boun, as will be emonstrate by the experiments Section 5. Thus, a solution may be approximate by minimizing the right-han sie of.0. To o so, an to give each quantity in the theorem a geometric interpretation, we must first unerstan the behavior of the perturbation as a function of the scale parameter. 3. Bouning the Effects of oise an Curvature. In this section we stuy the behavior of each term in.6 as a function of the scale parameter. irst, we provie insight as to their leaing orer behavior. As explaine by ukunaga 0] Chapter 5, estimator bias an estimator variance epen on the Hessian an graient, respectively, of the function being estimate. Consier the local manifol moel.. This secon orer approximation is presente in a coorinate system such that its graient is zero an its Hessian is a iagonal matrix with the principal curvatures as its entries. We therefore expect perturbation terms associate with variance to ten to zero as the scale parameter increases. Likewise, we expect pure curvature terms to grow with. ormal calculations will show that CCT, the term associate purely with curvature, has nonzero expectation that increases with. ote that while the iagonal entries of EET also have nonzero expectation, these terms o not grow with an are therefore associate with a noise-floor rather than with estimator bias. All other terms in.6 are zero in expectation, an thus only carry variance. Accoringly, these terms ecay as /..

9 3.. Preliminaries. OPTIMAL TAGET PLAE RECOVERY Sampling a Linear Subspace. Consier sampling a linear subspace by uniformly sampling points insie B x 0 r, the -imensional ball of raius r centere at x 0. We rop the epenence on x 0 from our notation for the remainer of this analysis. Because we are sampling from a noise-free linear subspace, the number of points capture insie B r is a function of the sampling ensity ρ: 3. = ρv r, where v is the volume of the -imensional unit ball. As we wish to maintain a local analysis, we must enforce that r be small. To make this explicit, enote by r max the largest raius within which the local moel. hols an compute the number of points capture in B r max : 3. max = ρv r max. Then rescale 3. by iviing by 3. an solve for r: 3.3 r = r max. max Remark. Equation 3.3 suppresses the epenence on sampling that is capture by the ρv term. ote that because r is small, the sampling ensity ρ may have to be large to allow for large, as is explicitly seen in equation 3.. The volume of the unit ball, v, is very small for even reasonable values of, further necessitating a large sampling ensity. The analysis in this section may be performe entirely in the context of equation 3. provie that r is taken to be small. In oing so, the epenence on sampling ensity ρ is clear in all steps. We prefer to instea perform the analysis in the context of the rescale equation 3.3, explicitly forcing the analysis to the local scale by consiering r to be a fraction of the largest raius allowe by the local moel otation. In this section an throughout the remainer of this work, we will make use of the following efinitions involving the principal curvatures: K i = K = n= D κ i n, i=+ K i,

10 0 D.. KASLOVSKY AD. G. MEYER 3.6 K ij nn = n= κ i n κ j n, K ij mn = m,n= m n κ i m κ j n. The constant K i quantifies the curvature in coimension i, for i = +,..., D. ote that given our choice of coorinate system in Section., K i is the trace of the Hessian in the ith coimension. The overall curvature of our local moel is quantifie by K an is a natural result of our use of the robenius norm. We note that K i K j = Knn ij + Kmn. ij By the choice of coorinate system, U is the D D matrix whose columns are the last D columns of I D, the ientity matrix of orer D. Due to the specific form of each matrix, we have U T C, CT U, U T L, an L T U are all zero matrices of the appropriate size. Because is a symmetric matrix, we have that U T U = U T U. inally, we will work with projections of vector a onto U an U, where a takes the form of l, c, or e equations.3.5, an enote such projections by 3.7 U T p a = a up, for p = {, }. 3.. Analysis of Perturbation Terms. We begin by presenting our general strategy for bouning terms of the form U T p à B T U q for p, q = {, } where A an B are general matrices of size D. The key observation is that à B T is a sample mean of outer proucts of vectors a an b, each sample from a given istribution: à B T = Êa Êa]b Êb]T ], where ÊY ] is the sample mean efine in.. We therefore expect that à B T will converge towar the centere outer prouct of a an b. We will use the following result of Shawe-Taylor an Cristianini 8] to boun, with high probability, the norm of the ifference between this sample mean an its expectation, 3.8 EUp T a Ea]b Eb] T U q ] ÊU p T a Êa]b Êb]T U q ] where EY ] is the expectation of the ranom variable Y Y. Theorem 3.. Shawe-Taylor & Cristianini, 8]. Given samples of a ranom variable Y generate inepenently at ranom from Y accoring to the istribution P Y, with probability at least e η over the choice of the samples, we have 3.9 EY ] ÊY ] R + η

11 OPTIMAL TAGET PLAE RECOVERY where R = sup supppy Y an suppp Y is the support of istribution P Y. Remark. With a slight abuse of notation, we note that the robenius norm of a vector is equivalent to the vector s Eucliean norm, an thus we use for both matrices an vectors. Remark. The choice of R in 3.9 nee not be unique. Our analysis will procee by using upper bouns for Y which may not be suprema. Continuing from 3.8, EUp T a Ea]b Eb] T U q ] ÊU p T a Êa]b Êb]T U q ] = Ea up b T u q ] Êa u p b T u q ] + Êa u ]ÊbT p u q ] Ea up ] Eb T u q ] 3.0 Ea up b T u q ] Êa u p b T u q ] + Ea up ] Eb T u q ] Êa u ]ÊbT p u q ]. Because El] = 0 an Ee] = 0, Ea up ] Eb T u q ] is nonzero only for the case a = b = c, p = q =. In this case, ÊU T p a Êa]b Êb]T U q ] = ÊU T c Êc]c Êc]T U ] is an empirical covariance matrix. As shown in 8], such a matrix is unchange when the origin is shifte by a fixe translation. Therefore we may assume that the origin has been shifte to the center of mass of the istribution an we may take Ec u ] an Ec T u ] to be zero. ote that we may only o so in the context of this calculation, an in general Ec u ] an Ec T u ] are nonzero. Then for all choices of a, b, p, q, we have Ea up ] Eb T u q ] = 0 an the right-han sie of 3.0 becomes Ea up b T u q ] Êa u p b T u q ] + 3. Ea up b T u q ] Êa u p b T u q ] + Êa u ]ÊbT p u q ] Ea up ] Êa u p ] Eb T uq ] ÊbT u q ]. We now use Theorem 3. to boun each of the three terms in 3.. or this analysis, the ranom variable Y in Theorem 3. takes one of the following two forms: Y = a up b T u q or Y = a up for p, q = {, }. Thus there are two corresponing efinitions for R: R pq ab = R p a = sup suppp a suppp b a up b T u q sup a up suppp a

12 D.. KASLOVSKY AD. G. MEYER where a an b are sample accoring to istributions P a an P b, respectively. Directly applying Theorem 3. to each of the three terms in 3. an using a stanar union boun argument yiels 3.4 Ea up b T u q ] Êa u p b T u q ] Rpq ab + η ab with probability greater than + + Rp ar q b Ea up ] Êa u p ] + η a + η b 3.5 e η ab e η a e η b Eb T uq ] ÊbT u q ] over the ranom sampling of a an b. or the case that a = b we instea simply have the result holing with probability greater than 3.6 e η aa e η a over the ranom sampling of a. The probability constants may be chosen to ensure such an event hols with high probability. or example, in 3.5, letting η ab = η a = η b = η, we have probability greater than for η = an greater than for η = 3. Putting it all together, we have that EUp T a Ea]b Eb] T U q ] ÊU p T a Êa]b Êb]T U q ] 3.7 Rpq ab + η ab + Rp ar q b an we may conclue that 3.8 U p T where µ = Ã B T + η a + η b U q Ea up b T u q ] Ea up ] Eb T u q ] an Γ = Rpq ab + η ab ] µ Γ, µ + Γ, + Rp ar q b with probability greater than { e η ab e η a e η b for a b e η ab e η a for a = b + η a + η b,

13 OPTIMAL TAGET PLAE RECOVERY 3 over the ranom sampling of a an b. Before computing the constants R pq ab an Rp a, we must ensure that either the suprema 3. an 3.3 exist or that finite bouns may be given in place of suprema. oting that 3.9 a up b T u q a up b uq, it suffices to show that the projections l up, c up, an e up are boune. The vectors l an c are functions of the coorinates of points rawn uniformly from B r. Therefore their entries are boune, as are the norms of their projections. The entries of e, while unboune in general, are Gaussian ranom variables an are therefore boune over a set with large measure. Let Ω e be the set in R D for which both 3.0 e u σ + ξe D 3. e u σ + ξe hol. A formal construction of Ω e is given in Appenix A an the measure γ D Ω e of this set is shown to be large, γ D Ω e > e ξ e. Therefore, on Ω e, we may state bouns for Re p p =,. All results involving projections of the vector e will be given over this set. We apply Theorem 3. by conitioning on e Ω e. If we consier the joint probability of the inepenent ranom variables a an e, then we can estimate the probability that the eviation of the outer-prouct of a up an e uq from its expectation is large. This probability is given by: ] Prob Ea up e Tuq ] Êa up e Tuq ] > Rpq ae + η ae ] = Prob Ea up e Tuq ] Êa up e Tuq ] > Rpq ae ] + η ae e Ωe Prob e Ω e ] + Prob Ea up e Tuq ] Êa up e Tuq ] > Rpq ae ] + η ae e Ωe Prob e Ω e e η ae e ξ e + e ξ e = e η ae + e ξ e e η aee ξ e Thus we have that 3. Ea up e T u q ] Êa u p e T u q ] Rpq ae + η ae

14 4 D.. KASLOVSKY AD. G. MEYER with probability greater than e η ae e ξe +e ηae e ξe over the ranom sampling of a an ranom realization of e, an an ientical calculation hols when applying the theorem to Ee up ] Êe u p ]. The probability given in 3.5 can be boune by 3.3 e η ae e η a e η e e ξ e + e ξ e e η ae + e η e > e η ae e η a e η e e ξ e an when a = b = e the probability given in 3.6 can be boune by 3.4 e η ee e η e e ξ e + e ξ e e η ee + e η e > e η ee e η e e ξ e where we have neglecte the positive contribution of the prouct of exponentially small terms Suprema R pq ab an Rp a. We now compute bouns for the Ra p terms. Simple norm calculations give l u = l i rmax, 3.5 c u = D i=+ r 4 max 4 c i = 4 max D i=+ 4 i= max κ i l κ i l D i=+ n= κ i n = K r 4 max 4 max 4, where we have assume all principal curvatures have the same sign. Combining with 3.0 an 3. yiels bouns for Ra p terms, an 3.9 may be use to boun the R pq ab terms. The results are liste in Appenix B. Remark. The calculation 3.5 for c u requires that all principal curvatures have the same sign for the inequality to hol. This requirement will carry through as an assumption in the statement of Theorem 4. Main Result. To avoi this requirement we must work with moments of l i rather than with norms, specifically when computing CL T an LC T terms. While not possible here, it is possible to o so as emonstrate in Main Result see Section Despite this assumption, it will be seen in Section 5 that this current analysis oes in fact provie meaningful results for principal curvatures of mixe signs except for when K i = 0, inicating that tighter R pq ab bouns are possible. We note that Main Result will require no such assumption an will hol for any value of K i.

15 OPTIMAL TAGET PLAE RECOVERY Expectations. We are almost in position to efine confience intervals of the form 3.8, where à an B may be the centere matrices L, C, an Ẽ. All that remains is to compute the true expectation term of equation 3.7: 3.6 EUp T a Ea]b Eb] T U q ] = Ea up b T u q ] Ea up ] Eb T u q ]. As the coorinates of l an c are functions of points sample uniformly from B r an e 0, σ I D, the expectation terms are zero for cl T, le T, an ce T. Only the pure curvature cc T an pure noise ee T terms may have nonzero expectations an their calculations are given in Appenix B. We list here only the results. Pure Curvature Term: Ec u c T u ] Ec u ] Ec T 3.7 u ] = rmax 4 4 D max D i=+ j=+ + K ij nn Kmn ij ] We note that later, for the purpose of interpretation, we will replace this exact expectation with an upper boun. Using that El ml n] < El 4 n] an Knn ij + Kmn ij = K i K j, we boun 3.8 Ec u c T u ] Ec u ] Ec T u ] < K r4 max max. 4. The nonzero expectation 3.7 an its boun 3.8 grow with an in this way may be thought of as the source of the estimator bias see the beginning iscussion of Section 3. Pure oise Terms: Ee up e T u q ] Ee up ] Ee T u q ] = σ if p, q =,, σ D if p, q =,, 0 if p, q =,. ote that unlike the nonzero expectation of the pure curvature term, the nonzero expectations of the pure noise terms p = q are constant an o not grow with. Thus they represent a noise floor rather than a source of estimator bias.

16 6 D.. KASLOVSKY AD. G. MEYER orm Bouns. We may now use the right-han sie of the confience interval 3.8 to boun the size of the perturbation norms. When consiering noise terms, recall that we must conition on e Ω e. To ai in interpretation, we recall the rescale notation r = r max / max /. As each curvature term c has norm roughly of size Kr, we expect CCT to grow as K r 4. Concentration of Gaussian measure inicates that the norm of the noise matrix will have size that epens on the square root of the projection imension an the variance σ. All other terms are zero in expectation an thus we expect / ecay. The linear-curvature, linearnoise, an curvature-noise matrices shoul have norm Kr 3 /, σr/, an Kσr /, respectively. The leaing orer behavior of all norm bouns may be foun in Table 4. an the full expressions with associate probabilities are given in Appenix C. 4. Optimal Scale Selection an Subspace Recovery. Our main result, a boun on the angle between the recovere an true tangent planes, is formulate in this section. irst we use the triangle inequality to boun the norms appearing in Theorem.. We then inject the perturbation norms compute in Section 3 an Appenix C to formulate the main result. We have: U T U U T LẼT U + U T ẼẼT U U T U U T C C T U + U T CẼT U + U T ẼẼT U U T U U T C L T U + U T Ẽ L T U + U T CẼT U + U T ẼẼT U. As is a symmetric matrix, we also have that U T U = U T U. Using a stanar union boun argument, the bouns for each term hol simultaneously with probability greater than 4. e η cc 3e η ee e η lc e η le e η ce e η l e η c e η e e ξ e over the joint ranom selection of the sample points an ranom realization of the noise. We may pick a constant η an set 4. η cc = η ee = η lc = η le = η ce = η l = η c = η e = η

17 such that 4. becomes OPTIMAL TAGET PLAE RECOVERY e η e ξ e. inally, recall from Theorem. that 4.4 δ = λ U T U U T U. In orer to boun the size of δ we must compute λ, the th eigenvalue of L L T. This matrix is a centere covariance matrix an therefore its th eigenvalue is the variance in the th coorinate. rom our moment calculations in Appenix D, we let 4.5 λ = Varl ] = r max. + max 4.. Main Result: Bouning the Angle Between Subspaces. We are now in position to apply Theorem. an state our main result. irst, efine the following constants: K = p η = p ξ e = p 3 ξ e = ζk, σ, η, ξ e = D D i=+ j=+ + K ij nn Kmn ij ] + η + + η, + ξe, D + ξe, K rmax 4 max +σp ξ e r max Then we have our main result: 4 + σp3 ξ e Kr max max, max + σ p ξ e + p 3 ξ e ]. Theorem 4.. Main Result. Let the following conitions hol: Conition δ = λ U T U U T U > 0, Conition U T U < δ.

18 8 D.. KASLOVSKY AD. G. MEYER or each i = +,..., D, let principal curvatures κ i n n =,...,. Then we have have the same sign, 4.6 P P p η K r3 + σp ξ e K r + σp 3 ξ e r + σ p ξ e p 3 ξ e ] r + Kr σ + D p η ζk, σ, η, ξ e with probability greater than 3e η e ξ e over the joint ranom selection of the sample points an ranom realization of the noise, where the rescale notation r = r max / max / has been use. Proof. Applying the norm bouns compute in Appenix C to Theorem. an choosing the probability constants as given in 4. yiels the result. The optimal scale may be selecte as the for which 4.6 is minimize. As will be analyze in Section 4.., 4.6 is either monotonically ecreasing for the curvature-free case, monotonically increasing for the noise-free case, or ecreasing at small scales an increasing at large scales for the general case. We therefore expect a unique minimizer in all cases. ote that the constants η an ξ e nee to be selecte to ensure that this boun hols with high probability. As iscusse in Section, we may interpret δ as the conition number for tangent plane recovery, an we may analyze it by bouning 4.4 using the bouns for U T U an U T U. We note that the enominator in 4.6 is a lower boun on δ an we therefore analyze the conition number via this boun. rom Main Result 4.6, we see that when δ is small, we may recover a tight approximation to the true tangent space. Likewise, when δ becomes large, the angle between the compute an true subspaces becomes large. The notion of an angle loses meaning as δ tens to infinity, an we are unable to recover an approximating subspace. Conition imposes that the spectrum corresponing to the linear subspace λ must be well separate from the spectra of the noise U U an curvature U U perturbations. In this way, conition quantifies our requirement that there exists a scale such that the linear subspace is sufficiently ecouple from the effects of curvature an noise. When the spectra are not well separate, the angle between the subspaces becomes ill-efine. In this case, the approximating subspace contains an eigenvector corresponing to a irection orthogonal to the true tangent plane. In the language of 4], there is a crossover between the spectrum of the linear

19 OPTIMAL TAGET PLAE RECOVERY 9 subspace an the spectrum of the perturbation, an we will observe a loss of tracking of the true tangent plane. Unlike the result of 4] where the crossover results from noise perturbation, we will emonstrate a crossover ue to geometry in Section 5. As we will see, conition inee imposes a geometric requirement for tangent plane recovery. With conition imposing restrictions on the effects of curvature an noise, conition may be interprete as a control on sampling. This conition may be satisfie by increasing the sampling ensity, as such an increase allows to take on large values. Recall that r = r max / max /. While / max terms are boune by one, the numerator of 4.6 is compose of terms that behave as /. Thus, provie that the enominator is well-conitione, conition may be satisfie by allowing for large enough. Before numerically emonstrating our main result, we give a practical interpretation of its conitions an implications. In oing so, we quantify the separation neee between the linear structure an the noise an curvature with a geometric uncertainty principle. Then in Section 4.3, we present evience of a tighter main result through the Central Limit Theorem. 4.. Interpreting the Boun. The boun 4.6 is ifficult to interpret ue to many complicate terms. or subspace tracking, the behavior of the boun as a function of scale is of more practical use than the boun s actual value. The scale at which the boun reaches its minimum an the scales at which it becomes ill-conitione are the quantities of interest. Thus we now analyze our main result using more practical an interpretable, albeit less sharp, bouns. The following boun is not as sharp as Main Result owing mainly to a less precise treatment of the principal curvatures. We replace equation 3.7 with 3.8 for the expectation of the pure curvature term, an we analyze only leaing-orer behavior of each term. eglecting the probability-epenent constants an the contributions of the LET an CET terms, we may write our main result as: Interpretable Main Result. 4.7 P P an we recall that r = r max / max /. K r3 + σ ] D r + K r σ + D 4... Revisiting the oise-curvature Trae-off Through the Conition umber. We may now use this more compact, yet less sharp, form 4.7 of

20 0 D.. KASLOVSKY AD. G. MEYER our main result to provie some interpretation for the boun 4.6. Consier first the enominator. Assume that sampling is sufficiently ense such that may become large. Doing so ensures conition is met an we may focus our attention on conition an our neglect of the LET an CET terms in the enominator of 4.6 is also justifie. The enominator of 4.7 is of the form 4.8 δ = r + K r σ + D. It is now easy to see that δ quantifies the separation between the linear subspace Or an the perturbation ue to the curvature OK r 4 an the noise level σ + D. To approximate the appropriate linear subspace, we must at least have that δ > 0 as require by conition, an the approximation improves for larger δ. We note the similarity of this conition to that of equation.0 in 4]. The noise-curvature trae-off is now reaily apparent. The linear an curvature contributions, controlle by r = r max / max /, are small for small values of. Thus for small, the enominator 4.8 is either negative or ill conitione for most values of σ. This makes intuitive sense as we have not yet encountere much curvature but the linear structure has also not been explore. Therefore the noise ominates the early behavior of this boun an an approximating subspace may not be recovere from noise. As increases, the conitioning of the enominator improves, an the boun is controlle by the / behavior of the numerator. This again correspons with our intuition, as the aition of more points serves to overcome the effects of noise as the linear structure is explore. Thus, the boun becomes tighter. Eventually, becomes large enough such that the curvature contribution approaches the size of the linear contribution, an δ becomes large. The / term is overtaken by the ill conitioning of the enominator an the boun is force to become large. The noise-curvature trae-off, seen analytically here in 4.8, will be emonstrate numerically in Section Geometric Uncertainty Principle for Subspace Recovery. Imposing conition on 4.8 yiels a geometric uncertainty principle quantifying the amount of curvature an noise we may tolerate. Solving for the range of scales such that δ > 0, the following conition naturally arises. To recover an approximating subspace, we must have that:

21 OPTIMAL TAGET PLAE RECOVERY σ D σ D a b ig. color online] Illustration of the geometric uncertainty principle 4.9. or a manifol of fixe curvature K, a shows an acceptable noise level such that the geometry of the ata remains intact an a tangent plane may be approximate from the noisy ata. b illustrates a violation of the uncertainty principle as the manifol geometry may be estroye by the noise. In this case a tangent plane approximation cannot be recovere. Geometric Uncertainty Principle Kσ < + + D. By preventing curvature an noise from simultaneously becoming large, this requirement ensures that the geometry of the ata is not estroye by noise. With high probability, the noise concentrates on a sphere with mean curvature /σ D. Intuitively, we expect to require that the curvature of the manifol be less than the curvature of this noise-ball. Recalling the efinitions of K i an K from equations 3.4 an 3.5, K i / is the mean curvature in coimension i. The quaratic mean of the D mean curvatures is then given by K/ D an we enote this normalize version of curvature as K. Then 4.9 requires that K < O/σD 3 4. oting that > D 4, the uncertainty principle 4.9 inee may be interprete as a requirement that the mean curvature of the manifol be less than that of the perturbing noise-ball. igure provies an illustration. It is important to keep in min that equation 4.9 is compute using the compact boun 4.7, an is thus meant for interpretation. or the precise expression represente by 4.9, the erivation must start with the full boun Towars a Tighter Boun: Chasing the Constants. Thus far we have presente a rigorous analysis bouning the norm of each perturbation term. The analysis captures leaing orer behavior with high probability by utilizing Theorem 3., but oes so at the cost of attaching large constants to each

22 D.. KASLOVSKY AD. G. MEYER term. Theorem 3., a result erive from the boune ifference an Hoeffing inequalities 8], introuces constants base on suprema of functions of ranom variables taken over the support of their istributions. Accoringly, each perturbation term is shown to eviate from its expectation by factors larger than constant multiples of its variance. In this section we use the Central Limit Theorem CLT to show that the variance of the perturbation terms controls the eviation from their expectations. Doing so yiels tighter bouns for each term an a tighter overall main result. urther, by working with moments of the unerlying ranom variables rather than norms, a more precise treatment of curvature terms is possible, allowing a relaxation of the assumption in Main Result that all principal curvatures in coimension i have the same sign. Despite the fact that our analysis is most often to be applie to sample sizes on the orer of = 0 5 or 0 6, we must acknowlege that the sample means with which we work have a Gaussian istribution only in the limit as tens to infinity. This finite-sample analysis can be mae rigorous through the use of Bernstein-type inequalities an concentration of measure in fact such approaches yiel only sightly larger constants. However, we procee with a CLT-base analysis, treating convergence in istribution as equality, to provie evience that a tight boun may be achieve an to motivate future analyses that may rigorously yiel tighter constants Central Limit Theorem an Gaussian Tail Bouns. As previously seen, each entry of a matrix ABT is a sample mean of i.i.. ranom variables. The CLT an a Gaussian tail boun yiel a confience interval for such an entry. Using a union boun to simultaneously control all of the entries of this matrix, we may give an overall confience interval for the value of its robenius norm. While such an analysis yiels a tighter result than that using Theorem 3., it hols with lower probability ue to the use of many union bouns. Remark. It is important to note that the probability attache to this analysis correspons to the result serving as an upper boun for the true subspace recovery error. The probability is increase by choosing large values for the constants appearing in the exponential terms. This in turn loosens the result so as to boun any ranom fluctuation of the true error from above. In many practical applications, we are most intereste in tracking the recovery error regarless of whether we guarantee an upper boun. When the nee for an upper boun is relaxe, we will emonstrate in Section 5 that the leaing orer behavior of the following analysis tightly tracks the tren of the true error curve.

23 OPTIMAL TAGET PLAE RECOVERY 3 The analysis procees as follows. An entry of matrix à B T has the form 4.0 à B T i,j = A i,k B j,k k= ÊA i]êb j], where A an B represent the matrices L, C, an E from equation.7. We use the CLT to assert that for the i.i.. ranom variables A i,k B j,k with mean µ an variance σ, k =,...,, an for large, 4. A i,k B j,k k= Let Y µ, σ. Then we have 4. Prob{ Y µ ɛ} exp µ, σ. } { ɛ σ an we may pick ɛ Oσ to ensure that Y µ < ɛ with high enough probability. Setting ɛ = ησ gives Y µ ησ, µ + ησ ] with probability greater than e η, an we have a confience interval that epens on the variance of Y an whose probability is controlle by the constant η > 0. We compute the size of the entries in an etail the norm bouns in Appenix E. The main result is now state after efining some constants A Tighter Main Result. Just as we neee notation for the sake of the reaability of Main Result, we now efine new constants for Main Result : CC ij = LC ij = ] + Knn ij Kmn ij + 4ηC K ik j η CC K ij nn η CC Kmn ij η CK i K j + ]}, η LC 3 K j + η L κ j i 3 ηlc 5 ηlc

24 4 D.. KASLOVSKY AD. G. MEYER ] η +K j L η C +, CE i = η CE 3Knn ii + Kmn ii + η E K i + η ] C +, EEx = σ x η EE + ηee x + η ] x E + x, Aitionally, we will nee D D K = K = K = i=+ j=+ D CC ij LC ij i= j=+ D i=+ CE i,, L = ] D η LE + η L η E,, E = D ] η EE + ηe, an ζ = K σ D + r max + EE + EED. max + L σ + r max max After using the triangle inequality in the exact same manner as in the beginning of Section 4, combining the norm bouns compute in Appenix E with Theorem. yiels a new main result. Assume that conitions an hol. Then using the rescale notation r = r max / max /, we have: Main Result. 4.3 P P + + L σr r + K r 4 4+ σ + D ζ K r 3 + K σr + ] + + σ E

25 OPTIMAL TAGET PLAE RECOVERY 5 with probability greater than ] D e η CC + e η CC D De η EE D D e η EE e η LC + e η LC ] De η LE DD e η CE e η L e η C De η E over the joint ranom selection of the sample points an ranom realization of the noise. The comments following Main Result apply here as well. In particular, conitions an have the same interpretation an the enominator δ controls the conitioning of the recovery problem. urther, we note that the leaing orer behavior of the perturbation norms has not change. Main Result exhibits the same behavior as Main Result, but provies a tighter tracking of the true subspace recovery error, as will be shown in Section 5. Thus this analysis gives rise to the same interpretable boun as 4.7, up to multiplicative constants. The same geometric uncertainty principle applies as well. Table 4. shows the leaing orer behavior for each perturbation norm. ote that r max / max has been replace by r an only the leaing orer in is shown. This sie-by-sie comparison reveals the reasons why Main Result is a tighter boun. As suprema terms are replace by variance terms, the CLT result introuces powers of / that reuce the size of many norms. Aitionally, the approach of Main Result allows for a more precise treatment of the principal curvatures, most importantly for the CL T term. This precise treatment allows Main Result to relax the assumption that all principal curvatures in coimension i have the same sign, as require by Main Result. inally, notice that Theorem 3. introuces probability constants of the form + η, whereas the CLT introuces probability constants of the form η. Thus the CLT yiels tighter bouns Consistency with Previously Establishe Results. In 9], Singer an Wu stuy local PCA for tangent plane recovery in the absence of noise. The covariance matrix is ecompose following the assumption that for a given r, the number of points in a ball of raius r is large, implying a moel with variable ensity. ixing a ensity allows us to translate their results to our moel. Then the covariance matrix ecomposition yiels error terms corresponing to curvature of size Or 4 an finite-sample variance of sizes Or /, Or 3 /, an Or 4 /. Recalling that CCT

26 6 D.. KASLOVSKY AD. G. MEYER orm Main Result top / Main Result bottom U T C C T U ] 4 D D r + K ij 3 nn Kmn] ij i=+ j=+ ] D D + Knn ij Kmn] ij r 4 3 U T ẼẼT U σ σ i=+ j=+ U T σ D ẼẼT U σ D U T σ D ] ẼẼT U + ξe + D + ξe + ηee σ D ηee U T C L T r 3 U K + η lc D r 3 3 i= j=+ U T Ẽ L T σ r + ξe + ηle U σ r η LE U T Ẽ L T U σ r D + ξ e + ηle σ r D η LE U T σ r CẼT U K + ξe + ηce D σ r η CE 3K ii nn + K ii +4 i=+ ] K jη LC + ηl 3 + κ j i η LC 5 ηlc mn + U T σ r CẼT U K D + ξ e + ηce D σ r D η CE 3K ii nn + K ii +4 i=+ ] η E K i + mn + ] η E K i + Table 4. Comparison of leaing orer perturbation terms for Main Result top an Main Result bottom. otationally, r max / max has been replace by r an only leaing orer terms are shown.

27 OPTIMAL TAGET PLAE RECOVERY 7 Or 4 + Or 4 / an CLT Or 3 /, our analysis recovers the same leaing orer behavior as reporte by Singer an Wu in the noise-free setting. As previously iscusse, aler presents a finite-sample PCA analysis in 4] assuming a linear moel. Setting curvature terms in Main Results an to zero recovers aler s leaing orer boun on the angle between the finite-sample eigenvectors an the true eigenvectors. In our notation, aler reports that, to leaing orer, the angle is boune by sin θû,u σ λ D + Oσ, where is taken to be one. We now show that our results recover this leaing orer behavior. irst, set all curvature terms to zero. ext, assume conition hols such that the enominator δ is sufficiently well conitione an we may neglect all terms other than λ. Using the more compact notation of r in place of r max / max an following the approach in 4] of retaining only leaing orer terms an ropping probability constants, we have sin θû,u D σr + σ r + = σ r + D + Oσ. Taking = an noticing that λ r yiels aler s boun. 5. umerical Results. In this section we emonstrate that the bouns of Main Results an accurately an efficiently track the true subspace recovery error an may therefore be use to obtain the optimal scale for tangent plane recovery. We then aress the case of ata sample from a sale such that the principal curvatures are of mixe signs that brings to light a particular ifference between the two main results. We explain an numerically emonstrate the behavior of the true subspace recovery error at large scales an the corresponing lack of subspace tracking an connect this observation to the crossover phenomenon reporte in 4]. inally we emonstrate the accurate an efficient recovery of local curvature. 5.. Subspace Tracking an Recovery. We generate a ata set sample from a 3-imensional manifol embee in R 0 accoring to the local moel.. The raius of the local moel is set to r max = an = points are uniformly sample from the tangent plane. ote that 0 0 6, an thus this choice of represents reasonable sampling in R 0. Curvature

28 8 D.. KASLOVSKY AD. G. MEYER an the stanar eviation σ of the ae Gaussian noise will be specifie in each experiment. We compare our bouns with the true tangent plane recovery error. The tangent plane at reference point x 0 is compute at each scale via PCA of the nearest neighbors of x 0. The true tangent plane recovery error P P is then compute at each scale. ote that computing the true error requires SVDs. A true boun is compute by applying Theorem. after measuring each perturbation norm irectly from the ata. While no SVDs are require, this true boun utilizes information that is not practically available, an therefore represents the best possible boun that we can hope to achieve. We will compare the mean of the true error an mean of the true boun over 0 trials with error bars inicating one stanar eviation to the following three curves:. Main Result holing with probability 0.5 magenta,. Main Result holing with probability 0.5 black, an 3. Main Result with all probability constants set to green. κ j i i = i = i = 3 j = 4,..., j = 7,..., Table 5. Principal curvatures of the manifol for igure -c. The thir curve abanons any guarantee of proviing an upper boun in favor of capturing the tren of the true error. This curve represents the case where we may not care to upper boun the error, but instea wish to track the height of the true error curve as closely as possible. We will refer to these curves as Main Result, Main Result, an Main Result -tren. The results are isplaye in figure. Panel a shows the noisy σ = 0.0 curvature-free linear subspace result. As the only perturbation is ue to noise, we expect the error to ecay as / as the scale increases. The curves are shown on a logarithmic scale for the Y-axis an ecrease monotonically, inicating the expecte ecay. As expecte, Main Result black is tighter than Main Result magenta an both accurately track the behavior of the true error blue. Main Result -tren green tightly tracks the same behavior, in this case sitting on top of the true boun re. Panel b shows the results for a noise-free manifol with principal curvatures given in Table 5. such that K =.605. otice that three of the coimensions experience high curvature while the others are flatter, giving a tube-like structure to the manifol. In this case, perturbation is