onto the column space = I P Zj. Embed PZ j

Transcription

1 EIGENVALUES OF AN ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING CHI-KWONG LI, REN-CANG LI, AND QIANG YE Abstract Alignment algorithm is an effective method recently proposed for nonlinear manifold learning (or dimensionality reduction By first computing local coordinates of a data set, it constructs an alignment matrix from which a global coordinate is obtained from its null space In practice, the local coordinates can only be constructed approximately and so is the alignment matrix This together with roundoff errors requires that we compute the the eigenspace associated with a few smallest eigenvalues of an approximate alignment matrix For this purpose, it is important to know the first nonzero eigenvalue of the alignment matrix or a lower bound in order to computationally separate the null space This paper bounds the smallest nonzero eigenvalue, which serves as an indicator of how difficult it is to correctly compute the desired null space of the approximate alignment matrix Key words Smallest nonzero eigenvalues, alignment matrix, overlapped partition, nonlinear manifold learning, dimensionality reduction subject classifications 65F15, 15A18 1 Introduction Given an N l matrix Z and s submatrices Z j C kj l (for 1 j s consisting of certain rows of Z, let P Zj be the orthogonal projector in C k j onto the column space of Z j, and PZ j = I P Zj Embed PZ j into C N N according to the position of the rows of Z j in Z and denote the resulting N N matrix by Φ j (see (3 in Section for details The matrix P s Φ j (11 j=1 is called an alignment matrix This definition is abstracted from and slightly more general than the one in [9], where Z s first column is all ones Nonetheless most analysis and the results there regarding the null space of P can be carried over in a straightforward way For example, it is proved under a condition called fully overlap among {Z j } that the null space of P is the span of Z [9] With this property of the alignment matrix, we can reconstruct the rows of Z, up to a linear transformation, from the local projectors P Zj This forms a theoretical basis for the LTSA (Local Tangent Space Alignment algorithm of [11] recently developed for the problem of nonlinear manifold learning In nonlinear manifold learning [5, 8], one is concerned with determining a suitable parametrization for a set of given high-dimensional data points lying in a nonlinear Department of Mathematics, College of William and Mary, Williamsburg, VA (ckli@mathwmedu C-K Li is an honorary professor of the University of Hong Kong, and an honorary professor the Heilongjiang University His research was partially supported by a USA NSF grant, and a HK RGC grant Department of Mathematics, University of Texas at Arlington, PO Box 19408, Arlington, TX (rcli@utaedu Supported in part by the National Science Foundation CAREER award under Grant No CCR and by the National Science Foundation under Grant No DMS Department of Mathematics, University of Kentucky, Lexington, KY (qye@msukyedu Supported in part by the National Science Foundation under Grants CCR and DMS

2 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING manifold, which is also known as (nonlinear dimensionality reduction Several methods have been proposed recently for this problem [1, 3, 5, 7, 11] The alignment matrix was first introduced in the LTSA method [11] in which a local coordinate system is first constructed for a small neighborhood (ie, a patch around each sample points and all local coordinates are then aligned together to arrive at a global coordinate The process of aligning the local coordinates together is achieved through the alignment matrix We note, however, that the alignment matrix can be used in a more general setting to align coordinates for subsets of data points that are not necessarily local [9] In this context, the rows of Z are the unknown global coordinates of the high-dimensional data points and the rows of Z j correspond to the coordinates of the data points in a subset (eg, a local patch Then the rows of Z, up to a linear transformation, is constructed from the projectors P Zj by computing the null space of the alignment matrix In practice, only an approximation of the alignment matrix is available This together with roundoff and/or data errors require that we compute the eigenspace associated with a few smallest eigenvalues that are considered the perturbations of the zero eigenvalues However, if the perturbations cause the zero eigenvalues to become as large in magnitude as the smallest nonzero eigenvalue of the alignment matrix, it is not possible to determine how many smallest eigenvalues and which of them should be considered zeros For this purpose, it is important to investigate the first nonzero eigenvalue of the alignment matrix or a lower bound in order to computationally separate the null space This paper presents a lower bound on the smallest nonzero eigenvalue, which serves as an indicator of how difficult it is to correctly compute the desired null space of the alignment matrix An implication of our bound is that the smallest nonzero eigenvalue depends on the amount of overlap among Z j and hence it is necessary to maintain sufficient overlap among Z j in practice Our study is based on an ideal situation, namely P is uncontaminated, while contaminated P in practice likely has no nonzero eigenvalues Thus such simplification becomes somewhat necessary Nevertheless our effort here represents a step forward to acquire better understanding towards instructively how much overlaps among Z j for robust recovery of Z, which, translated into the language of nonlinear manifold learning [9, 11], how much overlaps among local patches for robust recovery of global coordinates Our investigation into the null space and eigenvalues of this so-called alignment matrix P, abstracted from and slightly more general than its counterpart in nonlinear manifold learning, may be of interest in its own right from matrix theoretical point of view The rest of this paper is organized as follows In Section, we set up our framework to study the alignment matrix We then derive the lower bound at stages, first for the case s = in Section 3 and then for the general case in Section 4 We shall also discuss when the fully overlap condition is a necessary condition in Section 4 Notation As we have done already, denote by C m n the set of all m n complex matrices, C n = C n 1, and C = C 1 Denote by I n the n n identity matrix, and sometimes simply I when its size is clear from the context Let X be the spectral norm of a matrix X, ie, its largest singular value, and eig(x be the set of the eigenvalues of a square X X and X T denote the conjugate transpose, the transpose of a matrix or vector X, respectively X Y for two Hermitian matrices X and Y means that Y X is positive semi-definite, and accordingly X Y means Y X For 1 i j n, i : j is the set of integers from i to j inclusive and i : i = {i} For

3 CHI-KWONG LI, REN-CANG LI, and QIANG YE 3 vector u and matrix X, u (j is u s jth entry, X (i,j is the (i,jth entry of X Moreover, subvector u (I consists of all entries i I; submatrices X (I,J, X (I,:, and X (:,J consist of intersections of all rows i I and all columns j J, all rows i I and all columns, and all rows and all columns j J, respectively Alignment Matrix Material in this section is essentially taken from [9], but stated in a slightly more general term, namely in [9] Z s first column is all ones, which is not required here Also to allow a bit more generality, we assume all involved numbers are complex, unless otherwise explicitly stated In the context of nonlinear manifold learning [9], most likely they are real Let Z C N l, and N > l Suppose Z j C kj l for 1 j s are submatrices of Z and each consists of certain rows and all columns Let I j = {j 1,j,,j kj } be the index set for the rows of Z j as the rows of Z, ie, Assume throughout this paper Z j = Z (Ij,: = (I N (Ij,: Z C k j l (1 s I j = {1,,,N}, ( j=1 ie, each row of Z appears in at least one of the Z j Let P Zj be the orthogonal projector in C kj onto the column space span(z j of Z j, and P Z j = I P Zj is the orthogonal projector also in C kj but onto the orthogonal complement of span(z j It is known that P Zj = Z j Z j, where Z j is the Moore-Penrose inverse [6] of Z j In particular P Zj = Z j (Z j Z j 1 Z j if Z j has full column rank Let Φ j be the embedding of P Z j into C N, ie Φ j = [ (I N (Ij,:] T P Zj (I N (Ij,: C N N (3 Finally an N N matrix P is constructed as P = s Φ j (4 It can be verified that P Z = 0, ie, span(z null(p, the null space of P j=1 Definition 1 This definition is recursive 1 Z i always fully overlaps itself regardless of its rank; Z i and Z j for i j are fully overlapped, if Z T (Ii Ij,: has full column rank; 3 The collection Z = {Z j,1 j s} for s 3 is fully overlapped, if it can be partitioned into two nonempty disjoint subsets Z 1 and Z each of which is a fully overlapped collection and that Z ( e I1,: and Z ( e I,: are fully overlapped, where Ĩ i = I j (5 Z j Z i

4 4 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING This definition is rather general For example it encompasses the following case: the partitioning graph of Z is connected By the partitioning graph of Z we mean a graph whose vortices are submatrices Z j and there is an edge connecting two vortices Z i and Z j if and only if Z i and Z j for i j are fully overlapped Theorem Assume ( holds If {Z j,1 j s} is fully overlapped, then null(p = span(z In nonlinear manifold learning, Z j is not known but an approximations to PZ j can be computed; so is an approximation to P whose eigenspace associated with a few smallest eigenvalues will then give an approximation to the column space of Z This theorem says if PZ j is exactly known, the column space of Z can be recovered exactly as null(p Theorem is an extension of the main result in [9] and can be proved by a minor modification to the argument in [9] Later our method for deriving the eigenvalue bound will lead to another proof of the result Corollary 3 Under the conditions of Theorem, λ + min (P P Z P λ max (P P Z, where λ + min (P is the smallest nonzero eigenvalue of P, and λ max(p is the largest eigenvalue of P Proof Since P is Hermitian, it has eigen-decomposition P = UΛU, where U is unitary and Λ is diagonal with the last l diagonal entries being zero Thus which implies N l ( λ + min (P N l I l l 0 Λ λ max (P ( N l l N l I l 0 λ + min (P U (:,1:N l[u (:,1:N l ] P λ max (P U (:,1:N l [U (:,1:N l ] Notice null(p = span(z = null(pz by Theorem to conclude that span(u (:,1:N l = span(pz and thus P Z = U (:,1:N l[u (:,1:N l ] By construction, it is clear λ max (P = P s However, there is not much we can say about λ + min (P at this point The main contribution of this paper is to present a lower bound of it 3 The case of two submatrices Without loss of generality, upon permuting rows of Z we may take ( l ( l m Z 1 = 11 Z 11 m, Z m 1 Z = 1 Z 1, (31 1 m Z where Z 1 = Z 1 is the common part in Z 1 and Z, m 1 = m 1 Then ( m 11 +m 1 m ( m 11 m 1 +m m P = 11 +m 1 PZ 1 0 m m 0 0 m 1 +m 0 PZ (3 Theorem says if Z 1 has full column rank (ie, Z 1 and Z are fully overlapped, then dimnull(p = l and in fact null(p = span(z which implies P has exactly l zero eigenvalues We would like to know more about its nonzero eigenvalues, too We shall start by looking into the eigen-structure of P without assuming Z 1 and Z are fully overlapped and then specialize the results to the fully overlapped case,

5 CHI-KWONG LI, REN-CANG LI, and QIANG YE 5 31 Z 1 and Z not necessarily fully overlapped The case when m 1 = 0, ie, there is no overlap at all between Z 1 and Z is not interesting, because then ( P P = Z1 a direct sum of two orthogonal projectors whose eigenvalues are either 1 or 0; the case when either m 11 = 0 or m = 0, ie, one of Z i is part of the other, is not particularly interesting, either, because, say, if m 11 = 0, then P Z P P Z So we shall assume m 1 1, m 11 1, and m 1 in the rest of this section The key idea of our analysis below is to find an N N unitary matrix Q so that Q P Q has simple structure to allow us to determine the null space and the eigenvalues of P Theorem 31 Assume m 1 1, m 11 1, and m 1 Z 11, Z 1 = Z 1 and Z admit the following decompositions P Z ( r 1 r l r 1 r r Z 11 = U M1 Σ 0 m 11 r M ( r 1 l r 1 I 0 0 V V1, (33 ( r 1 l r 1 r Z 1 = Z 1 = U 1 1 Σ 1 0 m 1 r ( r 1 r 3 l r 1 r 3 r Z = U 3 3 M Σ 3 0 m r 3 M 0 0 V 1, (34 ( r 1 l r 1 I 0 0 V3 V1, (35 where U 1 (m 1 m 1, U (m 11 m 11, U 3 (m m, V 1 (l l, and V and V 3 (both (l r 1 (l r 1 are unitary, Σ 1 and Σ are diagonal with positive diagonal entries In particular r 1 = rank(z 1, r = rank((z 11 V 1 (:,r1 +1:l, r 3 = rank((z V 1 (:,r1 +1:l (36 Proof Equation (34 is the singular value decomposition (SVD of Z 1 Consider the submatrix of the last l r 1 columns of Z 11 V 1 and let its SVD be ( r l r 1 r r (Z 11 V 1 (:,r1 +1:l = U Σ 0 m 11 r 0 0 V (37 Now notice U Z 11 V 1 = (U (Z 11 V 1 (:,1:r1 U (Z 11 V 1 (:,r1 +1:l, together with (37, to arrive at (33 with M 1 and M 1 being the top r rows and the bottom m 11 r rows of U (Z 11 V 1 (:,1:r1, respectively Similarly let the SVD of the submatrix consisting of the last l r 1 columns of Z V 1 be ( r 3 l r 1 r 3 r (Z V 1 (:,r1 +1:l = U 3 3 Σ 3 0 m r 0 0 V 3 (38 to lead to (35 with M and M being the top r 3 rows and the bottom m r 3 rows of U 3 (Z V 1 (:,1:r1, respectively

6 6 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING In what follows, we shall use X cols Y to mean span(x = span(y for convenience Note from (34 that ( r 1 l r 1 r Z 1 = Z 1 = U 1 1 Σ 1 0 m 1 r ( r 1 l r 1 I 0 0 V V1 for any (l r 1 (l r 1 matrix V In particular set V = V and V 3 to get where Set ( ( U Z11 cols U 1 Z 1 r 1 r r M1 Σ m 11 r M 1 0 r 1 Σ 1 0 m 1 r cols Z def 1 = cols r 1 r r 0 I m 11 r W 1 0 R1 r 1 I 0 ( I m 1 r r 1 r r W1 I m 11 r W 1 0 r 1 I 0 m 1 r 1 0 0, (39 W 1 = M 1 Σ 1 1, R 1 = (I +W 1 W 1 1/ (310 m 11 r m 1 r 1 r 0 0 Z 1 m = 11 r I 0 r 1 W1 0 m 1 r 1 0 I ( D1 I with D 1 = (I +W 1 W 1 1/ Then ( Z 1 Z 1 is unitary Thus, the column space of Z 1 is a basis for the orthogonal complement of span( Z 1 in C k1 Similarly, where and ( U 1 U 3 ( Z1 Z cols r 1 r 3 r 1 r 3 r 1 Σ 1 0 r 1 I 0 m 1 r r 3 M Σ 3 cols Z m = 1 r R r 3 0 I (, I m r 3 M 0 m r 3 W 0 (311 W = M Σ 1 1, R = (I +W W 1/, (31 r 1 W 0 Z m = 1 r 1 0 I r m r 3 I 0 m r 3 m 1 r 1 ( D I with D = (1+W W 1/

7 CHI-KWONG LI, REN-CANG LI, and QIANG YE 7 has orthonormal columns spanning the orthogonal complement of span( Z in C k Let ( m G 1 = 11 +m 1 Z 1 m 0 ( m, G = 11 0, m 1 +m Z and r m 11 r D r G = (G 1 G = 1 W1 D 1 0 W D 0 m 1 r 1 0 I 0 I r m r D 0 m 11 r m 1 r 1 m r 3 m 1 r 1 Set Q def = Then Q is a unitary matrix, and m 11 U m 11 m 1 m m 1 U 1 m U 3 (313 P def = Q P Q = Q Φ 1 Q+Q Φ Q = G 1 G 1 +G G = GG Note also that the null space of P is the same as the null space of G, which is the same as the orthogonal complement of the column space of G Let r r 1 r 3 r I 0 0 m 11 r 0 W 1 0 r G 3 = 1 0 I 0 m 1 r r I m r 3 0 W 0 Note that in G, the 4th block column is the same as the nd one, and the first 3 block columns are linear independent Therefore rank(g = m 11 +m 1 +m (r 1 +r +r 3 which implies dimnull(g = r 1 +r +r 3 Evidently, rank(g 3 = r 1 +r +r 3 Therefore null( P = null(g = span(g 3 because G G 3 = 0 Theorem 3 Let all symbols keep their assignments so far in this section Then 1 dimnull(p = dimnull( P = r 1 +r +r 3 ; null( P is the column space of G 3 and null(p = Qnull( P 3 Suppose Z 1 and Z have full column rank Then null(p = span(z if and only if Z 1 and Z are fully overlapped Proof Only Item 3 needs a proof If Z 1 and Z are fully overlapped, then r 1 = l and r = r 3 = 0 which imply dimnull(p = l by Item 1 Now dimspan(z = l and span(z null(p as noted before imply null(p = span(z Suppose Z 1 and Z are

8 8 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING not fully overlapped Then r 1 < l Noticing that r 1 +r = l = r 1 +r 3 because Z 1 and Z have full column rank, we have dimnull(p = r 1 +r +r 3 > l+(l r 1 > l, and thus null(p span(z Remark 31 The third assertion in Theorem 3 was also obtained by Zha and Zhang [10] for Z j whose first column is all ones Now let us look at the eigenvalues of P, which are the same as those of P = GG Apart from additional zeros, they are the same as those of m 11 r I 0 D 1 W 1 W D 0 G m G = 1 r 1 0 I 0 I m r 3 D W W1 D 1 0 I 0 m 1 r 1 0 I 0 I m 11 r m 1 r 1 m r 3 m 1 r 1 which is permutationally similar to a direct sum of ( ( I I I D and 1 W 1 W D I I D W W1 D 1 I The former matrix has nonzero eigenvalue with multiplicity m 1 r 1 ; the latter matrix has eigenvalues 1±σ j for j = 1,,k, where σ 1,,σ k are the nonzero singular values of D 1 W 1 W D, and the remaining eigenvalues equal to 1 Thus, we have the following Theorem 33 Let the nonzero singular values of D W W 1 D 1 be σ 1,σ,,σ k Then eig(p consists of 1±σ j, for 1 j k,, with multiplicity m 1 r 1, 1, with multiplicity m 11 +m r r 3 k, 0, with multiplicity r 1 +r +r 3 We shall now bound the singular values σ j of D W W 1 D 1 First we have σ j D W W 1 D 1 = D W D 1 W 1, (314 D i W i = W i ( Wi It follows from (33, (34, and (310 that ( Z 11 Z Σ 1 1 = Z 1 11V 1 U1 = ((Z 0 11 V 1 (:,1:r1 Σ 1 1 0U1, ( M1 ( M1 U (Z 11 V 1 (:,1:r1Σ 1 1 = Σ 1 Σ 1 = 1 1 W 1 They yield M 1 W 1 U (Z 11 V 1 (:,1:r1Σ 1 1 = Z 11 Z 1 with equality if M 1 = 0 or r = 0 (316

9 CHI-KWONG LI, REN-CANG LI, and QIANG YE 9 Since the construction of W is similar to the construction of W 1, one can give a similar bound to W, namely, Combine (314 (317 to get W Z Z 1 with equality if M = 0 or r 3 = 0 (317 We have proved σ j Z 11Z 1 Z Z 1 1+ Z 11 Z 1 1+ Z Z 1 Theorem 34 The nonzero eigenvalues of P is no smaller than 1 τ where τ def = Z 11 Z 1 1+ Z 11 Z 1 Z Z 1 ( Z Z 1 Its largest eigenvalue is no greater than 1+τ if m 1 = r 1 and it is if m 1 > r 1 3 Z 1 and Z fully overlapped When Z 1 and Z are fully overlapped, results in the previous subsection are still valid, only simpler Here is the list of a few notably changes to what in the previous subsection: r 1 = l and r = r 3 = 0; Decompositions (37 and (38 are not needed, and consequently the decompositions in Theorem 31 are simplified to Z 11 = M 1 V 1, Z 1 = U 1 Σ 1 V 1, Z = M V 1 (39 and (311 remain valid with U = I and U 3 = I; (316 and (317 are equalities, and in fact W i = Z ii Z 1 U 1 for i = 1,; Theorems 33 and Theorems 34 are valid as they are, and furthermore Theorem 34 has a stronger version Theorem 35 below Theorem 35 Let τ be defined by (318 If Z 1 and Z are fully overlapped, then Furthermore, λ + min (P 1 (1 τp Z P { } (1+τ if m1 = l, P if m 1 > l Z (319 ( σ min (Z 1 σmax(z 11 + σ min (Z /( 1 σmax(z 1+ σ min (Z 1 σmax(z 11 + σ min (Z 1 σmax(z, (30 where σ min and σ max denote the smallest and the largest singular value respectively Proof (319 is a consequence of Item 3 of Theorem 3 and the proof of Corol-

10 10 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING lary 3 From (318, we have as expected λ + min (P Z 11 Z 1 1+ Z Z 1 (1+ Z 11 Z 1 (1+ Z Z 1 1 (1+ Z 11 Z 1 (1+ Z Z 1 Z 11 Z 1 + Z Z 1 (1+ Z 11 Z 1 + Z Z 1 σmin = (Z 1/σmax(Z 11 +σmin (Z 1/σmax(Z [1+σmin (Z 1/σmax(Z 11 +σmin (Z 1/σmax(Z ], The theorem implies that if Z 1 has full column rank but with nearly linearly dependent columns, σ min (Z 1 is small and the smallest nonzero eigenvalue λ + min (P may also and can be nearly zero In particular, λ + min (P may be of order σ min (Z 1 Remark 3 Independently, Zha and Zhang [10, Theorem 51] obtained, in our notation, the following result (original version was for Z j whose first column is all ones: Let PZ j = Q j Q j (j = 1, and partition Q 1 = ( m 11 Q 11 m 1 Q 1, Q = ( m 1 Q 1 m Q conformally to those in (31 If Z 1 and Z are fully overlapped, then the smallest nonzero eigenvalue of P is given by 1 σ max (Q 1Q 1 This will obviously give the same smallest nonzero eigenvalue of P as one can deduce from Theorem 33, but since Q j depends on Z j in a nontrivial way, ie, there is no explicit expression to write down Q j in terms of Z j, it is not clear if one could establish any lower bound based on 1 σ max (Q 1Q 1 in terms of Z j, as we did in Theorem 35 based on Theorem 33 Zha and Zhang also extended their result for more than two submatrices, the case we will be dealing with in the next section Next we give an example to show that the bound in Theorem 35 can be asymptotically attained and hence sharp Example 31 Consider m 11 = 1 = m, m 1 =, and l = : ( 1 a ( 1 c 1 Z11 Z 1 = Z1 1 c Z 1, Z = 1 c 1 Z, 1 c 1 b assuming c 1 c, ie, Z 1 Z 1 is nonsingular All numbers are real Z as such comes from nonlinear manifold learning [9] Calculation by Maple 1 shows that the characteristic polynomial of P is ( 1 λ λ λ+ (c 1 c, 1 1

11 CHI-KWONG LI, REN-CANG LI, and QIANG YE 11 where 1 = (a c 1 +(a c +(c 1 c, = (b c 1 +(b c +(c 1 c, = (a c 1 +(a c +(c 1 c +(b c 1 +(b c +(a b So there are two zero eigenvalues and two nonzero ones, as expected The two nonzero eigenvalues are 1 (c 1 c 1 (c 1 c = (c 1 c, 1 (c 1 c 1+ 1 (31 Calculations also show 1 (c 1 c = [(c a(c b+(c 1 a(c 1 b] Now let us look at what our bounds by Theorem 35 say We have Z 1 Z 1 1 = 1 ( c c 1, c c Z 11 Z 1 1 = 1 c c 1 (c a a c 1, Z Z 1 1 = 1 c c 1 (c b b c 1 The lower and upper bounds by Theorem 34 are (a c1 1± +(a c (b c1 +(b c (3 1 which can be verified to be exactly the two values in (31 if (c a(c b+(c 1 a(c 1 b = (a c 1 +(a c (b c 1 +(b c, ie, when two vectors (c a a c 1 and (c b b c 1 are parallel, which happens when c 1 = c When c 1 = c, however, {Z 1,Z } is not fully overlapped But by making c 1 c while as close as needed, {Z 1,Z } is fully overlapped and at the same time the lower and upper bounds by Theorem 34 can be made as close to the two values in (31 as wished 4 The case of more than two submatrices In general for s 3, our approach in the previous section appears to break down In what follows, we shall describe a way to recursively bound the smallest nonzero eigenvalue λ + min (P from below To do so, we define a function τ which takes two submatrices of Z with all columns as arguments Given Z i = Z ( e Ii,:, i = 1,, we define τ( Z 1, Z def t 1 t =, t 1+t 1 1+t i = Z (J i,: Z ( e T I 1 ei,:, (41

12 1 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING where J i is the complement set of Ĩ1 Ĩ in Ĩi Throughout the rest of this section, we adopt in whole the notation associated with Z and Z j as introduced in Section, and we assume that Z = {Z j,1 j s} is fully overlapped, and ( holds From Definition 1, Z can be partitioned into two nonempty disjoint subsets Z 1 and Z each of which is a fully overlapped collection and that Z 1 = Z ( e I1,:, Z = Z ( e I,: (4 are fully overlapped, where Ĩ1 and Ĩ are defined as in (5 By Theorem 35, we have [ 1 τ( Z 1, Z ] PZ (I N T ( e P I j,: Zj e (I N ( e Ij,: j=1 Now recursively bound P e Zj in exactly the same way because Z j is fully overlapped until the right-hand side becomes P The following procedure recursively computes α(z that satisfies α(zpz P : α({z i } = 1, (43 α({z i,z j } = 1 τ(z i,z j, (44 [ α(z = 1 τ( Z 1, Z ] min{α(z 1,α(Z } (45 The smallest nonzero eigenvalue λ + min (P is then no smaller than α(z Theorem 41 Suppose Z = {Z 1,Z,,Z s } is a fully overlapped collection, where Z j are submatrices of Z C N l as defined by (1 and ( Let α(z be computed recursively by (43 (45 Then α(zpz P, where alignment matrix P is defined by (4 Example 41 Consider s = 3 Suppose Z 1 and Z def = Z S (I I3,: are fully overlapped Let Ĩ def = I I3 Then we have by Theorem 35 [ 1 τ(z 1, Z ] PZ (I N T (I 1,: P Z 1 (I N (I1,: +(I N T ( e P I,: Z e (I N ( e I,:, and 3 [1 τ(z,z 3 ](I N T ( e P I,: Z e (I N ( e I,: (I N T (I j,: P Z j (I N (Ij,: j= Put the two inequalities together to get α(zpz P with [ α(z = 1 τ(z 1, Z ] [1 τ(z,z 3 ] Example 4 Consider s = 4 Suppose Z 1 and Z, Z 3 and Z 4, and Z def 1 = Z S (I1 I,: and Z def = Z S (I3 I4,: are fully overlapped pairs Let Ĩ1 def = I 1 I and Ĩ def = I 3 I4 Then we have by Theorem 35 [ 1 τ( Z 1, Z ] PZ (I N T ( e P I 1,: Z1 e (I N ( e I1,: +(I N T ( e P I,: Z e (I N ( e I,:,

13 and CHI-KWONG LI, REN-CANG LI, and QIANG YE 13 [1 τ(z 1,Z ](I N T ( e P I 1,: Z1 e (I N ( e I1,: (I N T (I P j,: Z j (I N (Ij,:, 4 [1 τ(z 3,Z 4 ](I N T ( e P I,: Z e (I N ( e I,: (I N T (I j,: P Z j (I N (Ij,: Put the three inequalities together to get α(zpz [ P with α(z = 1 τ( Z 1, Z ] min{[1 τ(z 1,Z ],[1 τ(z 3,Z 4 ]} j=1 j=3 Example 43 This is for the sequentially fully overlapping case, ie, Z i and Z j are fully overlapped if i j = ±1 and Z i and Z j have no overlap at all if i j Write Z j = m j1 m j m j3 l Z j1 Z j Z j3, m 11 = m s3 = 0, k j = m j1 +m j +m j3, where Z j1 = Z j 13 is the overlapped part between Z j 1 and Z j Recursively, we have α({z 1,,Z s } = [ 1 τ(z (1:p+mj 13,:,Z (p+1:s,: ] where p = j 1 i=1 (m i1 +m i, and t 1 = Z (1:p,: Z j 13, t = min{α({z 1,,Z j 1 }, α({z j,,z s }}, Z (p+mj1 +1:s,:Z j 1, τ(z (1:p+mj 13,:,Z (p+1:s,: = t 1 1+t 1 t 1+t The ending conditions (43 and (44 still apply For shortest recursion, j should be picked about s/, eg, the smallest integer that is no less than s/ To see how good our recursive bound is, we have tested on random Z and random Z with its first column being all ones to mimic cases from nonlinear manifold learning Figure 41 plots λ + min (P vs α(z for s 10 and l 5 Our bounds for small s (about s 4 here, especially for s = look pretty good; however, they very much underestimate λ + min (P for big s (about s > 4 here Example 44 Consider Z C N with the first column Z (:,1 being all ones and the second column Z (:, being 1,,,N, and Z j = Z (j:j+,: (1 j s = N This corresponds to the one-dimensional case in manifold learning with data points equidistantly sit on a straight line The case is so special that it allows us to estimate more precisely our bound and λ + min (P through analytical means It can be seen that 1 1 PZ j = , P =

14 14 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING l= l=3 l= l= s s s s l=4 l=5 l=4 l= s s s s Fig 41 λ + min (P (red vs α(z (green : m j1 = m j3 = l and m j = l 1 (except m 11 = m s3 = 0 Left 4 plots: random Z; Right 4 plots: random Z (except Z (:,1 all ones It can be verified that 6P = U T U, where U = ( e 1 T e s C s N, e 1 and e s are the first and last column of the identity matrix I s, and T C s s is the famous tridiagonal Toeplitz matrix with diagonal entries and off-diagonal entries 1 Thus 6λ + min (P is the smallest eigenvalues of UU T = e 1 e T 1 +T +e s e T s T The eigenvalue system of T is explicitly known [, 4], and so is T : T = QΛQ T, where Λ = diag(λ 1,,λ s with λ j = +cosθ j, θ j = j s+1 π, Q (i,j = ( sin π ij s+1 s+1 for 1 i, j s Therefore λ + min (P 1 6 ( cosθ 1 = 8 3 sin4 θ 1 π 4 6(s+1 4 = π 4 6(N 1 4 (46 for large N We now establish an upper bound on λ + min (P Note that UU T = T (I +XX T T (1+ X T, X = (T 1 e 1 T 1 e s This implies λ + min (P 1 6 ( cosθ 1 (1+ X (47

15 We now bound X We have CHI-KWONG LI, REN-CANG LI, and QIANG YE 15 X T 1 e 1 + T 1 e s = T 1 e 1 = 8 s sin θ j s+1 λ = s cot θ j j s+1 j=1 s+1 cot θ π = θ s+1 cot π = θ s+1 cot π 8(N 1 π, π/ for large N Combine this with (47 to get π (s+1 j=1 cot tdt ( cott+ π t π/ π (s+1 ( π cot (s+1 + π (s+1 π λ + min (P 1 6 ( cosθ 1 (1+ X 4π 3(N 1 3 (48 Next we estimate what we may expect from our bound α(z It can be seen that a key step in our recursive procedure for α(z is for 1 i 1 m Z 1 = 1 m, 1 m+1 Z =, 1 m+1 1 j where i < m < m+1 < j with m about half way between i and j Let Z 1 be their common part, Z 11 the part in Z 1 excluding Z 1, and Z the part in Z excluding Z 1 also We have m i+1 (m i 1 Z 11 Z 1 = 3 1, Z Z 1 = For large j i and m about half way between i and j, Z 11 Z 1 Z Z 1 1 ( 3/ j i 3 Consequently for large j i k=1 α({ Z 1, Z } ( j i 3 (j m 1 j m ( j i This implies α(z, modulo a constant factor, is approximately log N ( 3 3 ( 3 N/ k 3 log N log N (log N / 1 N log N = N, (49 (log N/ 3 log 3

16 16 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING where log N is the smallest integer that is no less than log N Compared to (47 and (48, this very much underestimates λ + min (P for large N, a conclusion similar to what we have made at the end of Example Necessary Condition In the case s =, Theorem 3 states that the fully overlapped condition is also a necessary condition for null(p = span(z, provided that all Z i have full column rank It turns out it is not a necessary condition in general when s 3 We shall first give a counterexample to illustrate this and then give a result on when null(p = span(z does not hold Example 45 Consider the 7 4 matrix Z = , Z 1 = Z (1:5,:, Z = Z (3:7,:, Z 3 = Z ({1,,5:7},: Then all Z i have full column rank and they are pairwise not fully overlapped Moreover, {Z 1,Z,Z 3 } is not fully overlapped Computation by Maple s nullspace(p gives a basis of 4 vectors, ie, dimnull(p = 4 which implies null(p = span(z, since P Z = 0 and rank(z = 4 While the fully overlapped condition is not a necessary condition, each Z i in this example is fully overlapped with the union of the remaining Z j s The following theorem shows that this is indeed necessary for null(p = span(z Theorem 4 Assume that Z j has full column rank for 1 j s and that Z = {Z j,1 j s} can be partitioned into two nonempty disjoint subsets Z 1 and Z such that the union set of Z 1 and that of Z are not fully overlapped, ie, Z 1 = Z ( e I1,:, Z = Z ( e I,:, (410 are not fully overlapped, where Ĩ1 and Ĩ are defined as in (5 Then span(z is a proper subspace of null(p Proof Without loss of generality, let Z 1 = {Z j,1 j p} and Z = {Z j,p+1 j s} Since Ĩ1 and Ĩ are not fully overlapped, it follows from Theorem 3 that dim null (I N T ( e P I j,: Zj e (I N ( e Ij,: > l j=1 Utilizing the fact that null(x +Y = null(x null(y for two positive semi-definite

17 X,Y 0, we also have CHI-KWONG LI, REN-CANG LI, and QIANG YE 17 null (I N T ( e P I j,: Zj e (I N ( e Ij,: = null j=1 ( ( (I N T ( e P I 1,: Z1 e (I N ( e I1,: null (I N T ( e P I,: Z e (I N ( e I,: null(φ 1 + +Φ p null(φ p+1 + +Φ s = null(p Thus, dimnull(p > l = dimspan(z and the theorem is proved The following is an interesting corollary that is not obvious from Definition 1 Corollary 43 Suppose Z = {Z 1,Z,,Z s } is a fully overlapped collection, where Z j are submatrices of Z C N l as defined by (1 and ( Then for any two nonempty disjoint subsets Z 1 and Z of Z, Z 1 = Z ( e I1,:, Z = Z ( e I,:, must be fully overlapped, where Ĩ1 and Ĩ are defined as in (5 Remark 41 Zha and Zhang [10, Theorem 3] also established some necessary conditions for null(p = span(z in terms of full overlap as well as connected overlap for Z j whose first column is all ones 5 Conclusions We have studied the eigenstructure of the alignment matrix P in a slightly more general context than in nonlinear manifold learning It is proved that α(zpz P under the condition that Z is a fully overlapped collection, where α(z > 0 is computed recursively For s =, the bound is no worse than proportional to the square of the ratio of the smallest singular value of the matrix in the overlapped part to the largest singular value of the matrix in the non-overlapped part and this ratio can be considered as a measure of the amount of overlap From the computational point of view, the bigger the smallest nonzero eigenvalue λ + min (P, the less difficult it is to recover null(p numerically Our lower bound can be used as an indicator on the difficulty of numerically recovering null(p Another implication of our result is concerned with how to make λ + min (P bigger increase the overlaps as one naturally expects But we provide a quantitative measure Our present study contributes to the theoretical foundation of the LTSA algorithm [11] But further studies are necessary We mention two unanswered issues that need to be addressed in the future 1 For s =, our bound α(z is tight and asymptotically achievable, but for s 3, the recursively computed α(z > 0 depends on how Z is partitioned as in Definition 1 and could very much underestimate λ + min (P How do we improve the bound? Any practical use of our result here remains to be investigated because in practice data errors may and will complicate the analysis and must be taken into account We shall leave these issues, among others, to our future studies on the subject

18 18 ALIGNMENT MATRIX IN NONLINEAR MANIFOLD LEARNING Acknowledgment Theorem 31 was explicitly formulated at the suggestion from an anonymous referee who also suggested us to add a simpler example Example 44 Previously Theorem 31 was immersed in the text The authors wish to thank his comments that improved the presentation of this paper REFERENCES [1] M Brand, Charting a manifold, in Advances in Neural Information Processing Systems 15, S T S Becker and K Obermayer, eds, MIT Press, Cambridge, MA, 003, pp [] J Demmel, Applied Numerical Linear Algebra, SIAM, Philadelphia, 1997 [3] D Donoho and C Grimes, Hessian eigenmaps: new tools for nonlinear dimensionality reduction, Proceedings of National Academy of Science, 100 (003, pp [4] A E Naiman, I M Babuška, and H C Elman, A note on conjugate gradient convergence, Numer Math, 76 (1997, pp [5] S T Roweis and L K Saul, Nonlinear dimensionality reduction by locally linear embedding, Science, 90 (000, pp [6] G W Stewart and J-G Sun, Matrix Perturbation Theory, Academic Press, Boston, 1990 [7] Y W Teh and S Roweis, Automatic alignment of local representations, in Advances in Neural Information Processing Systems 15, S T S Becker and K Obermayer, eds, MIT Press, Cambridge, MA, 003, pp [8] J B Tenenbaum, V de Silva, and J C Langford, A global geometric framework for nonlinear dimensionality reduction, Science, 90 (000, pp [9] Q Ye, H Zha, and R-C Li, Analysis of an alignment algorithm for nonlinear manifold learning, BIT, to appear [10] H Zha and Z Zhang, Spectral analysis of alignment in manifold learning, in Proceedings of Acoustics, Speech, and Signal Processing (ICASSP 05, vol 5, March 005, pp [11] Z Zhang and H Zha, Principal manifolds and nonlinear dimension reduction via local tangent space alignment, SIAM J Sci Comput, 6 (004, pp