Near-Isometry by Relaxation: Supplement

Transcription

1 Near-Isometry by Relaxation: Supplement Proof of Proposition,3. We first prove the following Lemma. Anonymous Author(s) Affiliation Address Proposition. If f : S R D R is convex, non-negative and f exists for all x int S, then f (x) is convex. Proof ( f ) = f f; ( f ) = f f + f f which is positive definite whenever f f is.. Using the above Lemma, and the fact that H k I n is non-negative and infinitely differentiable almost everywhere, we obtain the desired result. Proof of Proposition sup u R s u Gu G G0+εI s = sup u 0 u G 0 u + ɛ u () u T Gu u T G 0 u + ɛ u = sup u 0 v G ɛ For (), we first prove the following fact v v with G ɛ = (G 0 + ɛi) / and v = G ɛ u () GG ɛ = G with G = (G 0 + ɛi) / G(G 0 + ɛi) / (3) = sup u Null G u T Gu u T G 0u+ɛ u if Null(G) = Null(G 0 ) max α +β = β λ (G)+α λ max(g)+αβθ max(g,g 0) β ɛ+α (λ min (G0)+ɛ) if Null(G) Null(G 0 ) (4) where λ max (G) is the spectral radius of G, Θ(G, G 0 ) = sup u = v =,v Null G,u Null G0 u Gv is the cosine of the principal angle between Null G and Null G 0, and λ min (G 0) is the smallest nonzero eigenvalue of G 0. u Denote for simplicity g) = T Gu u T G 0u+ɛ u. () If Null(G) = Null(G 0 ) then for u Null G the value is 0, which cannot be the sup. Let u = v u 0 with u 0 Null G, v Null G. Then u T G 0 u + ɛ u = v T G 0 v + ɛ v + ɛ u 0 > v T G 0 v + ɛ v. Hence, the u which attains the supremum must be in Null G. Now note that, if Null G Null G 0, R s = Null G 0 Null G 0, and Null G 0 = (Null G 0 Null G) V, with V the orthogonal complement of Null G 0 Null G in Null G 0 and the supremum of g) = is attained on U = V Null G 0 (as adding any component along the orthogonal complement of this space only adds a positive value to the denominator, increasing g)). Any u U can be written as u = αu 0 βv 0 with u 0 Null G 0 and v 0 V unit vectors. By upper bounding every term in

2 the numerator and lower bounding u 0G 0 u 0 we obtain the result. Note that for ɛ small enough, the expression in 4 is close to ɛ λ (G). For (), let v V and compute g(v) as above, with α = 0. It follows that g(v) = v Gv ɛ v and by taking the supremum over v V we obtain that sup V g(v) = ɛ λ (G) < r, from which the result follows. For (3), it is obvious that when ɛ 0, g(v) on V, but remains finite for u V. More precisely, G G0 = iff Null G 0 G. To verify that G0 is a norm, we must verify the triangle inequality, since the other two properties obviously hold. If A G0 = or B G0 =, triangle inequality holds trivially. Assume then that A G0, B G0 <. Since A G0+εI s + B G0+εI s A + B G0+εI s for every ɛ > 0, then in the limit we will have that A G0 + B G0 A + B G0. The norm for comparing Riemannian metric The norm of a bilinear functional f : R R R is defined as sup u = v = f, v), or since for a fixed orthonormal base of R s f, v) = u Av, f = sup u = v = u Av. If A is hermitian, then f = max λ(a) λ i where λ(a) denotes the spectrum of A. One can define the norm with respect to any metric G 0 on R s where G 0 is a symmetric, positive definite matrix by f G0 = sup u G0 = v G0 = u Av = sup ũ = ṽ = ũ G / 0 AG / 0 ṽ = max / λ(g λ 0 AG / i In other words, the appropriate operator norm we seek can be expressed as a (generalized) matrix spectral norm. In our cases G 0 = I d 0 and A = H k I d 3 Proof of Propositions 3 Note that we can write the loss as: n Π ky L k YΠ k Π k U k U k Π k k= Where Π k = (U k U k + (ε orth ) k I s ) /. We take the Π k matrices to be fixed and don t depend on the data points Y (in practice they do, however, after taking a gradient step we update the Π k in an E-M style algorithm). Since U k U k and Π k are the identity matrix (the latter multiplied by /( + ε orth )) when s = d we can compute the derivative when s > d without loss of generality. 3. Proof of Derivative Since the derivative is a linear operator it s sufficient to show that the derivative of a single loss function is of the form: l k Y = ( λ k )sgn(λ k)l k YΠ k u k u kπ k To compute the derivative we will make use of the chain rule. First define the function l k as a composition of functions: l k (Y) ρ(p k (H k (Y)) C k ) With C k = Π k U k U k Π k and ρ(u) = (max λ k (U) ) k P k (H) = Π khπ k H k (Y) = Y L k Y Where U, H are both symmetric. Here we note that the matrix spectral norm reduces to the spectral radius if U is symmetric. Since H k (Y) is defined to be symmetric and C k is symmetric this is the case. By the chain rule: Taking these from left to right: Dl k (Y) = Dρ(P k (H k (Y)) C k )DP k (H k (Y))DH k (Y)

3 Dρ Since ρ is defined to be the largest (in absolute value) eigenvalue of U (squared) the derivative is the kronecker product between the corresponding eigenvector and itself multiplied by the sign of the eigenvalue: D ρ(u) = sgn(λ k) k u k) Where λ k = ρ(u) and Uu k = λ k u k Then since we square the spectral radius we add the factor of ( λ k ) so that: D(ρ(U) = ( λ k )sgn(λ k) k u k) 3.. DP k DPk(H) = (Π k Π k) P k (H) = Π khπ k dp k (H) = Π kdhπ k vec(dp k (H)) = vec(π kdhπ k ) = (Π k Π k)dvec(h) 3..3 DH k DHk(Y) = Ns(Is Y Lk) Where N s = I s + K ss for K ss the commutation matrix defined in Magnus & Neudecker ch H k (Y) = Y L k Y dh k (Y) = [(dy) L k Y + Y L k dy] vec(dh k (Y)) = [(Y L k I s )dvec(y ) + (I s Y L k )dvec(y)] 3..4 Dc k Putting it all together = [(Y L k I s )K ns dvec(y) + (I s Y L k )dvec(y)] = [K ss(i s Y L k)dvec(y) + (I s Y L k )dvec(y)] = [(K ss + I s )(I s Y L k )dvec(y)] L k is symmetric = [N s(i s Y L k )dvec(y)] = N s (I s Y L k )dvec(y) Dc k (Y) = ( λ k )sgn(λ k) k u k)(π k Π k)n s (I s Y L k ) = vec ( ) ck Y see Matrix Differential Calculus With Applications in Statistics And Economics by Magnus & Neudecker ch. 9 for proof 3

4 We can simplify this to get the claim: c k Y = ( λ k )sgn(λ k)l k YΠ k u k u kπ k Dc k (Y) = ( λ k )sgn(λ k) k u k)(π k Π k)n s (I s Y L k ) Then note that: = ( λ k )sgn(λ k) k u k)(π k Π k) (K ss + I s )(I s Y L k ) = ( λ k )sgn(λ k) kπ k u kπ k)(k ss + I s )(I s Y L k ) = ( λ k )sgn(λ k) [ k Π k u kπ k)k ss (I s Y L k ) + kπ k u kπ k)(i s Y L k ) ] = ( λ k )sgn(λ k) [ k Π k u kπ k)(y L k I s )K ns + kπ k u kπ k)(i s Y L k ) ] = ( λ k )sgn(λ k) [ k Π ky L k u kπ k)k ns + kπ k u kπ ky L k ) ] = ( λ k )sgn(λ k) [ K kπ k u kπ ky L k ) + kπ k u kπ ky L k ) ] = ( λ k )sgn(λ k) kπ k u kπ ky L k ) K = = ( λ k )sgn(λ k)(π k u k L k YΠ k u k ) vec(( λ k )sgn(λ k)l k YΠ k u k u kπ k) = ( λ k )sgn(λ k)vec([l k YΠ k u k ][][u kπ k]) = ( λ k )sgn(λ k)(π k u k L k YΠ k u k )vec() = ( λ k )sgn(λ k)(π k u k L k YΠ k u k ) = (Dc k (Y) So that c k Y = ( λ k )sgn(λ k)l k YΠ k u k u kπ k The proposition then follows by removing the absolute value and multiplication by the sign. 4

5 Figure : The average number of neighbors m(r) vs the neighborhood radius r, on a log-log scale, for the SDSS spectra data, computed on the whole sample of 675,000 galaxies. The blue regression line, is fitted to the graph points in the shown r range, and has slope.87. The absence of a linear region on this graph suggests that the data dimension varies with the scale. The analysis and visualization in this paper corresponds to the largest meaningful scale. 5