Appendix to: On the Relation Between Low Density Separation, Spectral Clustering and Graph Cuts

Transcription

1 Appendix to: On the Relation Between Low Density Separation, Spectral Clustering and Graph Cuts Hariharan Narayanan Department of Computer Science University of Chicago Chicago IL 6637 ikhail Belkin Department of Computer Science and Engineering The Ohio State University Columbus, OH 43 Partha Niyogi Department of Computer Science University of Chicago Chicago IL 6637 A Regularity conditions on p and S We make the following assumptions about p:. p can be extended to a function p that is L Lipshitz and which is bounded above by p max.. For < t < t, min(p(x), K t (x, y)p(y)dy) p min. Note that this is a property of both of the boundary and p. We note that since p is L Lipshitz over R d, so is K t(x, z)p (z)dz. We assume that S has condition number /τ. We also make the following assumption about S:- The volume of the set of points whose distance to both S and is R, is O(R ) as R. This is reasonable, and is true if S is a manifold of codimension. B Proof of Theorem This follows from Theorem 4 (which is proved in a later section), by setting µ to be equal to ɛ d+. C Proof of Theorem In the proof we will use a generalization of cdiarmid s inequality from [7, 8]. We start with the with the following Definition Let Ω,..., Ω m be probability spaces. Let Ω = m Ω k and let Y be a random variable on Ω. We say that Y is strongly difference-bounded by (b, c, δ) if the following holds: there is a bad subset B Ω, where δ = Pr(ω B). If ω, ω Ω differ only in the kth coordinate, and ω B, then Y (ω) Y (ω ) c.

2 Furthermore, for any ω and ω differing only in the kth coordinate, Y (ω) Y (ω ) b. Theorem ([7, 8]) Let Ω,..., Ω m be probability spaces. Let Ω = m Ω k and let Y be a random variable on Ω which is strongly difference-bounded by (b, c, δ). Assume b c >. Let µ = E(Y ). Then for any r >, By Hoeffding s inequality P [ z x K t(x, z) N Pr( Y µ r) ( exp ( r 8mc ) + mbδ c ). E(K t (x, z)) > ɛ E(K t (x, z))] < e (N )E(Kt(x,z)) ɛ t e (N )p min ɛ t. We set ɛ to be t /N µ. Let e (N )p min ɛ t be δ/n. By the union bound, the probability that the above event happens for some x X is δ. The set of all ω Ω for which this occurs shall be denoted by B. Also, for any X, the largest possible value that /N π/t K t (x, y) {( y X z x K t(x, z))( z y K t(y, z))} / could take is π/t(n ). Then, x X E[β] α < ( ɛ ) α + δ π/t(n ). () Let q = (p min / t ). β is strongly difference-bounded by (b, c, δ) where c = O((qN t) ), b = O(N/ t). We now apply the generalization of cdiarmid s inequality in Theorem. Using the notation of Theorem, ( ( ) r Pr[ β E[β] > r] exp 8mc + Nbδ ) ( exp ( O(Nr q t) ) + O ( N 3 q exp ( O(Nqɛ c ) ))). () Putting this together with the relation between E[β] and α in (), the theorem is proved. We note that in (), the rate of convergence of E[β] to α is controlled by ɛ, which is t /N µ, and in (), the rate of convergence of β to E[β] depends on r, which we set to be t / tn µ. We note that in (), the dependence on r of the probability is exponential. Since we have assumed that u = t / (tn µ ) = o(), t /N µ = O(t d+ u). Thus the result follows. D Proof of Theorem 3 We shall prove theorem 3 through a sequence of lemmas. Without a loss of generality we can assume that τ = by rescaling, if necessary. Let R = dt ln(/t) and ɛ = z >R K t(, z)dx. Using the inequality ( ) d/ td K t (, z)dx R e R 4t + d = (et ln(/t)) d/ z >R (3)

3 E_ B_ O_ A H_ F_ R R R^ D_ G_ C Figure : A sphere of radius outside that is tangent to S A H_ F_ R C (R^)/ D_ G_ O_ B_ E_ Figure : A sphere of radius inside that is tangent to S we know that ɛ (et ln(/t)) d/. For any positive real t, ln(/t) t e. Therefore the assumption that ( ) t τ, (d) e e implies that R dt /e <. 3

4 Let the point y (represented as A in figures D and D) be at a distance r < R from. Let us choose a coordinate system where y = (r,,..., ) and the point nearest to it on is the origin. There is a unique such point since r < R <. Let this point be C. Let D lie on the segment AC, at a distance R / from C. Let D lie on the extended segment AC, at a distance R / from C. Thus C is the midpoint of D D. Definition. Denote the ball of radius tangent to at C that is outside by B.. Denote the ball of radius tangent to at C which is inside by B. 3. Let H be the halfspace containing C bounded by the hyperplane perpendicular to AC and passing through D. 4. Let H be the halfspace not containing C bounded by the hyperplane perpendicular to AC and passing through D. 5. Let H 3 be the halfspace not containing A, bounded by the hyperplane tangent to at C. 6. Let B be the ball with center y = A, whose boundary contains the intersection of H and B. 7. Let B be the ball with center y = A, whose boundary contains the intersection of H and B. Definition 3. h(r) := H 3 K t (x, y)dx.. f(r) := H B K t (x, y)dx. 3. g(r) := H B K t (x, y)dx. It follows that and K t (x, y)dx = h(r R /) H K t (x, y)dx = h(r + R /). H Observation Although h(r) is defined by an d-dimensional integral, this can be simplified to e (r x ) /4t h(r) = dx, 4πt by integrating out the coordinates x,..., x d. Lemma If r > R, the radius of B is R. x < Proof: By the similarity of triangles CF D and CE F in figure D, it follows that CF CE = CD CF. CE = and CD = R /. Therefore CF = R. Since CD F is right angled at D, and CD = R /, this proves the claim. Lemma The radius of B is R. Proof: By the similarity of triangles CF E and CD F in figure D CF = R. However, the distance of point y := A from F is CF. Therefore, the radius of B is R. Definition 4 Let the set of points x such that B(x, R) be denoted by. Let be S and S be S. Let =, be and S be. We shall denote ( + L/p min )R) by l. 4

5 Consider a point x, Then, K t (x, y)p(y)dy y x <R K t (x, y)p(y)dy ( ɛ)(p(x) LR) ( ɛ)p(x)( LR/p min ) = p(x)( O(l)) On the other hand, K t (x, y)p(y)dy K t (x, y)p(y)dy + K t (x, y)p(y)dy y x R y x >R p(x)( + l) + K t (, R) = p(x)( + O(( + L/p min )R)) Therefore, ψ t (x) = p(x)( ± O(( + L p min )R)). d Lemma 3 B(x, 5R) implies that dx Kt (x, z)p(z)dy = O(L). Proof: Consider the function p, which is equal to p on, but which has a larger support and is L Lipshitz as a function on R d. K t (x, z)p (z)dy is L Lipshitz and on points x where B(x, 5R), the contribution of points z outside, is o(). Therefore d dx Kt (x, z)p(z)dy = O(L). This implies that on the set of points x such that B(x, 5R), ψ t (X) is O(L) Lipshitz. We now estimate K t (y, z)p(z)dz for y S. Definition 5 For a point y S, such that d(y, ) < R < τ = let π(y) be the nearest point to y in S. Note that by the assumption that the condition number of S is, since R is smaller than, there is a unique candidate for π(y). Let y be as in Definition 5. Lemma 4 h(r + R /) ɛ < f(r) K t (y, z)dz. Proof: K t (x, y)dx > H B K t (x, y)dx(since H B ) K t (x, y)dx H > h(r + R /) ɛ B c K t (x, y)dx The last inequality follows from lemma. Lemma 5 > p(π(y))( O(l))(h(r + R /) ɛ). 5

6 Q P Q P Figure 3: The correspondence between points on and [ ] r Proof: (since H B ) H B > p(π(y))( O(l))(h(r + R /) ɛ). Lemma 6 Let r > R. Then, < ( + O(l))(h(r R /)p(π(y)) + ɛp max ). Proof: < R d B H R d B H B + B c < h(r R /)p(π(y))( + O(l)) + ɛp max ( + O(l)) < ( + O(l))(h(r R /)p(π(y)) + ɛp max ) The last inequality follows from lemma. Definition 6 Let [ ] r denote the set of points at a distance of r to [ ]. Let π r be map from [ ] r to [ ] that takes a point P on [ ] r to the foot of the perpendicular from P to. (This map is well-defined since r < τ =.) Lemma 7 Let y [ ] r. Let the Jacobian of a map f be denoted by Df. ( r) d Dπ r (y) ( + r) d. 6

7 Proof: Let P Q be a geodesic arc of infinitesimal length ds on joining P and Q. Let πr (P ) = P and πr (Q) = Q (see Figure D.) The radius of curvature of P Q is. Therefore the distance between P and Q is in the interval [ds( r), ds( + r)]. This implies that the Jacobian of the map π r has a magnitude that is always in the interval [( + r) d, ( r) d ]. Lemma 8 Proof: R d [ ] R dy ɛvol p max ( + O(l)). dy = R d [] R K t (x, y)ψ t (x)ψ t (y)dydx R d [S ] R K t (, z)p max ( + O(l))dzdx z >R < vol p max ( + O(l)). in line holds because the distance between x and y in the double integral is always R. Lemma 9 ( e α /4t ) π/t α h(r)dr π/t. Proof: Using observation, Setting y x := r, this becomes α r h(r)dr = aking the substitution r α := z, we have α α α e (x y ) /4t 4πt dx dy. e r /4t e r /4t dy dr = (r α)dr. 4πt α 4πt e (z+α) /4t 4πt zdz = e α /4t e z /4t zdz 4πt t π e α /4t Equality holds in the above calculation if and only if α =. Hence the proof is complete. Definition 7 Let [S ] [ ] r be r. Let [S ] [ ] r be r and [S ] [ ] r be r. We assume that vol( R ) < C R for some absolute constant C. Since the thickness of ( R ) is O(R) in two dimensions, this is a reasonable assumption to make. The assumption that has a d dimensional volume implies that vol = O(R). Putting these together to prove Theorem 3: 7

8 dy = dydr r O(p max/p min ) K t (x, y)dxdydr r ( R ) O(p max/p min ) K t (x, y)dxdydr + volsɛ r O(p max/p min ) ( vol ( R ) + ɛvol ). O(p max/p min ) ( C t µ + O(t µ vol ) ) S dy = dydr r R = ( dydr) r S + ( dydr) R r R ( dydr) r + ɛvol p max ( + O(l)). }{{} E (from lemma 8) R p max ( + O(l))dydr r R + ( + O(l))(h(r R /)p(π(y) + ɛp max )) + E R r The last line follows from Lemma 6. E + R ( + R ) d p max ( + O(l))vol ( )dr(from lemma 7) R + ( + R) d ( + O(l))( h(r)dr p(y)dy + ɛp max R). ( + O(l))( t/π p(y)dy + p max ((R + ɛr)vol ( ) + ɛvol )). ( + O(l))( p(y)dy( t/π + p max o(t µ )) + p max ɛvol ) p min Similarly, we see that S dy 8

9 = dydr r R > ( dydr) r R > p(π(y))( + O(l))f(r)dxdr r R > ( R) d ( O(l))( p(y)dy)(h(r + R /) ɛ)dr R > ( R) d ( O(( L/p min )R))( p(y)dy)(( (h(r)dr) ɛr R /) ( O(l))( p(y)dy)(( e R /4t ) t/π ɛr R /) ( O(l))( p(y)dy)( t/π o(t µ )). Noting only the dependence of the rate on t, and introducing the condition number τ, π ) dy = ( + o((t/τ )) µ t S S p(s)ds. Proof of Theorem 4: This follows directly from Theorem and Theorem 3. The only change made was that the t d+ term was eliminated since it is dominated by t ɛ when t is small. References []. Belkin and P. Niyogi (4). Semi-supervised Learning on Riemannian anifolds. In achine Learning 56, Special Issue on Clustering, []. Belkin and P. Niyogi. Toward a theoretical foundation for Laplacian-based manifold methods. COLT 5. [3] A. Blum and S. Chawla, Learning from labeled and unlabeled data using graph mincuts, ICL. [4] O.Chapelle, J. Weston, B. Scholkopf, Cluster kernels for semi-supervised learning, NIPS. [5] O. Chapelle and A. Zien, Semi-supervised Classification by Low Density Separation, AISTATS 5. [6] Chris Ding, Spectral Clustering, ICL 4 Tutorial. [7] Samuel Kutin, Partha Niyogi, Almost-everywhere Algorithmic Stability and Generalization Error., UAI, 75-8 [8] S. Kutin, TR--4, Extensions to cdiarmid s inequality when differences are bounded with high probability. Technical report TR--4 at the Department of Computer Science, University of Chicago. [9] S. Lafon, Diffusion aps and Geodesic Harmonics, Ph. D. Thesis, Yale University, 4. []. Hein, J.-Y. Audibert, U. von Luxburg, From Graphs to anifolds Weak and Strong Pointwise Consistency of Graph Laplacians, COLT 5. [] J. Shi and J. alik. Normalized cuts and image segmentation. [] U. von Luxburg,. Belkin, O. Bousquet, Consistency of Spectral Clustering, ax Planck Institute for Biological Cybernetics Technical Report TR 34, 4. [3] X. Zhu, J. Lafferty and Z. Ghahramani, Semi-supervised learning using Gaussian fields and harmonic functions, ICL 3. 9