18.409: Topics in TCS: Embeddings of Finite Metric Spaces. Lecture 6

Transcription

1 Massachusetts Institute of Technology Lecturer: Michel X. Goemans 8.409: Topics in TCS: Embeddings of Finite Metric Spaces September 7, 006 Scribe: Benjamin Rossman Lecture 6 Today we loo at dimension reduction in l. Suppose X is a metric space in l of size n. From previous lectures, we now that X embeds isometrically into l n. We as the question: for < n, what is the minimal distortion D needed to embed X in l d? We will see that there is a tradeoff between distortion and dimension. To achieve distortion close to, we need only logarithmic many dimensions. Theorem (Johnson-Lindenstrauss, 984). For all ε > 0, X embeds into l O( ε log n) with distortion + ε. We also prove a theorem of Alon which shows that the Johnson-Lindenstrauss Lemma (as Theorem is nown) is tight. Theorem (Alon []). If v,..., v n+ R d are such that v i v j + ε for all i j, then d = Ω( log n ). ε log ε We give two proofs of the Johnson-Lindenstrauss Lemma. The idea in both proofs is to project X onto a random -dimensional subspace of R n where = O( ε log n). The proofs differ in the way the projection is randomly chosen. Measure Concentration and Levy s Lemma Let S n = {x R n : x = } and let µ be the unique rotation-invariant (Haar) measure on S n such that µ(s n ) =. For points x, y S n, d(x, y) denotes the geodesic distance between x and y defined by d(x, y) = arccos( x, y ). For a point a S n and r 0, B a (r) denotes the cap of radius r around a defined by B a (r) = {x S n : d(a, x) r}. We will need the following lemma: Lemma 3 (Levy s Lemma). Let A S n be a closed set and let B S n be a cap such that µ(a) = µ(b). Then, for all t 0, µ({x : d(a, x) t}) µ({x : d(b, x) t}). In particular, if B = B a (r) then µ({x : d(a, x) t}) µ(b a (r + t)). We remar that Levy s Lemma also holds when d(, ) denotes Euclidean instead of geodesic distance. Lemma 4. Consider a function f : S n R which is -Lipschitz, meaning that f(x) f(y) d(x, y) for all x, y S n. We define m(f) R, called the median of f, such that µ(a + ) and µ(a ) where A+ = {x : f(x) m(f)} and A = {x : f(x) m(f)}. Then µ({x : f(x) m(f) > ε}) ( + o())e ε n.

2 This lemma says that -Lipschitz functions are highly concentrated around the mean. Before we prove the lemma, we need a bound on µ(b a ( π s)). One can show (for the derivation see, for example, Barvino [, p. 58]) that, for any 0 s π/, µ(b a ( π s)) π s (n ) 8 e, or since we are interested in large values of n that ( π )) ( ) µ (B a s + o() e s n. Proof. By Levy s Lemma and the inequality above, we have This implies µ({x : d(a ±, x) ε}) µ(b a ( π + ε)) ( + o())e ε n. µ({x : d(a +, x) ε} {x : d(a, x) ε}) ( + o())e ε n. Using the fact that f is -Lipschitz, it is easy to see that {x : f(x) m(x) > ε} lies inside the complement of {x : d(a +, x) ε} {x : d(a, x) ε}. Therefore, µ({x : f(x) m(f) > ε}) ( + o())e ε n. First Proof of Johnson-Lindenstrauss Lemma Rather than project onto a random -dimensional subspace of R n, we apply a random rotation of R n and then project onto the first coordinates. Choose v R n at random where the direction v v S n is distributed with respect to µ, and let f(v) = i= v i. We argue that the value f(v) is close to v with high probability when = Θ( log n). Specifically, we show there exists ε a constant c > 0 such that Pr[c v f(v) c( + ε) v ] n. ( ) Once we prove ( ), the Johnson-Lindenstrauss Lemma follows easily. For points x,..., x n R n, we let v ij = x i x j for all i j. Then f(v ij ) equals the distance between x i and x j after projecting onto a random -dimensional subspace. Applying a union bound to inequality ( ), we get ( n ) Pr[ i j, c v ij f(v ij ) c( + ε) v ij ] Since (n ) > 0, there exists a projection R n R for which the l n -metric space on points x,..., x n has distortion + ε. To prove the inequality ( ), we invoe Lemma 4. We first note that f is -Lipschitz. We then note that m(f) is close to n since E[f(v) ] = n ; one can argue for example that m(f) = n + O(/ n) for all. Lemma 4 now gives us Pr[ f(v) m(f) > εm(f)] = µ({x S n : f(x) m(f) > εm(f)}) = ( + o())e (εm(f)) n = c 0 ( + o())e ε n.

3 for some constant c 0. Since = Θ( ε log n), we have ε = Θ(log n). Therefore, c 0 e ε n for suitably chosen constant in the expression for. This proves the inequality ( ) where c = m(f) n. Second Proof of Johnson-Lindenstrauss Lemma We now give a different proof of the Johnson-Lindenstrauss Lemma due to Indy and Motwani (998). The elementary presentation we follow is due to Dasgupta and Gupta (003). Let x,..., x n R n. The idea is to project X = {x,..., x n } onto independently generated directions. We define random vectors r,..., r R n where r ij N(0, ) are independent { Gaussian 0 if j random variables for all i and j n. Thus, E[r ij ] = 0 and E[r ij r i ] = if j =. We define a projection R n R by f : x ( x, r i ) i=,...,. Our goal is to show that the random embedding f of X into l probability. has distortion + ε with positive Theorem 5. For = O( log n ) and v R n, ε [ Pr ε f(v) ] + ε v n. Once we prove Theorem 5, the J-L Lemma follows by the same argument as in the first proof. Proof. We shall assume that v =, since the fraction f(v) v is invariant under scaling of v. For random variables X i = v, r i = n j= v jr ij, we have E[X i ] = n v j E[r ij ] = 0, j= E[X i ] = ( n j= v j E[r ij ] ) + ( j, {,...,n} j v j v E[r ij r i ] Therefore, E[ f(v) ] = n i= E[X i ] =. We now use Chernoff bounds to prove the inequalities Pr [ ] f(v) + ε v n and Pr ) = n vj = v =. j= [ ] f(v) ε v n, which together imply the theorem. We give the argument for the lefthand inequality only (the argument for the righthand inequality is similar). Since v =, this means we must show Pr[ f(v) ( + ε) ] n. 3

4 Let Y be the random variable f(v) and let α = ( + ε). For every s > 0, we have Pr[Y > α] = Pr[e sy > e sα ]. Recall Marov s inequality: E[X β] E[x] β where X is a nonnegative random variable and β > 0. We apply Marov s inequality to get Pr[Y > α] = Pr[e sy > e sα ] E[esY ] e sα = e sα E[e s P i= X i ] = e sα E[e sx i ] ( ) i= where the last equality follows from independence of the random variables X,..., X. Each X i is Guassian with mean 0 and variance, so by elementary calculus + E[e sx i ] = e st e t / dt = + e (s )t dt. π π We now apply a change of variables, letting u = ( s)t so that dt = u t Thus, + E[e sx i ] = e (s + )t dt = π s π Plugging this into ( ), we get Pr[Y > α] = e sα ( s). We now choose s = α, so that s = α. This gives e u du = s. s du = s du. e sα ( s) = e α ( α ) ( ) ( α = e α α). Recall that α = ( + ε). Thus, we have: e α ( ) α = e ε ε e ln( α ) = e ε ε e ln( (+ε) ) = e ε ε e ln(+ε) = e ( ε ε +ε ε +O(ε 3 )) = e ε +O(ε 3), using the taylor s expansion ln( + x) = x x + O(x3 ). Taing = Θ( log n ), we have ε Pr[f(v) ( + ε) ] = e ε +O(ε 3) = O( n ). Alon s Theorem (to be continued) In the next lecture, we will give a proof of Alon s theorem. For now, we give a setch of the proof. Let v,..., v n+ R d be such that v i v j + ε for all i j. The theorem states that d = Ω( log n ). ε log ε Clearly, we can assume that v n+ = (0,..., 0) by translating al vectors v i. We then scale vectors v i to obtain new vectors v i such that v i =. After scaling, we have v i v j = O(ε). We 4

5 now loo at the symmetric matrix B = ( v i, v j ), which has the form i,j n [ ε, +ε] [ ε, +ε]... i.e., ones along the diagonal and all other entries between ε and + ε. This matrix has ran d. Alon s theorem is proved by establishing a lower bound on d in terms of n and ε. References [] N. Alon, Problems and results in extremal combinatorics, I, Discrete Math. 73 (003), [] A. Barvino, Lecture Notes on Measure Concentration, available from barvino/total70.pdf. 5