Lower bounds on Locality Sensitive Hashing

Transcription

1 Lower bouns on Locality Sensitive Hashing Rajeev Motwani Assaf Naor Rina Panigrahy Abstract Given a metric space (X, X ), c 1, r > 0, an p, q [0, 1], a istribution over mappings H : X N is calle a (r, cr, p, q)-sensitive hash family if any two points in X at istance at most r are mappe by H to the same value with probability at least p, an any two points at istance greater than cr are mappe by H to the same value with probability at most q. This notion was introuce by Iny an Motwani in 1998 as the basis for an efficient approximate nearest neighbor search algorithm, an has since been use extensively for this purpose. The performance of these algorithms is governe by the parameter ρ = log(1/p) log(1/q), an constructing hash families with small ρ automatically yiels improve nearest neighbor algorithms. Here we show that for X = l 1 it is impossible to achieve ρ 1 2c. This almost matches the construction of Iny an Motwani which achieves ρ 1 c. 1 Introuction In this note we stuy the complexity of fining the nearest neighbor of a query point in certain high imensional spaces using Locality Sensitive Hashing (LSH). The nearest neighbor problem is formulate as follows: Given a atabase of n points in a metric space, preprocess it so that given a new query point it is possible to quicly fin the point closest to it in the ata set. This funamental problem arises in numerous applications, incluing ata mining, information retrieval, an image search, where istinctive features of the objects are represente as points in R. There is a vast amount of literature on this topic, an we shall not attempt to iscuss it here. We refer the intereste reaer to the papers [6, 5, 4, 7], an especially to the references therein, for bacgroun on the nearest neighbor problem. While the exact nearest neighbor problem seems to suffer from the curse of imensionality, many efficient techniques have been evise for fining an approximate solution whose istance from the query point is at most c times its istance from the nearest neighbor. One of the most versatile an efficient methos for approximate nearest neighbor search is base on Locality Sensitive Hashing, as introuce by Iny an Motwani in 1998 [6]. This metho has been refine an improve in several papers- the most recent algorithm can be foun in [4]. We also refer the reaer to the LSH website, where more information on this algorithm can be foun, incluing its implementation an coe- all this can be foun at The LSH approach to the approximate nearest neighbor problem is base on the following concept. Definition 1.1. Let (X, X ) be a metric space, r, R > 0 an p, q [0, 1]. A istribution over mappings H : X N is calle a (r, R, p, q)-sensitive hash family if for any x, y X, X (x, y) r = Pr H [H (x) = H (y)] p. X (x, y) > R = Pr H [H (x) = H (y)] q. 1

2 Given c 1 an q (0, 1) we efine ρ X (c, q) to be the smallest constant ρ > 0 such that for every r > 0 there exists p (0, 1) an a (r, cr, p, q)- sensitive hash family H : X N with log(1/p) log(1/q) ρ. In other wors { } log(1/p) ρ X (c, q) = sup inf : (r, cr, p, q) sensitive hash family H : X N. (1) r>0 log(1/q) Of particular interest is the case X = l s, for some s > 0 an N. Here, an in what follows, l s enotes the space R equippe with the l s norm (x 1,..., x ) s = ( x 1 s + + x s ) 1/s (this is only a quasi-norm when 0 < s < 1). In this case we efine ρ s (c) = sup lim sup ρ l s (c, q). 0<q<1 The importance of these parameters stems from the following application to approximate nearest neighbor search. It will be convenient to iscuss it in the framewor of the following ecision version of the c-approximate nearest neighbor problem: Given a query point, fin any element of the ata set which is at istance at most cr from it, provie that there is a ata point at istance at most r from the query point. This ecision version is nown as the (r, cr)- near neighbor problem. It is well nown that the reuction to the ecision version as only a logarithmic factor in the time an space complexity [6, 5]. The following theorem was prove in [6]; the exact formulation presente here is taen from [4]. Theorem 1.2. Let (X, X ) be a metric on a subset of R. Suppose that (X, X ) amits a (r, cr, p, q)-sensitive hash family H, an write ρ = log(1/p) log(1/q). Then for any n 1 q there exists a ranomize algorithm for (r, cr)- near neighbor on n-point subsets of X which uses O ( n + n 1+ρ) space, with query time ominate by O (n ρ ) istance computations an O ( n ρ log 1/q n ) evaluations of hash functions from H. Thus, obtaining bouns on ρ X (c) is of great algorithmic interest. It is prove in [6] that ρ 1 (c) 1/c, an for small values of c, namely c [1, 10], is was shown in [4] that this inequality is strict. We refer to [4] for numerical ata on the best now estimates for ρ 1 (c) for small c. For s = 2 a recent result of Anoni an Iny [1] shows that ρ 2 (c) 1/c 2, an for general s (0, 2] the best nown bouns [4] are ρ s (c) max{1/c, 1/c s }. The main purpose of this note is to obtain lower bouns on ρ 1 (c) an ρ 2 (c) which nearly match the bouns obtaine from the constructions in [6, 4, 1]. Our main result is: Theorem 1.3. For every c, s 1, ρ s (c) e 1 c s 1 e 1 c s + 1 e 1 e c s c s. (2) The secon to last inequality in (2) follows from concavity of the function t et 1 e t +1 on [0, ). Observe also that as c, e1/c 1 e 1/c c. It woul be very interesting to etermine lim sup c c ρ 1(c) exactly- ue to Theorem 1.3 an the results of [6] we currently now that this number is in the interval [1/2, 1]. 2 Proof of Theorem 1.3 The basic iea in the proof of Theorem 1.3 is simple. Choose a ranom point x {0, 1} an consier the ranom subset A of the cube {0, 1} consisting of points u for which H (u) = H (x). The secon conition 2

3 in Definition 1.1 forces A to be small in expectation. But, when A is small we can boun from above the probability that after r steps, the ranom wal starting at a ranom point in A will en up in A. We obtain this upper boun using a Fourier analytic argument, an in combination with the first conition in Definition 1.1 we euce the esire boun on ρ 1 (c). Theorem 1.3 follows from the following result: Proposition 2.1. Let H be a (r, R, p, q)- sensitive hash family on the Hamming cube ({0, 1}, 1 ). Assume that r is an o integer an that R < 2. Then p (q + e 1 ( 2 R)2) e2r/ 1 e 2r/ +1. Choosing R 2 log an r R/c in Proposition 2.1, an letting, yiels Theorem 1.3 in the case s = 1. The case of general s 1 follows from the fact that for x, y {0, 1}, x y s = x y 1/s 1. Remar 2.1. Proposition 2.1 implies non-trivial lower bouns on log(1/p) log(1/q) for any (r, cr, p, q)- sensitive hash family on ({0, 1}, 1 ) even if q is allowe to epen on. Observe that with the efinition given in (1), Theorem 1.3 implies such a lower boun only for constant q. But, Proposition 2.1 is much stronger, an implies a boun which asymptotically coincies with the lower boun in 1.3 for every q 2 o(). The proof of Proposition 2.1 will be broen into a few lemmas. Lemma 2.2. Let H be a (r, R, p, q)-sensitive hash family on the Hamming cube ({0, 1}, 1 ), an fix x {0, 1}. Then E H 1 (H (x)) R ( ) ( ) + q. Proof. We simply write E H 1 (H (x)) = = =0 = R +1 Pr[H (u) = H (x)] u {0,1} {u {0, 1} : u x 1 R} + q {u {0, 1} : u x 1 > R} R ( ) + q =0 = R +1 Corollary 2.3. Assume that R < 2. Then, using the notation of Lemma 2.2, we have that E H 1 (H (x)) ( 2 q + e ( 1 R)2) 2. ( ) Proof. This follows from Lemma 2.2 an the stanar estimate 2 a ( ) 2 e a2. Lemma 2.4 (Ranom wal lemma). Let r be an o integer. Given B {0, 1}, consier the ranom variable Q B {0, 1} efine as follows: Choose a point z B uniformly at ranom, an perform r-steps of the stanar ranom wal on the Hamming cube starting from z. The point thus obtaine will be enote Q B. Then ( ) e 2r/ 1 e Pr[Q B B] 2r/

4 Proof. We begin by recalling some bacgroun an notation on Fourier analysis on the Hamming cube. Given S {1,... }, the Walsh function W S : {0, 1} { 1, 1} is efine by For f : {0, 1} R we set so that f can be ecompose as follows: W S (u) = ( 1) j S u j. f (S ) = 1 2 f (u)w S (u), u {0,1} f = f (S )W S. For every f, g : {0, 1} R we write f, g = 1 2 u {0,1} f (u)g(u). By Parseval s ientity, f, g = f (S )ĝ(s ). For ε [0, 1] the Bonami-Becner operator T ε is efine as T ε f = ε S f (S )W S. The Bonami-Becner inequality [3, 2] states that for every f : {0, 1} R, ε 2 S f (S ) 2 = T ε f 2 2 = 1 2 (T ε f (u)) 2 f 2 1 = 1+ε 2 2 f (u) 1+ε2 u {0,1} u {0,1} Specializing to the inicator of B {0, 1} we get that ε 2 S 1 B (S ) ε 2. ( ) 2 1+ε 2 2. (3) Now, let P be the transition matrix of the stanar ranom wal on {0, 1}, i.e. P uv = 1/ if u an v iffer in exactly one coorinate, P uv = 0 otherwise. By a irect computation we have that for every S {1,..., }, ( PW S = 1 2 S ) W S, 4

5 i.e. W S is an eigenvector of P with eigenvalue 1 2 S. The probability that the ranom wal starting form a ranom point in B ens up in B after r steps equals 1 ( Pr[Q B B] = P r ) ab = 2 = 2 a,b B Pr 1 B, 1 B 2 S /2 ( 1 B (S ) S 1 B (S ) 2 ( 1 2 S where we use the fact that r is o (i.e. we roppe negative terms). Thus, using (3) we see that ) r ) r, Pr[Q B B] 2 1 B (S ) 2 e 2r S / 2 ( ) 2 1+e 2r/ 2 = ( ) 1 e 2r/ 1+e 2r/ 2. Proof of Proposition 2.1. Assume that r is an o integer an R < 2. For x {0, 1} let W r (x) {0, 1} be the ranom point obtaine by preforming a ranom wal for r steps starting at x. Since x W r (x) 1 r we now that Pr [H (W r (x)) = H (x)] p. Taing expectation with respect to the uniform probability measure on {0, 1} we euce that p E x {0,1} n Pr [H (W r (x)) = H (x)] = E H Pr [ x {0, 1} n : W r (x) H 1 (H (x)) ] = E H Pr [ x {0, 1} n : W r (x) H 1 (H (x)) H (x) = ] N = E H N E H N H 1 () 2 Pr [ Q H 1 () H 1 () ] H 1 () 2 H 1 () 2 H 1 (H (x)) = E H E x {0,1} E x {0,1} 2 E H 1 H (H (x)) 2 e 2r/ 1 e 2r/ +1 e 2r/ 1 e 2r/ +1 e 2r/ 1 e 2r/ +1 (4) (5) (q + e 1 ( 2 R)2) e2r/ 1 e 2r/ +1, (6) where in (4) we use Lemma 2.4, in (5) we use Jensen s inequality, an in (6) we use Corollary

6 Acnowlegements. We are grateful to Piotr Iny an Jira Matouše for helpful suggestions. References [1] A. Anoni an P. Iny. Faster algorithms for high imensional nearest neighbor problems. Manuscript, [2] W. Becner. Inequalities in Fourier analysis. Ann. of Math. (2), 102(1): , [3] A. Bonami. Étue es coefficients e Fourier es fonctions e L p (G). Ann. Inst. Fourier (Grenoble), 20(fasc. 2): (1971), [4] M. Datar, N. Immorlica, P. Iny, an V. S. Mirroni. Locality-sensitive hashing scheme base on p- stable istributions. In SCG 04: Proceeings of the Twentieth Annual Symposium on Computational Geometry (Broolyn, NY, 2004), pages ACM Press, New Yor, NY, [5] S. Har-Pele. A replacement for Voronoi iagrams of near linear size. In 42n IEEE Symposium on Founations of Computer Science (Las Vegas, NV, 2001), pages IEEE Computer Soc., Los Alamitos, CA, [6] P. Iny an R. Motwani. Approximate nearest neighbors: towars removing the curse of imensionality. In STOC 98: Proceeings of the Thirtieth Annual ACM Symposium on Theory of Computing (Dallas, TX, 1998), pages ACM Press, New Yor, NY, [7] R. Panigrahy. Entropy base nearest neighbor search in high imensions. In SODA 06: Proceeings of the seventeenth annual ACM-SIAM symposium on Discrete algorithm (Miami, FL, 2006), pages ACM Press, New Yor, NY,