Min-Wise independent linear permutations

Transcription

1 Min-Wise independent linear permutations Tom Bohman Department of Mathematical Sciences, Carnegie Mellon University, Pittsburgh PA523, USA Colin Cooper School of Mathematical Sciences, University of North London, London N7 8DB, UK Alan Frieze Department of Mathematical Sciences, Carnegie Mellon University, Pittsburgh PA523, USA, Submitted: January 2, 2000; Accepted: April 23, 2000 Abstract A set of permutations F S n is min-wise independent if for any set X [n] and any x X, whenπ is chosen at random in F we have P (min{π(x)} = π(x)) = X This notion was introduced by Broder, Chariar, Frieze and Mitzenmacher and is motivated by an algorithm for filtering near-duplicate web documents Linear permutations are an important class of permutations Let p be a (large) prime and let F p = {π a,b : a p, 0 b p } where for x [p] ={0,,,p }, π a,b (x) =ax + b mod p For X [p] weletf (X) = Supported in part by NSF Grant DMS Research supported by the STORM Research Group Supported in part by NSF grant CCR-9884

2 the electronic journal of combinatorics 7 (2000), #R26 2 max x X {P a,b (min{π(x)} = π(x))} where P a,b is over π chosen ( uniformly ) at random from F p We show that as, p, E X [F (X)] = +O (log ) 3 confirming 3/2 that a simply chosen random linear permutation will suffice for an average set from the point of view of approximate min-wise independence Introduction Broder, Chariar, Frieze and Mitzenmacher [3] introduced the notion of a set of min-wise independent permutations We say that F S n is min-wise independent if for any set X [n] and any x X, whenπ is chosen at random in F we have P (min{π(x)} = π(x)) = X () The research was motivated by the fact that such a family (under some relaxations) is essential to the algorithm used in practice by the AltaVista web index software to detect and filter near-duplicate documents A set of permutations satisfying () needs to be exponentially large [3] In practice we can allow certain relaxations First, we can accept small relative errors We say that F S n is approximately min-wise independent with relative error ɛ (or just approximately min-wise independent, where the meaning is clear) if for any set X [n] and any x X, whenπ is chosen at random in F we have P( min{π(x)} = π(x) ) X ɛ X (2) In other words we require that all the elements of any fixed set X have only an almost equal chance to become the minimum element of the image of X under π Linear permutations are an important class of permutations Let p be a (large) prime and let F p = {π a,b : a p, 0 b p } where for x [p] ={0,,,p }, π a,b (x) =ax + b mod p, where for integer n we define n mod p to be the non-negative remainder on division of n by p For X [p] welet F (X) =max a,b(min{π(x)} = π(x))} x X where P a,b is over π chosen uniformly at random from F p The natural questions to discuss are what are the extremal and average values of F (X) asx ranges over A = {X [p] : X = } The following results were some of those obtained in [3]:

3 the electronic journal of combinatorics 7 (2000), #R26 3 Theorem (a) Consider the set X = {0,, 2 }, as a subset of [p] As, p, with 2 = o(p), P a,b (min{π(x )} = π(0)) = 3 ( ln 2 π 2 + O p + ) (b) As, p, with 4 = o(p), ( ) 2+ 2( ) E X[F (X)] + O, 2 2 where E X denotes expectations over X chosen uniformly at random from A In this paper we improve the second result and prove Theorem 2 As, p, E X [F (X)] = + O ( (log ) 3 3/2 ) Thus a simply chosen random linear permutation will suffice for an average set from the point of view of min-wise independence Other results on min-wise independence have been obtained by Indy [6], Broder, Chariar and Mitzenmacher [4] and Broder and Feige [5] 2 Proof of Theorem 2 Let X = {x 0,x,,x } [p] Let β i = ax i mod p for i =0,,, Let i = i(x, a) =min{β 0 β j mod p : j =, 2,, } (3) Let and note that Then A i = A i (X) ={a [p] : i(x, a) =i} A i, i =, 2,,p min{π(x)} = π(x 0 )iff0 {β 0 + b, β 0 + b,,β 0 + b i +} mod p Thus if p Z = Z(X) = i A i, i=

4 the electronic journal of combinatorics 7 (2000), #R26 4 P a,b (min{π(x)} = π(x 0 )) = Z p(p ) (4) Fix a {, 2,,p } and x 0 Then P(a A i )=( ) p 2 t= ( i + t ) p t (5) We write Z = Z 0 + Z where Z 0 = i 0 i= i A i where i 0 = 4p log Now, by symmetry, E X (P a,b (min{π(x)} = π(x 0 )) = (6) and so It follows from (5) that for large, p E(Z ) ( ) E X (Z) = p i=i 0 + p(p ) { } 4( 2) log i exp p2 3 (7) We continue by using the Azuma-Hoeffding Martingale tail inequality see for example [,2,7,8,9]Letx 0 be fixed and for a given X let ˆX be obtained from X by replacing x j by randomly chosen ˆx j Forj let d j =max X { Eˆx j (Z(X) Z( ˆX)) } Then for any t>0wehave { } 2t 2 P( Z 0 E(Z 0 ) t) 2exp d (8) d2 We claim that d j i 0 i= i + i 0 i= ( )i 2 p i i3 0 3p + O(p) (9) 30(log )3 p 2 2 (0)

5 the electronic journal of combinatorics 7 (2000), #R26 5 Explanation for (9): If a A i (X) because ax j = ax 0 i mod p then changing x j to ˆx j changes A i by one This explains the first summation The second accounts for those a A i (X) for which ax 0 aˆx j mod p<i, changing the minimum in (3) We then use A i andp(ax 0 aˆx j mod p<i)= i p Using (0) in (8) with t = ε p2 P we see that ( Z 0 E(Z 0 ) ε p2 ε=0 ) { exp ε 2 450(log ) 6 It now follows from (4), (6), (7) and the above that E X [F (X)] = ( + O + { { }} ) ε 2 min,exp dε 2 450(log ) 6 and the result follows 2 } References [] N Alon and JH Spencer, The Probabilistic Method, Wiley, 992 [2] B Bollobás, Martingales, isoperimetric inequalities and random graphs, in Combinatorics, A Hajnal, L Lovász and VT Sós Ed, Colloq Math Sci Janos Bolyai 52, North Holland 988 [3] AZ Broder, M Chariar, AM Frieze and M Mitzenmacher, Min-Wise Independent permutations, Proceedings of the 30th Annual ACM Symposium on Theory of Computing (998) [4] AZ Broder, M Chariar and M Mitzenmacher, A derandomization using min-wise independent permutations, Proceedings of Second International Worshop RAN- DOM 98 (M Luby, J Rolim, M Serna Eds) (998) 5 24 [5] AZ Broder and U Feige, Min-Wise versus Linear Independence, Proceedings of the th Annual ACM-SIAM Symposium on Discrete Algorithms (2000) [6] P Indy, A small approximately min-wise independent family of hash-functions, Proceedings of the 0th Annual ACM-SIAM Symposium on Discrete Algorithms (999) [7] CJH McDiarmid, On the method of bounded differences, Surveys in Combinatorics, 989, Invited papers at the Twelfth British Combinatorial Conference, Edited by J Siemons, Cambridge University Press, 48 88

6 the electronic journal of combinatorics 7 (2000), #R26 6 [8] CJH McDiarmid, Concentration, Probabilistic methods for algorithmic discrete mathematics, (MHabib, C McDiarmid, J Ramirez-Alfonsin, B Reed, Eds), Springer (998) [9] MJ Steele, Probability theory and combinatorial optimization, CBMS-NSF Regional Conference Series in Applied Mathematics 69, 997