String Hashing for Linear Probing

Transcription

1 Strng Hashng for Lnear Probng Mkkel Thorup Abstract Lnear probng s one of the most popular mplementatons of dynamc hash tables storng all keys n a sngle array. When we get a key, we frst hash t to a locaton. Next we probe consecutve locatons untl the key or an empty locaton s found. At STC 07, Pagh et al. presented data sets where the standard mplementaton of 2-unversal hashng leads to an expected number of Ωlog n probes. They also showed that wth 5-unversal hashng, the expected number of probes s constant. Unfortunately, we do not have 5-unversal hashng for, say, varable length strngs. When we want to do such complex hashng from a complex doman, the generc standard soluton s that we frst do collson free hashng w.h.p. nto a smpler ntermedate doman, and second do the complcated hash functon on ths ntermedate doman. ur contrbuton s that for an expected constant number of lnear probes, t s suffces that each key has 1 expected collsons wth the frst hash functon, as long as the second hash functon s 5-unversal. Ths means that the ntermedate doman can be n tmes smaller, and such a smaller ntermedate doman typcally means that the overall hash functon can be made smpler and at least twce as fast. The same doublng of hashng speed for 1 expected probes follows for most domans bgger than 32-bt ntegers, e.g., 64-bt ntegers and fxed length strngs. In addton, we study how the overhead from lnear probng dmnshes as the array gets larger, and what happens f strngs are stored drectly as ntervals of the array. These cases were not consdered by Pagh et al. 1 Introducton A hash table or dctonary s the most basc non-trval data structure. We want to store a set S of keys from some unverse U so that we can check membershp, that s, f some x U s n S, and f so, look up satellte nformaton assocated wth x. ften we want the set S to be dynamc so that we can nsert and delete keys. We are not nterested n the orderng of the elements of U, and that s why we can use hashng for effcent mplementaton of these tables. Hash tables for strngs and other complex objects are central to the AT&T Labs Research, Shannon Laboratory, 180 Park Avenue, Florham Park, NJ 07932, USA. mthorup@research.att.com. analyss of data, and they are drectly bult nto hgh level programmng languages such as python, andperl. Lnear probng s one of the most popular mplementatons of hash tables n practce. We store all keys n a sngle array T, and have a hash functon h mappng keys from U nto array locatons. When we get a key x, we frst check locaton hx nt.ifadfferentkeysn T [hx], we scan next locatons sequentally untl ether x s found, or we get to an empty spot concludng that key x s new. If x s to be nserted, we place t n ths empty spot. To delete a key x from locaton, wehave to check f there s a later locaton j wth a key y such that hy, and n that case delete y from j and move t up to. Recursvely, we look for a later key z to move up to j. Ths deleton process termnates when we get to an empty spot. Thus, for each operaton, we only consder locatons from hx and to the frst empty locaton. Above, the successor to the last locaton n the array s the frst locaton, but we wll generally gnore ths boundary case. It can also be avoded n the code f we leave some extra space at the end of the array that we do not hash to. If the keys are complex objects lke varable length strngs, we wll typcally not store them drectly n the array T. Instead of storng a key x, we store ts hash value hx and a ponter to x. Storng the hash value serves two purposes: 1 durng deletons we can quckly dentfy keys to be moved up as descrbed above, and 2 when lookng for a key x, we only need to check keys wth the same hash value. We wll, however, also consder the opton of storng strngs drectly as ntervals of the array that we probe, and thus avod the ponters. Practce The practcal use of lnear probng dates back at least to 1954 to an assembly program by Samuel, Amdahl, Boehme c.f. [10]. It s one of the smplest schemes to mplement n dynamc settngs where keys can be nserted and deleted. Several recent expermental studes [2, 7, 14] have found lnear probng to be the fastest hash table organzaton for moderate load factors 30-70%. Whle lnear probng s known to requre more nstructons than other open addressng methods, the fact that we access an nterval of array entres means that lnear probng works very well wth modern archtectures for whch sequental access s much faster than random access assumng that the 655 Copyrght by SIAM.

2 elements we are accessng are each sgnfcantly smaller than a cache lne, or a dsk block, etc.. However, the hash functons used to mplement lnear probng n practce are heurstcs, and there s no known theoretcal guarantee on ther performance. Snce lnear probng s partcularly senstve to a bad choce of hash functon, Heleman and Luo [7] advce aganst lnear probng for general-purpose use. Analyss Lnear probng was frst analyzed by Knuth n a 1963 memorandum [9] now consdered to be the brth of the area of analyss of algorthms [15]. Knuth s analyss, as well as most of the work that has snce gone nto understandng the propertes of lnear probng, s based on the assumpton that h s a truly random functon, mappng all keys ndependently. In 1977, Carter and Wegman s noton of unversal hashng [3] ntated a new era n the desgn of hashng algorthms, where explct and effcent ways of choosng provably good hash functons replaced the unrealstc assumpton of complete randomness. They asked to extend the analyss to lnear probng. Carter and Wegman [3] defned unversal hashng as havng low collson probablty, that s 1/t f we hash to a doman of sze t. Later, n [20], they defne k-unversal hashng as a functon mappng any k keys ndependently and unformly at random. Note that 2-unversal hashng s stronger than unversal hashng n that the dentty s unversal but not 2-unversal. ften the unformty does not have to be exact, but the ndependence s crtcal for the analyss. The frst analyss of lnear probng based on k- unversal hashng was gven by Segel and Schmdt n [16, 17]. Specfcally, they show that log n- unversal hashng s suffcent to acheve essentally the same performance as n the fully random case. Here n denotes the number of keys nserted n the hash table. However, we do not have any practcal mplementaton of log n-unversal hashng. In 2007, Pagh et al. [12] studed the expected number of probes wth worst-case data sets. They showed that wth the standard mplementaton of 2- unversal hashng, the expected number of lnear probes could be Ωlog n. The worst-case s one or two ntervals somethng that could very well appear n practce, possbly explanng the experenced unrelablty from [7]. It s nterestng to contrast ths wth the result from [11] that smple hashng works f the nput has hgh entropy. The stuaton s smlar to the classc one for non-randomzed quck sort, where we get nto quadratc runnng tme f the nput s already sorted: somethng very unlkely for random data, but somethng that happens frequently n practce. Lkewse, f we were hashng characters, then n practce t could happen that they were mostly letters and dgts, and then they would be concentrated n two ntervals. n the postve sde, Pagh et al. [12] showed that wth 5-unversal hashng, the expected number of probes s 1. From [19] we know that 5-unversal hashng s fast for small domans lke 32-bt ntegers. Man contrbuton ur man contrbuton s that for an expected constant number of lnear probes, we can salvage 2-unversal hashng and even unversal hashng f we follow t by 5-unversal hashng. The unversal hashng collects keys wth the same hash value n buckets. The subsequent 5-unversal hashng shuffles the buckets. Clearly ths s weaker hashng than f we appled the 5-unversal hashng drectly to all keys, but t may seem that the ntal unversal hashng s wasted work. ur motvaton s to get effcent hashng for lnear probng of complex objects lke varable length strngs. The pont s that we do not have any reasonable 5- unversal hashng for these domans. When we want complex hashng from a complex doman, the generc standard soluton s that we frst do collson free hashng w.h.p. nto a smpler ntermedate doman, and second do the complcated hash functon on ths ntermedate doman. ur contrbuton s that for expected constant lnear probes, t suffces that each key has 1 expected collsons wth the frst hash functon, as long as the second hash functon s 5-unversal. Ths means that the ntermedate doman can be n tmes smaller. Such a smaller ntermedate doman typcally means that both the frst and the second hash functon are smpler and run at least twce as fast. The same doublng of hashng speed for 1 expected probes follows for most domans bgger than 32-bt ntegers, e.g., fxed length strngs and even the mportant case of 64-bt ntegers whch may be stored drectly n the array. A dfferent way of vewng our contrbuton s to consder the extra prce pad for constant expected lnear probng on worst-case nput. For any effcent mplementaton of hash tables, we need a hash functon h 1 nto say [2n] ={0,..., 2n 1} where each key has 1 expected collsons. ur result s that for lnear probng wth constant expected cost, t suffces to compose h 1 wth a 5-unversal hash functon h 2 :[2n] [2n]. Usng the fast mplementaton of h 2 from [19], the extra prce of h 2 s much less than that of hashng a longer strng or that of a sngle cache mss. ur general contrbuton here s to prove good performance for a specfc problem for a low-complexty hash functon followed by a hgh complexty one, where the frst hash functon s not expected collson-free. We hope that ths approach can be appled elsewhere to 656 Copyrght by SIAM.

3 speed up hashng for specfc problems. Addtonal results We extend our analyss to two scenaros not consdered by Pagh et al. [12]. ne s f the sze of the array s large compared wth the number of elements. The other s when we store strngs drectly as ntervals of the array. ur results for these cases are thus also new n the case of a sngle drect 5-unversal hash functon from the orgnal doman to locatons n the array. Contents In Secton 2 we state our formal result andnsectons3-4weprovet. InSecton5weshow how to apply our result to dfferent domans and how the tradtonal soluton wth a frst collson free hashng nto a larger ntermedate doman would be at least twce as slow. In Secton 6 we consder the opton of storng strngs as ntervals of the lnear probng array. 2 The formal result As descrbed above, we use lnear probng to place a dynamc set of keys S U of sze S n n an array T wth t> S locatons [t] ={0,..., t 1}. We wll often refer to α = n/t as the fll. Each key x S s gven a unque locaton x [t] ={0,..., t 1}. Locaton arthmetc s done modulo t, sof<jthen [j, ] ={j,.., t 1, 0,.., }. IfQ s a set of locatons, then + Q = { + q q Q}. The locatons depend on a hash functon h : U [t]. For lnear probng, the assgnment of locatons has to satsfy the followng nvarant: Invarant 2.1. For every x S, the nterval [hx, x] s full. Invarant 2.1 mples that to check f x s n S, wecan start n locaton hx and search consecutve locatons untl ether x or an empty locaton s found. In the later case, we can add x, settng x =. Invarant 2.1 also mples that a locaton s full f and only f there s an l such that S h 1 [l] l. Here S h 1 [l] = {y S l<hy }. For each key x U, weletfull h,s x denotethe number of locatons from hx to the nearest empty locaton, or more precsely, the largest l such that hx +[l] s full. We can now bound the number of locatons consdered by the dfferent operatons: nsert h,s x fnds empty locaton for x S n full h,s x+1probes. delete h,s x consders full h,s x + 1 locatons, movng keys up to fll gaps created by the deleton. fnd h,s x uses at most full h,s\{x} x +1 probes. If x S, t fnds an empty locaton. If x S, toget to x, t only to passes locatons that would be full wthout x n S. ur hash functon h : U [t] wll always be composed of two functons va some ntermedate doman A; namely h 1 : U A and h 2 : A [t]. Then hx =h 2 h 1.Foreachx U, let the bucket sze b 1 x be the number of elements y S wth h 1 y =h 1 x. Note that b 1 x countsx f x S and that b 1x s the sum of the squares of the bucket szes. ur man techncal result can now be stated as follows: Theorem 2.1. Let h 1 be fxed arbtrarly, and let h 2 s be a 5-unversal hash functon from A to [t] where for every a A and [t], the probablty that h 2 a = s at most α/n. Then, for any x 0 U, b1 x 0 E [full h,s x 0 ] = 1 α + b 1x/n 1 α 5 Concernng the factor 1 α 5,weknowfromKnuth [9] that a factor 1 α 2 s needed even when h 1 s the dentty and h = h 2 s a truly random functon. Applyng Theorem 2.1 to an unversal hash functon h 1, we get the corollary below, whch s what was clamed n the ntroducton. Corollary 2.1. Let h 1 be a hash functon from U to an ntermedate doman A such that the expected bucket sze b 1 x of any key x s constant. Let h 2 be a 5- unversal hash functon from A to [t] where for every a A and [t], the probablty that h 2 a = s at most α/n for some constant α < 1. Then for every x 0 U, we have E [full h,s x 0 ] = Small fll We wll also consder cases wth fll α = o1. It s then useful to consder coll h,s x =full h,s x [x S] Here the Boolean expresson x S has value 1 f x S; 0 otherwse. We can thnk of coll h,s x asthenumber of elements from S \{x} that x colldes wth n the longest full nterval startng n hx. Smlarly, we defne c 1 x =b 1 x [x S], whch s the standard number of collsons between x and S \{x} under h 1.Notethat we may have full h,s\{x} x coll h,s x because x may fll a an empty gap between two full ntervals. Theorem 2.2. Let h 1 be fxed arbtrarly, and let h 2 s be a 5-unversal hash functon from A to [t] where for every a A and [t], the probablty that h 2 a = s at most α/n wth α 0.9. Then, for any x 0 U, E [full h,s x 0 ] = b 1 x α b 1 x 0 + b 1 x/n E [coll h,s x 0 ] = c 1 x 0 +α b 1 x/n 657 Copyrght by SIAM.

4 Usng the last bound of Theorem 2.2, we get Corollary 2.2. Let n 0.9 t and t t. Let h 1 be a unversal hash functon from U to [t ] and let h 2 be a 5-unversal hash functon from [t ] to [t]. Then for every x 0 U, E [coll h,s x 0 ] = n/t. Proof. Snce h 1 s unversal, we get E [c 1 x] <n/t n/t = α for all x U. If h 1 s the dentty and h = h 2, then ths s the scenaro studed by Pagh et al. [12], except that we have subconstant fll α = n/t = o1. ne can thnk of coll h,s x as the hashng overhead from not havng each key mapped drectly to a unque locaton n the array. Havng E [coll h,s x] = α s clearly best possble wthn a constant factor. 3 Proof of Theorem 2.1 We wll now start provng Theorem 2.1. Let [hx H, hx +L be the longest full nterval contanng hx. Then full h,s x =L and we want to lmt the expectaton of L. As n Theorem 2.1 we have h = h 2 h 1 where h 1 s fxed and h 2 s 5-unversal. Defne S 0 = {x S h 1 x h 1 x 0 }. Then S 0 + b 1 x 0 = S. Snce[hx H, hx+l sexactly full, we have H + L = S h 1 [hx 0 H, hx 0 +L = b 1 x 0 + S 0 h 1 [hx 0 H, hx 0 + S 0 h 1 [hx 0,hx 0 +L It follows that at least one of the followng cases are satsfed: 1. b 1 x 0 1 α/3 L. 2. S 0 h 1 [hx 0,hx 0 +L α +1 α/3 L. 3. S 0 h 1 [hx 0 H, hx 0 H 1 α/3 L. We are gong to pay L n each of the above cases. In fact, n each case separately, we are gong to seek the largest L satsfyng the condton, and then we wll use 3 =1 L as a bound on L. Incase1,wehave L 1 3b 1 x 0 /1 α, correspondng to the frst term n the bound of Theorem 2.1. To complete the proof, we need to bound the expectatons of L 2 and L 3. We call L 2 the tal because t only nvolves keys hashng to hx 0 and later, and we call L 3 the head because t only nvolves keys hashng before hx A basc lemma The followng lemma wll be used to bound the expectatons of both L 2 and L 3. Lemma 3.1. For any Q [t] of sze q, 3.1 Pr [ αq + d S 0 h 1 hx 0 +Q <D ] αq 0 [b 1 x <D]b 3 1x d 4 n + 3α2 q 2 0 [b 1 x <D]b 1 x 2 d 4 n 2 Above, the Boolean expresson [b 1 x <D]hasvalue1 f b 1 x <D; 0 otherwse. Before contnung, note that f h 1 s the dentty, that s, f we just used 5-unversal hashng, then b 1 x = 1 for all x S, and then the rght hand sde reduces to αq, whch s what s proved n [12]. Here we need to handle dfferent bucket szes b 1 x. For our later analyss, t wll be crucal that we have ntroduced the cut-off parameter D. d + 3q2 α 2 4 d 4 Proof of Lemma 3.1. Apart from the cut-off parameter D, our proof s very smlar to one used n [12]. Snce h 2 s 5-unversal, t s 4-unversal when we condton on any partcular value of hx 0 =h 2 h 1 x 0, so we can thnk of Q 0 = hx 0 +Q as a fxed set and h 2 as 4-unversal on h 1 S 0 =h 1 S \{h 1 x 0 }. For each a A, setb a = {x S 0 h 1 x =a} and b a =[b a <D]b a, observng that S 0 h 1 Q 0 <D S 0 h 1 Q 0 = a A[h 2 a Q 0 ]b a Therefore, the event n 3.1 mples 3.2 αq + d a A[h 2 a Q 0 ]b a For each a A, letp a =Pr[h 2 a Q 0 ] αq/n, and consder the random varables X a = b a [h 2 a Q 0 ] p a Then E [X a ]=0. SetX = a X a.then a A[h 2 a Q 0 ]b a = X + a A p a b a X + αq. Hence 3.2 mples 3.3 d X To bound the probablty of 3.3, we use the 4th moment nequalty Pr [X d] E [ X 4] /d 4. Snce E [X ]=0andtheX are 4-wse ndependent, we get E [ X 4] = E [X a1 X a2 X a3 X a4 ] a 1,a 2,a 3,a 4 A E [ X 4 ] +3 a A a A E [ Xa 2 ] Copyrght by SIAM.

5 Here E [ X 4 ] < and a A I E [ X 2 ] a A p a b 4 a αq n p a b 2 a a A 2 a S 0 [b 1 a <D]b 3 1 a αq 2 [b 1 x <D]b 1 x n 0 For the probablty bound of the lemma, we dvde the total expectaton of E [ X 4] by d The tal L 2 We want to show that E [L 2 ]= b 1x/n α 5. Let W l = {S 0 h 1 [hx 0,hx 0 +l} Then L 2 s the largest value n [t] such that W L 2 α +1 α/3l 2. In [12] they have b 1 x = [x S] 1 for all x U, and they use that E [L 2 ]= =1 Pr[L 2 ]. However, wth some large b 1 x, we have no good bound on Pr[L 2 ]. Ths s why we ntroduced our cut-off parameter D n Lemma 3.1. Let l 0 = 27/1 α and l = l 1 /1 1 α/9. Thenl 1 > 1 1 α4/27l. We say that >0sgood f W l [α +1 α/9l, α +1 α/3l. Lemma 3.2. If l 1 L 2 <l,then s good. Proof. If L 2 < l then W l < α +1 α/3l. Also, we have W l W L 2 α +1 α/3l 2 α+1 α/3l 1 > α+1 α/31 1 α4/27l > α +1 α/9l. Hence E [L 2 ] <l 0 + Pr[ good]l. =1 Here l 0 satsfes 3.4. To bound Pr[ good] we use Lemma 3.1 wth q = l, Q =[q], d =1 α/9 l, and D = l.then Pr[ good] = αq 0 [b 1 x <D]b 3 1 x d 4 n 2 + 3α2 q 2 0 [b 1 x <D]b 1 x 2 d 4 n 2 = 0 [b 1 x <l ]b 3 1 x 1 α 4 l 3 n 0 [b 1 x <l ]b 1 x 2 + Hence Pr[ good]l = + = 0 = 0 = 1 α 4 l 2 n2 0 [b 1 x <l ]b 3 1 x 1 α 4 l 2 n 0 [b 1 x <l ]b 1 x 2 :l >b 1x 1 α 4 l n 2 b 3 1x 1 α 4 l 2 n b 1 x + 1 α 4 l n 2 b 1 x 0 b 1 x 1 α 5 n α 5 n 2 0 b 1 x 1 α 5 n. 0 b 1 x Above, the marks the pont at whch we explot our cut-off [b 1 x < l ]. Ths completes the proof of 3.4. Below we are gong to need several smlar calculatons, but wth dfferent parameters, carefully chosen dependng on the context. The presentaton wll be focused on the dfferences, only sketchng the smlar parts. 3.3 The head L 3 We want to show that E [L 3 ]= b 1x/n α 5. Here L 3 s the largest value such that from some H, we have S 0 h 1 [hx 0 H, hx 0 H 1 α/3 L 3. We want to bound the expectaton of L 3. The argument s bascally a messy verson of that n Secton 3.2 for the tal L 2. Let UH = S 0 h 1 [hx 0 H, hx 0. ThenH maxmzes D = UH H, andthenl 3 =3D/1 α, so we are really tryng to bound D. Losng at most a factor 2, t suffces to consder cases where UH 2H, for suppose UH > 2H and consder the largest H such that UH 1 > 2H 1. Then H Copyrght by SIAM.

6 H + D/2 souh 1 H 1 > H + D/2, mplyng UH 1 H 1 D/2+1. Consequently, D = UH H UH 1 H D/2. Thus there s an H such that UH 2H andsuchthat UH H 1 α/6 L 3. Fnally, defne R such that UH =α + R 1 αh.then1 R 2/1 α and L 3 6R 1H = R H. For I =0, 1,.., log 2 1 α,leth I be the largest value such that UH I α +2 I 1 αh I. Let I be such that R /2 < 2 I R. Then H I H,so 2 I H I >R H /2. Thus we conclude that L 3 = max I {2 I H I }= log 2 1 α I=0 2 I H I. As n Secton 3.2, let l 0 = 27/1 α, and l = l 1 /1 1 α/9. Wesaythat s I-good f Ul [α +2 I 1 α/3l, α +2 I 1 αl. Smlar to Lemma 3.2, we have that s I-good f l 1 H I <l. Hence log 2 1 α E [L 3 ] = 2 I l 0 + Pr[ I-good]l I=0 = 1/1 α 2 log 2 1 α + Pr[ I-good]2 I l I=0 The frst term satsfes 3.5 so we only have to consder the double sum. To bound Pr[ I-good] we use Lemma 3.1 wth q = l, Q =[q], d =2 I 1 α/3 l, and D =2 I l. Wth calculatons smlar to those n Secton 3.2, we get Pr[ I-good]2 I l = = It follows that 0 [b 1 x < 2 I l ]b 3 1 x 1 α 4 2 3I l 2 n 0 [b 1 x < 2 I l ]b 1 x b 1 x 2 I 1 α 5 n log 2 1 α I=0 1 α 4 2 3I l n 2. Pr[ I-good]2 I l = log 2 1 α I=0 = 0 b 1 x 1 α 5 n 0 b 1 x 2 I 1 α 5 n. Ths completes the proof of 3.5, hence of Theorem Proof of Theorem 2.2 We wll now prove Theorem 2.2. The bound concdes wth that of Theorem 2.1 when α<1saconstant,so we may assume α = o1. We wll use the same basc defntons as n the proof of Theorem 2.1. Recall that h = h 2 h 1 where h 1 s fxed and h 2 s 5-unversal. Defne S 0 = {x S h 1 x h 1 x 0 }. Welet[hx H, hx+l be the longest full nterval contanng hx. Then full h,s x =L and we want to lmt the expectaton of L. We ntroduce a parameter β, and note that at least one of the followng cases are satsfed: 4. b 1 x 0 1 α1 β L. 5. S 0 h 1 [hx 0,hx 0 +L α+1 αβ/2 L. 6. S 0 h 1 [hx 0 H, hx 0 H 1 αβ/2 L. Wth β =2/3, the above concdes wth cases 1-3 from Secton 3. However, n our analyss below, we wll have α<β/8andβ 2/5. In each case separately, we are gong to seek the largest L satsfyng the condton, and then we wll use 6 =4 L as a bound on L. In case 4, we trvally have 4.6 L 4 b 1 x 0 /1 α1 β. We are gong to bound L 5 and L 6 by α 4.7 L 5 + L 6 = β 2 b 1 x/n. We wll now show how 4.6 and 4.7 mply Theorem 2.2. For the bound on full h,s x 0 wesetβ = α 1/3. Then L 4 b 1 x 0 /1 α1 3 α = b 1 x α whle L 5 + L 6 = 3 α b 1x/n. Gettng the bound on coll h,s x 0 s a bt more subtle. We use β =2/5 andα < 1/6. Then L 4 b 1 x 0 /1 α1 β <b 1 x 0 /5/6 3/5 = 2b 1 x 0. Snce L 4 and b 1 x 0 are nteger, we get that L 4 = b 1 x 0 for b 1 x 0 1, and L 4 1 3b 1 x 0 1 for b 1 x 0 > 1. Snce x 0 S mples b 1 x 0 1, we get Hence L 4 [x 0 S] 3b 1 x 0 [x 0 S] = 3c 1 x 0. coll h,s x 0 = full h,s x 0 [x 0 S] 660 Copyrght by SIAM.

7 L 4 [x 0 S]+L 5 + L 6 3c 1 x 0 +α b 1 x/n. Thus Theorem 2.2 follows from 4.6 and 4.7. It remans to bound the expectatons of L 5 and L 6 as n 4.7. We call them the small head and tal because they are much smaller now that α = o The small tal L 5 We want to show that α 4.8 E [L 5 ]= β 2 b 1 x/n. usng α β<1. The proof s very smlar to that n Secton 3.2. Let W l = {S 0 h 1 [hx 0,hx 0 +l} Then L 2 s the largest value n [t] such that W L 2 α +1 αβ/2l 5 >β/2 L 5. Ths tme, for =0, 1,.., we defne l =2,whchs a much faster growth than t had n Secton 3.2 We say that >0sgood f W l [β/4 l,βl. Smlar to Lemma 3.2, we have that s good f l 1 L 5 <l. Hence E [L 5 ] Pr[ good]l. =0 To bound Pr[ good] we use Lemma 3.1 wth q = l, Q = [q], d = β/4 αl β/8 l for α β/8, and D = βl. Wth calculatons smlar to those n Secton 3.2, we get Pr[ good]l = α = α 0 [b 1 x <βl ]b 3 1 x β 4 l 2 n + α2 0 [b 1 x <βl ]b 1 x 2 β 4 l n 2. 0 b 1 x β 2 n Ths completes the proof of The small head L 6 We want to show that α 4.9 E [L 6 ]= b 1 x/n. β Here L 6 s the largest value such that for some H 6,we have S 0 h 1 [hx 0 H 6,hx 0 H 6 1 αβ/2 L 6. Defne UH = S 0 h 1 [hx 0 H, hx 0. We wll look for the largest H such that UH H.Note that UH 6 +1 αβ/2 L 6 UH 6 H 6 +1 αβ/2 L 6. Hence H H 6 +1 αβ/2 L 6, and therefore L 6 = H /β. As n the prevous secton, we defne l =2. Ths tme we say that s good f Ul [l /2,l. Smlar to Lemma 3.2, we have that s good f l 1 H <l. Hence E [L 3 ]=E[H ] /β = Pr[ good]l /β. =0 To bound Pr[ good] we use Lemma 3.1 wth q = l, Q = [q], d = 1/2 αl 1/3 l for α 1/6, and D = l. Wth calculatons smlar to those n Secton 3.2, we get Pr[ good]l /β = α 0 [b 1 x < 2l ]b 3 1 x l 2 n + α2 0 [b 1 x < 2l ]b 1 x 2 α = 0 b 1 x βn. l n 2 Ths completes the proof of 4.9, hence of Theorem Applcaton We wll now show how to apply our result to dfferent domans and how the tradtonal soluton wth a frst collson free hashng nto a larger ntermedate doman would be at least twce as slow. Frst we consder 64-bt ntegers, then fxed length strngs, and fnally varable length strngs. We assume that the mplementaton s done n a programmng language lke C [8] or C++ [18] leadng to effcent and portable code that can be nlned from many other programmng languages. We note that ths secton n tself has no theoretcal contrbuton. It s demonstratng how the theoretcal result of the prevous secton plays together wth the fastest exstng technques, yeldng smpler code runnng at least twce as fast. For the reader less famlar wth effcent hashng, ths secton may serve as a mnsurvey of hashng that s both fast and theoretcally good. 661 Copyrght by SIAM.

8 Recall the general stuaton: We are gong to hash a subset S of sze n of some doman U. The hash range should be ndces of the lnear probng array. For smplcty, the range wll be [2n], and we assume that 2n s a power of 4. ur hash functon h : U [2n] wll always be composed of two functons va some ntermedate doman [m]; namely h 1 : U [m] andh 2 :[m] [2n]. Then hx =h 2 h 1 x. Wth the tradtonal method, we want h 2 to be collson free w.h.p., whch means m old = ωn 2. From our Corollary 2.1, t follows that for expected constant number of probes, t suffces wth expected constant bucket sze, and hence t suffces wth m new = n. Thus m new = o m old. As a concrete example, we wll thnk of m new =2 30 and m old =2 60. We denote the correspondng bt-lengths l new =30andl old = unversal hashng from the ntermedate doman Below we consder two dfferent methods for 5-unversal hashng. Degree 4 polynomal The classc method [20] for 5-unversal hashng s to use a random degree 4 polynomal h a0,..,a 4 x = 4 =0 a x over Z p. To get afast modp operaton, Carter and Wegman [3] suggest usng a Mersenne prme p such as , , , or The pont s that wth p =2 a 1, we can explot that y =y & p+y>>a mod p where>> s rght shft and & s bt-wse and. At the end, we extract the l least sgnfcant bts. A typcal stuaton would be that m new <p new = whle < m old < p old = For multplcatons n Z , we can use that 64- bt multplcaton provdes exact 32-bt multplcaton. However, for multplcaton n Z , we have the problem that 64-bt multplcaton dscards overflow, so we need 4 64-bt multplcatons to get the full answer. For larger n, we mght have p new = and p old {2 81 1, }, but, we stll need more than twce as many 64-bt multplcatons wth the large old doman. Tabulaton More recently [19] suggested a tabulaton based approach to 5-unversal hashng whch s roughly 5 tmes faster than the above approach 1. The smplest case s that we dvde a key x [m] nto 2 characters x 0 x 1. The hash functon s computed as T 0 [x 0 ] T [x 1 ] T [x 0 +x 1 ]. Here s bt-wse xor and T 0,T 1,T 2 are ndependently tabulated 5-unversal hash functons. T 0 and T 1 have m entres, and T 2 has 2 m entres. Ths s fast f m s small enough to ft n 1 The method s desgned for 4-unversal hashng, but as ponted out n [12] t works unchanged for 5-unversal hashng. fast memory. The experments n [19] found ths to be the case for m =2 32, ganng a factor 5 n speed over the polynomal-based method above. Ths s perfect for our new small ntermedate doman m = m new =2 30, but not wth the old large ntermedate doman m = m old =2 60. The method n [19] s generalzed so that we can dvde the key nto q characters, and then perform 2q 1 look-ups n tables of sze 2m 1/q, but the method s more complcated for q>2. Assumng that we want the same small table sze wth the old and the new method, ths means that q old 2q new, hence that the large old doman wll requre more than twce as many look-ups and twce as much space bt ntegers We only need unversal hashng from 64-bt ntegers to l-bt ntegers, so we can use the method from [6]: We pck a random odd 64-bt nteger a, and compute h a x =a x >>64 l. Ths method actuallyexplots that the overflowfroma x s dscarded. The probablty of collson between any two keys s at most 1/2 l. We use ths method for h 1 usng l new =30 n the new approach and l old = 60 n the old approach. The value of l does not affect the speed of h 1, but h 1 s very fast compared wth the 5-unversal h 2.Theoverall hashng tme s domnated by h 2 whch we above found more than twce as fast wth our smaller ntermedate doman. 5.3 Fxed length strngs We now consder the case that the nput doman s a strng of 2r 32-bt characters. For h 1 we wll use a straghtforward combnaton of a trck for fast sgnatures n [1] wth the 2-unversal hashng from [4]. The combnaton s folklore [13]. Thus, our nput s x = x 0 x 2r 1,wherex s a 32- bt nteger. The hash functon s defned n terms of 2r random 64-bt ntegers a 0,..., a 2r 1. The hash functon s computed as h a0,...,a 2r 1 x 0 x 2r 1 r 1 = x 2 + a 2 x a 2+1 >> 64 l =0 Ths method only works f l 33, but then the collson probablty for any two keys s 2 l. The cost of ths functon s domnated by the r 64-bt multplcatons. We can use the above method drectly for our l new = 30. However, for l old = 60, the best we can do s to apply the above type of hash functon twce, usng a new set of random ndces a 0,..., a 2r 1 for the second applcaton, and then concatenate the resultng hash values, thus gettng an output of 2l new = l old bts. Thus for the ntal hashng h 1 of fxed length strngs, we gan 662 Copyrght by SIAM.

9 a factor 2 n speed wth our new smaller ntermedate doman. 5.4 Varable length strngs For the ntal hashng h 1 of varable length strngs x = x 0 x 1 x v,wecanuse the method from [5]. We vew x 0,..., x v as coeffcents of a polynomal over Z p, assumng x 0,..., x v [p]. We pck a sngle random a [p], and compute the hash functon v h a x 0 x v = x a. =0 If y s another strng that s no longer than x, then Pr[h a x =h a y] v/p. As usual, the computatons are faster f we use Mersenne prmes. In ths case, we would probably use p = and 32-bt characters x, regardless of the sze of the ntermedate doman. We wll hash down to the rght sze usng the 64-bt hashng from Secton 5.2. As stated above, our smaller ntermedate doman does not make any dfference n speed, but f we want fast hashng of longer varable length strngs, then the above method s too slow. For hgher speed, we dvde the nput strng nto chunks of, say, ten 32-bt characters, and apply the fxed length hashng from Secton 5.3 to each chunk, thus gettng a reduced strng of chunk hash values. If we have more then one chunk, we apply the above varable length strng hashng to the strng of chunk hash values. Note that two strngs reman dfferent under ths reducton f there s a chunk n whch they dffer, and the hashng of that chunk preserves the dfference. Hence we can apply exactly the same hash functon to each chunk. The overall tme bound s domnated by the length reducng chunk hashng, and here, wth the old method, we should hash chunks to 64 bts whle we wth the new method can hash chunks to 32 bts. As dscussed n Secton 5.3, the new method gans a factor 2 n speed. 6 Storng varable length strngs drectly Aboveweassumedthateachstrngx had a sngle array entry contanng ts hash value and a ponter to x. Pagh [13] suggested analyzng the alternatve where we store varable length strngs drectly as ntervals of the array we probe. Each strng s termnated by an end-of-strng character \0. If a strng x s n the array, t should be preceded by ether \0 or an empty locaton, and t should be located between hx andthefrstempty locaton after hx, as stated by Invarant 2.1. We nsert anewkeyx startng at the frst empty locaton after hx. We may have to push some later keys further back to make room, but all n all, we only consder locatons between hx and the frst empty locaton after hx n the state after x has been nserted. The advantage to the above drect representaton s that we do not need to follow a ponter to get to x. Deletons are mplemented usng the same dea. As n the prevous sectons, we let full h,s x denote the largest l such that hx+[l] s full. Then full h,s x+ 1 bounds the number of locatons consdered by any operaton assocated wth x. In ths measure we should nclude x n S f x s nserted or deleted. By a smple reducton to Theorem 2.2, we wll prove Theorem 6.1. Let S U be the set of varable length strngs x. Letlengthx be the length of x ncludng an end-of-strng character \0. Suppose the total length of the strngs n S s at most αt for some α<0.9. Leth 1 be a unversal hash functon from U to [t], andleth 2 be a 5-unversal hash functon from [t] to [t]. Then for every x 0 U, we have 6.10E [full h,s x 0 ] = lengthx α lengthx 0 + Moreover, when x 0 S, 6.11 E [full h,s x 0 ] = α length2 x lengthx length2 x lengthx To bound the number of cells consdered when nsertng or deletng x 0, we use 6.10 and get a bound of lengthx α lengthx 0 + length2 x lengthx. For fndng x 0 we can use the stronger bound from 6.11, yeldng 1+ α length2 x lengthx f x 0 S lengthx 0 + α length2 x lengthx f x 0 S Proof of Theorem 6.1. To apply Theorem 2.2, we smply thnk of all characters of a key x as hashng ndvdually to hx. The fllng of the array s ndependent of the order n whch elements are nserted, so as long as we satsfy Invarant 2.1, t doesn t matter for full that we want characters of x to land at consecutve locatons. More specfcally, we let the hash functon h 1 apply ths way to the ndvdual characters. Recall here that h 1 s 663 Copyrght by SIAM.

10 arbtrary n Theorem 2.2. We get that E [full h,s x 0 ] b 1 x 0 = 3 a α b 1 x 0 + b 1x {a x S} = 3 α b 1 x 0 + lengthxb 1x lengthx, Now b 1 x s the total length of strngs y S wth h 1 y =h 1 x. Snce h 1 s unversal, for each strng x, E [b 1 x] = [x S]lengthx+ [x S]lengthx+α. y S\{x} lengthy/t By lnearty of expectaton, 6.10 follows of the theorem. For the bound 6.11 when x 0 S, we use the bound on coll h,s x 0 from Theorem 2.2. When x 0 S, we have coll h,s x 0 =full h,s x 0 andc 1 x 0 =b 1 x 0 so E [c 1 x 0 ] = α. We therefore get E [full h,s x 0 ] = E [coll h,s x 0 ] a = c 1 x 0 +α b 1x {a x S} = α lengthx2 lengthx. It s easy to see that the bound of Theorem 6.1 s tght wthn a constant factor even f a truly random hash functon h s used. The frst term s trvally needed. For the second term, the basc pont n the square s that the probablty of httng y s proportonal to ts length, and so s the expected cost of httng y. Ths drect storage of strngs was not consdered by Pagh et al. [12], so ths s the frst proof that lmted randomness suffces n ths case. The bound of Theorem 6.1 gves a large penalty for large strngs, and one may consder a hybrd approach where one for strngs of length bgger than some parameter l store a ponter to the suffx, that s, f the strng does not end after l characters, then the next characters represent a ponter to a locaton wth the remanng characters. Then t s only for longer strngs that we need to follow a ponter, and the cost of that can be amortzed over the work on the frst l characters. References [1] J. Black, S. Halev, H. Krawczyk, T. Krovetz, and P. Rogaway. UMAC: fast and secure message authentcaton. In Proc. 19th CRYPT, pages , [2] J. R. Black, C. U. Martel, and H. Q. Graph and hashng algorthms for modern archtectures: Desgn and performance. In Proc. 2nd WAE, pages 37 48, [3] J. Carter and M. Wegman. Unversal classes of hash functons. J. Comp. Syst. Sc., 18: , Announced at STC 77. [4] M. Detzfelbnger. Unversal hashng and k-wse ndependent random varables va nteger arthmetc wthout prmes. In Proc. 13th STACS, LNCS 1046, pages , [5] M. Detzfelbnger, J. Gl, Y. Matas, and N. Pppenger. Polynomal hash functons are relable extended abstract. In Proc. 19th ICALP, LNCS 623, pages , [6] M. Detzfelbnger, T. Hagerup, J. Katajanen, and M. Penttonen. A relable randomzed algorthm for the closest-par problem. J. Algorthms, 25:19 51, [7] G. L. Heleman and W. Luo. How cachng affects hashng. In Proc. 7th ALENEX, pages , [8] B. Kernghan and D. Rtche. The C Programmng Language 2nd ed.. Prentce Hall, [9] D. E. Knuth. Notes on open addressng, Unpublshed memorandum. Avalable at [10] D. E. Knuth. The Art of Computer Programmng, Volume III: Sortng and Searchng. Addson-Wesley, [11] M. Mtzenmacher and S. P. Vadhan. Why smple hash functons work: explotng the entropy n a data stream. In Proc. 19th SDA, pages , [12] A. Pagh, R. Pagh, and M. Ruzc. Lnear probng wth constant ndependence. In Proc. 39th STC, pages , [13] R. Pagh. Personal communcaton, [14] R. Pagh and F. F. Rodler. Cuckoo hashng. J. Algorthms, 51: , [15] H. Prodnger and W. Szpankowsk. Preface for specal ssue on average case analyss of algorthms. Algorthmca, 224: , [16] J. P. Schmdt and A. Segel. The analyss of closed hashng under lmted randomness. In Proc. 22nd STC, pages , [17] A. Segel and J. Schmdt. Closed hashng s computable and optmally randomzable wth unversal hash functons. Techncal Report TR , Currant Insttute, [18] B. Stroustrup. The C++ Programmng Language, Specal Edton. Addson-Wesley, Readng, MA, [19] M. Thorup and Y. Zhang. Tabulaton based 4- unversal hashng wth applcatons to second moment estmaton. In Proc. 15th SDA, pages , [20] M. Wegman and J. Carter. New hash functons and ther use n authentcaton and set equalty. J. Comp. Syst. Sc., 22: , Copyrght by SIAM.