Bloom Filters. filters: A survey, Internet Mathematics, vol. 1 no. 4, pp , 2004.

Transcription

1 Bloo Filters References A. Broder and M. Mitzenacher, Network applications of Bloo filters: A survey, Internet Matheatics, vol. 1 no. 4, pp , Li Fan, Pei Cao, Jussara Aleida, Andrei Broder, Suary Cache: A Scalable Wide-Area Web Cache Sharing Protocol, IEEE/ACM Transactions on Networking, Vol. 8, No. 3, June o Oiin Origin of counting Bloo filters Bloo Filters (Sion S. La) 1 1

2 Origin and applications Randoized data structure introduced by Burton Bloo [CACM 1970] o It represents a set for ebership queries, with false positives o Probability of false positive can be controlled by design paraeters o When space efficiency is iportant, a Bloo filter ay be used if the effect of false positives can be itigated. First applications in dictionaries and databases Bloo Filters (Sion S. La) 2 2

3 First application in networking: distributed cache (2000) Proxy 1 Cache 1 Suary 2 Suary 3 Proxy 2 Cache 2 Suary 1 Suary 3 Proxy 3 Cache 3 Suary 1 Suary 2 Nuerous applications in networking since 2000 Bloo Filters (Sion S. La) 3 3

4 Standard Bloo Filter A Bloo filter is an array of bits representing a set S = { x 1, x 2,, x n n} of n eleents o Array set to 0 initially k independent hash functions h 1,, h k with range {1, 2,, } o Assue that each hash function aps each ite in the universe to a rando nuber uniforly over the range {1, 2,, } For each eleent x in S, the bit h i (x) in the array is set to 1, for 1 i k, o A bit in the array ay be set to 1 ultiple ties for different eleents Bloo Filters (Sion S. La) 4 4

5 A Bloo filter exaple (three hash functions) Insert X 1 and X 2 Check Y 1 and Y 2 Bloo Filters (Sion S. La) 5 5

6 Standard Bloo Filter (cont.) To check ebership of y in S, check whether h i (y), 1 i k, k, are all set to 1 o If not, y is definitely not in S o Else, we conclude that y is in S, but soeties this conclusion is wrong (false positive) For any applications, false positives are acceptable as long as the probability bilit of a false positive is sall enough We will assue that kn < Bloo Filters (Sion S. La) 6 6

7 False positive probability After all ebers of S have been hashed to a Bloo filter, the probability that a specific bit is still 0 is 1 p ' = (1 ) kn e kn/ = p For a non eber, it ay be found to be a eber of S (all of its k bits are nonzero) withfalse positive probability k (1 p ') (1 p ) k Bloo Filters (Sion S. La) 7 7

8 False positive probability (cont.) Define f ' = (1 p') 1 = (1 (1 ) ) k f = (1 p) kn/ k = (1 e ) k kn k Two copeting forces as k increases (1 p ') k o Larger k -> is saller for a fixed p (1 1 / ) kn o Larger k -> p = is saller -> 1-p larger Bloo Filters (Sion S. La) 8 8

9 False positive rate vs. k n = Nuber of bits per eber 8 Nuber of Bloo Filters (Sion S. La) 9 9

10 Optial nuber k fro derivative Rewrite f as / / exp(ln(1 kn k kn ) ) exp( ln(1 )) f = e = k e kn/ Let g = kln(1 e ) Miniizing g will iniize f = exp( g) kn/ g kn/ k (1 e ) = ln(1 e ) + kn / k 1 e k kn/ k n kn/ = ln(1 e ) + e = ln(2) + ln(2) = 0 kn/ 1 e if we plug k = ( / n) ln 2 which is optial (It is in fact a global optiu) Bloo Filters (Sion S. La) 10 10

11 Optial k fro syetry / p = e kn Alternatively, fro we get k = ln( p ) n Fro previous slide, we have g = k e = p p n kn/ ln(1 ) ln( )ln(1 ) Fro above, syetry indicates that the iniu value for g occurs when p=1/2. Thus k opt = ln(1/ 2) = n n ln(2) Bloo Filters (Sion S. La) 11 11

12 Optial k fro syetry using the precise probability of false positive f ' = (1 p') k = exp( k ln(1 p')) kn Fro p ' = (1 1/ ), solving for k 1 k= ln( p') n ln(1 1/ ) Let g' = k ln(1 p') ( q f ) (in equation for f ' above) = 1 ln( p')ln(1 p') n ln(1 1/ ) Bloo Filters (Sion S. La) 12 12

13 Using the precise probability of false positive to get optial k (cont.) Fro previous slide g 1 ' = ln( ')ln(1 ') nln(1 1/ ) p p By syetry, g (also f ) iniized at p =1/2 Optial k is 1 1 k' opt = ln( p') = ln(1/2) nln(1 1 / ) nln(1 1 / ) Bloo Filters (Sion S. La) 13 13

14 Optial nuber of hash functions Using k opt = n ln(2) the false positive rate is ln(2) ln(2) n n n / (1 p) = (0.5) (0.6185), where ln(2) = In practice, k should be an integer. May choose an integer value saller than k opt to reduce hashing hi overhead /n denotes False positive rate bits per entry Bloo Filters (Sion S. La) 14 14

15 False positive rate vs. bits per entry False positive rate 4 hash functions Using optial nuber of hash functions /n Bloo Filters (Sion S. La) 15 15

16 Standard Bloo Filter tricks Two Bloo filters representing sets S 1 and S 2 with the sae nuber of bits and using the sae hash functions. o A Bloo filter that represents the union of S 1 and S 2 can be obtained by taking the OR of the bit vectors A Bloo filter can be halved in size. Suppose the size is a power of 2. o Just OR the first and second halves of the bit vector o When hashing to do a lookup, the highest order bit is asked Notation: OR denotes bitwise or Bloo Filters (Sion S. La) 16 16

17 Counting Bloo filters Proposed by Fan et al. [2000] for distributed caching Every entry in a counting Bloo filter is a sall counter (rather than a single bit). o When an ite is inserted into the set, the corresponding counters are each increented by 1 o When an ite is deleted d fro the set, the corresponding counters are each decreented by 1 To avoid counter overflow, its size ust be sufficiently large. It was found that 4 bits per counter are enough. Bloo Filters (Sion S. La) 17 17

18 Counter overflow probability Consider a set of n eleents, k hash functions, and counters o C(i) is the count for the i th counter nk 1 1 Pci [() = j] = 1 j nk 1 P[() ci j] j j j nk j enk j j (a very loose upper bound) Bloo Filters (Sion S. La) 18 18

19 Counter overflow probability (cont.) Choose k such that k /n (ln 2) Then j enk e ln 2 Pci [() j] j j e ln 2 P[ax c( i) j] 1 i j j j for soe i Using 4 bits, each counter counts fro 0 to 15 P[ax c( i) 16] i 15 Bloo Filters (Sion S. La) 19 19

20 Counter overflow consequences When a counter does overflow, it ay be left at its axiu u value. This can later cause a false negative only if eventually the counter goes down to 0 when it should have reain at nonzero. The expected tie to this event is very large but is soething we need to keep in ind for any application that does not allow false negatives Bloo Filters (Sion S. La) 20 20

21 Conclusions Wherever a list or set is used, and space is at a preiu, a Bloo filter ay be used if fthe effect of false positives can be itigated o No false negative With a counting Bloo filter, false negatives are possible, albeit highly unlikely Bloo Filters (Sion S. La) 21 21

22 The End Bloo Filters (Sion S. La) 22 22