CSCB63 Winter Week 11 Bloom Filters. Anna Bretscher. March 30, / 13

Transcription

1 CSCB63 Winter 2019 Week 11 Bloom Filters Anna Bretscher March 30, / 13

2 Today Bloom Filters Definition Expected Complexity Applications 2 / 13

3 Bloom Filters (Specification) A bloom filter is a probabilistic dictionary and stores a summarized set F S of a set S. It requires elements (keys) to be hashable. Operations: insert(f S, k): Add element k to F S. search(f S, k): Return no" if k S and probably yes" if (with high probability) x S. Note. probably yes" can be returned even though k S. What is this called? false positive. k present k absent return true return false We try to keep Pr(true absent) > 0 small. No delete usually. (If supported, false negatives possible too.) 3 / 13

4 Bloom Filters (Implementation) Let U be the universe of keys. Let BF be an array of m bits BF[0..m 1]. There are t hash functions, h 1, h 2,..., h t that map from U {0,..., m 1}. Idea. To insert key k j, for each hash function h i, set BF[h i (k j )] = 1. Suppose we hash n keys. Q. How can we search for some key k U whether it has been inserted into BF? A. We simply check if all the bits BF[h i (k)] = 1 for 1 i t. Then k is not in BF if any bit is 0 and probably in if all bits are 1. Q. When will this answer be wrong? correct? A. Imagine that we have two hash functions, and inserting x sets bits 1 and 3, and inserting y sets bits 2 and 4. Then if it happens z would set bits 2 and 3, if we search for z we will get probably yes. 4 / 13

5 Summary. insert(f S, k): for i = 1 to t: BF[h i (x)] = 1 search(f S, k): for i = 1 to t: if BF[h i (x)] = 0 return no return probably yes Q. What is the complexity of each of these functions? A. Θ(t) time. Q. Why use multiple hash functions? A. To reduce the probability of false positives. 5 / 13

6 Probability of False Positive Q. What does the probability of a false positive depend on? A. size m of the filter n the number of items inserted t the number of hash functions Intuition. If α = n/m goes up, then collisions go up, false positives go up. If t is too low, more collisions, more false positives If t is too high, too many bits set to 1, more collisions, more false positives Notes. If 1/α = m/n < key length (e.g., keys are IP addresses, addresses, URLs) this uses much less memory than storing the keys verbatim. 6 / 13

7 Probability of False Positive Q. Insert n keys into an empty filter. Now lookup an absent key. What is the probability of returning true? Assumptions. Hash values are uniformly random and mutually independent. Insert n keys can be modelled by: Randomly set a bit to 1 nt times. Search for an absent key can be modelled by: Randomly pick t bits to read, What is the probability of getting all 1 s? After inserting n keys, probability that an individual bit is 1: Q. What is the probability that h i (k) hashes to j? A. 1/m. Q. What is probability that BF[j] = 0 after one hash function applied to 1 key? A. 1 1/m. 7 / 13

8 Probability of False Positive Q. What is the probability that BF[j] = 0 after t hash functions applied to 1 key? A. (1 1/m) t Q. What is the probability that BF[j] = 0 after t hash functions applied to n keys? A. (1 1/m) nt Pr(BF[i] = 1) = 1 Pr(BF[i] = 0) ( = ) nt m ( = nt/m ) nt nt 1 e nt/m = 1 e αt 8 / 13

9 Probability of False Positive Pr(BF[i] = 1) = 1 e αt To model search(f S, k) we randomly check t bits in BF. Q. What is the probability of getting all 1 s? A. Pr(false positive) = (1 e αt ) t Note. We now can minimize the probability of false positive based on the number of hash functions t. The value of t that minimizes Pr(false positive) is t = ln 2 1. α Plug back in for optimal probability: ( ) ln 2 1/α α ln 2 α (1 e α ) = (1 2 1 ) ln α 2 1 = /α 2 ln 2 1 see Hadzilakos notes 9 / 13

10 Examples Example 1. 1/α = 32. Optimal t? t = ln 2/α = ln , use t = 22. Probability of false positives: Pr(false positive) = (1 e αt ) t = (1 e 22/32 ) Example 2. Suppose we want a probability 10 7, find α and t /α = /α t = ln 2/α 23 Q. What about delete(f S, k)? can it be implemented? A. Usually don t. Would cause false negatives - ruins the whole concept. However, if occasionally necessary, could increment a counter at each location in BF. Then decrement counters for delete. BF is no longer just a bit array. 10 / 13

11 Application 1 Blacklisting Consider we have a huge blacklist of domain names, stored on a slow disk. Too long to keep in faster memory. Use the fact most lookups turn out to be not blacklisted. Implementation. Use a Bloom filter for quick 1st-stage check. Q. How can we quickly check whether an input domain name is blacklisted? A. Every time a domain d is blacklisted, insert d into the Bloom filter BF. To check whether a domain d is blacklisted, search(bf, d). False means for sure not blacklisted. Fast and happens most of the time because most inputs are not blacklisted. True means probably blacklisted. We need to check for sure, so check disk file. Slow but happens rarely because few inputs are blacklisted or false positives. Note. A key is bits long, but a Bloom filter just needs 10 bits per key for pretty low probability (< 1%) of false positives. 11 / 13

12 Approximate Counting of Visitors Count how many IP addresses have visited your website. Do not have enough space to store all of them. Implementation. Store a counter and a Bloom filter. When an IP address visits your website, ask the Bloom filter. If false, insert into Bloom filter, and increment counter. If true, do nothing. Q. Is the counter exact? A. This undercounts, but you re probably OK with it. Technically we say that the website has at least counter visitors. An IPv4 address is 32 bits long, IPv6 even worse. But 10 bits per key buys you a 99%-accurate answer. 12 / 13

13 Application: Program Verification SPIN verifies a program by converting to a graph (like finite state automaton) and DFS to see whether bad states are reachable. DFS stores and lookups visited vertices. Each vertex is maybe 160 bits because your program has 5 32-bit variables. Too big. SPIN uses a Bloom filter for that with t = 2. Much fewer bits per vertex and still low false-positive rate. (Also, adjacency list not stored permanently, instead recomputed from state transition function which is small and fast.) Not perfect but still much more effective than handwritten tests. 13 / 13