Filters. Alden Walker Advisor: Miller Maley. May 23, 2007

Transcription

1 Filters Alden Walker Advisor: Miller Maley May 23, 2007 Contents 1 Introduction Motivation Mathematical Abstraction Example (Single Bit Hashing) The Distribution of Items and Queries Two Types of Filters Off-Line Filters On-Line Filters Off-Line Filters Introduction Example Comparing Off-Line Filters An Efficient Construction A Bound to Match The Construction

2 3 On-Line Filters Example (Single Bit Hashing On-Line) On-Line Error Rate and Efficiency Number of Insertions On-Line Error Rate On-Line Efficiency Optimization of a Two-State On-Line Filter 22 6 A Class of Nice On-Line Filters S as a Poset Regular Filters Error Rate for Regular Filters Less Ambitious Properties The Big Theorem Nice Consequences of Regularity Numerical Optimization of Regular Filters Phrasing the Problem Our Method Defining the Objective Function and Feasible Set Finding Derivative Information The Optimization Algorithm Finding a Direction Optimizing Along a Line Putting it Together

3 8 Results of Numerical Optimization A Conjecture Multiple Local Minima of the Error Rate Function Finding Local Minima Some Optimized Filters Our Best Known Regular Filters Truncated Boolean Algebras Truncated Boolean Algebras Asymptotically Asymptotic Error Rate Number of States in the Limit Asymptotic Efficiency A Minimizing Along a Line 61 B Generating Random Lattices 66 C Evidence for the Local Minimum 69 3

4 1 Introduction 1.1 Motivation Storing a lot of data on a computer requires a lot of memory: if one piece of data takes b bits to record, then any method for storing n pieces of data without losing information must use bn bits of memory. Conceptually, a storage algorithm takes a collection X of data and organizes it into a representation. Once this is complete, the representation can be used to answer the question is x in X? for any new piece of data x (a membership test, or query). Generally, storage algorithms are lossless the data is organized into a representation which responds yes if and only if x actually is in the collection X. Generating a balanced search tree is one way to store data in a lossless manner (it s not important what a balanced search tree is, just that its memory use is typical for lossless algorithms). If our collection X is a list of text files, where an average size might be one million bits, then generating a balanced search tree gives us: Memory Errors (average per query) Search Tree 1, 000, 000n 0 It is also possible to develop lossy algorithms storage algorithms which produce representations which respond incorrectly in some cases. In this paper, we develop a mathematical model of lossy storage algorithms, which we call filters. These algorithms can be very useful: we ll show how to create a filter which behaves as follows on the above example of n text files: Memory Errors (average per query) Filter 2n < 1/2 If we are limited to a small amount of memory and are willing to accept an imperfect representation, a comparison between the above tables clearly shows why studying filters is 4

5 useful. In this paper, we give a mathematical abstraction for a filter algorithm, and we show that there is a tradeoff between the amount of memory required and the average number of errors per query. We ll show how to measure the error rate of a filter, and we ll exhibit basic bounds on how well a filter can do. In certain cases, we can prove stronger bounds and exhibit filter constructions which do as well as possible and match the bound. We also define a natural class of filters and show how to find the best filter in this class for certain parameters. 1.2 Mathematical Abstraction In this section, we ll make a series of abstractions and show that we can move from the Computer Science notion of a filter algorithm to a mathematical model without a loss of generality. We ll refer to a piece of data as an item. We think of a filter (or any other storage algorithm) as an algorithm which takes a list of items and produces a representation from it: List of items Filter Representation Implicit in the representation is an understanding of how to use it to answer membership queries. Often, we design storage methods which produce representations for which a querying algorithm is obvious, but technically any storage algorithm needs to specify how to answer membership queries with the representation it produces. In other words, a filter is an algorithm which outputs a representation algorithm. The output produced by the filter for a given input list X is an algorithm which takes a single item x and answers yes or no based on whether the representation regards x in X. Since we are considering filters, which are lossy, the representation algorithms produced by the filter will be imperfect. 5

6 Item List of items Filter Representation algorithm Yes No The above is a precise definition of a filter from a Computer Science perspective. In order to study filters in a mathematical way, we can make some abstractions. First, we need to define a hash function: a hash function is a algorithm for generating deterministic, pseudorandom bits based on input data. We give a hash function an item (some bits on a computer), and it produces either a 0 or a 1. 1 A good hash function f keeps the probability that f(x) = f(y) at 1/2 for x y, regardless of how similar x and y are. In other words, f looks chaotic. We assume that we have an arbitrary number of independent hash functions {f i }. With each hash function, we produce one bit, and thus we can associate any item on a computer with a sequence of 0 s and 1 s: Item f 1 f 2 f 3 f Interpreting the binary sequence as a binary decimal expansion of a real number, we see that any item can be associated with a real number in the interval [0, 1]. In addition, no matter how similar two items may be (two text files differing in a single letter, for instance), the real numbers associated with the items will be completely independent. Associating items with real numbers does not reduce the generality with which we work; it is simply a mathematical abstraction. Thus now we think of a filter as an algorithm which takes a sequence of real numbers (x 1,..., x n ) with x i [0, 1] and produces a representation algorithm which takes a single number x [0, 1] and says yes or no: 1 Here we consider hash functions with output in {0, 1}. It is certainly possible to consider hash functions with different images 6

7 x [0,1] (x,..., x ) 1 n Filter Representation algorithm Yes No In fact, the operation of the representation algorithm is completely described by recording which numbers in [0, 1] it says yes to. In other words, we can associate an entire representation algorithm with a subset of [0, 1]. We now make two assumptions: first, the subsets of [0, 1] which are associated with representation algorithms are measurable. We will denote the collection of measurable subsets of [0, 1] by M([0, 1]). Second, in a real-world situation, we want our representation algorithm generated from a list X of items to respond correctly for every item in X: f x X and we generate a representation for X, then the representation must say yes when given x as input. In terms of the mathematical abstraction we just made, this means that if we input the list (x 1,..., x n ) to a filter, then the output (now just a subset of [0, 1]) must contain all the x i : Filter (x 1,..., x n) S M([0,1]) with {x 1,..., x n} S At this point, defining a filter in our mathematical abstraction is a small step: Definition 1.1 A filter is a function F : i I[0, 1] i M([0, 1]), where I N. It is required that Im(F) be finite and that for all x 1,..., x n [0, 1] with n I, we have {x 1,..., x n } F(x 1,..., x n ). We will abuse notation slightly: take F(x 1,..., x n ) to mean F((x 1,..., x n )) and X F(X) to mean that if X = (x 1,..., x n ), then {x 1,..., x n } F(X). Also, the above definition completely describes all filters, but in this paper, we will typically be interested in two cases: filters which take sequences of items (we will continue to refer to the real numbers 7

8 input to the filter as items ) of some fixed length k, so the index set I is just the singleton set {k}, and the domain of the filter is [0, 1] k, and filters which can take as input a sequence of any length, so I = N and the domain is [0, 1] i Example (Single Bit Hashing) i=0 We ll define a filter which takes input sequences of length n. Define F : [0, 1] n M([0, 1]) as follows: Divide [0, 1] into 2n pieces, letting A i = [ i, ) { } i+1 2n 2n for 0 i < 2n (include 1 in A 2n 1 ). Let B = A j J n. Note that B is a poset under inclusion containing j J all unions of size less than or equal to n of the intervals A j, and B has a meet operation (intersection). Let F(x 1,..., x n ) = {C B {x 1,..., x n } C}. In other words, the filter produces the smallest union containing all n items. To see what this filter looks like as a real-world implementation, set n = 8. Then the filter has 16 bins. When it is given a list of items, it uses a hash function to produce the first four bits of the real number associated with each item. For each item, these bits give the bins into which it lands. Then, the filter just records which bins are hit. To respond to queries, this representation uses the same hash functions to generate the first four bits of the queried item. If the bin labeled with those bits is recorded as hit, then the representation says yes. Of course, it is likely that an item not in the original list is falsely reported as a member by the representation. Here we see the lossyness of this filter The Distribution of Items and Queries Since the list of real numbers input to the filter is essentially the output of a collection of hash functions, the real numbers appear to be random. In our abstraction, we will assume that the input to a filter is a list of random variables X = (X 1,..., X n ), where the X i are independent and X i U([0, 1]). In addition, we assume that the items with which the representation is queried are almost never actually in the original list X (we could also just 8

9 assume that the queries are random and uniform on [0, 1] since the set of input items has measure zero, the probability of querying one of them will be zero). This second assumption means that a representation algorithm (subset of [0, 1]) R will respond incorrectly whenever it responds yes at all! Therefore, R will respond incorrectly with probability m(r), the measure of R. We can already see the space/error trade off. The memory required to store the representation generated by the filter is the number of bits required to represent F(X), so the memory required is log 2 Im(F). Also, once we have generated F(X), the probability of an error is the measure of F(X). If we think about filters whose domain is [0, 1] 2 (filters which take as input pairs of items), then we see that if we want to use very little memory, the size of the image of F must be small, but still we must have X F(X) for all X [0, 1] 2. This translates to the restriction that any pair of items must be contained in some set in Im(F), and it s fairly obvious that if we need to use a small number of sets, then they must be big. Thus, the probability of an error using a representation F(X) will be high. When we study certain classes of filters in detail, we ll then make the notion of the probability of an error precise for those classes. 1.3 Two Types of Filters The general definition of a filter is very broad in order to cover all possible algorithms. Often, we ll make additional restrictions on the filters we study. Here we describe some of the restrictions and why they make sense. The rest of the paper is devoted to studying these types of filters in detail Off-Line Filters It is easy to imagine a situation in which our filter will be required to produce a representation for fixed-length lists of items i.e. our filter has domain [0, 1] k for some fixed k. In addition, 9

10 we don t expect the order in which the items are listed to matter. Definition 1.2 A filter F is commutative if when (x 1,..., x k ) [0, 1] k and σ S k is any permutation, then F(x 1,..., x k ) = F(x σ(1),..., x σ(k) ). We call a commutative filter which has as its domain fixed length sequences (really, a map from multisets since it is commutative) an off-line filter On-Line Filters There is another real world situation which can motivate our interest in a certain class of filters. First, we ll discuss the motivation in the language of real-world filtering algorithms. Then, we ll translate these requirements into our mathematical model. We might require that a filter cannot see all of X at once suppose we can fit only a single item in memory at a time for processing. Then we must be able to build F(X) incrementally. In other words, the real world filter algorithm would process each item in X in turn and would produce a suitable set representation after processing each item. This requirement is also the result of another real-world desire: what if we want to create a representation F(X 0 ) of a list, query some items, and then go back and add some items to X 0 to get X 1 and thus F(X 1 ). We d like to be able to go directly from F(X 0 ) to F(X 1 ) without looking at the items, X 0, that we ve already processed. We call a filter which must produce a representation after processing each element of X (and must thus build the representation incrementally) on-line. We make this precise with our definition in the mathematical language of filters. Definition 1.3 A filter is on-line if the following hold: 1. The domain of F is i=0 [0, 1]i. 2. For all (x 1,..., x i 1 ) [0, 1] i 1 and x [0, 1], we have F(x 1,..., x i 1 ) {x} F(x 1,..., x i 1, x). 10

11 3. If F(x 1,..., x k ) = F(y 1,..., y j ), then F(x 1,..., x k, z) = F(y 1,..., y j, z). This restriction allows a simpler, equivalent definition of an on-line filter (see Section 3) which makes analysis easier. Since the domain of an on-line filter is i=0 [0, 1]i and we must have X F(X), it seems that almost always, the filter will produce a set which is very large (since the set must contain a large number of items). When we study on-line filters in detail, we ll motivate the assumption that the length of the list of input items will be random with a fixed mean. In this way, even though a list of length 1 million, for instance, might be possible, it almost never occurs if the distribution of the length of the sequence has a small mean. 2 Off-Line Filters 2.1 Introduction In this section, we consider off-line filters maps F from finite multisets of fixed size of real numbers to measurable subsets of [0, 1] such that X F(X). We ll refer to them simply as filters. A filter of size k will be a filter which takes inputs of size k Example We ll define a filter F : [0, 1] 2 M([0, 1]) much like the example above. Let A 1 = [0, 1/3), A 2 = [1/3, 2/3), and A 3 = [2/3, 1]. As in our example before, let F(x, y) = {A i x A i or y A i }. Then the image of F is {A 1, A 2, A 3, A 1 A 2, A 1 A 3, A 2 A 3 }. Note how F is forced to cover [0, 1] in such a way that any pair of numbers is contained in one of the sets in its image. 11

12 2.2 Comparing Off-Line Filters We need to define how well these filters do we need to define an error rate. Recall that we think of the input string as a set of random variables (X 1,..., X k ), each uniform on [0, 1]. Definition 2.1 The error rate E(F) of a filter F is the expected value, denoted E, of m(f(x 1,..., X k )), where m is the Lebesgue measure. This definition captures all off-line filters, and there is no simple formula for the error rate of a filter in general, but typically, figuring out the error rate of a filter is quite easy. For the construction we ll discuss, the probability that F(X 1,..., X k ) = T for some T in the 1 image will be, and every set in the image will have the same measure µ, so the error ImF rate will just be µ. It is easy to make the error rate go down when we make the size of the image of the filter large. However, we d like a way to measure how well a filter does independent of the size of its image how well does it use the subsets in its image to cover [0, 1]? Definition 2.2 The efficiency of an off-line filter F of size k is F(F) = log 2 E(F) log 2 Im(F) k Intuitively, this is the bits of information we get per query, divided by the bits per item our filter has to create the representation. Theorem 2.3 The efficiency of an off-line filter is bounded above by 1. Proof: Let F be an off-line filter of size k. Enumerate the elements of Im(F) as A 1,..., A N, and let µ i = m(a i ). Then if we let P i =Pr[F(X) = A i ], we know that P i (µ i ) k since X F(X), and Pr[X A i ] (m(a i )) k. Thus (P i ) 1/k µ i, so E(F) = E(m(F(X))) = N P i m(a i ) i=1 N P i (P i ) 1/k = i=1 N (P i ) 1+1/k i=1 12

13 We can minimize the last quantity as follows: define f(p 1,..., P N ) = N (P i ) 1+1/k. We want to minimize f subject to the constraint that N P i = 1 and that P i 0. Note that this region is a simplex, call it S To see that f is convex, note that each function g i (x) = x 1+1/k i is convex in the positive octant, so their sum (i.e. f) is convex. Since S is a simplex (a convex set) inside the positive octant, f restricted to S is convex. In addition, any local minimum of f in S is a global minimum in S (see [4], p. 119). The gradient of f is f = (1 + 1/k)((P 1 ) 1/k,..., (P N ) 1/k ). i=1 i=1 By a standard calculus argument, f will attain a minimum or maximum on the interior of S only if f is perpendicular to the level curves of the constraint function g(p 1,..., P N ) = N P i, i.e. when f = c g = c(1, 1,..., 1). This occurs exactly when all the components of f are equal, i.e. when P i = 1/N. The Hessian matrix at this point is just ( k+1 )(1/N) (k+1)/k times the k 2 identity. Each of these entries is positive, so this matrix is positive definite, so the point P i = 1/N is local minimum. By the concavity argument above, this point is therefore a global minimum on S. Thus E(F) N(1/N 1+1/k ) = N 1/k. We can plug this into our formula for efficiency to get the bound, noting that decreasing the error rate decreases the (already negative) log, thus increasing the negated log: F(F) = log 2 E(F) log 2 N k log 2 N 1/k log 2 N k = (1/k) log 2 N log 2 N k = 1 i=1 2.3 An Efficient Construction Combinatorial constructions lend themselves well to off-line filters, because designing a good off-line filter means finding a way for the sets in the image of F to cover [0, 1] well while keeping the sets small and the size of the image of F small. There should be a certain combinatorial optimality to functions which do this. 13

14 In this section, we describe an off-line filter with high efficiency: the subspace filter of size k. Divide [0, 1] into 2 k equal intervals A i = [ i 2 k, i+1 2 k ), where 0 i 2 k 1 and A 2 k 1 contains 1. For simplicity, assume that a set X always selects exactly k of the A i (if it does not, select enough of the remaining A i at random to make k). If we associate an interval A i with the vector in GF(2) k which is the binary representation of i, then we see that a set X [0, 1] k selects k vectors in GF(2) k. There is exactly one affine hyperplane in the vector space GF(2) k which contains any given k affinely independent vectors, and we can record which hyperplane easily: a hyperplane is the set of solutions x to x p = b, where p is the vector perpendicular to the hyperplane (since the vector space is over GF(2), there s only one perpendicular vector). Our filter F maps a point X in [0, 1] k to the union of the intervals which are associated with the points in the hyperplane in GF(2) k containing the k vectors determined by X. We know the image of F comprises subsets of [0, 1] and that these subsets are in oneto-one correspondence with affine hyperplanes in GF(2) k. There are two nice ways to count affine hyperplanes in GF(q) k. First, note that specifying a vector plus what the dot product should equal (a scalar) can be done in q k+1 ways. We don t want to count the zero vector, which gets rid of q of our possibilities, and for each good vector, there are q 1 multiples of it which have the same perpendicular space, so there are 1 q 1 (qk+1 q) hyperplanes. We could also use q-binomial coefficients (see [6], p. 26): since there are k hyperplanes, and k 1 q for each hyperplane there are q parallel planes, there must be q k = q(qk 1) total q 1 k 1 affine hyperplanes. Thus in our case, there are k k 1 2 = 2k = 2 k+1 2 sets in Im(F). Because an affine hyperplane will contain exactly 1/2 of the vectors, a set in the image q 14

15 of F will have measure 1/2, and thus this filter has error rate 1/2. We can easily compute the efficiency, which is F(F) = log 2(1/2) log 2 (2 k+1 2) k = k log 2 (2 k+1 2) > k k + 1 As k gets large, the efficiency clearly gets close to 1, and it gets close quickly when k = 20, the efficiency is above If for some reason we require a filter with a lower error rate, we can do this same construction over larger fields. Using GF(q) k, the error rate will be 1/q, k log and the efficiency 2 q. This gets efficient even faster: if q = 3, we only need log 2 (q k+1 q) log 2 (q 1) k = 8 to break 0.95 efficiency, and we have a lower error rate to boot! 2.4 A Bound to Match The Construction We will show that the construction in the last section is optimal in a certain sense. Definition 2.4 An n-covering S of [0, 1] is a finite collection of measurable subsets, called states, of [0, 1] such that any n elements of [0, 1] are contained in at least one of the sets in S. An n-covering of uniform measure µ is an n-covering such that each set in the collection has measure µ. Here, if the set which an n-covering covers if not mentioned, it is understood to be [0, 1]. Notice that the image of an off-line filter of size k is a k-covering. Theorem 2.5 Let q Z +. Any n-covering S of uniform measure µ 1/q must have S qn+1 q. This inequality is strict when µ < 1/q. q 1 Proof: By induction on n. The theorem is clearly true for n = 1 because we need at least q = q2 q q 1 sets to cover [0, 1] with sets of measure less than or equal to 1/q and strictly more than q if µ < 1/q. Now assume it is true for (n 1)-coverings. Suppose we are given an n-covering S with uniform measure µ 1/q and S < qn+1 q. We may assume that S = qn+1 q 1 because q 1 q 1 15

16 we could add extra sets of measure µ to S. For x [0, 1], define f(x) to be the number of states which contain x. Because the states in S are measurable, it can be shown that f is measurable. Then 1 f(x) dx, which is the average value of f, will be µ times the number of 0 ( ) q states. That is, the average value of f is µ n+1 q 1. Since f is integer-valued, there must q 1 ( ) ( ) q be some set A of nonzero measure w for which f(x) µ n+1 q 1 q 1 n+1 q 1 q 1 q q 1 for all x in A. Because there are only a finite number (2 S ) of possible combinations of states and complements of states in S, a measurable subset B of A must have the property that every x in B is contained in exactly the same states. Let T be the collection of states in ( ) 1 q S which contain B. Note T n+1 q 1. Any measure-preserving map from [0, 1] q q 1 to itself induces a map between n-coverings, so we can suppose without loss of generality that B = [0, w] because we can construct a measure-preserving map under which [0, w] is the image of B. Define g : [w, 1] [0, 1] by g(x) = x/(1 w). Since T is a (n 1)-covering of [w, 1], T = {g(s) S T} is an (n 1)-covering of [0, 1]. In addition, the states in T have measure µ w 1/q w 1 w 1 with q Then < 1/q. Thus we have created an (n 1)-covering of uniform measure 1 w µ < 1/q ( ) q n+1 q 1 states. Note that qn 1 = 1+q+ q 1 q 1 +qn 1, so qn+1 q = q+q 2 + +q n. q 1 ( ) 1 q n+1 q 1 q q 1 = = = 1 ( q + q q n 1 ) q 1 + q + + q n 1 1 q ( 1 1 ) + q + + q n 1 q = q + + q n 1 = qn q q 1 Thus T is an (n 1)-covering with uniform measure µ strictly less than 1/q with qn q q 1 states. By the induction hypothesis, this is impossible. Therefore, we cannot have an n- 16

17 covering S of uniform measure µ 1/q with fewer than qn+1 q states. q 1 To prove that the inequality is strict when µ < 1/q, assume that S = qn+1 q. By the q 1 same argument, we can construct an (n 1)-covering T of uniform measure µ < 1/q with ( ) strictly fewer than 1 q n+1 q states. Thus T qn 1 1 = qn q. Since by the induction q q 1 q 1 q 1 hypothesis T > qn q, this is impossible. Therefore S > qn+1 q. q 1 q 1 Suppose we only consider off-line filters of size k of uniform measure µ filters for which the measure of every state in the image is fixed at µ. These filters have images which are k- coverings. If 1/µ is a power of a prime, then we can construct the subspace filter described in the previous section, and it has an image whose size exactly matches the bound in Theorem 2.5, so it has the fewest number of states possible. Since the error rate of such a filter is fixed at µ, the subspace filter is the most efficient filter possible among all filters of uniform measure µ. 3 On-Line Filters On-line filters are a natural next step after studying off-line filters. The first step in understanding on-line filters is to provide an equivalent, simpler definition for this class of filters. We will show an equivalent definition for an on-line filter with the size of X random and the domain of the filter is [0, 1] i. i=0 Definition 3.1 An on-line filter is a pair (S, ), where S is a collection of measurable subsets of [0, 1], called states, and : S [0, 1] S is a function, the insertion function, satisfying T {x} T x for all T S and x [0, 1]. The intersection T 0 = S must be an element of S, called the start state, and for every state S S, there must be some sequence x 1,..., x n with T 0 x 1 x n = S. Lemma 3.2 This definition is equivalent to our previous one. 17

18 Proof: Given a function F : i=0 [0, 1] i M([0, 1]) satisfying the previous definition of online, we let S = Im(F) and define F(x 1,..., x k ) x = F(x 1,..., x k, x). Then for any T S, T = F(x 1,..., x k ) since it is in the image of F. Therefore, T x T x by condition 2 of the previous definition. The third condition guarantees that is well-defined: if the preimage of T under F contains more than one set of items, then we will always get the same thing for T x, regardless of what preimage of T we choose. By condition 2, F( ) F(X) for all X, so T 0 = F( ) = S is an element of S. From the other direction, define F(x 1,..., x k ) = (T 0 x 1 ) x k. All the conditions on F are satisfied fairly trivially. Thus specifying a pair in this definition is equivalent to specifying an on-line filter as previously defined. This definition makes it much easier to gain intuition and study the probability of errors in a representation produced by an on-line filter. A very important note to keep in mind is that we haven t said that the insertion function must be nice at all: order of insertion can affect the final state, and even if x S, we might insert x to S and find that S x S, and so on. 3.1 Example (Single Bit Hashing On-Line) We ll consider single bit hashing again; this time, we ll actually use only a single bit. There is a slight difference between this example and our previous ones, in that our start state is nonempty. It makes this example less trivial. Define a filter F = (S, ) as follows: our set of states S is {T 0 = [0, 1/2], T 1 = [0, 1]}, and T 0 x is defined to be T 0 if x 1/2 and T 1 otherwise. Recall that everything we define equivalently defines a filter mapping from [0, 1] i to M([0, 1]), but we will be thinking about i=0 the filter in our new on-line language. Finding the representation generated by the filter from a pair of numbers (x 1, x 2 ) is equivalent to finding T 0 x 1 x 2. Suppose that our pair is 18

19 (1/4, 1/8). Then T 0 1/4 1/8 = T 0 by definition, so that is the representation generated by the filter. The power of the on-line nature of the filter is that if we decide that we would like the representation generated by (1/4, 1/8, 2/3), we don t need to start over; we simply find T 0 2/3 = T 1. 4 On-Line Error Rate and Efficiency As we did for off-line filters, we define an error rate and efficiency for on-line filters in order to compare them. In order to do this, we need to decide how many items we will be inserting: if we insert two items, our filter will probably produce subsets with a much smaller expected value than if we insert 20. We will not specify exactly how many items we will insert, but will instead motivate our assumption that the number of items inserted is random under a Poisson distribution. 4.1 Number of Insertions We are designing filters to use very little memory. In real computers, memory is divided into chunks. Currently, basic chunks are 32 bits or 64 bits, but it s not important here exactly how big a chunk is. Our problem is that it takes a long time to retrieve a chunk for processing (called probing ). Because probing is expensive, we would like to minimize the number of probes per query for our filters. Unfortunately, this limits the size of the images of our filters to the number of subsets which can be enumerated by a chunk (currently 2 32 or 2 64 ). The only way to build a bigger filter without a sacrifice in speed is to create one big filter whose representation can use M chunks, but for which an insertion or a query requires a single probe. This is accomplished by dividing the interval[0, 1] into M intervals and hashing items to one of the M chunks based on which interval they lie in. Each of the M chunks will actually hold the representation of a small filter, which will insert and query 19

20 as normal. This big filter is really just M small filters working in concert to handle a larger set of items. We can t optimize or change the behavior of the large filter, so our goal is to optimize the M small filters as much as possible. We will now show that the number of items inserted into a small filter is random under a distribution which is approximately Poisson. If we have n items to insert into the big filter and M chunks of memory, then a particular chunk can be used for k of the items in exactly ( n ) k ways. For each item, the probability that a particular chunk will be hashed to is 1/M, so the probability that the number of items inserted, I, to a particular chunk is: ( ) ( ) k ( n 1 Pr[I = k] = 1 1 ) n k k M M As n and M get large with constant ratio λ = n, this distribution converges quickly to M the Poisson distribution with mean λ, so P (I = k) λk k! e λ This motivates our assumption that the number of insertions to our on-line filters will be exactly Poisson of known mean λ. We call λ the load. 4.2 On-Line Error Rate We can define much the same error rate for on-line filters as we did with off-line filters. Now, though, we need to consider the number of insertions, I, to be a random variable in addition to the items, which are already random and uniform on [0, 1]. We motivated in the previous section that I =Poisson(λ), so we define the error rate as follows, which is essentially the expected value of the measure of the state produced under the process of random insertions. Definition 4.1 The error rate of an on-line filter F = (S, ) under load λ is E λ (F) = S S m(s)pr[f(x) = S] = S S m(s) e λ 20 k=0 λ k k! Pr[ S = T 0 X 1 X k ]

21 4.3 On-Line Efficiency Similarly, we define the efficiency of an on-line filter. Definition 4.2 The efficiency of an on-line filter F = (S, ) under load λ is F λ (F) = log 2 E λ (F) log 2 S λ As with off-line filters, we can bound the efficiency of on-line filters. Corollary 4.3 The efficiency of on-line filters is bounded above by 1. Proof: This is a corollary of Theorem 2.3. We will use on-line filters to construct a large on-line filter with a fixed number of insertions. Since it has a fixed number of insertions, this filter will be off-line as well, and then we use the theorem. Assume that we have a filter on N states whose efficiency is larger than one for load λ. Consider the filter with N M states which is just the filter on N states repeated M times. To insert into this filter, we hash each item to a particular copy of the efficient filter and insert it. Into this large filter we will insert λm items. Note here we re referring to the actual single number λ that is the mean of the random distribution and not using shorthand to indicate that the quantity λm is random. As we let M gets large, the number of items inserted into each of the little sub-filters approaches a Poisson distribution with mean λ, so each of the little filters will have efficiency greater than one, and thus the whole filter will have efficiency greater than one. Since this is impossible by Theorem 2.3, we conclude that cannot have had an on-line filter with efficiency greater than one. 21

22 5 Optimization of a Two-State On-Line Filter When we discussed off-line filters, we exhibited a construction which achieves a very high efficiency. On-line filters are more complicated, but we can optimize to find the best filter in certain cases. Here we show how to find the best two-state filter. First, we parameterize all possible two-state filters. Then we show which values of these parameters minimize the error rate. Without loss of generality, we assume that the start state is [0, x]. This turns out to be our only degree of freedom in choosing states, because: Lemma 5.1 Any on-line filter has [0, 1] as a state. Proof: Let F = (S, ) be an on-line filter. Enumerate the states in S as {A 1,..., A n }. Then define B i = [0, 1] \ A i for all 1 i n. If for any i we have B i empty, then A i = [0, 1] and [0, 1] S. If each B i is nonempty, we derive a contradiction. Let x i be some point in B i for all i. Now let T = T 0 x 1 x n. By the restrictions on, T must contain every x i, but no A i contains x i, so no state in S can contain every x i. Since T S, this is a contradiction, and thus one of the B i must be empty, and [0, 1] must be a state. Thus we have only one parameter governing the states: the measure of the start state. We must also parameterize all possible functions. Since T {y} T y for all y and T, we know that [0, 1] y = [0, 1]. By the same restriction, T 0 y = [0, 1] for all y in [0, 1] \ T 0. The only decision we can make about is whether T 0 y should be T 0 or [0, 1] for y in T 0. Without loss of generality, we assume that T 0 y = T 0 for all y with 0 y b for some b x, and T 0 y = T 1 otherwise. We are left with this parameterization in x and k of all two-state filters: T 0 if y b S = {T 0 = [0, x], T 1 = [0, 1]} and T 0 y = otherwise 22 T 1

23 To find the optimal values for x and b, we must find the error rate for load λ. definition, Here, Pr[T 0 E λ (F) = e λ k=0 λ k k! m(s) Pr[ S = T 0 X 1 X k ] S S = T 0 X 1 X k ]= b k, since the probability of k random uniform variables all being less than b is b (the measure of the set [0, b]) to the k th power. Thus Pr[T 1 = T 0 X 1 X k ]= 1 b k. Therefore we simplify our error rate (recall m(t 0 ) = x): E λ (F) = e λ k=0 λ k k! (xbk + 1(1 b k )) = e (x λ (bλ) k + k! k=0 = xe λ(b 1) + 1 e λ(b 1) k=0 λ k k! k=0 ) (bλ) k ) k! The partial with respect to b is λxe λ(b 1) λe λ(b 1) = (x 1)λe λ(b 1). Since x 1 must be negative, and the exponential is always positive, this partial is always negative, meaning b should be as large as possible. The largest we can make b is x, so we know that b = x. This isn t too surprising: it says we should not go to [0, 1] unless we have to. Now we must find the optimal value for x. To do this, we take the partial with respect to x and solve for zero to find critical points: E λ (F) x However, x must be in [0, 1]. = 0 = e λ ( e xλ + xλe xλ λe xλ) 0 = (1 λ + λx)e (x 1)λ 0 = (1 λ + λx) λ 1 = x λ When this critical point lies outside [0, 1], we check the endpoints, noting that the partial with respect to x is always positive when λ 1, so we should decrease x as much as possible (set it to zero). By When this critical point does lie in [0, 1], then we use the second derivative test to verify that it is a minimum: the second 23

24 partial is (1 λ + λx)e (x 1)λ + e (x 1)λ. At x = (λ 1)/λ, the first part is zero, and the second part is always positive, so the error rate function is concave up, and x = (λ 1)/λ is a minimum. Summarizing, we have shown that the optimal filter on two states is: T 1 if T = T 0 and y T 1 \ T 0 S = {T 0, T 1 } and T y = T otherwise if λ 1 Where T 1 = [0, 1] and T 0 = [0, λ 1] if λ > 1 λ A plot of the efficiency of the optimal filter and the optimal measure of the bottom state as functions of λ is shown in Figure F ( ) λ Optimal measure λ Figure 1: Efficiency of the optimal filter and the optimal measure of the start state as functions of λ This optimization works out nicely. With more states, both the measures of the states and the insertion function become too complicated to parameterize easily. In order to enable us to optimize larger filters, we must make restrictions on our filters. 24

25 6 A Class of Nice On-Line Filters In this section, we ll introduce several simple properties than an on-line filter might have. We ll show the logical relationships between the properties, and we ll show that these properties set off a natural class of filters, called regular, and that this class has a simple error rate formula. Then we ll describe attempts to find the best regular filter and prove nontrivial upper bounds on the efficiency of regular filters. 6.1 S as a Poset The restrictions we put on regular filters have very much to do with the structure of the states of a filter as a poset. Since each state is a subset of [0, 1], the states form a finite poset under inclusion. Recall that an element x of a poset P is a minimal state with property G if x has property G and for all y < x, y does not have property G. Also, x is the unique minimal, or minimum element of a finite poset P with property G if it is the only minimal element with G in P. Note that this means that if z has G, then x z. If x and y are elements of a poset P then x covers y if y < x and there is no z P with y < z < x. A poset P is a lattice if every pair of elements x, y has a least upper bound x y, called a join and a greatest lower bound x y, called a meet. A meet semilattice is a poset with a meet operation, and likewise for a join semilattice. 6.2 Regular Filters As our poset S of states in a filter gets large, it is inevitable that the error rate formula becomes complicated. The following type of filter has a simple error rate formula and is therefore a nice type of filter to study. Definition 6.1 An on-line filter (S, ) is regular if for all (x 1,..., x n ) [0, 1] i, there is 25 i=0

26 a unique minimal state T in the poset S containing {x 1,..., x n }, and T = T 0 x 1 x n. This definition requires two things: first, there is always a minimum element in the poset S containing a given set of items, and second, that inserting those items from the start state yields the minimum state. Notice that the poset of states implicitly defines the insertion function, so it isn t necessary to specify the insertion function for a regular filter. Thus we can think about a regular filter as an abstract poset structure together with measures associated with each element Error Rate for Regular Filters In this section, we show how the structure of the poset of states in a regular filter can give us a clean version of the error rate formula for regular filters. The major issue in finding a simple formula for the error rate of an on-line filter is determining the probability that the state produced by the filter after a random number of random insertions is some given state. We ll first prove a more general fact about posets and then exhibit as a corollary a simple error rate formula for regular filters. A useful function on posets is the Möbius function µ, where µ is defined on a poset P recursively as: µ(x, x) = 1 for all x P µ(x, y) = µ(x, z) for all x < y in P. x z<y The Möbius function facilitates a very interesting relationship between functions defined on a poset. Lemma 6.2 (Möbius inversion formula, p. 116 in [6]) Let P be a finite poset, and let f, g : P R. Then g(y) = f(z) for all y P z y 26

27 if and only if f(x) = y x µ(y, x)g(y) for all x P In order to prove the Möbius inversion formula, we ll need the following fact: Lemma 6.3 Let P be a poset, and let x, z P. Then 1 if z = x µ(y, x) = 0 otherwise z y x Proof: We ll show that the lemma is true for any given pair by induction. Given x, z P, the lemma is clearly true for z = x since the sum reduces to µ(x, x) = 1. Now suppose that the formula holds for all p with z p < x and that z < x. Rewrite µ(y, x) = µ(x, x) + µ(y, x) z y x = 1 + z y<x = 1 z y<x ( z p<x z y p y p<x µ(y, p) µ(y, p) By the induction hypothesis, the inside sum is 0 except when z = p (when the sum is 1). Since this happens exactly once in the outer sum, the entire double sum is just 1. Thus µ(y, x) = 0. z y x Proof of the Möbius inversion formula: In [6], Stanley provides a clean, fancy proof. Here we ll show the lemma more directly. µ(y, x) f(z) = µ(y, x)f(z) y x z y y x z y = µ(y, x)f(z) z x z y x = f(z) µ(y, x) z x z y x ) 27

28 so versa. By Lemma 6.3, the inner sum is 0 except when z = x. When z = x, the inner sum is 1, y x µ(y, x) z y f(z) = f(z). Now the lemma is an immediate consequence of substituting z y f(z) for g(y), and vice We can use Möbius inversion to show the following. Corollary 6.4 For a regular filter F = (S, ), E λ (F) = S S m(s) T S λ(m(t ) 1) µ(t, S)e Proof: We ll define functions f and g as in the Möbius inversion formula and apply the lemma to get this simple result. First, note that in a regular filter, Pr[ T 0 X 1 X k S ] = Pr[ X 1 S and X 2 S and and X k S ] = m(s) k. Thus Pr[ F(X) S ] = e λ k=0 λ k k! Pr[ T 0 X 1 X k S ] = e λ k=0 λ k k! m(s)k = e λ(m(s) 1) Now define f : S R by f(s) =Pr[ F(X) S ]= e λ(m(s) 1). Define g : S R by g(s) = Pr[ F(X) = S]. Since the events of ending up in different states are disjoint, f(s) = T S g(t ) So by Möbius inversion, Pr[ F(X) = S ] = g(s) = T S µ(t, S)f(T ) = T S λ(m(t ) 1) µ(t, S)e All that is left is to plug this in to the error rate formula for on-line filters, and we achieve the corollary. 28

29 6.3 Less Ambitious Properties We would like to study regular filters because of the clean error rate formula. However, regularity is a strong, global condition. Perhaps by looking at regular filters we exclude some obvious good examples and restrict our search too much. We ll now turn to some local properties which seem much more natural at first to assume. As we go, we ll prove pieces of a theorem that ties everything together. We start with two very natural properties. Definition 6.5 A filter (S, ) is stable if for all S S and for all x S, we have S x = S. Definition 6.6 A filter (S, ) is commutative if for all S S and for all x, y [0, 1] the equation (S x) y = (S y) x) holds. These are natural local conditions and it s very plausible that they are good properties to have. As we might expect, regularity is a strong condition, and the properties defined so far are implied by regularity: Lemma 6.7 A regular filter is stable and commutative. Proof: Given a regular filter F = (S, ), we ll go through the properties as listed. To see that regularity implies stability, note that if S S and x S, then there is a sequence of items x 1,..., x n such that T 0 x 1 x n = S, so S is the minimum state containing {x 1,..., x n }. Since x S, S contains {x 1,..., x n, x}, and it is the minimum state which does. Thus S = T 0 x 1 x n x = S x. Since regularity is defined using sets, a regular filter is commutative: again let S = T 0 x 1 x n. Then S x y = T 0 x 1 x n x y, which is defined to be the state T which is the minimum state containing {x 1,..., x n, x, y}. Since the set is the same, T is the minimum state containing {x 1,..., x n, y, x}, and thus S x y = T 0 x 1 x n x y = T 0 x 1 x n y x = S y x. 29

30 Commutativity and stability are obvious local conditions on the insertion function. Another natural condition governs both the poset S and the insertion function. Definition 6.8 A filter (S, ) has the canonical insertion property if for all states S S and all x [0, 1], S x is the minimum element of {T S : T S {x}} This is a clear extension of stability, but this is a stronger condition, almost akin to regularity. It says not only that there is a unique minimal element of {T S : T S {x}} but also that S x is this minimum element. Canonical insertion is a strong condition, but we see that it is implied by the properties we have seen before: Lemma 6.9 A commutative, stable filter has the canonical insertion property. Proof: Given F = (S, ) and S S and x [0, 1], we need to show S x is the unique minimal element of S containing S {x}. To this end, suppose we have a state W such that W S {x}. We need to show that S x W. We have sequences (s 1,..., s k ) and (w 1,..., w n ) such that S = T 0 s 1 s k and W = T 0 w 1 w n. Also, we have stability, so since S {x} W we know: S x = T 0 s 1 s k x (T 0 s 1 s k x) w 1 w n = (T 0 w 1 w n ) s 1 s k x by commutativity = W s 1 s k x = W by stability Thus S x is the minimum state containing S {x}, so F has the canonical insertion property. Note that this lemma also in turn shows that regularity implies canonical insertion. 30

31 6.4 The Big Theorem At this point, we ve seen that the fairly simple properties of stability and commutativity imply the more structural property of canonical insertion, and we know that regularity implies them both. Now we see that everything is equivalent: Theorem 6.10 For a filter F, the following are equivalent: (i) F is regular (ii) F is stable and commutative (iii) F has the canonical insertion property We re most of the way toward proving this theorem already. First, we ll prove that canonical insertion extends to sequences of insertions. The proof of Theorem 6.10 will then be trivial. Lemma 6.11 If F = (S, ) is a filter with the canonical insertion property, then for all states S S and finite sequences (x 1,..., x n ) of elements of [0, 1], S x 1 x n is the unique minimal element of {T S : T S {x 0,..., x n }}. Proof: We ll do this by induction on the length of the insertion sequence. The base case of a single insertion is immediate from the definition of canonical insertion. Now assume that for all insertion sequences (x 1,..., x k ) we have that S x 1 x k is the unique minimal state containing S {x 1,..., x k }. Let A = (S x 1 x k x k+1 ). By canonical insertion, A is the unique minimal state containing (S x 1 x k ) {x k+1 }. Therefore, A contains S {x 1,..., x k, x k+1 }, and we want to show it s the minimum state which does. Suppose that we have a state W with W (S {x 1,..., x k, x k+1 }). We need to show that A W. Naturally, W (S {x 1,..., x k }). By the inductive hypothesis, S x 1 x k is the unique minimal state containing S {x 1,..., x k }, so (S x 1 x k ) W. Since W also contains x k+1, W (S x 1 x k ) {x k+1 } and since A is the unique minimal element containing this union, we know that A W, as desired. 31

32 Proof of Theorem 6.10: From Lemma 6.7 we know that (i) (ii), and from Lemma 6.9 we know that (ii) (iii). We now show that (iii) (i). Since F has canonical insertion, we can apply Lemma 6.11 with state T 0 to note that for all finite sequences (x 1,..., x n ) of elements of [0, 1], T 0 x 1 x n is the unique minimal element of S containing {x 1,..., x n }. This is just the statement of regularity. 6.5 Nice Consequences of Regularity Not only do the equivalent properties in Theorem 6.10 give us the nice error rate formula in Section 6.2.1, we can deduce some very nice structural properties of the poset of states in a regular filter: Theorem 6.12 If F = (S, ) is a regular filter, then S is closed under intersection. Proof: Given a regular filter F = (S, ) and two states S, T S, we want to show that S T S. Now let {S 1,..., S k } be the states in S such that S i S T. This set is not empty because T 0 is a subset of both S and T, so T 0 S T. Now let x i be some point in (S T ) \ S i for each i, and let A = T 0 x 1 x k. First, A S, and since x i S and x i T, A S and A T, so A S T. However, A cannot be S i for any i because A contains x i and S i does not. Therefore, A = S T, and we are done. This theorem says that intersection is a meet operation on S. We ll use the following lemma to show that S is actually a lattice. Lemma 6.13 If a meet semilattice has a greatest element, it is a lattice. 32

33 Proof: Let P be a meet semilattice with a greatest element g. We ll show that P also has a join. Given a, b P, let A = {x P a x and b x}. Because a, b g, A is nonempty, and P has a meet, so we can let d = A. Since a x for all x A, a d, and likewise for b. Thus d is an upper bound on a and b, and if a, b c, then c A, so d c. Therefore d is the least upper bound for a and b, and d = a b, so P has a join and is a lattice. Corollary 6.14 If (S, ) is a regular filter, then S is a lattice under intersection. Proof: By Lemma 5.1, S has a greatest element, and by Theorem 6.12, S is closed under intersection and thus has a meet. Since S is a meet semilattice with a greatest element, it is a lattice by Lemma Thus, the poset S in a regular filter has a very nice lattice structure. By the previous theorem, the meet operation on S is just intersection, and we know S has a join, but this does not mean that the join operation is union in general, S T S T. The state S T can contain more than just S T ; it can add a little measure. It is useful to characterize this formally. Definition 6.15 For a filter F = (S, ) with S S, let W (S) = S \ weight of a state S to be m(w (S)). And as we expect, regular filters have very nice properties here: Theorem 6.16 For a regular filter F, the following hold: (i) T 0 x = S if and only if x W (S). (ii) W (S) W (T ) = if S T. (iii) S = T S W (T ) 33 T S T. Define the