Sparse Structured Associative Memories as Efficient Set-Membership Data Structures

Transcription

1 Sparse Strutured Assoiative Memories as Effiient Set-Membership Data Strutures Vinent Gripon Eetronis Department Tééom Bretagne, Brest, Frane Vitay Skahek Institute of Computer Siene University of Tartu, Estonia Mihae Rabbat Eetria and Computer Engineering MGi University, Montréa, Canada Abstrat We study the use of sparse strutured assoiative memories as a memory-effiient and omputationay-effiient data struture for representing a set of eements when one wishes to perform set-membership ueries and some errors fase positives are toerabe. Assoiative memories, when viewed as representing a set, enjoy a number of interesting properties, inuding that set membership ueries an be arried out even when the input uery eement is ony partiay known or is partiay orrupted. The assoiative memories onsidered here initiay proposed in [Gripon and Berrou, 0] enode the set in the edge struture of a graph. In this paper we generaize this onstrution to enode the set in the edge struture of a hypergraph. We derive bounds on the fase positive rates the probabiity that the assoiative memory erroneousy deares that an eement is in the stored set when it, in fat, was not. Interestingy, the proposed strutures enjoy many of the same properties as Boom fiters e.g., they have zero fase negative rate, the time to perform an insert and ookup does not depend on the number of eements stored, and the fase positive rate an be redued by using additiona memory for storage, whie aso offering the properties of assoiative memories aowing for ueries on partia or orrupted inputs. I. INTRODUCTION A wide variety of appiations reuire the abiity to uiky determine whether a given eement is a member of a pre-speified subset of possibe eements. For exampe, in network seurity and intrusion detetion, a paket must be uiky tested to see if it mathes patterns of maware or known attaks []. Simiary, ueries in database systems an be resoved or approximated set representations and operations e.g., ounting [], []. In genera, we onsider the probem of representing a subset S of a finite universe of eements U with U = n and S = m < n. The aim is to represent S using a data struture whih an uiky test whether a partiuar eement u U is a member of S i.e., in Θog n time, whie not reuiring substantia memory. Observe that storing S as a sorted array woud reuire Θm og n bits of memory and eah uery woud have a ompexity of Θog m og n time. Boom fiters [4], and their variations [5], are a popuar randomized data struture for representing sets in order to effiienty perform set membership ueries when some errors are toerabe. They store the subset S in suh a way that set membership ueries have no fase negatives i.e., if u S, then the fiter aways reognizes u as being in S, but fase positives may our i.e., an eement u U, u / S may be fasey deared to be a member of S. Speifiay, Boom fiters represent S using an array of b bits and k hash funtions h,..., h k, with eah h j mapping eements of U to [b] def = {,..., b}. The empty set S = is represented by setting a bits to 0. To add an eement x S, the bits indexed by {h j x} k j= are set to. One a bit is set to, it remains set at regardess of whether other eements of S aso hash to the same bit. After a eements x S have been inserted, to test whether an eement x is in S, one heks the status of the bits indexed by {h j x } k j= ; if a bits are set to one, then we deare x S, and otherwise we deare x / S. Ceary, as more eements are inserted in the data struture, more bits wi be set to, eading to more fase positives. The fase positive rate an be ontroed by appropriate hoie of the number of bits b and the number of hash funtions k. In partiuar, the fase positive rate is guaranteed to be no more than ɛ > 0 if b m og e og /ɛ bits and k = b/m og e hash funtions are used, regardess of the size of the universe. Insertions and ueries both have ompexity of Θk og n time if the hash funtions are evauated one after the other, or Θog n if the hash funtions an be evauated in parae. In this paper we onsider an aternative to Boom fiters for representing sets. Our approah buids on an assoiative memory onstrution proposed by Gripon and Berrou [6] whih an be viewed as sparse, strutured binary graphia modes with binary-vaued edge-potentias. We refer to these as sparse, strutured assoiative memories SSAMs. SSAMs enjoy many of the same properties as Boom fiters. Insertions and ueries an be performed in Θog n time. They never produe fase negatives, and their fase positive rate an be redued through the use of additiona bits of memory. In addition, SSAMs enjoy the error-orreting properties of assoiative memories; that is, they an be used to test if an eement x is in the stored set, even if a portion of the binary representation of x has been erased or orrupted. On the other hand, the anaysis of SSAMs assumes that the set S being represented is drawn uniformy from a subsets of m eements from U. The ontributions of this artie are as foows. First, we present detaied error bounds on the fase positive rate for

2 uster uster uster uster 4 Fig.. Iustration of the storing proess of the message in a graph using the prinipes desribed in [6]. SSAMs. The SSAM mode desribed in [6] represents the set S in terms of the edges of a graph see Setion II for detais. In this paper we present an extension of the SSAM mode to hypergraphs and demonstrate how this eads to improved error rates, at the ost of higher storage. We show that using a -reguar hypergraph is more effiient in terms of the amount of memory reuired when the number of messages m to be stored is in the regime Θn og n /. II. GRAPH-BASED SPARSE STRUCTURED ASSOCIATIVE MEMORIES This setion desribes the onstrution of SSAMs foowing the desription in [6]. Rea that eah eement x U an be represented using og n bits. Euivaenty, eah eement an be represented as a string of symbos from an aphabet A = [] as ong as og og n. For an eement x U, et w i x A, i [], denote the ith symbo in its representation as a string of symbos in A. An SSAM is represented as a simpe undireted graph G = V, E, and in partiuar, the set S = {x,..., x m } U of eements to be stored is enoded in the edge set of the graph. Let and be positive integers suh that og og n. The graph G has V = verties; more preisey, G is -partite with eah partition ontaining verties. Let v i,j, i [], j [], denote the jth vertex in the ith partition. Initiay the graph is empty; i.e., E =. Then, for eah eement x S, we add edges between a distint pairs of verties v i,wix, v i,w i x, where i, i [], i i. One an edge has been added to the graph for one eement x S, it remains in the graph i.e., nothing hanges if another eement x S aso inudes this edge. Observe that, after a eements have been added, eah eement x S orresponds to a iue in G. Let Cx = { v i,wix, v i,w i x V V : i, i, i i } denote the edges omprising the iue orresponding to x U. Then, after adding a eements in S to the SSAM we have that E = m i= Cx i. See Figure for an iustration. Now, given the graph G representing the set S, the SSAM deares that a given eement x is in S if a edges v i,wix, v i,w i x, i, i, i i, are in E; i.e., if Cx E. Ceary, there are no fase negatives, sine x S impies that Cx E. However, there may be fase positives. In partiuar, as S grows, more edges are added to E and the fase positive rate wi inrease. We provide bounds on the fase positive rate in the foowing setion. Before proeeding, we note that representing G reuires bits sine G is -partite and eah partition ontains verties. The operations of adding an eement to the SSAM and uerying an eement for set membership both have ompexity of Θ time sine they invove setting or heking the status of bits. III. FALSE POSITIVE RATE Next we derive bounds on the fase positive rate of the SSAMs desribed in the previous setion. We assume that the set S = {x,..., x m } is omprised of m random eements x i U euivaenty, m words in [] drawn uniformy and independenty. Let P denote the orresponding probabiity measure. To avoid triviaities, we assume that m >. Rea that there are possibe edges that may be in the graph. Eah time a word is inserted, one edge is added if it isn t aready present between eah partition of G. Therefore, after inserting m eements, the probabiity that a partiuar edge has not been added is m. The expeted density the proportion of edges present in G after a m eements are inserted is thus ρ = m. Our main resut haraterizes the error rate when, for a fixed density, we et the size of the network grow as the number of eements inserted inreases. Theorem : Let ρ and K be fixed onstants satisfying 0 < ρ < and 0 < K <, and et and be positive integers satisfying og og n and hosen suh that as. Then ρ K / 4 P Cx E x / S exp{ Θ}. 5 Remark : Note that the ondition 4 states that binary messages are spit into submessages whose number,, and ength, og, are of the same order of magnitude. The onstant K is a saing fator, the roe of whih wi beome ear in the seue. It aso foows from that the number of messages m to be stored is of the order of ρ K. Theorem states that the fase positive rate deays exponentiay in, the number of partitions in G, as ong as and are hosen to satisfy 4, whih ditates the rate at whih the size of the network shoud grow as a funtion of the number of eements whih have been added. To prove Theorem we need to bound the probabiity that G ontains a iue Cx for some eement x / S. For two eements, x, x U, et dx, x denote the Hamming distane between wx and wx, where wx = w x, w x,..., w x is the representation of the eement using symbos from

3 the aphabet A = []. Hene dx, x is a non-negative integer between 0 and. With sight abuse of notation, for a set S U, et dx, S = min x S dx, x. The proof of Theorem hinges on the foowing emma. Lemma : Let ρ and K be fixed onstants satisfying 0 < ρ < and 0 < K <, and et and be positive integers satisfying og og n and hosen suh that ρ K / 6 as. Let δ [] be an integer. Then, for x 0 suh that dx 0, S = δ, P Cx 0 E x 0 / S exp{ Θ}. 7 Proof: [Sketh of proof] For the eements {x,..., x m } omprising the set S, et C i = Cx i denote the orresponding iue. Let us aso write C 0 = Cx 0 for the eement x 0 / S. Without oss of generaity, suppose that dx 0, x = δ, and suppose that wx 0 and wx differ in the ast δ symbos, where wx = w x,..., w x. To simpify notation, et us further assume without oss of generaity that C is the set of a edges between pairs of verties in the set {v,, v,,..., v, }, and C 0 is the set of a edges between pairs of verties in the set {v,, v,,..., v δ,, v δ+,,..., v, }. In other words, C 0 differs from C in edges invoving the ast δ verties. Let E = C 0 \ C. Sine the expeted probabiity that a speifi edge is present in G is ρ, and sine v,, v δ+, / C, we have P v,, v δ+, E C ρ. Next, onsider the event that the edge v,, v δ+, E, and et C be a iue that ontains it. Then the edge v,, v δ+, is in E if it either beongs to C or if it beongs to a iue different from C and C. For the first ase, we have P v,, v δ+, C C, v,, v δ+, C =. For the seond ase, simiar to above, we obtain P v,, v δ+, E C, v,, v δ+, C, v,, v δ+, / C ρ. Continuing aong this ine of reasoning, we an bound the probabiity that eah of the edges in E are in E. Let and et V both = {v,, v,...., v δ, }, V diff = {v δ+,, v δ+,,..., v, }. Observe that E is exaty E = V both V diff V diff V diff \{v i,j, v i,j : v i,j V diff }. Therefore, there are E = δδ + δδ = δ δ + edges in E, and we obtain that P E E δ+ δ C ρ + }{{} ρ δ δ+. Rea that, by assumption, ρ is a fixed onstant and = exp{θ}. Thus, for suffiienty arge, we have P C 0 E C ρ δ+ δ δ+ δ ρ, where ρ approahes ρ as sine, from 6, grows exponentiay as. Now, fix x 0 U euivaenty, fix the iue C 0 or the word w 0 = wx []. For a random eement x U, P dx, x = δ > 0 δ δ δ. Sine <, it foows that the joint probabiity that C E, C 0 E, and dc 0, C = δ is at most δ δ+ δ ρ. δ There are m eements x,..., x m in S, and we assume eah is drawn independenty and uniformy from U. Therefore, appying the union bound, the probabiity that at east one x i is at Hamming distane δ from x 0 is at most Sine m δ δ δ+ δ ρ. ρ K /, and sine m og ρ f. Euation, we onude that the probabiity that there is an eement x i in S at distane δ from x 0 and that C 0 E is bounded from above by δ δ+ δ m ρ δ < og ρ ρ K δ +δ δ+δ. δ There are possibe ways to hoose x 0. By the union bound, we obtain that the probabiity of reating a fase positive iue C 0 E at distane δ from one of the iues C,..., C m orresponding to the set S is upper bounded by og ρ δ = og ρ δ = og ρ ρ K δ +δ δ+δ ρ K δ +δ δ+δ ρ og δ+ ρ hδ/ K +δ δ+δ, where h denotes the binary entropy funtion.

4 Sine 0 < ρ <, to show that this ast expression goes to zero exponentiay uiky as, it suffies to show that the foowing hods: δ + δ + δ og ρ h δ/ K +δ δ. 8 [Note that the right hand side of 8 approahes infinity for any δ > 0 as.] It remains to show that 8 hods. Sine K < and sine δ, we have δ + K > δ + + δ + δ δ + δδ + δ 7 δ δ δ δ, where the ast ineuaity hods based on the foowing reasoning: if δ = Ωand δ by definition, then the two dominating terms are 7 δ > δ /, and the other two terms are sma; if δ = o, then 6 δ > sine δ and the two other terms are sma ompared with δ. Moreover, for any δ, og ρ h δ/ + δ approahes infinity as. Therefore, 8 hods, and so we have shown that P Cx 0 E x0 / S exp{ Θ} as. The proof of Theorem now foows by bounding the fase positive rate over a possibe Hamming distanes. Proof: [Proof of Theorem ] Lemma gives that the probabiity of introduing a iue orresponding to an eement x 0 at a distane dx 0, S = δ from S deays exponentiay uiky as. Taking the union bound over a distanes δ =,,..., gives that the probabiity of introduing a fase positive iue into the graph is upper bounded by exp{ Θ} = exp{ Θ}. i= IV. EXTENSION TO HYPERGRAPHS The SSAMs introdued in Setion II are based on a graph onstrution. Speifiay, eah eement x i inserted into the memory is represented as a iue in a -partite graph. In this setion we generaize this onstrution so that the set S is represented by the edge struture of a hypergraph. Speifiay, et us onsider a -reguar hypergraph H with the same set of verties V as above, and with eah hyperedge being a set of verties for some positive integer between and. Let V x = {v,wx, v,wx,..., v,wx} uster uster uster uster 4 Fig.. Iustration of the storing proess of the message in a tridimensiona hypergraph =. denote the verties impiated by the eement x U. For >, when inserting x into the assoiative memory, we add a hyperedges orresponding to subsets of verties in V x. In genera this orresponds to a hyperiue ontaining hyperedges. The ase = is euivaent to the graph onstrution desribed in Setion II. Another interesting ase is =, for whih there is a bijetion between the sets of input messages and the possibe networks. Figure shows an exampe for = 4, =, and =. Sine H is sti -partite, and there are verties in eah partition, and sine H is aso -reguar, the memory reuired to store suh a hypergraph is bits. Simiar to the graphbased SSAM, it is ear that the hypergraph SSAM H- SSAM wi not produe fase negatives. As more eements are inserted, there wi be more hyperedges in the network, and so the fase positive rate inreases. The fase positive rate for H-SSAMs an be omputed using simiar arguments to those as in Setion III. Here we sketh the main ideas. The probabiity that a given hyperedge is in H after inserting m eements is now ρ = m. Thus, for a fixed density ρ, the number of messages stored shoud grow ike m og. ρ The probabiity that a random message x 0 is aepted by the network is p e = ρ, whih is the probabiity that the orresponding hyperedges are in network after having inserted the m eements. The expeted number of messages aepted by the network is { } m e = p e = exp ogρ + og. Thus, m e = o if and ony if ogρ + og. Given and fixed 0 < ρ <, a suffiient ondition is that og K og, ρ

5 where K < is a onstant. Euivaenty, it is suffiient to have ρ K. Now, et us re-express the number of bits, L, used to represent H in memory in terms of ρ and K. Sine there are possibe hyperedges, eah of whih is either present or not in H, we have that L ρ K. Ceary, L oud be redued by using some ompression tehniue, espeiay if ρ is far from 0.5. Sine m = og ρ K ρ and n = = ρ K, it foows that m = on. It an be shown that the number of bits reuired to represent a random set S of size S = m of eements drawn from a universe of size U = n is ower bounded by the entropy of suh a set whih saes as HS n og n n m og n m L HS m og n as n. Ceary, sine n = and, we have n as. Observe that the ratio of L to HS obeys ρ K K og ρ og ρ K ρ og K ogρ og ρ as. This ratio is minimum, and hene the H-SSAMs are most effiient, when ρ = 0.5. In this ase, when K, the memory reuired by H-SSAMs is within a fator of / og.44 of the optima vaue. Fixing ρ = 0.5 and fixing, we an derive the number of messages m that gives the best effiieny. It is eua to og K. Rea that n = K. Combining these expressions gives that the orresponding H-SSAM is most effiient i.e., the number of bits reuired to represent the H-SSAM is osest to the ower bound when the number of messages stored is m = ogn og = Θn n /. V. COMPARISON WITH OTHER SET IMPLEMENTATION TECHNIQUES Let us fix ρ = 0.5, and rea that, in this ase, m = Θ and og n = Θ. Tabe ompares the ompexities of various set impementation tehniues expressed as a funtion of. In these omparisons we have assumed that eah approah is impemented on a mahine whih performs operations on up to og bits in onstant time. Rea that a Boom fiter is guaranteed to have a fase positive rate of no more than ɛ if b m og e og /ɛ bits and k = b/m og e hash funtions are used. For the given density ρ = 0.5, the fase positive rate of the H-SSAM is. Thus, to ahieve a omparabe fase positive rate whie storing the same number of messages, a Boom fiter reuires Θ bits of memory, whih is of the same order as that reuired by H-SSAMs. Hene, Boom fiters and H-SSAMs are euivaent from the perspetive of ompexity. VI. CONCLUSION This paper proposes the use of sparse strutured assoiative memories SSAMs as a mehanism for effiient representation for sets when one needs to perform set membership ueries and some errors are toerabe. The basi onstrution represents the set of eements in terms of the edge struture of a graph. We aso propose an extension where the set is enoded in the struture of a hypergraph H-SSAMs, and we show that this approah has advantages in terms of redued error rate in appropriatey arge regimes. In omparison to Boom fiters, SSAMs and H-SSAMs have the advantage that they an be used in situations where error orretion is reuired. For exampe, if the ueried eement x is represented as a string of symbos from an aphabet of size, if some symbos are erased or orrupted, Boom fiters are no onger appiabe sine the entire desription of x is needed for hashing. Assoiative memories, on the other hand, are designed to aow for reovery of orrupted or missing data, and their abiity to orret for errors is studied in [6], [7]. SSAMs and H-SSAMs may be of interest for use in distributed appiations, e.g., by storing the eements of the adjaeny matrix/tensor at different agents. In this manner, no singe agent oud independenty determine if an eement is in the set, and they must ooperate when responding to ueries. This may be of interest in distributed databases with sensitive ontent. By using hash funtions to map eements to random bit indies, Boom fiters provide universa ompression of sets in the sense that the fase positive rates hod regardess of how the eements of S are drawn. However, the anaysis in this paper is onduted under the assumption that eements are drawn uniformy and independenty. We are urrenty investigating the performane of SSAMs under more genera distributions on S. One simpe way to irumvent orreation is to append the binary representation of eah eement x with additiona bits whih are drawn uniformy and independenty; of ourse, this redues the orreation but aso inreases the amount of memory reuired. Finay, we note that assoiative memories with strutures simiar to SSAMs have reenty been shown to enjoy nie faut toerane properties [8], in the sense that the error rates are not severey impated if the graph struture is perturbed i.e., edges are randomy added or removed. Ahieving this faut toerane reuires a sighty more sophistiated, iterative ookup proedure for performing ueries. In genera there are many interesting simiarities between Boom fiters and SSAMs, and estabishing more onretey the onnetion between these two strutures is the subjet of future work.

6 TABLE I COMPARISON OF THE COMPLEXITIES OF VARIOUS SET IMPLEMENTATION TECHNIQUES. Impementation Average ompexity Worst ase ompexity Memory ompexity Insert Query Insert Query Naive array Θ Hash tabe Θ Trie Θ Sef-baaning binary searh tree Θ Boom fiter Θ H-SSAM Θ ACKNOWLEDGEMENTS This work was funded in part by the European Researh Couni ERC AdG 0 NEUCOD, the Natura Sienes and Engineering Researh Couni of Canada NSERC, and the Fonds Québéoos de a reherhe sur a nature et es tehnoogies FQRNT. REFERENCES [] A. Papadogiannakis, M. Poyhronakis, and E. P. Markatos, Improving the auray of network intrusion detetion systems under oad using seetive paket disarding, in Proeedings of the Third European Workshop on System Seurity, ser. EUROSEC 0, New York, NY, USA, 00, pp. 5. [] C. S. Lin, D. C. P. Smith, and J. M. Smith, The design of a rotating assoiative memory for reationa database appiations, ACM Trans. Database Syst., vo., pp. 5 65, Mar [] A. Rajaraman and J. Uman, Mining of Massive Datasets. Cambridge University Press, 0. [4] B. Boom, Spae/time trade-offs in hash oding with aowabe errors, Communiations of the ACM, vo., no. 7, pp. 4 46, 970. [5] A. Broder and M. Mitzenmaher, Network appiations of Boom fiters: A survey, Internet Mathematis, vo., no. 4, pp , 005. [6] V. Gripon and C. Berrou, Sparse neura networks with arge earning diversity, IEEE Transations on Neura Networks, vo., no. 7, pp , Juy 0. [7], Neary-optima assoiative memories based on distributed onstant weight odes, in Proeedings of Information Theory and Appiations Workshop, San Diego, CA, USA, February 0, pp [8] F. Ledu-Primeau, V. Gripon, M. Rabbat, and W. Gross, Faut-toerant assoiative memories based on ustered graphs, May 0, submitted.