CBFM cutted Bloom filter matrix for multi-dimensional membership query

Size: px

Start display at page:

Download "CBFM cutted Bloom filter matrix for multi-dimensional membership query"

Aldous Burns
5 years ago
Views:

1 37 3 Vol.37 No Journal on Communications March 2016 doi: /j.issn x CBFM 1 1, CBFM(cutted Bloom filter matrix) BFM(Bloom filter matrix) CBFM CBFM CBFM BFM 3 TP393 A CBFM cutted Bloom filter matrix for multi-dimensional membership query WANG Yong 1, YUN Xiao-chun 1,2, WANG Shu-peng 1, WANG Xi 1 (1.Institute of Information Engineering, Chinese Academy of Sciences, Beijing , China; 2. National Computer Network Emergency Response Technical Team/Coordination Center of China, Beijing , China) Abstract: In order to improve the flexibility and accuracy of mu i-dimensional membership query, a new indexing structure called CBFM(cutted Bloom filter matrix) was proposed. CBFM built the bit matrix by the Cartesian product of different bloom filters, each representing one attribute of primary data. In this way, the proposed matrix supported by-attribute membership query. Besides, the attribute combinations in the bit matrix could be reduced and weighted on demand to further enhance memory utilization rate. Theoretical analysis proves that CBFM utilizes memory more efficiently than BFM, the current state of art. Experiments also show that, on the scenario of memory size fixed, the false positive rate of CBFM is lower than that of all other indexin ethods. Especially on the scenario of memory constrained, the false positive rate of CBFM can be 3 orders of magnitude lower than that of BFM(Bloom filter matrix) indexing method. CBFM is an accurate data structure for multi-dimensional membership query. Key words: query algorithm, multi-dimensional membership query, Bloom filter, bit matrix 1 Web2.0 true/false [2~6] [1] [7] PB No , No , No , No ( 863 ) No. 2012AA013001, No.2013AA013205, No.2013AA01A213 Foundation Items: The National Natural Science Foundation of China (No , No , No , No ), The National High Technology Research and Development Program of China (863 Program) (No.2012AA013001, No.2013AA013205, No.2013AA01A213)

2 Google BigTable [2] HBase [3] CBFM CBFM MapReduce [4] Hive [5] Impala [6] 2) CBFM BFM CBFM TCP 3) CBFM [7] [8,9] CBFM BFM [10,11] [12] 3 CBFM [8, 9] [10,11] 2 [12] MDBF [13] CMDBF [14] 2.1 [12] SBF 1 BFM [14] S ={x 1, x 2,L, x n } n k (h 1, h 2,L, h k ) m x i h j (x i ) (1 j k) BF h j (x i ) 1 xk h 1 (x), h 2 (x),l, h k (x) (schema-free) 1 x x CBFM cutted Bloom filter matrix CBFM 1)

3 3 CBFM 141 O(n) O(log n) (false positive rate) 2.2 SBF EDMI Guo [13] KD MDBF ZPR MD-HBase [11] Z Z-Value CMDBF [14] MDBF KD CBF 2 MDBF 1 2 CBF {n, m, k, L} CMDBF ( f BF (m, k, n)) L +1 MDBF ( f BF (m, k, n)) L MDBF CMDBF Adina [18] 2015 [14] Bloofi B+ (BFM) Flat-Bloofi KEY BF-Matrix [19] BFM 1 B+ BFM 3 CBFM 3.1 [9] Zhou [10] [15] [16] (CBF) S ={x 1, x 2,L, x n } n x i, x i d, x i = [x i1, x i 2,L, x id ] BFM x = [x 1, x 2,L, x d ], 1 j d : x j x = [x [8,9] 1, x 2,L, x d ], [10,11] 1 j d : x [18,19] j true/ false q[q [8] ZFS 1,q 2,L,q d ] S : x j (1 j d ), q j = x j true q[q 1, q 2,L, q d ] S : x j (1 j d), q j x j false

4 CBFM CBFM 1 1 d n e c CBFM r i, c i R,1 i c r i d i=1 m M 3 CBFM CBFM 4 CBFM 2 CBFM 2 CBFM 2(a) CBFM 1 elementinsert(object Element) : A=(a 1,a 2,, a d) :VOID begin assign 0 to ResultArray; for each attribute a x in A { for each hashfunction hashx in HashSet { record hashx(a x) to hashperfield[][]; } // k for each DC of (2 d - 1) combinations { for each hash value hashv in hashperfield { BFM 2(b) YoZ YoZ // calculateindex invoke calculateindex(dc, hashv, ResultArray); } // 1 for each idx of ResultArray { set bit of ResultArray [idx] to 1 end elementinsert XYZ 3 CBFM A(A x, A y ) 4 CBFM CBFM CBFM 5 CBFM WCBFM -Weighted CBFM

5 3 CBFM elementsearch(object Element) : A=(a 1,a 2,, a d), SDC null : begin assign 0 to ResultArray; for each field a x in A { for each hashfunction hashx in HashSet { record hashx(a x) to hashperfield[][]; } // k for each EDC(does not match any dimension combinations to cut off) of SDC { for each hash value hashv in hashperfield { // calculateindex invoke calculateindex(edc, hashv, ResultArray); } for each idx of ResultArray { if bit of ResultArray [idx] equals to 0 { return FALSE; } return TRUE; end elementsearch 5 CBFM 3.3 S ={x 1, x 2,L, x n } x i, x i d CBFM n e d c k r i c d 2 d 1 :VOID if DC match some dimension combinations to cut off { f =e end calculateindex; // [17] kn 1 f BF (m, k, n) = (1 (1 ) kn ) k 1 e m (1) m f n k m k = lbf (2) 1 m = lbelb n (3) f k m+1 r i r i (m +1) r i m r i CBFM (4) BFM m d c d i=1 M = (m+1) 2 d 1 BFM CBFM e r i c [(m + 1) r i m r i ] (4) i=1 d A = (a 1, a 2,L, a d ) CBFM k dk c d d 6 2d 2 c k 2 CBFM O((d + d 3 2 c )k ) d 3 calculateindex(object DC, Object hashv, Object ResultArray) : A=(a 1,a 2,,a d); ; begin } assign 0 to currentbitindex; // DC for each dimension a x in DC(from high to low) { // a x split a x to special bit array and normal bits array; calculate non-bulk offset in special bit array, and add it to currentbitindex; calculate non-bulk offset in normal bits array, and add offset*(hashv[a x]- 1) to currentbitindex; add currentbitindex to ResultArray; end calculateindex CBFM

6 n M 2 1 MB a =0 CBFM 2 (0x x3 1 2) CBFM BFM 14 CBFM 3 n=1 000 CBFM BFM 3 (M=16 MB) 7(a) CBFM SBF BFM CBFM CBF 2 ( M=1 MB) e d c n(bfm) n(cbfm) (0x5,0x3) (0x5,0x3) (0x5,0x3) (0x5,0x3) (0x5,0x3) a =1 6 (k=6) CMDBF MDBF a a =1 SBF BFM CBFM CBF CMDBF MDBF 3 SBF BFM CBFM MDBF a 1 3 ( n=1 000) e d c M-BFM/MB M-CBFM/MB (0x5,0x3) (0x5,0x3) (0x5,0x3) (0x5,0x3) (0x5,0x3) CBFM SBF MDBF CMDBF BFM CBF 4.1 SBF [12] MDBF [13] CMDBF [14] CBF [15] BFM [16] CBFM 6 CBFM 1 32 {0,2 32-1} SBF CMDBF MDBF BFM CBFM CBF 7 CBFM

7 3 CBFM 145 7(b) BFM CBF 100% CBFM ( 0.21%) BFM 3 CBFM 2 BFM 3 CBFM 9 9 WCBFM CBFM 2 CBFM 1.69 DNS 2 { IP IP CBFM } 8(a) SBF CBFM BFM { IP } 8(b) CBFM CBFM 8 CBFM CBFM BFM MDBF CBF 2~3 CBFM

8 CBFM CBFM CBFM CBFM d={1,2,3,4} n=100 CBFM e = CBFM CBFM CBFM CBFM 3 4 BFM 5 CBFM 1 BFM CBFM BFM 4 SBF [12] MDBF [13] CMDBF [14] CBF [15] BFM CBFM 2 BFM [16] CBFM 12 CBFM CBFM(Cutted Bloom Filter Matrix)

3 CBFM 147 CBFM, 2008, 29(1): -64. 56 [1] LYNCH C. Big data: how do your data grow[j]. Nature, 2008, 455(7209): 28-29. [2] CHANG F, DEAN J, GHEMAWAT S, et al.

9 3 CBFM 147 CBFM, 2008, 29(1): [1] LYNCH C. Big data: how do your data grow[j]. Nature, 2008, 455(7209): [2] CHANG F, DEAN J, GHEMAWAT S, et al. Bigtable: a distributed storage system for structured data[j]. ACM Transactions on Computer Systems (TOCS), 2008, 26(2): 4. [3] VORA M N. Hadoop-HBase for large-scale data[j]. IEEE Computer Science and Network Technology, 2011, (1): [4] DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters[j]. Communications of the ACM, 2008, 51(1): [5] THUSOO A, SARMA J S, JAIN N, et al. Hive: a warehousin solution over a map-reduce framework[j]. Proceedings of the VLDB Endowment, 2009, 2(2): [6] KORNACKER M, BEHM A, BITTORF V, et al. Impala: a modern, open-source SQL engine for Hadoop[C]. Conference on Innovative Data Systems Research (CIDR'15). c2015. [7] TARKOMA S, ROTHENBERG C E, LAGERSPETZ E. Theory and practice of bloom filters for distributed systems[j]. Communications Surveys & Tutorials, IEEE, 2012, 14(1): [8] RODEH O, TEPERMAN A. zfs-a scalable distributed file system using object disks[c]//mass Storage Systems and Technologies (MSST), c2003: [9] DEBNATH B, SENGUPTA S, LI J. FlashStore: high throughput persistent key-value store[j]. Proceedings of the VLDB Endowment, 2010, 3(1-2): [10] ZHOU X, ZHANG X, WANG Y, et al. Efficient distributed multi-dimensional index for big data management[m]. Springer Berlin Heidelberg, [11] NISHIMURA S, DAS S, AGRAWAL D, et al. MD-HBase: a scalable multi-dimensional data infrastructure for location aware services[j]. IEEE Mobile Data Management (MDM), 2011, 1: [12] BLOOM B H. Space/time trade-offs in hash coding with allowable errors[j]. Communications of the ACM, 1970, 13(7): [13] GUO D, WU J, CHEN H, et al. Theory and network applications of dynamic bloom filters[c]//infocom. c2006: [14],,,. XIE K, QIN Z, WEN J G, et al.combine multi-dimension Bloom filter for membership queries[j]. Journal on Communications ( 1) : [15] WANG Z, LUO T, XU G, et al. A new indexing technique for supporting by-attribute membership query of multidimensional data[m]. Springer Berlin Heidelberg, [16] WANG Z, LUO T, XU G, et al. The application of cartesian-join of bloom filters to supporting membership query of multid mensional data[c]//ieee Big Data. c2014: [17] BRODER A, MITZENMACHER M. Network applications of bloom filters: a survey[j]. Internet Mathematics, 2004, 1(4): [18] CHENG X, LI H, WANG Y, et al. BF-matrix: a secondary index for the cloud storage[m]. Springer International Publishing, [19] CRAINICEANU A, LEMIRE D. Bloofi: multidimensional Bloom filters[j]. Information Systems, 2015, 54:

Introduction to Randomized Algorithms III

Introduction to Randomized Algorithms III Joaquim Madeira Version 0.1 November 2017 U. Aveiro, November 2017 1 Overview Probabilistic counters Counting with probability 1 / 2 Counting with probability