VParC: A Compression Scheme for Numeric Data in Column-Oriented Databases

Transcription

1 The Intenatonal Aab Jounal of Infomaton Technology VPaC: A Compesson Scheme fo Numec Data n Column-Oented Databases Ke Yan, Hong Zhu, and Kevn Lü School of Compute Scence and Technology, Huazhong Unvesty of Scence and Technology, Chna College of Busness, Ats and Socal Scences, Bunel Unvesty, UK Abstact: Compesson s one of the most mpotant technques n data management, whch s usually used to mpove the quey effcency n database. Howeve, thee ae some estctons on exstng compesson algothms that have been appled to numec data n column-oented databases. Fst, a compesson algothm s sutable only fo columns wth cetan data dstbutons not fo all knds of data columns; second, a data column wth egula dstbuton s had to be compessed; thd, the data column compessed by usng heavyweght methods cannot be opeated befoe decompesson whch leads to neffcent quey. Based on the fact that t s moe possble fo a column to have sub-egulaty than have global-egulaty, we developed a compesson scheme called Vetcally Pattonng Compesson (VPaC). Ths method s sutable fo columns wth dffeent data dstbutons, even fo egula columns n some cases. The moe mpotant thng s that data compessed by VPaC can be opeated dectly wthout decompesson n advance. Detals of the compesson and quey evaluaton appoaches ae pesented n ths pape and the esults of ou expements demonstate the pomsng featues of VPaC. Keywods: Column-stoes, data management, compesson, quey pocessng, analytcal wokload. Receved August 8, 03; accepted Octobe 3, 04; publshed onlne Mach 8, 05. Intoducton Column-oented database systems (often efe to as column-stoes) that stoe the content by columns (such as MonetDB, Sybase IQ and C-Stoe) have been shown to pefom bette than tadtonal ow-oented database management systems (efe to as ow-stoes, such as Oacle, IBM DB and SQL Sevce) on analytcal wokloads n data waehouses, decson suppot and busness ntellgence applcatons []. Compesson s a technque known fo mpovng system pefomance fo DBMSs and othe applcatons [6, 9]. Column-stoes ae moe sutable to compesson schemes than ow-stoes [] due to ts stoage mode. Compesson technques futhe mpove the system pefomance of column-stoes and gadually become one of the ndspensable technques. Recently, thee ae seveal studes epoted on compesson algothms fo column-stoes [, ]. Numec type (ncludng: Intege, float, decmal and date typed data) s a commonly used data type and s dffeent fom othe data types (e.g., stng type). Howeve, thee ae some lmtatons n the exstng compesson methods fo numec data columns n column-stoes: A compesson algothm s sutable only fo columns wth cetan data dstbutons. The sze of a data column compessed wth unsutable schemes could even become lage than that of the ognal. A data column wth egula dstbuton s had to be compessed by usng any exstng compesson algothms. Wthout dect opeatons on compessed data, compesson becomes a tade off: Systems must pay the exta CPU cost to decompess n exchange fo the I/ O savngs of eadng less data fom stoage [3, 8]. In ode to, allevate the lmtatons descbed above, we examned the ssues elated the egulates n data dstbutons and found a new egulaty n data dstbuton called Sub-Regulaty. Consequently, a compesson scheme called Vetcally Pattoned Compesson (VPaC) s poposed, n whch sub-egulaty ae used to compess numec data n column-oented databases and the dect opeatons on compessed data columns ae suppoted. In summay, the man contbutons of ths study ae: We ntoduced an appoach to measue the egulates of data dstbuton though extactng thee chaactestc values fo captung the featues of data dstbutons. Base on the egulates of data dstbuton, we poposed and mplemented a compesson scheme fo column-stoes, called VPaC. We developed dect opeatons on compessed data. Expemental studes have been pefomed. Ou expements assessed the compesson ato, tme equed fo compesson and decompesson, as well as ssues elated to dect opeatons. The est of ths pape s oganzed as follows: Secton shows the elated wok. Defntons and analyss elated to the egulaty of data dstbuton ae pesented n secton 3. In secton 4, we gve the ntoducton about the new compesson algothm VPaC and the dect seachng on compessed data column. Secton 5 shows the expements esults. Fnally, secton 6 concludes the pape.

2 The Intenatonal Aab Jounal of Infomaton Technology. Related Woks Loss and lossless compesson technques ae wdely used to save stoage space and mpove access speed [4, 0, 3, 6]. Lossless compesson technques, ncludng lghtweght and heavyweght compesson technques, had been appled to column-stoes [, 3]. Howeve, thee ae some lmtatons o estctons on applyng these compesson technques dectly to numec data columns. One knd of lghtweght compesson technque cannot be sutable fo all knds of data columns. Run- Length Encodng (RLE) [7], whee epeats of the same element ae expessed as (value, un-length) pas, s an attactve appoach fo compessng soted data n a column-stoe. Nevetheless, t s only suted fo columns that have easonable-szed uns of the same value. Dctonay encodng and ts mpoved algothms ae pehaps the most pevalent compesson scheme n use n data management today [5, 5]. In these schemes, fequent pattens ae eplaced wth smalle codes. Bt packng s then used on top of dctonaes to futhe compess the data [7]. Howeve, ths appoach s only appecated fo columns that have a lmted amount of dffeent values. FOR compesson scheme [0] uses a base value, a fame of efeence and stoes all values as offsets fom the efeence value. Ths method s useful when the values ae not wde spead, snce ths wll esult n small offset values that can be epesented by a small numbe of bts. Addtonally, t s not sutable fo float data column. Btmap encodng s anothe wdely used compesson scheme [4]. It uses bt vectos to encode the values. One bt vecto s equed fo each value, whch makes t only useful when thee s a lmted amount of dffeent values. Thus, each method s sutable fo only one case. And, f a column wthout any of the chaactestcs descbed above, t cannot be compessed by any lghtweght technques. In ode to addess ths poblem, we poposed VPaC to ft fo all knds of data columns. Zv and Lempel [9] Encodng s the most wdely used Heavyweght technque fo lossless compesson. The basc dea of ths method s to pase the nput sequence nto non-ovelappng blocks of dffeent lengths. Then, a dctonay of blocks s constucted, whch leads to the subsequent appeaances of these blocks beng eplaced by a ponte to an eale occuence of the same block. Howeve, ecent eseaches [] llustated that the data columns compessed usng heavyweght compesson algothms ae dffcult to be opeated dectly. That s, the data columns compessed by usng heavyweght algothms should be decompessed befoe beng opeated. Ths wll degade the pefomance n the case that the exta CPU cost of decompesson s moe than the savng I/O cost of eadng less data fom stoage. Dffeent fom heavyweght technques, VPaC could suppot the dectly opeatons on compessed data, thus, we popose the coespondng method of dectly quey fo VPaC. In C-Stoe [, 3] each column s dvded nto blocks and each block s compessed usng the decson tee compesson scheme. Ths appoach s used to ad the database desgne to decde how to compess patcula columns. Howeve, ths method also cannot suppot the dect opeatons on compessed data, just lke heavyweght technque. And t costs much tme on the decson of compesson method. Ou method s moe effcent n compesson than C-Stoe whch can be llustated n ou expement. 3. Regulaty of Data Dstbuton Compesson algothms ae useful only when data dstbutons of a column contan cetan egulates. Thus, ssues elated to data dstbuton egulates ae mpotant to compesson. 3.. Vetcally Pattoned Dffeent types of data may be stoed n dffeent ways. Fo example, float data s stoed usng 4 bytes, double data s stoed usng 8bytes and ntege data s stoed usng 4 o 8 bytes; decmal data and date data n dffeent system may be stoed n dffeent ways. Howeve, data n dffeent types can be vetcally pattoned nto seveal sub-columns. Example : Fgue shows the vetcal pattonng of an ntege 3bt data column. Ths ntege 3bt column s pattoned vetcally nto 4 sub-columns. Obvously, RLE s qute sutable fo the fst subcolumn. Fgue. Vetcal patton of an ntege 3bt column. 3.. Chaactestc Values Dffeent compesson schemes wll be ft fo dffeent data columns. RLE s sutable fo columns wth few uns. The columns whch have less numbe of dffeent values pefe to dctonay and btmap encodng. If the values of a column ae not wde spead, FOR would be moe appopated. Thus, thee key chaactestc values of a column detemne the compesson effectveness of these algothms. Defnton. Dffeence Degee: The numbe of dffeent values n a column.

3 VPaC: A Compesson Scheme fo Numec Data n Column-Oented Databases 3 Defnton. Contnuty Degee: The numbe of uns n a column. A un s a sequence n whch the same data value occus consecutvely. Defnton 3. Max Dstance: The dffeence between the maxmal value and the mnmal value n a column. Example : Fgue shows a column wth 5 values. Dffeence Degee of ths column s 4, because thee ae 4 dffeent values n ths column. These 4 dffeent values ae 0, 53, 5 and 75. Contnuty Degee of ths column s 7, snce thee ae 7 uns n all (the fst un s 0, ts length s 3; the second un s 53, ts length s and so on). The max dstance s 75. Because the maxmal value n ths column s 75 and the mnmal value s 0, so the dffeence between them s 75. These chaactestc values of a column can be obtaned by tavesng all the values n a column Fgue. An nstance of a column Compesson Ratos A compesson algothm s sutable fo a column only when the compesson ato s smalle than. Clealy, the smalle the compesson ato s the moe sutable the compesson algothm wll be fo a data column. Assumed that, the numbe of values n a data column t s n(t) and each value s assgned b(t) bts. RLE [7] compesses uns n a column to a compact sngula epesentaton. A un s eplaced wth a tuple (value, un-length), whee each element of a tuple s gven a fxed numbe of bts. When a data column t s compessed usng RLE, the element value s gven b(t) bts. The element un-length s assumed to be g bts. dg c (t) denotes the Contnuty Degee of the column t. space (t) denotes the data sze of the column t compessed by RLE and s defned as follows: space (t) = dg (t) ( g + b (t)) c FOR [0] uses a base value, a fame of efeence and stoes all values as offsets fom the base value. ds m (t) denotes the max dstance of a column t. Thus, when the mnmum value n a column t s the base value, the offsets fom the base value need log ds m (t) bts. space fo (t) denotes the data sze of the column t compessed by FOR and s defned as follows: m space (t) = n(t) log ds (t) + b (t) fo In btmap encodng [4] a bt-stng s assocated wth each value wth a n the coespondng poston f that value appeaed at that poston and a 0 othewse. dg d (t) denotes the dffeence degee of a column t.space bt (t) denotes the data sze of the column t beng compessed usng btmap encodng and s defned as follows: space (t) = n (t) dg (t) + b (t) dg (t) bt d d () () (3) Dctonay compesson schemes [7, 5] ae pehaps the most pevalent compesson schemes found n data management systems today. These schemes eplace each value n a column t wth a smalle dctonay code. Each code needs log dg d (t) bts. space dct (t) denotes the data sze of the column t compessed by dctonay (wth the addton of bt packng) and s defned as follows: = + space (t) b (t) dg (t) n (t) log dg (t) dct d d space no (t) denotes the data sze of a column t wthout compesson. It s equal to n(t)*b(t). (t), fo (t), bt (t) and dct (t) denote the compesson ato of RLE, FOR, Btmap and Dctonay encodng fo a column t and equal to space (t), space fo (t), space bt (t) and space dct (t) dvded by space no (t) espectvely. Thus, based on Equatons,, 3 and 4, compesson atos of each algothm ae: fo dct space ( t ) dg ( t ) g ( t ) = = c + space ( t ) n ( t ) b ( t ) no space ( t ) log ds ( t ) fo m ( t ) = = + space ( t ) b ( t ) n ( t ) bt no (5) (6) space ( t ) dg ( t ) dg ( t ) bt d d ( t ) = = + (7) space ( t ) b ( t ) n ( t ) no space ( t ) log dg ( t ) d dg ( t ) dct d ( t ) = = + space ( t ) b ( t ) n ( t ) no (8) 3.4. Global-Regulaty and Sub-Regulaty Based on the compesson atos of these lghtweght compesson algothms, we defne the data dstbuton egulaty of a data column s poposed RLE-Regulaty and Sub-RLE-Regulaty Defnton 4. RLE-Regulaty: Gven a data column t, f (t)<, we call that the column t has the (t) RLE-Regulaty. Defnton 5. Sub-RLE-Regulaty: A column t s vetcally pattoned nto s sub-columns, whch ae denoted as t(),, t(s). If exstng,,..., s}, the th sub-column t() has RLE-Regulaty, t s consdeed to have Sub-RLE-Regulaty. Lemma. Gven a data column t wth n(t) elements, each element value s stoed n b(t) bts and each un length s stoed n g bts. The pobablty of the column t havng RLE-Regulaty s: p β C = = whee b ( t ) b ( t ) n ( t ) β ( ) b ( t ) n ( t ) n ( t ) b ( t ) = b ( t ) + g (4) (9)

4 4 The Intenatonal Aab Jounal of Infomaton Technology Poof: Accodng to Equaton 5, RLE-Regulaty mples that: That s dg ( t ) g c ( t ) = + < n ( t ) b ( t ) n ( t ) b ( t ) dg ( t ) < c b ( t ) + g dg c (t) s an ntege snce t s the contnuty degee of column t. We can obtan that: n ( t ) b ( t ) dg ( t ) β, whee β = c b ( t ) + g Thee ae b(t) possble values fo each element n column t snce, each element s stoed n b(t) bts. Thus, the numbe of possble outcomes of column t s b(t)n(t). dg c (t)= means that the numbe of uns n column t s. Thus, the numbe of possble outcomes that dg c (t) equals to s: Whee m C n ( ) C b ( t ) b ( t ) n ( t ) denotes the numbe of combnatons. Futhe, the pobablty of dg c (t) equal to s: C p( dgc ( t ) = ) = b ( t ) b ( t ) n ( t ) ( ) b ( t ) n ( t ) The pobablty of a column t havng RLE-egulaty s the pobablty of dg c (t) smalle and equal to β. That s: ( ) β ( ) β ( ( ) ) p = p dg t = p dg t = c c = As a esult, we can obtan the Equaton BM-Regulaty and Sub-BM-Regulaty Defnton 6. BM-Regulaty: Gven a data column t, f bt (t)<, the column t has the bt (t) BM- Regulaty. Defnton 7. Sub-BM-Regulaty: A column t s vetcally pattoned nto s sub-columns. If exstng, the th sub-column t() has BM-Regulaty, the column t s consdeed to have Sub-BM-Regulaty. Lemma. Gven a data column t wth n(t) elements, each element value s stoed n b(t) bts. The pobablty of the column t havng BM-egulaty s: p = β P S bt b ( t ) n ( t ) bt b ( t ) n ( t ) = b (t) n(t) β Whee bt = b (t) + n(t) m S n : Denotes a stlng numbe of the second knd, and m P Denotes the numbe of m-pemutatons of n. n : (0) Poof: Accodng to Equaton 7, the stuaton n whch column t has BM-Regulaty mples that: That s bt b ( t ) n ( t ) dg ( t ) < d b ( t ) + n ( t ) dg ( t ) dg ( t ) d d ( t ) = + < n ( t ) b ( t ) Whee dg d (t) must be an ntege snce, t s the Dffeence Degee of column t. We can obtan that: b ( t ) n ( t ) dg ( t ) < β, whee β = d bt bt b ( t ) + n ( t ) dg d (t)= means that the numbe of dffeent elements n column t s. Thus, the numbe of possble outcomes that dg c (t) equals to s: P S b ( t ) n ( t ) Because the numbe of possble outcomes of column t s b(t)n(t), the pobablty of dg d (t) equal to s: P p ( dg ( t ) = ) = b ( t ) n ( t ) d b ( t ) n ( t ) S Pobablty of column t havng BM-Regulaty s the pobablty of dg d (t) smalle and equal to β bt. That s: β p = p dg t = p dg t = ( ( ) β ) bt ( ( ) ) bt d bt d = As a esult, we can obtan the Equaton DICT-Regulaty and Sub-DICT-Regulaty Defnton 8. DICT-Regulaty: Gven a data column t, f dct (t)<, the column t has the dct (t) DICT-Regulaty. Defnton 9. Sub-DICT-Regulaty: A column t s vetcally pattoned nto s sub-columns. If exstng, the th sub-column t() has DICT-Regulaty, the column t s consdeed to have Sub-DICT- Regulaty. Lemma 3. Gven a data column t wth n(t) elements, and each element value s stoed n b(t) bts. The pobablty of the column t havng DICT-Regulaty s: Whee β β dct = of the equaton: p β P S dct b ( t ) n ( t ) dct b ( t ) n ( t ) = = (), and β : Is the appoxmate soluton n(t) log dg d (t)+b(t) dg d (t)=n(t) b(t) Poof: Accodng to Equaton 8, the stuaton n whch column t has DICT-Regulaty mples that: dct ( t ) = log dg ( t ) d dg ( t ) d + < b ( t ) n ( t ) If dct (t)=, we can get an equaton: n(t) log dg d (t)+b(t) dg d (t)=n(t) b(t) It s an non-lneaequaton about dg d (t) and we can obtan the appoxmate soluton β usng the method of

5 VPaC: A Compesson Scheme fo Numec Data n Column-Oented Databases 5 Newton. Thee ae thee easons fo dg d (t) satsfyng dg d (t) β dct, whee β dct= β : dg d (t) s an ntege; dct (t)<; dct (t) s monotoncally nceasng at dg d (t). Because the pobablty of the column t havng DICT- Regulaty s the pobablty of dg d (t) smalle and equal to β dct, we can get that: ( ) β ( ) β dct ( ( ) ) p = p dg t = p dg t = dct d dct d = As a esult, we can obtan the Equaton Fo-Regulaty and Sub-fo-Regulaty Defnton 0. Fo-Regulaty: Gven a data column t, f fo (t)<, we call that the column t has the fo (t) fo-egulaty. Defnton. Sub-Fo-Regulaty: A column t s vetcally pattoned nto s sub-columns. If exstng, the th sub-column t() has Fo-Regulaty, the column t s consdeed to have sub-fo-egulaty Global-Regulaty and Sub-Regulaty Fo a data column t, t could have seveal knds of egulates. Fo example, t has RLE-Regulaty and BM-Regulaty smultaneously. Defnton. Global-Regulaty: If a data column t has one o moe knds of egulates, that s, λ(t)=mn{ (t), bt (t), dct (t), fo (t)}<, the column t has λ(t) Global-Regulaty. If a data column has Global-Regulaty, t can be compessed. It s clealy that the smalle the λ(t) of a column s the moe egula the column s and the moe effectve that the column can be compessed. If λ(t), we say that the column t s egula. Defnton 3. Sub-Regulaty: If a data column t has Sub-RLE-Regulaty, Sub-BM-Regulaty, Sub- DICT-egulaty o sub-fo-egulaty, t s called to have sub-egulaty. A column t wth Sub-Regulaty can be compessed afte vetcally pattoned nto sub-columns. It s possble that column t has seveal knds of subegulates. Fo example, t could have sub-rleegulaty and sub-bm-egulaty. The key ssue to patton a column nto subcolumns s how many sub-columns a column should be vetcally pattoned nto. Too many sub-columns a column s vetcally pattoned nto would lead to moe compesson and decompesson costs. On the othe hand, f a column s vetcally pattoned nto too few sub-columns, thee s no much help to compesson. Byte s the basc unt of data pocessng. Thus, a column could be vetcally pattoned by byte. Tadtonal compesson methods take advantage of Global-Regulaty to compess data columns. Obvously, f a column t has Fo-Regulaty, t must have Sub-Fo-Regulaty. Addtonally, accodng to the lemmas, and 3, a data column t s moe lkely to have Sub-RLE-Regulaty, Sub-BM-Regulaty and Sub-DICT-Regulaty than to have RLE-Regulaty, BM-Regulaty and DICT-Regulaty. That s, subcolumns that ae vetcally pattoned fom a column t ae moe egula than the column t tself. Thus, we utlze Sub-Regulaty to compess. 4. Compesson and Dect Seachng Based on the analyss and poof above, we popose a compesson algothm called VPaC. A column that s compessed usng VPaC s pattoned nto seveal sub-columns. The key ssue of seachng dectly on the compessed column s how to obtan the esults by means of the dect seachng on the compessed sub-columns. 4.. Compesson Pocedues of VPaC The pocedue of compessng a data column t usng VPaC can be dvded nto 3 steps:. Vetcally patton the data column t nto seveal sub-columns by byte, and each sub-column s a byte typed column.. Obtan thee chaactestc values of each subcolumn. Thee chaactestc values of each subcolumn can be obtaned by tavesal of each subcolumn. 3. Compess each sub-column usng sutable lghtweght compesson algothms. Sub-columns wth dffeent egulates need to be compessed usng dffeent compesson algothms. A key ssue s how to choose a sutable compesson algothm fo a sub-column t(). Based on chaactestc values of a sub-column t(), (t()), fo (t()), bt (t()) and dct (t()) can be obtaned and the compessed sze of the sub-column t() can be obtan. The pupose of compesson s to mpove the pefomance of seachng [8, ]. Thus, we choose the algothm that s the most benefcal fo seachng. We assume that α s a constant facto egadng dsk I/O capablty. γ, γ fo, γ dc and γ bt denote the cost of etevng on data compessed by RLE, dctonay, FOR and Btmap encodng espectvely. Thus, an appoxmaton of the tme fo seachng on the sub-column t() compessed by fou knds of compesson methods can be obtaned: S ( X ) = ato n ( α γ ) X Whee X s RLE, FOR, dctonay o Btmap encodng. Thus, we choose X to compess t() whee X leads to the mnmum S(X). 4.. Equvalent Seach and Rang Seach Thee ae two knds of seachng on numec data n column-stoes, whch s dffeent fom stng typed data: Equvalent seach and ange seach. An equvalent seach means to fnd values n a column that ae equal to a gven value a. A ange seach means to fnd values n a column that ae n a gven nteval. Thee X ()

6 6 The Intenatonal Aab Jounal of Infomaton Technology ae seveal pattens of ntevals. Because the pocess of ange seach s smla fo dffeent knds of ntevals, only open nteval (a, b) s dscussed n ths pape. A column t s pattoned vetcally nto u(t) subcolumns by byte. Accodngly, a value x n the column s pattoned nto u(t) bytes (called sub-values) and t s epesented as <x, x,, x u(t) >. A gven value a s pattoned nto u(t) bytes as well, epesented as <a, a,, a u(t) >. x s equal to a f and only f each subvalue of x s equal to each coespondng sub-value of a, that s x =a, x =a,, and x u(t) =a u(t). In a ang seach, the lowe bound and uppe bound of an nteval (a, b) can be pattoned nto u(t) subvalues, whch ae denoted as <a, a,, a u(t) > and <b, b,, b u(t) > sepaately. Result of ange seach can be obtaned by means of compang x wth a and b (=,,, u(t)). Howeve, columns wth dffeent data types n dffeent systems ae stoed n dffeent ways. Thus, the methods of ange seach ae vaed dependng on data types and stoage fomats. In ths secton, seachng pocess s shown based on IEEE standad fomat Stoage Fomat of Data Columns An ntege (3bts) 3 stoed n IEEE standad fomat and n lttle-endan compute byte odeng s shown n Fgue 3. It s stoed n the fom of complement Fgue 3. An nstance of stoage fomat of ntege data (3bts). A float (3bts) -0.5 (to pesent as bnay numbe s -.0* - ) n IEEE standad fomat and n lttleendan compute byte odeng s shown n Fgue 4. Fom left to ght, the fst bt s sgn bt; the next eght bts epesent the shft code of exponent (means addng 7 to exponent) and the left bts epesent mantssa Fgue 4. An nstance of stoage fomat of float data (3 bts) Range Seach In the mplementaton of ange seach, the elatonshps among x, a and b show whethe the value x s n the nteval (a, b). Thee ae thee cases whch values of a and b can be n: The fst case s a 0 and b> 0, the second case s a< 0 and b 0, the thd case s a< 0 and b> 0. Howeve, thee s only the fst case fo date data. Case : a 0 and b> 0.. f a b, then: a. f x <a o x > b, x s not n (a, b). b. f x >a and x < b, x s n (a, b). c. f x =a, then t needs to compae x and a. f x >a, x s n the nteval (a, b). f x <a, x s not n the nteval (a, b). f x =a, then t needs to compae x 3 and a 3 n same way as c. d. f x =b, then need to compae x and b. f x >b, x s not n the nteval (a, b). f x <b, x s n the nteval (a, b). f x =b, then t needs to compae x 3 and b 3 n same way as d.. f a=b, then: f x a, x s not n the nteval (a, b). f x=a, then t needs to compae x, a and b n accodance wth the steps and. In summay, thee ae thee knds of compason esults: a =b and x =a, a b and x =a, a b and x =b. If the compason esult of a sub-value x s one of these thee knds, t needs to compae the next sub-value. Case : a<0 and b 0.. f a b, then: a. f x <a o x >b, x s not n (a, b). b. f x >a and x <b, x s n the nteval (a, b). c. f x =a, then t needs to compae x and a. f x >a, fo ntege numbe, x s n (a, b); fo othe data types, x s not n (a, b) f x <a, fo ntege numbe, x s not n (a, b); fo othe data types, x s n (a, b) f x =a, then t needs to compae x 3 and a 3 n same way as c. d. f x =b, then need to compae x and b. f x >b, fo ntege numbe, x s not n (a, b); fo othe types, x s n (a, b). f x <b, fo ntege numbe, x s n the nteval (a, b); fo othe types, x s not n the nteval. f x =b, then t needs to compae x 3 and b 3 n same way as d.. f a =b, then: a. f x a, x s not n the nteval (a, b). b. f x =a, then t needs to compae x, a and b n accodance wth the steps and. Case 3: a< 0 and b> 0.. f x= 0, x s n the nteval (a, b).. f x> 0. a. f x <b, x s n the nteval (a, b). b. f x >b, x s not n the nteval (a, b). c. f x =b, then t needs to compae x and b n the same way as. 3. f x< 0. a. f x <a, fo ntege numbe x s not n (a, b); fo othe data types, x s n the nteval. b. f x >a, fo ntege numbe x s n (a, b); fo othe data types, x s not n the nteval. c. f x =a, then t needs to compae x and b n same way as a, b and c fo x< Expements To assess the featues of VPaC, expemental studes have been pefomed. The expements ae conducted on.5ghz Pentum Daul-Coe, unnng Wndows XP,

7 VPaC: A Compesson Scheme fo Numec Data n Column-Oented Databases 7 wth 3GB of man memoy. Fo the expements, a data geneato called dbgen s used to geneate an nstance of the TPC-H data set at scale, whch yelds a total database sze of appoxmately GB wth 6 tables, named Supple, Custome, Pat, Patsupp, Odes and Lnetem espectvely. We test two types of numec columns: Intege and float columns. Fst, we compae the compesson data sze, the compesson tme and decompesson tme of VPaC wth othefou compesson schemes: RLE, dctonay encodng (DICT fo shot), FOR, decson tee (DT fo shot). BITMAP s not pesented because f a column t has BM-Regulaty, t must has DICT- Regulaty (the eason s that bt (t) s always lage than dct (t), accodng to the Equatons 7 and 8 pesented n secton 3.3). Second, we test the pefomance of seach on these two types of columns, whch ae compessed by usng dffeent compesson schemes. In VPaC, each column s vetcally pattoned nto fou sub-columns, snce ntege and float data ae stoed n fou bytes. Then, chaactestc values of each sub-column ae evaluated, whch ae ganed by scannng sub-columns. The scannng pocesses ae ncluded n the compesson. The best algothm s chosen to compess the sub-column accodng to the chaactestc values. Appaently, sub-columns fom the same column may be compessed by usng dffeent compesson schemes. 5.. Compesson Pefomance on Intege Data Thee ae 3 ntege columns n the nstance of the TPC-H dataset. We use dffeent compesson schemes to compess these columns and compae the compesson ato, tme fo compesson and decompesson Compesson Rato on Intege Columns We lst the sze of data compessed by VPaC and othe fou encodng schemes on each ntege column n Table. NC denotes the ognal sze of a data column wthout beng compessed. Table. Compessed data szes of ntege columns (n KByte). NC RLE FOR DICT DT VPaC The sze of a data column becomes lage than the ognal f the column s compessed by unsutable compesson algothm. Fo example, the sze of the fst data column wthout compesson s 40KB, but t nceases to 75KB afte beng compessed by RLE. Thus, tadtonal compesson scheme, such as RLE, FOR and DICT cannot be sutable fo all knds of columns. In DT, the best algothm s chosen fo the column, thus, t acheves bette compesson esults than RLE, FOR and DICT. Howeve, fo most columns VPaC obtans the smla compesson ato as DT; and fo othe thee columns, the columns, 7 and 3, VPaC obtans bette compesson esults than DT. Because of the stoage stuctue of ntege, the hgh bytes of the values n a column ae the same. Some have Sub-RLE- Regulaty although the column tself has no RLE- Regulaty. Thus, these sub-columns can be compessed effcently. Fo example, the column has 0.45 FOR-Regulaty, whle two sub-columns n t have RLE-Regulaty. Thus, fo column, VPaC s moe effcent than DT Tme fo Compesson on Intege Columns Table shows the compesson tme of each compesson schemes. The tme fo compesson s composed of the tme to obtan the chaactestc values and the tme to compess data columns. In DT, most of the tme s spent on the obtanng of the chaactestc values. In VPaC, although thee ae fou sub-columns need to be pocessed, the equed tme s much less than DT due to the byte natue of sub-columns. Thus, the VPaC cost less tme to compess than DT. Table. Tme fo compesson on ntege columns (n ms). RLE FOR DICT DT VPaC Tme fo Decompesson on Intege Columns The tme needed fo decompesson s shown n Table 3. RLE needs less tme fo decompesson than FOR and DICT due to the bt shft n the FOR and DICT algothms.

8 8 The Intenatonal Aab Jounal of Infomaton Technology Table 3. Tme fo decompesson on ntege columns (n ms). RLE FOR DICT DT VPaC In VPaC, decompesson s faste than DT except columns 9 and 3, because n these two columns, thee s moe than one sub-column compessed usng FOR o DICT, whch ae moe slowe than RLE. Fo example, the column 3 has RLE-Regulaty whle thee ae two sub-columns have Sub-RLE-Regulaty and othe two sub-columns have Sub-DICT-Regulaty. Theefoe, n the columns wth Sub-DICT-Regulaty o Sub-DICT-Regulaty, decompesson tme fo VPaC s moe than DT. 5.. Compesson Pefomance on Float Data Thee ae 8 float columns n the nstance of the TPC-H data n all Smla to the expements fo ntege columns, we compaed the compesson ato, compesson tme and decompesson tme of each algothm on each float columns. Notce that, FOR s not sutable fo float columns, thus, t s not tested n ths expement Compesson Rato on Float Columns The szes of the data columns compessed usng each algothm on each float columns ae llustated n Fgue 5. It can be seen that VPaC leads to moe sze fo the columns 4, 6, 7 and 8 than DT. Because of the stoage stuctue of float data, the column wthout RLE-Regulaty, t wll not have Sub-RLE-Regulaty. Fo example, the column 6 has 0.5 DICT-Regulaty, whle each sub-column n column 6 has DICT- Regulaty. Thus, VPaC s less effcent than DT fo the columns wth bette global-egulaty. Szes of Compessed Data (n:ms) Column Identfe Fgue 5. Data szes of compessed float columns. Howeve, VPaC shows bette compesson ato fo columns,, 3 and 5, because they ae not much egula columns. Especally fo columns, and 5, they ae egula and cannot be compessed by any lghtweght schemes. In geneal, although VPaC has no advantage fo float columns wth good Global- Regulaty, t shows bette compesson ato fo egula columns Tme fo Compesson on Float Columns The tme equed fo each compesson scheme s shown n Fgue 6. Just lke ntege columns, VPaC eques less tme to compess than DT because obtanng chaactestc values s tme-consumng fo DT. Compesson Tme (n:ms) Column Identfe Fgue 6. Tme fo compesson on float columns Tme fo Decompesson on Float Columns Fgue 7 llustates the decompesson tme fo each algothms. Because columns, and 5ae egula columns, they cannot be compessed by any lghtweght algothms except VPaC. Thus, VPaC cost moe tme to decompess fo these columns than DT. Fo columns 6 and 7, each sub-column n these two columns has DICT-Regulaty. DICT and FOR s moe tme-consumng than RLE, whch leads to moe tme fo VPaC to decompess than DT. Howeve, fo the columns that ae not much egula, such as the columns 3, 4 and 8, VPaC shows bette decompesson pefomance than DT. decompesson Tme (n:ms) Column Identfe Fgue 7. Tme fo decompesson on float columns. In concluson, we compae the compesson ato, compesson tme and decompesson tme of fou compesson schemes fo numec data columns. Obvously, RLE, DICT and FOR ae not sutable fo all columns. Thus, ou expements focus on the compason between DT and VPaC. The esults of ou expements llustated that VPaC has advantages n the compesson speed and VPaC s not always supeo to DT n the aspects of decompesson speed and compesson ato. Howeve, VPaC s moe unvesal than DT. That s, fo columns wth Global-Regulaty that can be compessed by DT,

9 VPaC: A Compesson Scheme fo Numec Data n Column-Oented Databases 9 they can be compessed by VPaC; fo egula columns that cannot be compessed by DT, they also can be compessed by VPaC Tme Requed fo Equvalent Seachng In equvalent seach, the seach esults needn t to be decompessed. Thus, the tme fo equvalent seach ncludes the tme of loadng compessed data fom dsk to memoy and the tme of seachng on compessed data. In ou expement, andom values ae used as seachng value and the tme of seachng s an aveage tme of 00 tmes. Notce than, RLE, FOR and DICT ae not sutable fo all columns, we only evaluate the tme fo equvalent seachng on columns compessed by usng DT and VPaC Equvalent Seach on Intege Columns Fgue 8 shows that seachng on data compessed by VPaC needs less tme than DT, thee ae two easons: VPaC s moe effcent fo ntege column compesson, whch s shown n Table. Thus, t costs less tme fo loadng data compessed by VPaC than by DT. In VPaC, fo the values that ae not equal to the seachng value, thee s no need to compae all of the sub-values. Thus, the tme of seachng on compessed data s less than n DT. seach Tme (n:ms) Column Identfe Fgue 8. Tme fo equvalent seach on ntege column. ange and the tme of seachng s an aveage tme of 00 tmes Range Seach on Intege Columns Fgue 0 shows the tme fo ange seach on ntege columns. Because of fewe dsks I/Os and less decompesson tme, ange seach on VPaC s quck than DT fo most ntege columns except column 3. Because VPaC need moe decompesson tme fo ths column than DT, whch s moe than the tme savng fom compesson. seach Tme (n:ms) Column Identfe Fgue 0. Tme fo ange seach on ntege column Range Seach on Float Columns The tme fo ange seach on float columns s showed n Fgue. Fo the columns 6 and 7, although the sze of data compessed by VPaC s lage than DT, VPaC eques moe o less the same tme as DT fo ange seach on ths two columns due to less tme of decompesson. Fo the columns, and 5, VPaC needs a lttle moe tme than DT, because the tme saved by less I/O s moe than the tme of seachng and seach caused by compesson. Howeve, fo columns 3, 4 and 8, VPaC equed less tme than DT, because of the less sze of compessed data and less tme fo decompesson Equvalent Seach on Float Columns In Fgue 9, the tme of equvalent seach on float columns fo VPaC and DT ae compaed. Just lke the equvalent seachng on ntege columns, seachng on data compessed by VPaC cost less tme than compessed by DT. seach Tme (n:ms) Column Identfe seach Tme (n:ms) Column Identfe Fgue 9. Tme fo equvalent seach on float columns Tme Requed fo Range Seachng Fo ange seach, the seach tme s composed of the tme of loadng, the tme of seachng and the tme of decompesson. Random ange s used as seachng Fgue. The tme of ange seach fo float columns. We just lst the expement esults of ntege and float data columns. Fom the esults, we conclude that fo much egula data column, VPaC s not bette than DT; Howeve, VPaC s bette than DT fo not much egula columns and egula columns. Date and decmal data ae not lsted because they show the smla esults to the ntege and float data. 6. Conclusons and Futue Wok Compesson s one of the most mpotant technques fo column-oented data management. In ths pape,

10 0 The Intenatonal Aab Jounal of Infomaton Technology we pove that a column s moe possble to have sub-egulaty than to have global-egulaty. Thus, a new compesson scheme, called VPaC, s poposed and elated ssues ae examned. Futhemoe, we mplemented t and evaluaton studes have been pefomed. In VPaC, a data column s vetcally pattoned nto seveal sub-columns; each sub-column s compessed by sutable compesson algothms accodng to ts egulaty. Only lghtweght compesson algothms ae used to compess sub-columns. In ths way, equvalent seach and ange seach can be mplemented dectly on compessed data columns. A seachng method to opeate on compessed sub-columns dectly s desgned to dectly opeate the data compessed by VPaC wthout decompesson n advance. The expement esults llustated that the VPaC and the coespondng seachng method ae vey pomsng. As a pat of ou futue wok, we plan to ntegate moe compesson scheme n VPaC fo sub-columns and leads to mpoved compesson ato. Also, paallel pocessng and ndexes on sub-columns ae consdeed to mpove seachng pefomance. Refeences [] Abad J., Madden R., and Feea M., Integatng Compesson and Executon n Column Oented Database Systems, n Poceedngs of SIG- MOD Intenatonal Confeence on Management of Data, Chcago, USA, pp , 006. [] Abad J., Madden R., and Hachem N., Column- Stoes vs. Row-Stoes: How Dffeent ae They Really, n Poceedngs SIGMOD Intenatonal Confeence on Management of Data, Vancouve, Canada, pp , 008. [3] Abad J., Quey Executon n Column-Oented Database Systems, MIT PhD thess, 008. [4] Akman I., Baynd H., Ozleme S., Akn Z., and Msa S., Lossless Text Compesson Technque usng Syllable Based Mophology, the Intenatonal Aab Jounal of Infomaton Technology, vol. 8, no., pp , 0. [5] Bnng C., Hldenband S., and Faebe F., Dctonay-based Ode-Pesevng Stng Compesson fo Man Memoy Column-Stoes, n Poceedngs of the SIGMOD Intenatonal Confeence on Management of Data, pp , 009. [6] Chen Y., Johannes G., and Flp K., Quey Optmzaton n Compessed Database Systems, n Poceedngs of the SIGMOD Intenatonal Confeence on Management of Data, Calfona, pp.7-8, 00. [7] Gaefe G., Effcent Columna Stoage n B- tees, ACM SIGMOD Recod, vol. 36, no., pp. 3-6, 007. [8] Hazopoulos S., Lang L., Abad J., and Madden S., Pefomance Tadeoffs n Read-Optmzed Databases, n Poceedngs of the 3 nd Intenatonal Confeence on Vey Lage Data Bases, Seoul, pp , 006. [9] Holloway L. and DeWtt J., Read-Optmzed Databases: In Depth, VLDB Endowment, vol., no., pp , 008. [0] Hsao-Png T., De-Nan Y., and Mng-Syan C., Explong Applcaton-Level Semantcs fo Data Compesson, IEEE Tansacton Knowledge and Data Engneeng, vol. 3, no., pp , 0. [] Lason A., Clncu C., Hanson N., Oks A., Pce L., Rangaajan S., Suna A., and Zhou Q., SQL Seve Column Stoe Indexes, n Poceedngs of the 3 st SIGMOD, pp , 0. [] Lemke C., Sattle U., Faebe F., and Zee A., Speedng up quees n Column Stoes: A Case fo Compesson, Data Waehousng and Knowledge Dscovey of Lectue Notes n Compute Scence, pp. 7-9, 00. [3] Nngde X., Guqang D., and Tong Z., Usng lossless Data Compesson n Data Stoage Systems: Not fo Savng Space, IEEE Tansacton Computes, vol. 60, no. 3, pp , 0. [4] O'Nel P. and Quass D., Impoved Quey Pefomance wth Vaant Indexes, n Poceedngs of SIGMOD Intenatonal Confeence on Management of Data, Tucson, USA, pp , 997. [5] Roth A. and VanHon J., Database Compesson, ACM SIGMOD Recod, vol., no. 3, pp. 3-39, 993. [6] Shah D. and Vthlan C., VLSI-Oented Lossymage Compesson Appoach usng DA- Based D-Dscete Wavelet, the Intenatonal Aab Jounal of Infomaton Technology, vol., no., pp , 04 [7] Tanaka H. and Gaca L., Effcent Run-Length Encodngs, IEEE Tansacton Infomaton Theoy, vol. 6, no. 8, pp , 98. [8] Tsoganns D., Hazopoulos S., Shah A., Wene L., and Gaefe G., Quey Pocessng Technques fo Sold State Dves, n Poceedngs of SIGMOD Intenatonal Confeence on Management of Data, Rhode Island, pp. 59-7, 009. [9] Zv J. and Lempel A., Compesson of Indvdual Sequences va Vaable-Rate Codng, IEEE Tansacton Infomaton Theoy, vol. 4, no. 5, pp , 978. [0] Zukowsk M., Héman S., Nes N. and Boncz A., Supe-Scala RAM-CPU Cache Compesson, n Poceedngs of the nd Intenatonal Confeence on Data Engneeng, pp. 59, 006.

11 VPaC: A Compesson Scheme fo Numec Data n Column-Oented Databases Ke Yan eceved he BE and MS degees fom HuNan Unvesty n Chna n 003 and 006 espectvely. Cuently, she s a PhD canddate n School of Compute Scence and Technology, Huazhong Unvesty of Scence and Technology n Chna. He eseach topc ncludes cloud data management and column-oented database. Hong Zhu eceved he BE, MS and PhD degees fom Huazhong Unvesty of Scence and Technology n 987, 990 and 00 espectvely. Cuently, she s a pofesso and a PhD supevso n School of Compute Scence and Technology, Huazhong Unvesty of Scence and Technology n Chna. He man eseach aeas ae database and secuty. Kevn Lu s a lectue n Bunel Unvesty n Unted Kngdom. He supevsed sx PhD student pojects to completon. He obtaned gants fom EU, KTP pogam, ORS and othe oganzatons. Hs man eseach aeas ae nfomaton management and busness analytcs.