A Highly Parallel Algorithm for Frequent Itemset Mining

Transcription

1 A Highly Paralll Algorithm or Frqunt Itmst Mining Aljandro Msa 1,2, Claudia Frgrino-Urib 2,Rné Cumplido 2, and José Hrnándz-Palancar 1 1 Advancd Tchnologis Application Cntr, CENATAV. La Habana, Cuba {amsa,jpalancar}@cnatav.co.cu 2 National Institut or Astrophysics, Optics and Elctronics, INAOE. Publa, México {amsa,crgrino,rcumplido}@inaop.mx Abstract. Mining rqunt itmsts in larg databass is a widly usd tchniqu in Data Mining. Svral squntial and paralll algorithms hav bn dvlopd, although, whn daling with high data volums, th xcution o thos algorithms taks mor tim and rsourcs than xpctd. Bcaus o this, inding altrnativs to spd up th xcution tim o thos algorithms is an activ topic o rsarch. Prvious attmpts o acclration using custom architcturs hav bn limitd bcaus o th natur o th algorithms that hav bn concivd squntially and do not xploit th intrinsic paralllism that th hardwar provids. Th innovation in this papr is a highly paralll algorithm that utilizs a vrtical bit vctor (VBV) data layout and its asibility or making support counting. Our rsults show that or dns databass a custom architctur or this algorithm can prorm astr than th astst architctur rportd in prvious works by on ordr o magnitud. 1 Introduction Nowadays, many data mining tchniqus hav mrgd to xtract usul knowldg rom larg amounts o data. Finding corrlations btwn itms, spciically rqunt itmsts, is a widly usd tchnic in data mining. Th algorithms that hav bn dvlopd in this ara rquir powrul computational rsourcs and a lot o tim to solv th combinatorial xplosion o itmsts that can b ound in a datast. Th high computational rsourcs rquird to procss larg databass can rndr th implmntation o this kind o algorithms impractical. This is mainly du to th prsnc o thousands o dirnt itms or th us o a vry low thrshold o support (minsup 1 ). Attmpts to acclrat th xcution o algorithms or mining rqunt itmsts hav bn rportd. Th most common practic in this ara is th us o paralll algorithms such as CD [1], DD [1], CDD [1], IDD [5], HD [5], Eclat [12] 1 Is th minimum numbr o tims in a databas an itmst must occur to b considrd as rqunt. J.A. Carrasco-Ochoa t al. (Eds.): MCPR 2010, LNCS 6256, pp , c Springr-Vrlag Brlin Hidlbrg 2010

2 292 A. Msa t al. and ParCBMin [7]. Howvr, all ths orts hav not rportd good xcution tims givn a rasonabl amount o rsourcs or som practical applications. Rcntly, hardwar architcturs hav bn usd in ordr to spd up th xcution tim o thos algorithms. Ths architcturs wr prsntd in [2, 3, 11, 10, 9], and improvd th xcution tim o sotwar implmntations by som ordrs o magnitud. Prviously proposd paralll algorithms usd a coars granularity paralllism. Gnrally a partition o th databas is mad in ordr to procss in a paralll ashion ach block o data. In this papr an algorithm that xploits th inhrnt task paralllism o hardwar implmntation and th asibility to prorm bitwis oprations is proposd. A hardwar architctur or mining rqunt itmsts is dvlopd to tst th icincy o th proposd algorithm. Th rmaindr o this papr is organizd as ollows. Sction 2 dscribs rlatd work. Sction 3 discusss th proposd algorithm. Sction 4 dscribs th systolic tr architctur that supports th algorithm. Th rsults ar discussd in Sction 5 and Sction 6 prsnts th conclusions. 2 Rlatd Work Algorithms that us data paralllism dal with load balancing, costs o communication and synchronization. Ths ar problms not commonly prsnt in algorithms that xploit task paralllism. This is bcaus in this typ o algorithms th icincy is basd on th high spd that can b achivd by a simpl task xcutd in a multiprocss nvironmnt. Thror, normally th data ar accssd squntially. Currntly, hardwar architcturs or thr rqunt itmsts mining algorithms (Apriori, DHP and FP-Growth) hav bn proposd in th litratur [2, 3, 11, 10], whr squntial algorithms ar usd. A systolic array architctur or th Apriori algorithm was proposd in [2] and [3]. A systolic array is an array o procssing units that procsss data in a piplind ashion. This algorithm gnrats potntially rqunt itmst (candidats itmsts) ollowing a huristic approach. Thn, it pruns th candidats known to b inrqunt, and inally it counts th support o th rmaining itmsts. This is don or ach siz o itmst until thr is no rqunt itmst lt. In th proposd architcturs, ach itm o th candidat st and th databas ar injctd to th systolic array. In vry unit o th array, th subst opration and th candidat gnration is prormd in a piplind ashion. Th ntir databas has to b injctd svral tims through th array. In [2], th hardwar approach provids a minimum o a 4 tim prormanc advantag ovr th astst sotwar implmntation running on a singl computr. In [3], an nhancmnt to th architctur is xplord, introducing a bitmapd CAM or paralll counting th support o svral itmsts at onc, achiving a 24 tim prormanc spdup. In [11] th authors us as a starting point th architctur proposd in [3] to implmnt th DHP algorithm [8]. This introducs two blocks at th nd o th systolic array to collct usul inormation about transactions and candidat

3 A Highly Paralll Algorithm or Frqunt Itmst Mining 293 itmsts to prorm a trimming opration ovr thm and to rduc th amount o data that th architctur has to procss. In [10] a systolic tr architctur has bn proposd to implmnt th FP- Growth algorithm [6]. This algorithm uss a compact rprsntation o th data in an FP-Tr data structur. To construct th FP-Tr, two passs through th databas ar rquird and th rmaining procssing is mad through this data structur. This tr architctur consists o an array o procssing lmnts, arrangd in a multidimnsional tr to implmnt th FP-Tr. With this architctur a rduction o data workload is achivd. All prsntd works in this sction us a horizontal itms-list data layout. This rprsntation rquirs a numbr o bits pr itm to idntiy thm (usually 32). Also, all architcturs implmnt algorithms that hav bn concivd or sotwar nvironmnts and do not tak ull advantag o th hardwar intrinsic paralllism. 3 Th Proposd Algorithm Concptually, a datast can b dind as a two-dimnsional matrix, whr th columns rprsnt th lmnts o th datast and th rows rprsnt th transactions. A transaction is a st o on or mor itms obtaind rom th domain in which th data is collctd. Considring a lxicographical ordr ovr th itms thy can b groupd by quivalnc classs. Th quivalnc class (E(X)) o an itmst X is givn by: E(X) ={Y Prix k 1 (Y )=X Y = k} (1) Th proposd algorithm is basd on a sarch ovr th solution spac through th quivalnc class, considring a lxicographical ordr ovr th itms. This is a two-dimnsional sarch, both bradth and dpth is prormd concurrntly. Using th sarch through th quivalnc class allows us to xploit a charactristic o VBV data layout. With this typ o rprsntation, th support o an itmst can b dind as th numbr o activ bits o th vctor that rprsnts it (X). This vctor X rprsnts th co-occurrnc o itms in th databas and can b obtaind as a conscutiv bitwis and opration btwn all th vctors that rprsnt ach itm o th itmst(s quation 2). X = {a,b,c,...,n} X n = a and b and c and... and n (2) Th procss to obtain X canbdonbystps.first,anand opration can b prormd btwn th irst two vctors (a and b) and thn th rsults ar accumulatd. This accumulatd valu can b usd to prorm anothr and opration with th third vctor to obtain X or X = {a, b, c}. ToobtainX n this must b don or all th itms in X. This procdur is shown in Algorithm 1. Dining th sarch spac as a tr (Figur 1 shows a 5 itms sarch spac) in which ach nod rprsnts an itm o th databas, an itmst is dtrmind by

4 294 A. Msa t al. th shortst path btwn two nods o th tr. Using th procdur prviously dscribd, th vctor X n o an itmst can b obtaind rcursivly as X n = X n 1 and n, bingx n 1 th accumulatd valu in th parnt nod. Onc X n 1 is calculatd, it can b usd to obtain all th accumulatd vctors on ach child nod. With this procss, ach nod provids partial rsults or ach on o th itmsts that includs it in th path o th tr. association ab ac ad a a abd abc ab ac c d b d c d d Architctur structur Sarch spac Fig. 1. Structur or procssing 5 itm solution tr In this algorithm thr is no candidat gnration, but th sarch spac is xplord until raching a nod or which th support is zro. This procss is sustaind by th downward closur proprty, which stablishs that th support o any itmst is gratr than or qual to th support o any o its suprsts. Bcaus o this, i this nod dos not hav activ ons in th accumulatd vctor, th nods in th lowr subtr will not rciv any contribution in th support valu rom that vctor. Th lxicographical ordr o th itms is stablishd according to thir rquncy, ordring irst th ons with th smallst valus. This ordr causs th valu o th support o itmsts to dcras rapidly whn dscnding through th tr architctur. Managing larg databass with VBV data layout can b xpnsiv bcaus thsizox n is dtrmind by th numbr o transactions o th databas. To solv this, a horizontal partition o th databas is mad. Each sction is procssd indpndntly, as shown in Algorithm 2, and ach nod accumulats th itmsts sction support. Evry tim a nod calculats a partial support, it is

5 A Highly Paralll Algorithm or Frqunt Itmst Mining 295 addd to th accumulatd support o th itmst. Whn th databas procssing inishs, ach nod has th global support o all th itmsts that it calculatd. A structur o procssing lmnts is ndd to implmnt th algorithm. This structur intrchangs data as dscribd in Algorithm 2. Two mods o data injction and on or data xtraction ar ndd to achiv th task o rqunt itmsts mining: Data In, Support Thrshold and Data Flush. Inthirstmod(Procss Data()) all data vctors ar d into th structur by th root lmnt to procss th itmsts. Th scond mod (Procss Minsup()) injcts th support thrshold so th procssing lmnts can dtrmin which itmst is rqunt and which is not. For th data xtraction Gt Data(), all th lmnts o th structur lush th rqunt itmsts through thir parnt and th data xits th structur through th root lmnt. To implmnt th algorithm w us a binary tr structur o procssing lmnts (PE ). Figur 1 shows a iv-itm solution tr procssing structur. Th structur is a systolic tr and it was chosn bcaus this typ o construction allows to xponntially incras th concurrnt oprations at ach procssing stp. For this, th numbr o concurrnt oprations can b calculatd as 1 + cc i=0 2i,bingcc th numbr o procssing stps that hav lapsd sinc th procss startd. Each nod o th systolic tr (n arq ) is associatd to a nod o th solution tr (n sol ). This n arq dtrmins an itmst (IS n ) and it is ormd with th path romthtrrootton sol.thn arq calculats th support o th suprsts o IS n. Thos suprsts ar th ons that ar dtrmind by all th nods in th path rom th root to a la, ollowing th nod that is most to th lt o th solution tr. A PE has a connction rom its parnt and it is connctd to a lowrpe (PEu, with an uppr ntry) and to a right PE (PEl, with a lt ntry). In gnral, th amount o PEs a structur has, can b calculatd as ollows: with PEu n and PEl n : ST n =1+PEu n 1 + PEl n 1,orn 2, PEu 2 =0, PEl 1 =0, PEu 3 =1, PEl 2 =1, PEu n =1+PEl n 2 + PEu n 1 PEl n =1+PEl n 1 + PEu n Procssing Elmnts Th itmsts that a PE calculats ar dtrmind by th data vctors that th parnt ds to it. This must b in such a way that th irst vctor to rach ach PE is th X n o th prix o th quivalnc class that th IS n blongs to. Each PE calculats th support o th vctor that rcivs rom th parnt, it accumulats th bitwis and opration in a local mmory and propagats th data vctor that it rcivs rom th parnt or th accumulatd valu corrspondingly. To achiv this, th bhavior o th two typs o PEs is dind. In Data In mod, th main dirnc btwn PEl (dscribd in Algorithm 3) and PEu

6 296 A. Msa t al. (dscribd in Algorithm 4) is that PEu dos not accumulat th rsult o th bitwis and opration o th scond data vctor that it rcivs. This opration provids th down PE th prix with th quivalnc class o its IS n. 3.2 Fdback Stratgy As th numbr o PEs o th tr is dirctly dpndnt on th numbr o rqunt itms that xist in th databas, it is impractical to crat this structur or databass with many rqunt itms. To solv this problm a structur or n itms can b dsignd and taking into account th rcursiv dinition o trs w could calculat th itmsts stpwis, s Figur 2. In th irst stp all th itmsts that can b calculatd with this structur ar obtaind. Sinc it is known which lvl o th solution tr is procssd or a givn structur (Figur 2 shows it as th architctur procssing bordr), data ar injctd back into this structur xcpt that this tim th irst vctor will not b th irst itm, but th prix o th quivalnc class o th itmst that is dtrmind by th bordr tr nod that was not procssd. In Figur 2, an xampl o a six-itms sarch spac is shown. Th dottd lin subtrs ar xampls o subtrs with dirnt parnts that hav to b procssd with th dback stratgy. In th xampl, to procss th irst subtr to th lt, th irst vctor that has to b injctd to th architctur is th vctor that rprsnts th itmst X = {a, b, c}. This dback is rpatd or ach solution tr nod that is in th bordr o th nods that wr not procssd. As ach o ths nods dins a solution subtr (all th subtrs that ar blow th procssing bordr in Figur 2), this structur is compatibl with th ntir tr and consquntly can procss a solution tr o any siz. This procss is dind rcursivly or th ntir solution tr and as a rsult, w obtain a partition o th tr and ach subtr o th partition rprsnts a solution tr o n itms.

7 A Highly Paralll Algorithm or Frqunt Itmst Mining 297 a 3 itms architctur procssing bordr b c d c d d d 6 itms sarch spac Fig. 2. Fdback stratgy 4 Architctur Structur Th proposd architctur has thr mods o opration: Data In, Support Thrshold and Data Flush. In th Data In mod, all X n o ach sction o th databas ar injctd. Whn th ntir databas is injctd, th architctur ntrs in th Support Thrshold mod and th support thrshold valu is providd and propagatd through th systolic tr. Onc a nod rcivs th support thrshold valu, all th PEs chang to Data Flush mod and th rsults ar xtractd rom th architctur. In this architctur, th amount o clock cycls or th ntir procss can b dividd into two gnral stags: Data In (CC data ) and Data Flush (CC lush ). Th CC data is mainly dind by th numbr o rqunt itms in th databas to spciic support thrshold (n ), th numbr o transactions o th databas (T ) and th siz o th data vctor that is chosn (bw). Th data vctor dins th numbr o sctions (S) in th partition o th databas. CC data can b calculatd as ollows: CC data = S (n +1)+1,whrS = T/bw For th data lush stag, CC lush is mainly dind by th numbr o PEs that hav rqunt itmsts and th numbr o mpty PEs in th path up to th nxt PE with usul data. In this stratgy, onc th data ar xtractd rom a PE, data must b xtractd rom th PEl, sinc thr is no inormation to dtrmin i thy will hav rqunt itmsts or not. In th cas o th PEu in th lowr subtr, i thr ar no rqunt itmsts in th currnt PE,thn no rqunt itmsts will b ound in th lowr subtr. This is bcaus all th itmsts calculatd in th subtr will hav as a prix th irst itmst o th currnt PE, and i this is not rqunt, thn no suprst o it will b rqunt. Th numbr o PE with and without usul data, and th position in th systolic tr that thy occupy, only dpnds on th natur o th data and th support thrshold. In th worst cas, all th PEs will hav rqunt itmsts and CC lush could b calculatd as ST n + FI st.

8 298 A. Msa t al. 4.1 Flush Stratgy To obtain th data rom th architctur, ach PE ntrs in Data Flush opration mod. All PE hav a connction to th parnt so all th rqunt itmsts can b lushd up (Parnt out). In this stratgy ach nod o th systolic tr lushs th rqunt itmsts stord in local mmory, thn it srvs as a gatway to data rom th lowr PE (Down in) and whn it nds, it srvs as a gatway to th data o th right PE (Right in). Extracting th data through th root nod o th systolic tr allows to tak advantag o th charactristics o th itmsts. Ths charactristics prmit to din huristics to prun subtrs in th lush stratgy and thror shortns th amount o data that is ncssary to lush rom th architctur. Bcaus o this, lss tim is ndd to complt th task. 5 Implmntation Rsults Th proposd architctur was modld using VHDL and it was vriid in simulation with ModlSim SE 6.5 simulation tool. Onc th architctur was validatd, it was synthsizd using ISE 9.2i. Th targt dvic was st to a Xilinx Virtx-4 XC4VFX140 with packag FF1517 and 10 spd grad. For th xprimnts, thr datasts rom [4] wr usd: Chss, Accidnts and Rtail. As xplaind in Sction 4, th siz o th architctur incrass ruld by th numbr o rqunt itms that it is capabl o procssing, so th siz o th dvic bing usd will highly act th tim it will tak to inish th task. For th xprimnts w usd a 32 bits data vctor and th biggst architctur that its th dvic was a structur to procss 11 itms with 264 PEs. This architctur consums 74.6% o th LUTs (94,248 out o 126,336) and 18, 6% o th lip-lops (23,496 out o 126,336) availabl in th dvic. Sinc th architctur is compltly dcntralizd, thr ar no global connctions and thus th maximum oprating rquncy is not actd by th numbr

9 A Highly Paralll Algorithm or Frqunt Itmst Mining 299 Proposd Architctur FP-Growth Architctur Tim (ms) Chss Numbr o uniqu rqunt itms. Accidnts Tim (ms) Numbr o uniqu rqunt itms. Tim (ms) Rtail Nunbr o uniqu rqunt itms. Fig. 3. Mining tim comparison o PEs o th architctur. Th maximum rquncy obtaind or this architctur was 137 MHz. Th proposd algorithm is snsitiv to th numbr o rqunt itms mor than th support thrshold, so to show th bhavior mor prcisly th xprimnts wr carrid out basd on this variabl. In Figur 3 w compar th mining tim o th architctur against th bst tim o th FP-Growth architctur prsntd in [9]. This Figur shows that th gratst improvmnt in xcution tim was or th Chss databas, whr mor than on ordr o magnitud was obtaind. In th othr two databass it is shown how th xcution tim o th proposd architctur grows slowr than th FP-Growth architctur whn incrasing th numbr o rqunt itms. Morovr, whn incrasing th ratio btwn th numbr o obtaind rqunt itmsts and th siz o th databas, th prcntag o CC data dcrass. In th cas o Accidnts and Rtail databass CC data is 99% and 97% o total tim corrspondingly and or Chss databas th prcntag is btwn 62% and 68%. This is bcaus th xcution tim o th Data in mod dpnds only on th amount o rqunt itms and th numbr o transactions that th databass hav and th rmaining tim will dpnd on th numbr o rqunt itmsts obtaind. This bhavior shows a atur o th algorithm and its scalability. Th architctur prorms bttr whn th dnsity o th procssd databas incrass and gnrally prorms bttr whn th numbr o rqunt itmsts obtaind also incrass. This atur (bttr prormanc at highr dnsity) has a grat importanc i it is considrd that th dnsr th databass th mor diicult to obtain rqunt itmsts and th highr th numbr o rqunt itmsts obtaind. For spars databass and high supports (low numbr o rqunt itms), this task can b solvd without th ncssity o appaling to hardwar acclration in most cass. This nd is mor vidnt whn daling with larg databass with low thrshold o support or high dnsity, or both. Bcaus o this, this atur is dsirabl in th algorithms or obtaining rqunt itmsts. 6 Conclusion In this papr w proposd a paralll algorithm that is spcially dsignd or nvironmnts which allow a high numbr o concurrnt procsss. This charactristic bst suits a hardwar nvironmnt and allows to us a task paralllism

10 300 A. Msa t al. with in granularity. Furthrmor, a hardwar architctur to validat th algorithm was dvlopd. Th xprimnts show that our approach outprorms th FP-Growth architctur prsntd in [10] by on ordr o magnitud whn procssing dns databass, and th mining tim grows slowr than th FP-Growth architctur mining tim whn procssing spars matrics. Rrncs 1. Agrawal, R., Shar, J.C.: Paralll mining o association ruls dsign, implmntation and xprinc. Tchnical Rport RJ10004, IBM Rsarch Rport (Fbruary 1996) 2. Bakr, Z.K., Prasanna, V.K.: Eicint Hardwar Data Mining with th Apriori Algorithm on FPGAs. In: Proc. o th 13th Annual IEEE Symposium on Fild Programmabl Custom Computing Machins 2005 (FCCM 05), pp (2005) 3. Bakr, Z.K., Prasanna, V.K.: An Architctur or Eicint Hardwar Data Mining using Rconigurabl Computing Systm. In: Proc. o th 14th Annual IEEE Symposium on Fild Programmabl Custom Computing Machins 2006 (FCCM 06), pp (2006) 4. Gothals, B.: Frqunt itmst mining datast rpository, 5. Han, E.H., Karypis, G., Kumar, V.: Scalabl paralll data mining or association ruls. In: Proc. o th ACM SIGMOD Conrnc, pp (1997) 6. Han, J., Pi, J., Yin, Y.: Mining rqunt pattrns without candidat gnration. In: 2000 ACM SIGMOD Intl. Con. on Managmnt o Data, pp ACM Prss, Nw York (2000) 7. Palancar, J.H., Tormo, O.F., Cárdnas, J.F., Lón, R.H.: Distributd and shard mmory algorithm or paralll mining o association ruls. In: Prnr, P. (d.) MLDM LNCS (LNAI), vol. 4571, pp Springr, Hidlbrg (2007) 8. Park, J., Chn, M., Yu, P.: An ctiv hash basd algorithm or mining association ruls. In: Cary, M.J., Schnidr, D.A. (ds.) SIGMOD Conrnc, pp ACM Prss, Nw York (1995) 9. Sun, S., Stn, M., Zambrno, J.: A rconigurabl platorm or rqunt pattrn mining. In: RECONFIG 08: Proc. o th 2008 Intl. Con. on Rconigurabl Computing and FPGAs, pp IEEE Computr Socity, Los Alamitos (2008) 10. Sun, S., Zambrno, J.: Mining association ruls with systolic trs. In: Proc. o th Intl. Con. on Fild-Programmabl Logic and its Applications (FPL), pp IEEE, Los Alamitos (2008) 11. Wn, Y., Huang, J., Chn, M.: Hardwar-nhancd association rul mining with hashing and piplining. IEEE Trans. on Knowl. and Data Eng. 20(6), (2008) 12. Zaki, M.J., Parthasarathy, S., Ogihara, M., Li, W.: Nw algorithms or ast discovry o association ruls. In: Proc. o th 3rd Intl. Con. on KDD and Data Mining (KDD 97), pp (1997)