Ternary Directed Acyclic Word Graphs

Transcription

1 Ternry Direted Ayli Wrd Grphs Stru Miymt, Shunsuke Ineng,, Msyuki Tked,, nd Ayumi Shinhr, Deprtment f Infrmtis, Kyushu University 33, Fukuk , Jpn PRESTO, Jpn Siene nd Tehnlgy Crprtin (JST) E-mil: {s-miy, s-ine, tked, yumi}@i.kyushu-u..jp Abstrt. Given set S f strings, DFA epting S ffers very time-effiient slutin t the pttern mthing prblem ver S. The key is hw t implement suh DFA in the trde-ff between time nd spe, nd espeilly the hie f hw t implement the trnsitins f eh stte is ritil. Bentley nd Sedgewik prpsed n effetive tree struture lled ternry trees. The ide f ternry trees is t implnt the press f binry serh fr trnsitins int the struture f the trees themselves. This wy the press f binry serh bemes visible, nd the implementtin f the trees bemes quite esy. The direted yli wrd grph (DAWG) f string w is the smllest DFA tht epts ll suffixes f w, nd requires nly liner spe. We pply the sheme f ternry trees t DAWGs, intrduing new dt struture nmed ternry DAWGs (TDAWGs). We perfrm sme experiments tht shw the effiieny f TDAWGs, mpred t DAWGs in whih trnsitins re implemented by tbles nd linked lists. 1 Intrdutin Due t rpid dvne in infrmtin tehnlgy nd glbl grwth f mputer netwrks, we n utilize lrge munt f dt tdy. In mst ses, dt re stred nd mnipulted s strings. Therefre the develpment f effiient dt strutures fr serhing strings hs fr dedes been prtiulrly tive reserh re in mputer siene. Given set S f strings, we wnt sme effiient dt struture tht enbles us t serh S very quikly. Obviusly DFA tht epts S is the ne. The prblem rising in implementing suh n utmtn is hw t stre the infrmtin f the trnsitins in eh stte. The mst bsi ide is t use tbles, with whih serhing S fr given pttern p is fesible in O( p ) time, where p dentes the length f p. Hwever, the signifint drwbk is tht the size f the tbles is prprtinl t the size f the lphbet Σ used. In prtiulr, it is ruil when the size f Σ is thusnds lrge like in Asin lnguges suh s Jpnese, Kren, Chinese, nd s n. Using linked lists is ne pprent mens f espe frm this wste f memry spe by tbles. Althugh this surely redues spe requirement, serhing fr pttern p tkes O( Σ p ) time in bth wrst nd verge ses. It is esy t imgine tht this shuld be serius disdvntge when serhing texts f lrge lphbet.

2 Bentley nd Sedgewik [3] intrdued n effetive tree struture lled ternry serh trees (t be simply lled ternry trees in this pper), fr string set f strings. The ide f ternry trees is t implnt the press f binry serh fr trnsitins int the struture f the trees themselves. This wy the press f binry serh bemes visible, nd the implementtin f the trees bemes quite esy sine eh nd every stte f ternry trees hs t mst three trnsitins. Bentley nd Sedgewik gve n lgrithm tht, fr ny set S f strings, nstruts its ternry tree in O( Σ S ) time with O( S ) spe, where S dentes the ttl length f the strings in S. They ls shwed severl nie pplitins f ternry trees [2]. We in this pper nsider the mst fundmentl pttern mthing prblem n strings, the substring pttern mthing prblem, whih is desribed s fllws: Given text string w nd pttern string p, exmine whether r nt p is substring f w. Clerly, DFA tht regnizes the set f ll suffixes f w permits us t slve this prblem very quikly. The smllest DFA f this kind ws intrdued by Blumer et l. [4], lled the direted yli wrd grph (DAWG) f string w, tht nly requires O( w ) spe. In this pper, we pply the sheme f ternry trees t DAWGs, yielding new dt struture lled ternry DAWGs (TDAWGs). By the use f TDAWG f w, serhing text w fr ptten p tkes O( Σ p ) time in the wrst se, but the time mplexity fr the verge se is O(lg Σ p ), whih is n dvntge ver DAWGs implemented with linked lists tht require O( Σ p ) expeted time. Therefre, the key is hw t nstrut TDAWGs quikly. Nte tht the set f ll suffixes f string w is f size qudrti in w. Nmely, simply pplying the lgrithm by Bentley nd Sedgewik [3] merely llws us t nstrut TDAWG f w in O( Σ w 2 ) time. Hwever, using mdifitin f the n-line lgrithm f Blumer et l. [4], plesingly, the TDAWG f w n be nstruted in O( Σ w ) time. We ls perfrmed sme mputtinl experiments t evlute the effiieny f TDAWGs, using English texts, by the mprisn with DAWGs implemented by tbles nd linked lists. The mst interesting result is tht the nstrutin time f TDAWGs is drmtilly fster thn thse f DAWGs with tbles nd linked lists. Plus, it is evluted tht serh time by TDAWGs is ls fster thn tht by DAWGs with linked lists. 2 Direted Ayli Wrd Grphs Let Σ be finite lphbet. An element f Σ is lled string. Strings x, y, nd z re sid t be prefix, substring, nd suffix f string w = xyz, respetively. The sets f prefixes, substrings, nd suffixes f string w re dented by Prefix(w), Substr(w), nd Suffix(w), respetively. The length f string w is dented by w. The empty string is dented by ε, tht is, ε = 0. Let Σ + = Σ {ε}. Let S Σ. The number f strings in S is dented by S, nd the sum f the lengths f strings in S by S. The fllwing prblem is the mst fundmentl nd imprtnt in string pressing.

3 Fig. 1. STrie() is shwn n the left, where ll the sttes re epting. By minimizing this utmtn we btin DAWG(), n the right. Definitin 1 (Substring Pttern Mthing Prblem). Instne: text string w Σ nd pttern string p Σ. Determine: whether p is substring f w. It is ler tht this prblem is slvble in time prprtinl t the length f p, by using n utmtn tht epts Substr(w). The mst bsi utmtn f this kind is the suffix trie. The suffix trie f string w Σ is dented by STrie(w). Wht is btined by minimizing STrie(w) is lled the direted yli wrd grph (DAWG)fw [9], dented by DAWG(w). In Fig. 1 we shw STrie(w) nd DAWG(w) with w =. The initil stte f DAWG(w) is ls lled the sure stte, nd the stte epting w is lled the sink stte f DAWG(w). Eh stte f DAWG(w) ther thn the sure stte hs suffix link. Assume x 1,...,x k re the substrings f w epted in ne stte f DAWG(w), rrnged in the deresing rder f their lengths. Let y = x k, where y Σ nd Σ. Then the suffix link f the stte epting x 1,...,x k pints t the stte in whih y is epted. DAWGs were first intrdued by Blumer et l. [4], nd hve widely been used fr slving the substring pttern mthing prblem, nd in vrius pplitins [7, 8, 15]. Therem 1 (Crhemre [6]). Fr ny string w Σ,DAWG(w) is the smllest (prtil) DFA tht regnizes Suffix(w). Therem 2 (Blumer et l. [4]). Fr ny string w Σ with w > 1, DAWG(w) hs t mst 2 w 1 sttes nd 3 w 3 trnsitins. It is trivil ft tht DAWG(w) n be nstruted in time prprtinl t the number f trnsitins in STrie(w) by the DAG-minimiztin lgrithm by Revuz [13]. Hwever, the number f trnsitins f STrie(w) is unfrtuntely qudrti in w. The diret nstrutin f DAWG(w) in liner time is therefre signifint, in rder t vid reting redundnt sttes nd trnsitins tht re

4 ε Fig. 2. The n-line nstrutin f DAWG(w) with w =. The slid rrws re the trnsitins, nd the dshed rrws re the suffix links. Nte tht the stte pinted by the suffix link f the sink stte will be the tive stte f the next phse. In the press f updting DAWG() t DAWG(), the stte epting {, } is seprted int tw sttes fr {} nd {}. deleted in the press f minimizing STrie(w). Blumer et l. [4] indeed presented n lgrithm tht diretly nstruts DAWG(w) nd runs in liner time if Σ is fixed, by mens f suffix links. Their lgrithm is n-line, nmely, fr ny w Σ nd Σ it llws us t updte DAWG(w) tdawg(w) in mrtized nstnt time, mening tht we need nt nstrut DAWG(w) frm srth. We here briefly rell the n-line lgrithm by Blumer et l. It updtes DAWG(w) t DAWG(w) by inserting suffixes f w int DAWG(w) in deresing rder f their lengths. Let z be the lngest string in Substr(w) Suffix(w). Then z is lled the lngest repeted suffix f w nd dented by LRS (w). Let z = LRS (w). Let w = l nd u 1,u 2,...,u l,u l+1 be the suffixes f w rdered in their lengths, tht is, u 1 = w nd u l+1 = ε. We tegrize these suffixes f w int the fllwing three grups.

5 (Grup 1) u 1,...,u i 1 (Grup 2) u i,...,u j 1 where u i = z (Grup 3) u j,...,u l+1 where u j = z Nte ll suffixes in Grup 3 re lredy represented in DAWG(w). We n insert ll the suffixes f Grup 1 int DAWG(w) by reting new trnsitin lbeled by frm the urrent sink stte t the new sink stte. It bviusly tkes nly nstnt time. Therefre, we hve nly t re but thse in Grup 2. Let v i,...,v j 1 be the suffixes f w suh tht, fr ny i k j 1, v k = u k.we strt frm the stte rrespnding t LRS(w) =z = v i in DAWG(w), whih is lled the tive stte f the urrent phse. A new trnsitin lbeled by is inserted frm the tive stte t the new sink stte. The stte t be the next tive stte is fund simply by trversing the suffix link f the stte fr v i,in nstnt time, nd new trnsitin lbeled by is reted frm the new tive stte t the sink stte. After we insert ll the suffixes f Grup 2 this wy, the utmtn represents ll the suffixes f w. We nw py ttentin t LRS (w) =z = u j. Let x be the lngest string in the stte where u j is epted. We then hve t hek whether x = u j r nt. If nt, the stte is seprted int tw sttes, where ne epts the lnger strings thn u j, nd the ther epts the rest. Assiting eh stte with the length f the lngest string epted in it, we n del with this stte seprtin in nstnt time. The n-line nstrutin f DAWG() is shwn in Fig Ternry Direted Ayli Wrd Grphs Bentley nd Sedgewik [3, 2] intrdued new dt struture lled ternry trees, whih re quite useful fr string set f strings, frm bth viewpints f spe effiieny nd serh speed. The ide f ternry trees is t implnt the press f binry serh fr linked lists int the trees themselves. This wy the press f binry serh bemes visible, nd the implementtin f the trees bemes quite esy sine eh nd every stte f ternry trees hs t mst three trnsitins. The left figure in Fig. 3 is ternry tree fr Suffix(w) with w =. We n see tht this rrespnds t STrie(w) in Fig. 1, nd therefre, the tree is lled ternry suffix trie (TSTrie) f string. Fr substring x f string w Σ, we nsider set ChrSet w (x) ={ Σ x Substr(w)} f hrters. In STrie(w), eh hrter f ChrSet w (x) is ssited with trnsitin frm stte x (see STrie() in Fig. 1). Hwever, in TSTrie f w, eh hrter in ChrSet w (x) rrespnds t stte. This mens tht we n regrd ChrSet w (x) s set f the sttes tht immeditely fllws string x in the TSTrie f w, where elements f ChrSet w (x) re rrnged in lexigrphil rder, tp-dwn. There re mny vritins f the rrngement f elements in ChrSet w (x), but we rrnge them in inresing rder f their leftmst urrenes in w, tp-dwn. Thus the rrngement f the sttes

6 Fig. 3. TSTrie(w) is n the left, nd TDAWG(w) n the right, with w =. is uniquely determined, nd the resulting struture is lled the TSTrie f w, dented by TSTrie(w). The stte rrespnding t the hrter in ChrSet w (x) with the erliest urrene, is lled the tp stte with respet t ChrSet w (x), sine it is rrnged n the tp f the sttes fr hrters in ChrSet w (x). Given pttern p, t ny nde f TSTrie(w) we exmine if the hrter in p we urrently fus n is lexigrphilly lrger thn the hrter b stred in the stte. If <b, then we tke the left trnsitin frm the stte nd mpre t the hrter in the next stte. If >b, then we tke the right trnsitin frm the stte nd mpre t the hrter in the next stte. If = b, then we tke the enter trnsitin frm the stte, nw the hrter is regnized, nd we mpre the next hrter in p t the hrter in the next stte. We give nrete exmple f serhing fr pttern using TSTrie() in Fig. 3. We strt frm the initil stte f the tree nd hve >, nd thus g dwn t the next stte vi the right trnsitin. At the next stte we hve =, nd thus we tke the enter trnsitin frm the stte nd rrive t the next stte, with the hrter regnized. We then mpre the next hrter in the pttern with in the stte where we re. Nw we hve <, we g dwn lng the left trnsitin f the stte nd rrive t the next stte, where we hve =. Then we tke the enter trnsitin nd rrive t the next stte, where finlly is epted. This wy, fr ny pttern p Σ we n slve the substring pttern mthing prblem f Definitin 1 in O(lg Σ p ) expeted time. We nw nsider t pply the bve sheme t DAWG(w). Wht is btined here is the ternry DAWG (TDAWG) f w, dented by TDAWG(w). The right figure in Fig. 3 is TDAWG(). Cmpre it t DAWG() in Fig. 1 nd TSTrie() in Fig. 3. It is quite bvius tht using TDAWG(w) we n exmine if p Substr(w) in O(lg Σ p ) expeted time, s well. Resnbly, TDAWG(w) n be nstruted by the n-line lgrithm f Blumer et l. [4] tht ws relled in Setin 2. In TDAWG(w) nly tp sttes hve suffix links, nd n be direted by suffix links f ther sttes. The n-line nstrutin f TDAWG() is shwn in Fig. 4.

7 ε Fig. 4. The n-line nstrutin f TDAWG(w) with w =. The dshed rrws re the suffix links. Ntie nly tp sttes hve suffix links, nd re pinted by suffix links f ther tp sttes. In the press f updting TDAWG() t TDAWG(), the stte epting {, } is seprted int tw sttes fr {} nd {}, s well s the se f DAWGs shwn in Fig 2. Therem 3. Fr ny string w Σ,TDAWG(w) n be nstruted n-line, in O( Σ w ) time using O( w ) spe. 4 Experiments In this setin we shw sme experimentl results tht revel the dvntge f ur TDAWGs, mpred t DAWGs with tbles (tble DAWGs) nd DAWGs with linked lists (list DAWGs). The tbles were implemented by rrys f length 256. The linked lists were linerly serhed t ny stte f the list DAWGs. All the three lgrithms t nstrut TDAWGs, tble DAWGs nd list DAWGs were implemented in the C lnguge. All lultins were perfrmed n Lptp PC with PentiumIII-650MHz CPU nd 256MB min memry running VineLinux2.6r2. We used the English text hsumed.91 vilble t http: //tre.nist.gv/dt.html.

8 [Mbyte] 30 [se] 200 tble_dawg Memry spe TDAWG list_dawg Time 100 list_dawg TDAWG Text length [Kbyte] Text length [Kbyte] Spe requirements f TDAWG nd list_dawg Cnstrutin times f TDAWG, list_dawg nd tble_dawg [µse] 0.6 list_dawg 0.4 TDAWG Time 0.2 tble_dawg Pttern length Serh times f TDAWG, list_dawg nd tble_dawg Fig. 5. The upper left hrt is the memry requirements (in Mbytes) f TDAWGs nd list DAWGs. The upper right hrt is the nstrutin times (in sends) f TDAWGs, list DAWGs nd tble DAWGs. The lwer hrt is the serhing time (in mir sends) f TDAWGs, list DAWGs nd tble DAWGs. The upper left hrt f Fig. 5 shws memry requirements fr TDAWGs nd list DAWGs, where memry spes fr bth grw linerly, s expeted. One n see tht TDAWGs require but 20% mre memry thn list DAWGs. The memry requirement f tble DAWGs is nt shwn sine it is t muh fr the sle f the hrt. The tble DAWG fr the text f size 64KB required 98.82MB f memry spe, nd tht fr the text f size 128KB needed MB. Thus tble DAWGs re rther unusble in relity. The send test ws the nstrutin times fr TDAWGs, tble TDAWGs, nd list DAWGs, shwn upper right f Fig. 5. One n see tht the TDAWGs were nstruted but twie fster thn the list DAWGs. This seems the effet f binry serh in the TDAWGs, while the linked lists were linerly serhed in the list DAWGs. As fr the tble DAWGs, thugh serhing the trnsitin n

9 be dne in nstnt time, the memry lltin fr the tbles seemed t tke t muh time. The third test ws serhing times fr ptterns f different lengths. We rndmly hse 100 substrings f the text fr eh length, nd serhed fr every f them 1 millin times. The result shwn in the lwer hrt in Fig. 5 is the verge time f serhing fr pttern ne. Remrk tht the TDAWGs re fster thn list DAWGs, even fr lnger ptterns. 5 Cnlusins nd Further Wrk In this pper we intrdued new dt struture lled ternry direted yli wrd grphs (TDAWGs). The press f binry serh fr the trnsitins is implnted in eh stte f TDAWGs. Fr ny string w Σ, TDAWG(w) n be nstruted in O( Σ w ) time using O( w ) spe, in n-line fshin. Our experiments shwed tht TDAWGs n be nstruted muh fster thn bth DAWGs with tbles nd DAWGs with linked lists, fr English texts. Mrever, serhing time f TDAWGs is ls better thn tht f DAWGs with linked lists. Thinking ver the ft tht TDAWGs re better in speed thn the tw ther types f DAWGs thugh the lphbet size f the English text is nly 256, TDAWGs shuld be lt mre effetive when pplied t texts f lrge lphbet suh s Jpnese, Kren, Chinese, nd s n. We emphsize tht the benefit f the ternry-bsed implementtin is nt limited t DAWGs. Nmely, it n be pplied t ny utmt-riented index struture suh s suffix trees [16, 12, 14, 10] nd mpt direted yli wrd grphs (CDAWGs) [5, 9, 11]. Therefre, we n ls nsider ternry suffix trees nd ternry CDAWGs. Cnerning the experimentl results n TDAWGs, ternry suffix trees nd ternry CDAWGs re prmising t perfrm very well in prtie. As previusly mentined, the serh time fr pttern p using DAWGs with linked lists is O( Σ p ) in the wrst se, but there is wy t imprve it t O(lg Σ p ) time with dditinl effrt fr perfrming binry serh. Hwever, it is nt prtil sine the dditinl wrk nsumes nsiderble munt f time, nd is nt implementtin-friendly. Further, it des nt llw us t updte the input text sting sine it is just n ff-line lgrithm. Hwever, s fr TDAWGs, we n pply the tehnique f AVL trees [1] fr blning the sttes f TDAWGs. In this sheme, the sttes fr ChrSet w (x) fr ny substring x re AVL-blned, nd thus the time t serh fr pttern p is O(lg Σ p ) even in the wrst se. The differene frm DAWGs with linked lists is tht we n nstrut AVL-blned TDAWGs in n-line mnner, diretly, nd n updte the input text string esily. Mrever, the nstrutin lgrithm runs in O(lg Σ w ) time unlike the se f linked lists requiring O( Σ w ) time. Therefre, TDAWGs re mre prtil nd flexible fr gurnteeing O(lg Σ p )-time serh in the wrst se. Mrever, there is vritin f TDAWGs tht is mre spe-enmil. Nte tht Fig. 3 f Setin 3 n be minimized by the lgrithm f Revuz [13], nd the resulting struture is shwn in Fig. 6, whih is lled the minimum

10 Fig. 6. The MTDAWG f string. TDAWG (MTDAWG) f the string. T use Revuz s lgrithm we hve t mintin the reversed trnsitin fr every trnsitin, nd it fr sure requires t muh spe. Thus we re nw interested in n n-line lgrithm t nstrut MTDAWGs diretly, but it is still inmplete. We expet tht serhing fr pttern strings using MTDAWGs will be fster thn using TDAWGs, sine memry lltin fr MTDAWGs is likely t be mre quik. Referenes 1. G. M. Andelsn-Velskii nd E. M. Lndis. An lgrithm fr the rgnistin f infrmtin. Sviet. Mth., 3: , J. Bentley nd B. Sedgewik. Ternry serh trees. Dr. Dbb s Jurnl, J. Bentley nd R. Sedgewik. Fst lgrithms fr srting nd serhing strings. In Pr. 8th Annul ACM-SIAM Sympsium n Disrete Algrithms (SODA 97), pges ACM/SIAM, A. Blumer, J. Blumer, D. Hussler, A. Ehrenfeuht, M. T. Chen, nd J. Seifers. The smllest utmtn regnizing the subwrds f text. Theretil Cmputer Siene, 40:31 55, A. Blumer, J. Blumer, D. Hussler, R. MCnnell, nd A. Ehrenfeuht. Cmplete inverted files fr effiient text retrievl nd nlysis. J. ACM, 34(3): , M. Crhemre. Trnsduers nd repetitins. Theretil Cmputer Siene, 45:63 86, M. Crhemre nd W. Rytter. Text Algrithms. Oxfrd University Press, New Yrk, M. Crhemre nd W. Rytter. Jewels f Stringlgy. Wrld Sientifi, M. Crhemre nd R. Vérin. On mpt direted yli wrd grphs. In Strutures in Lgi nd Cmputer Siene, vlume 1261 f LNCS, pges Springer-Verlg, D. Gusfield. Algrithms n Strings, Trees, nd Sequenes. Cmbridge University Press, New Yrk, 1997.

11 11. S. Ineng, H. Hshin, A. Shinhr, M. Tked, S. Arikw, G. Muri, nd G. Pvesi. On-line nstrutin f mpt direted yli wrd grphs. In A. Amir nd G. M. Lndu, editrs, Pr. 12th Annul Sympsium n Cmbintril Pttern Mthing (CPM 01), vlume 2089 f LNCS, pges Springer-Verlg, E. M. MCreight. A spe-enmil suffix tree nstrutin lgrithm. J. ACM, 23(2): , D. Revuz. Minimiztin f yli deterministi utmt in liner time. Theretil Cmputer Siene, 92(1): , E. Ukknen. On-line nstrutin f suffix trees. Algrithmi, 14(3): , E. Ukknen nd D. Wd. Apprximte string mthing with suffix utmt. Algrithmi, 10(5): , P. Weiner. Liner pttern mthing lgrithms. In Pr. 14th Annul Sympsium n Swithing nd Autmt Thery, pges 1 11, 1973.