Approximate String Matching. Department of Computer Science. University of Chile. Blanco Encalada Santiago - Chile

Transcription

1 Very Fas and Simple Approximae Sring Maching Gonzalo Navarro Ricardo Baeza-Yaes Deparmen of Compuer Science Universiy of Chile Blanco Encalada Saniago - Chile fgnavarro,rbaezag@dcc.uchile.cl Absrac We improve he fases nown algorihm for approximae sring maching. This algorihm can only be used for low error levels. By using a new algorihm o verify poenial maches and a new opimizaion echnique for biased exs (such as English), he algorihm becomes he fases one for medium error levels oo. This includes mos of he ineresing cases in his area. Key words: Approximae Maching, Tex Searching, Informaion Rerieval. 1 Inroducion Approximae sring maching is one of he main problems in classical sring algorihms, wih applicaions o ex searching, compuaional biology, paern recogniion, ec. The problem can be formally saed as follows: given a (long) ex of lengh n, a (shor) paern of lengh m, and a maximal number of errors allowed, nd all ex posiions ha mach he paern wih up o errors. The allowed errors are characer inserions, deleions and subsiuions. Tex and paern are sequences of characers from an alphabe of size. We call = =m he error raio or error level. In his wor we focus on on-line algorihms, where he classical soluion, involving dynamic programming, is O(mn) ime [8]. In he las years many? This wor has been suppored in par by FONDECYT gran Preprin submied o Elsevier Preprin 8 April 1998

2 algorihms have been presened ha improve he classical algorihm, for insance [12,5,4,11,3,14,15,6,2]. For low error levels, he fases nown algorihm is [3]. I is a lraion algorihm, i.e. i quicly discards large ex areas ha canno conain a mach and laer veries suspicious areas using a classical algorihm. As all lraion algorihms, i ceases o wor well for moderae error levels. In his paper we use a new echnique o verify poenial maches developed in [7]. We also improve he ler on biased exs such as naural language. The resul is he fases algorihm for approximae sring maching for all bu high error levels. 2 The Original Algorihm The original idea of his algorihm was rs presened in [14]: Lemma: If a paern is pariioned in + 1 pieces, hen a leas one of he pieces can be found wih no errors in any approximae occurrence of he paern. This propery is easily veried by considering ha errors canno aler all he + 1 pieces of he paern, and herefore a leas one of he pieces mus appear unalered. This reduces he problem of approximae sring searching o a problem of mulipaern exac search plus vericaion of poenial maches. Tha is, we spli he paern in + 1 pieces and search all hem in parallel wih a mulipaern exac search algorihm. Each ime we nd a piece in he ex, we verify a neighborhood o deermine if he complee paern appears. In he original proposal [14], a varian of he Shif-Or algorihm was used for mulipaern search. To search r paerns of lengh m 0, his algorihm is O(rm 0 n=w). Since in his case r = +1 and m 0 = bm=( +1)c, he search cos is O(mn=w), which is he same cos for exac searching using Shif-Or. Laer, in [3], he use of a mulipaern exension of an algorihm of he Boyer-Moore (BM) family was proposed. In [1,2] we esed an exension of he BM-Sunday algorihm [10]: we spli he paern in pieces of lengh bm=( + 1)c and dm=( + 1)e and form a rie wih he pieces. We also build a pessimisic d able wih all he pieces (he longer pieces are pruned o build his able). This able sores, for each characer, he smalles shif allowed among all he pieces. Now, a each ex posiion we ener in he rie using he ex characers from ha posiion on. If we end up in a leaf, we found a piece, oherwise we did no. In any case, we use he d 2

3 able o shif o he nex ex posiion. Suppose we nd a ex posiion i he end of a mach for he subpaern ending a posiion j in he paern. Then, he poenial mach mus be searched in he area beween posiions i? j + 1? and i? j m + of he ex, an (m + 2)-wide area. This checing mus be done wih an algorihm resisan o high error levels, such as dynamic programming. This algorihm is he fases in pracice when he oal number of vericaions riggered is low enough, in which case he search cos is close o O(n=m) = O(n). We nd ou now when he oal amoun of wor due o vericaions is no higher. An exac paern of lengh ` appears in random ex wih probabiliy 1=`. In our case, his is 1= bm=(+1)c 1= 1=. Since he cos o verify a poenial mach using dynamic programming is O(m 2 ), and since here are + 1 m pieces o search, he oal cos for vericaions is m 3 = 1=. This cos mus be O() so ha i does no aec he oal cos of he algorihm. This happens for < 1=(3 log m). On English ex we found empirically he limi < 1=5. 3 A New Vericaion Technique In [7] we presened a dieren vericaion echnique. We show now how o use i in his algorihm. The idea is o ry o quicly deermine ha he mach of he small piece is no in fac par of a complee mach. To explain he use of his echnique, a sronger version of he Lemma mus be used, which was proved in [7]. Sronger Lemma: If segm = T ex[a::b] maches pa wih errors, and pa = p 1 :::p j (a concaenaion of subpaerns), hen segm includes a segmen ha maches a leas one of he p i 's, wih ba i =Ac errors, where A = P j i=1 a i. We explain now he echnique. Firs assume ha + 1 is a power of 2. We use A = 2; j = 2; a 1 = a 2 = 1. Tha is, we spli he paern in wo halves (halving also he number of errors). The sronger lemma saes ha a leas one half mus mach wih d=2e errors. We recursively coninue wih his spliing unil we reach he pieces ha are o be searched direcly (wih no errors). Each ime a leaf repors an occurrence, is paren node checs he area looing for is paern (whose size is close o wice he size of he leaf paern). Only if he paren node nds he longer paern, i repors he occurrence o is paren, and so on. The occurrences repored by he roo of he ree are he answers. This consrucion is correc because he pariioning lemma applies o each 3

4 level of he ree, i.e. any occurrence repored by he roo node mus include an occurrence repored by one of he wo halves, so we search boh halves. The argumen applies hen recursively o each half. Figure 1 illusraes his concep. If we search he paern "aaabbbcccddd" wih hree errors in he ex "xxxbbbxxxxxx", and spli he paern in four pieces, he piece "bbb" will be found in he ex. In he original approach, we would verify he complee paern in he ex area, while wih he new approach we verify only if is paren "aaabbb" appears wih one error and we immediaely deermine ha here canno be a complee mach. aaabbbcccddd ( = 3) aaabbb cccddd ( = 1) aaa bbb ccc ddd ( = 0) Fig. 1. The hierarchical vericaion mehod. The boxes (leaves) are he elemens which are really searched, and he roo represens he whole paern. A leas one paern a each level mus mach in any occurrence of he complee paern. If he bold box is found, all he bold lines may be veried. If + 1 is no a power of wo we ry o build he ree as well balanced as possible (o avoid verifying a very long paren because of a very shor child). For insance, if = 4 ( + 1 = 5), we pariion he ree in, say, a lef child wih hree pieces and a righ child wih wo pieces. We hen search he lef subree wih b3=5c errors and he righ one wih b2=5c errors. Coninuing wih his policy we arrive o he leaves, which are searched wih b4=5c = 0 errors each as expeced. Alhough when here are few maches (i.e. low error level) he old and new mehods behave similarly, here is an imporan dierence for medium error levels: he new algorihm is more oleran o errors. Figure 2 illusraes he improvemen obained (he experimenal seup is described in Secion 5). As i can be seen, on random ex he new mehod wors well up o = 1=2, while he previous wors well up o = 1=3. On he oher hand, afer ha poin he vericaions cos much more han in he original mehod. This is because of he hierarchy of vericaions which is carried ou for mos ex posiions when he error level is high. On he oher hand, i is hard o improve he barrier of < 1=2 wih his mehod, since a his poin we are searching for single characers and performing a vericaion each ime some of he characers is found in he ex (which is oo probable). 4

5 Fig. 2. The old (dashed) versus he new (full) vericaion echnique. We use m = 60 and show random (lef) and English ex (righ). 4 Opimizing he Pariion When spliing he paern, we are free o deermine he + 1 pieces as we lie. This can be used o minimize he expeced number of vericaions when he leers of he alphabe do no have he same probabiliy of occurrence (e.g. in English ex). For example, imagine ha P r( 0 e 0 ) = 0:3 and P r( 0 z 0 ) = 0:003. Then, if we search for "eeez" i is beer o pariion i as "eee" and "z" (wih probabiliies 027 and 03 respecively) raher han "ee" and "ez" (wih probabiliies 9 and 009 respecively). More generally, given ha he probabiliy of a sequence is he produc of he individual leer probabiliies, we wan a pariion ha minimizes he sum of he probabiliies of he pieces (which is direcly relaed o he number of vericaions o perform). A dynamic programming algorihm o opimize he pariion of pa[0::m? 1] follows. Le R[i; j] = Q j?1 r=i P r(pa[r]) for every 0 i j m. I is easy o see ha R can be compued in O(m 2 ) ime since R[i; j + 1] = R[i; j] P r(pa[j]). Using R we build wo marices, namely P [i; ] = sum of he probabiliies of he pieces in he bes pariion for pa[i::m? 1] wih errors. C[i; ] = where he nex piece mus sar in order o obain P [i; ]. This aes O(m 2 ) space. The following algorihm compues he opimal pariion in O(m 2 ) ime. for (i = 0;i < m;i ++) P [i; 0] = R[i; m]; 5

6 C[i; 0] = m; for (r = 1;r ;r ++) for (i = 0;i < m? r;i ++) P [i; r] = min j 2 i+1::m?r (R[i; j] + P [j; r? 1]); C[i; r] = j ha minimizes he expression above; The nal probabiliy of vericaion is P [0; ] (noe ha we can use i o esimae he real cos of he algorihm in runime, before running i on he ex). The pieces sar a `0 = 0, `1 = C[`0; ], `2 = C[`1;? 1],..., ` = C[`?1 ; 1]. As we presened he opimizaion, he obained speedup is very modes and even counerproducive in some cases. This is because we consider only he probabiliy of verifying. The search imes of he exended Sunday algorihm degrades as he lengh of he shores piece is reduced, as i happens in an uneven pariion. We consider in fac a cos model which is closer o he real search cos. We opimize 1 minimum lengh + P (verifying) m2 Figure 3 shows experimenal resuls comparing he normal versus he opimized pariioning algorihms. The experimenal seup is described in Secion 5. However we repeaed his experimen 100 imes insead of 50 because of is very high variance.. This experimen is only run on English ex since i has no eec on random ex. Boh cases uses he original vericaion mehod, no he hierarchical one. As i can be seen, he achieved improvemens are especially noiceable in he inermediae range of errors Fig. 3. The opimized (solid line) and he normal spliing (dashed), for m = 10 and 30 on English ex. 6

7 5 Experimenal Comparison In his secion we experimenally compare he old and new algorihms agains he fases algorihms we are aware of. Since we compare only he fases algorihms, we leave aside [8,5,14,12,4,11,15] and many ohers, which are no compeiive in he range of parameers we sudy here. All he experimenal resuls were obained on a Sun UlraSparc-1 running Solaris 2.5.1, wih 32 Mb of RAM, which is a 32-bi machine. We esed random ex wih = 32, and lower-case English ex. The paerns were randomly seleced from he ex (a word beginnings in he English case). Excep when oherwise saed, each daa poin was obained by averaging he Unix's user ime over 50 rials on 10 megabyes of ex. We presen all he imes in seconds. The algorihms included in his comparison are: BYP [3] (i.e. he original version of his algorihm), BYN [2] and Myers [6] (he oher fases algorihms, based on bi-parallelism), and Ours (our modicaion o BYP wih hierarchical vericaion). The code is from he auhors in all cases. On English ex we add wo exra algorihms: Agrep [13] (he fases nown approximae search sofware), and a version of our algorihm ha includes he spliing opimizaion. On English ex he code \Ours" corresponds o our algorihm wih hierarchical vericaion and spliing opimizaion, while \Ours/NO" shows hierarchical vericaion and no spliing opimizaion. The algorihms included in his comparison are: Agrep [13] (he fases nown approximae search sofware, only for English ex), BYP [3] (i.e. he original version of his algorihm), BYN [2] and Myers [6] (he oher fases algorihms, based on bi-parallelism), Ours (our modicaion o BYP wih boh improvemens) and Ours/NO (he same, including he hierarchical vericaion bu no he spliing opimizaion). The spliing opimizaion is shown only for English ex, of course. The code is from he auhors in all cases. As seen in Figure 4, for = 32 he new algorihm is more ecien han any oher for < 1=2, while for English ex i is he fases for < 1=3. Noice ha alhough Agrep is normally faser han BYP (i.e. he original version of his echnique), we are faser han Agrep wih he hierarchical vericaion, and he spliing opimizaion improves a lile over his. 6 Conclusions and Fuure Wor We modied he fases nown algorihm for approximae sring maching wih he use of an improved echnique o verify poenial maches and a new 7

8 Ours Ours/NO BYP Agrep BYN Myers Fig. 4. Experimenal resuls for random (lef) and English ex (righ). From op o boom m =10, 20 and 30. opimizaion echnique for biased exs. As a resul, he area of applicabiliy of he original algorihm is enlarged o spread almos all he ineresing cases in approximae sring searching, where i is sill he fases algorihm. 8

9 We are woring on beer cos funcions for he spliing opimizaion echnique. We also plan o sudy he online eec of spliing he paern in more han + 1 pieces (so ha more han one piece has o mach), as suggesed in [9] for oine searching. References [1] R. Baeza-Yaes and G. Navarro. A faser algorihm for approximae sring maching. In Proc. CPM'96, LNCS 1075, pages 1{23, [2] R. Baeza-Yaes and G. Navarro. Faser approximae sring maching. Algorihmica, To appear. [3] R. Baeza-Yaes and C. Perleberg. Fas and pracical approximae paern maching. Informaion Processing Leers, 59:21{27, [4] W. Chang and J. Lampe. Theoreical and empirical comparisons of approximae sring maching algorihms. In Proc. CPM'92, LNCS 644, [5] G. Landau and U. Vishin. Fas parallel and serial approximae sring maching. Journal of Algorihms, 10:157{169, [6] G. Myers. A fas bi-vecor algorihm for approximae paern maching based on dynamic progamming. In Proc. CPM'98, New Jersey, July To appear. [7] G. Navarro and R. Baeza-Yaes. Analysis for algorihm engineering: improving an algorihm for approximae sring maching. Submied, [8] P. Sellers. The heory and compuaion of evoluionary disances: paern recogniion. Journal of Algorihms, 1:359{373, [9] F. Shi. Fas approximae sring maching wih q-blocs sequences. In Proc. WSP'96, pages 257{271. Carleon Universiy Press, [10] D. Sunday. A very fas subsring search algorihm. Communicaions of he ACM, 33(8):132{142, Augus [11] E. Suinen and J. Tarhio. On using q-gram locaions in approximae sring maching. In Proc. ESA'95, LNCS 979. Springer-Verlag, [12] Eso Uonen. Finding approximae paerns in srings. Journal of Algorihms, 6:132{137, [13] S. Wu and U. Manber. Agrep { a fas approximae paern-maching ool. In Proc. of USENIX Technical Conference, pages 153{162, [14] S. Wu and U. Manber. Fas ex searching allowing errors. Communicaions of he ACM, 35(10):83{91, Ocober [15] S. Wu, U. Manber, and E. Myers. A sub-quadraic algorihm for approximae limied expression maching. Algorihmica, 15(1):50{67,