Multiple patterns. Why? Algorithms. Aho Corasick (AC) Text Algorithms (4AP) Lecture 3.2: Multiple pattern matching. Jaak Vilo 2008 fall

Transcription

1 Multipl pattrn Txt Algoritm (AP) Lctur.: Multipl pattrn matcing Jaak Vilo 8 fall S {P} Jaak Vilo MTAT..9 Txt Algoritm Wy? Multipl pattrn Higligt multipl diffrnt arc word on t pag Viru dtction filtr for viru ignatur Spam filtr Scannr in compilr nd to arc for multipl kyword Filtr out top word or diallowd word Map many DNA primr on t gnom Algoritm Ao Coraick (arc for multipl word) Gnralization of Knut Morri Pratt Commntz Waltr Gnralization i of Boyr Moor & AC Wu and Manbr improvmnt ovr C W Ao Coraick (AC) Alfrd V. Ao and Margart J. Coraick (Bll Lab, Murray Hill, NJ) Efficint tring matcing. An aid to bibliograpic arc. Communication of t ACM, Volum 8, Iu 6, p (Jun 97) ACM:DOI PDF ABSTRACT Ti papr dcrib a impl, fficint algoritm to locat all occurrnc of any of a finit numbr of kyword in a tring of txt. T algoritm conit of contructing a finit tat pattrn matcing macin from t kyword and tn uing t pattrn matcing macin to proc t txt tring in a ingl pa. Contruction of t pattrn matcing macin tak tim proportional to t um of t lngt of t kyword. T numbr of tat tranition mad by t pattrn matcing macin in procing t txt tring i indpndnt of t numbr of kyword. T algoritm a bn ud to improv t pd of a library bibliograpic arc program by a factor of to. Gnralization of KMP for many pattrn Txt S lik bfor. St of pattrn P = { P,.., P k } Total lngt P = m = Σ i=..k m i Problm: find all occurrnc of any of t P i P from S

2 Ida Crat an automaton from all pattrn Matc t automaton PATRICIA tri D. R. Morrion, "PATRICIA: Practical Algoritm To Rtriv Information Codd In Alpanumric", Journal of t ACM (968). Abtract PATRICIA i an algoritm wic provid a flxibl man of toring, indxing, and rtriving information in a larg fil, wic i conomical of indx pac and of rindxing tim. It do not rquir rarrangmnt of txt or indx a nw matrial i addd. It rquir a minimum rtriction of format of txt and of ky; it i xtrmly flxibl in t varity of ky it will rpond to. It rtriv information in rpon to ky furnid by t ur wit a quantity of computation wic a a bound wic dpnd linarly on t lngt of ky and t numbr of tir propr occurrnc and i otrwi indpndnt of t iz of t library. It a bn implmntd in vral variation a FORTRAN program for t CDC 6, utilizing dik fil torag of txt. It a bn applid to vral larg information rtrival problm and will b applid to otr. ACM:DOI PDF Word tri a good data tructur to rprnt a t of word (.g. a dictionary). tri (data tructur) Dfinition: A tr for toring tring in wic tr i on nod for vry commonprffix prffix. T tring ar tord in xtra laf nod. S alo digital tr, digital arc tr, dirctd acyclic word grap, compact DAWG, Patricia tr, uffix tr. Not: T nam com from rtrival and i pronouncd, "tr." To tt for a word p, only O( p ) tim i ud no mattr ow many word ar in t dictionary... Tri for P={,, i, r} Tri for P={,, i, r} i r 7 6 8

3 How to arc for word lik, ila, i. Do t occur in t tri? r 8 i 6 7 W crat an automaton M P for a t of tring P. Finit tat macin: rad acaractr from txt, and cang t tat of t automaton bad on t tat tranition... Main link: goto[j,c] rad a caractr c from txt and go from a tat j to tat goto[j,c]. If tr ar no goto[j,c] link on caractr c from tat j, u fail[j]. Rport t output. Rport all word tat av bn found in tat j. 9 How to arc for word lik, ila, i. Do t occur in t tri? AC matcing goto[,i] = 6. ; fail[7] =, fail[8] =, fail[]=. Output tabl tat output[j], 7 i 9 r 9 r 8 NOT {, } i 6 7 Input: Txt S[..n] and an AC automaton M for pattrn t P Output: Occurrnc of pattrn from P in S (lat poition). tat =. for i=..n do. wil (goto[tat,s[i]]== ) and (fail[tat]!=tat) do. tat = fail[tat]. tat = goto[tat,s[i]] 6. if ( output[tat] not mpty ) 7. tn rport matc output[tat] at poition i Algoritm Ao Coraick prprocing I (TRIE) Input: P = { P,..., P k } Output: goto[] and partial output[] Aum: output() i mpty wn a tat i cratd; goto[,a] i not dfind. procdur ntr(a,..., a m ) /* P i = a,..., a m */ bgin. =; j=;. wil goto[,a j ] do // follow xiting pat. = goto[,a j ];. j = j+;. for p=j to m do // add nw pat (tat) 6. nw = nw+ ; 7. goto[,a p ] = nw; 8. = nw; 9. output[] = a,..., a m nd bgin. nw =. for i= to k do ntr( P i ). for a Σ do. if goto[,a] = tn goto[,a] = ; nd Prprocing II for AC (FAIL) quu = for a Σ do if goto[,a] tn nquu( quu, goto[,a] ) fail[ goto[,a] ] = wil quu r = tak( quu ) for a Σ do if goto[r,a] tn = goto[ r, a ] nquu( quu, ) // bradt firt arc tat = fail[r] wil goto[tat,a] = do tat = fail[tat] fail[] = goto[tat,a] output[] = output[] + output[ fail[] ]

4 Corrctn AC algoritm i corrct... Lt tring t "point" from initial tat to tat j. Mut ow tat fail[j] point to longt uffix tat i alo a prfix of om word in P. Look at t articl... AC matcing tim complxity Torm For matcing t M P on txt S, S =n, l tan n tranition witin M ar mad. Proof Compar to KMP. Tr i at mot n goto tp. Cannot b mor tan n Fail tp. In total tr can b l tan n tranition in M. Individual nod (goto) Full tabl Lit Binary arc tr(?) Som otr indx? AC tougt Scal for many tring imultanouly. For vry many pattrn arc tim (of grp) improv(??) S Wu Manbr articl Wn k grow, tn mor fail[] tranition ar mad (wy?) But alway l tan n. If all goto[j,a] ar indxd in an array, tn t iz i M P * Σ, and t running tim of AC i O(n). Wn k and c ar big, on can u lit or tr for toring tranition function. Tn, O(n log(min(k,c)) ). Advancd AC Prcalculat t nxt tat tranition corrctly for vry poibl caractr in alpabt Can b good for ort pattrn Commntz Waltr Gnralization of Boyr Moor for multipl qunc arc Bat Commntz Waltr A String Matcing Algoritm Fat on t Avrag Procding of t 6t Colloquium, on Automata, Languag and Programming. Lctur Not In Computr Scinc; Vol. 7, 979. pp. 8, Springr Vrlag

5 C W dcription Ao and Coraick [AC7] prntd a linar tim algoritm for ti problm, bad on an automata approac. Ti algoritm rv a t bai for t UNIX tool fgrp. A linar tim algoritm i optimal in t wort ca, but a t rgular tring arcing algoritm by Boyr and Moor [BM77] dmontratd, it i poibl to actually kip a larg portion of t txt wil arcing, lading to fatr tan linar algoritm in t avrag ca. Commntz Waltr [CW79] Commntz Waltr [CW79] prntd an algoritm for t multi pattrn matcing problm tat combin t Boyr Moor tcniqu wit t Ao Coraick algoritm. T Commntz Waltr algoritm i ubtantially fatr tan t Ao Coraick algoritm in practic. Hum [Hu9] dignd a tool calld gr bad on ti algoritm, and vrion. of fgrp by t GNU projct [Ha9] i uing it. Baza Yat [Ba89] alo gav an algoritm tat combin t Boyr Moor Horpool algoritm [Ho8] (wic i a ligt variation of t claical Boyr Moor algoritm) wit t Ao Coraick algoritm. Ida of C W Build a backward tri of all kyword Matc from t nd until mimatc... Dtrmin t ift bad on t combination of uritic Wat ar t poibl limitation for C W? Many pattrn, mall alpabt minimal kip Wat can b don diffrntly? Wu Manbr Wu S., and U. Manbr, "A Fat Algoritm for Multi Pattrn Sarcing," Tcnical Rport TR 9 7, Dpartmnt of Computr Scinc, Univrity of Arizona (May 99). Citr: ttp://citr.it.pu.du/wu9fat.tml [Potcript] W prnt a diffrnt approac tat alo u t ida of Boyr and Moor. Ouralgoritm i quitimpl, andtmainnginofit ngin it i givn latr in t papr. An arlir vrion of ti algoritm wa part of t cond vrion of agrp [WM9a, WM9b], altoug t algoritm a not bn dicud in [WM9b] and only brifly in [WM9a]. T currnt vrion i ud in glimp [MW9]. T dign of t algoritm concntrat on typical arc ratr tan on wort ca bavior. Ti allow u to mak om nginring dciion tat w bliv ar crucial to making t algoritm ignificantly fatr tan otr algoritm in practic. Ky ida Main problm wit Boyr Moor and many pattrn i tat, t mor tr ar pattrn, t ortr bcom t poibl ift... Wu and Manbr: cck vral caractr imultanouly, i.. incra t alpabt.

6 Multipl Sift AND Intad of looking at caractr from t txt on by on, w conidr tm in block of iz B. log c M; in practic, w u itr B = or B =. T SHIFT tabl play t am rol a in t rgular Boyr Moor algoritm, xcpt tat it dtrmin t ift bad on t lat B caractr ratr tan jut on caractr. P={P, P, P, P}. Gnraliz Sift AND Bit = Start = P P P P Matc= Wat about otr poibiliti? tring Gnralization of Factor arc (?) A poibility for a projct? tring tring 6

7 tring Factor Oracl Factor Oracl: af ift Factor Oracl: Sift to matc prfix of P? Factor oracl Contruction of factor Oracl 7

8 Factor oracl Allauzn, C., Crocmor, M., and Raffinot, M Factor Oracl: A Nw Structur for Pattrn Matcing. In Procding of t 6t Confrnc on Currnt Trnd in tory and Practic of informatic on tory and Practic of informatic (Novmbr 7 Dcmbr, 999). J. Pavlka, G. Tl, and M. Bartok, Ed. Lctur Not In Computr Scinc, vol. 7. Springr Vrlag, London, 9. ttp://portal.acm.org/citation.cfm?id= &coll=guide&dl=g UIDE&CFID=9&CFTOKEN=686# ttp://www igm.univ mlv.fr/~allauzn/work/ofm.p 8