SUBSTRING SEARCH BBM ALGORITHMS TODAY DEPT. OF COMPUTER ENGINEERING ERKUT ERDEM. Apr. 29, Substring search applications.

Transcription

1 M 22 - LGORITHMS TODY Substrng search DPT. OF OMPUTR NGINRING RKUT RDM rute force Knuth-Morrs-Pratt oyer-moore Rabn-Karp SUSTRING SRH pr. 29, 214 cknowledgement: The course sldes are adapted from the sldes prepared by R. Sedgewck and K. Wayne of Prnceton Unversty. Substrng search Substrng search applcatons Goal. Fnd pattern of length M n a text of length N. Goal. Fnd pattern of length M n a text of length N. typcally N >> M typcally N >> M pattern N D L text I N H Y S T K N D L I N pattern N D L text I N H Y S T K N match Substrng search D L I N match Substrng search omputer forenscs. Search memory or dsk for sgnatures, e.g., all URLs or RS keys that the user has entered

2 Substrng search applcatons Substrng search applcatons Goal. Fnd pattern of length M n a text of length N. pattern text N D L typcally N >> M I N H Y S T K N D L I N lectronc survellance. Need to montor all nternet traffc. (securty) No way (prvacy) match Identfy patterns ndcatve of spam. PROFITS LS W1GHT There s no catch. Ths s a one-tme malng. Ths message s sent n complance wth spam regulatons. 5 TTK T DWN substrng search machne found Well, we re manly nterested n TTK T DWN OK. uld a machne that ust looks for that. Substrng search applcatons Screen scrapng: Java mplementaton Screen scrapng. xtract relevant data from web page. x. Fnd strng delmted by <b> and </b> after frst occurrence of pattern Last Trade:. <tr> <td class= "yfnc_tablehead1" wdth= "48%"> Last Trade: </td> <td class= "yfnc_tabledata1"> <bg><b>452.92</b></bg> </td></tr> <td class= "yfnc_tablehead1" wdth= "48%"> Trade Tme: </td> <td class= "yfnc_tabledata1">... Java lbrary. The ndexof() method n Java's strng lbrary returns the ndex of the frst occurrence of a gven strng, startng at a gven offset. publc class StockQuote publc statc vod man(strng[] args) Strng name = " In n = new In(name + args[]); Strng text = n.readll(); nt start = text.ndexof("last Trade:", ); nt from = text.ndexof("<b>", start); nt to = text.ndexof("</b>", from); Strng prce = text.substrng(from + 3, to); StdOut.prntln(prce); % ava StockQuote goog % ava StockQuote msft

3 rute force Knuth-Morrs-Pratt oyer-moore Rabn-Karp SUSTRING SRH rute-force substrng search heck for pattern startng at each text poston D R 2 2 R pat 1 1 R entres n red are msmatches R 3 3 R entres n gray are for reference only R entres n black 5 5 match the text R 4 1 R return when s M match 1 rute-force substrng search: Java mplementaton heck for pattern startng at each text poston D R D R 5 5 D R publc statc nt search(strng pat, Strng ) nt M = pat.length(); nt N =.length(); for (nt = ; <= N - M; ++) nt ; for ( = ; < M; ++) f (.chart(+) = pat.chart()) break; f ( == M) return ; ndex n text where pattern starts return N; not found rute-force substrng search: worst case rute-force algorthm can be slow f text and pattern are repettve Worst case. ~ M N char compares pat rute-force substrng search (worst case) match 11 12

4 ackup rute-force substrng search: alternate mplementaton In many applcatons, we want to avod backup n text stream. Treat nput as stream of data. bstract model: standard nput. TTK T DWN substrng search machne found rute-force algorthm needs backup for every msmatch. matched chars msmatch backup pproach 1. Mantan buffer of last M characters. pproach 2. Stay tuned. shft pattern rght one poston 13 Same sequence of char compares as prevous mplementaton. ponts to end of sequence of already-matched chars n text. stores number of already-matched chars (end of sequence n pattern) D R 7 3 D R 5 D R publc statc nt search(strng pat, Strng ) nt, N =.length(); nt, M = pat.length(); for ( =, = ; < N && < M; ++) f (.chart() == pat.chart()) ++; else -= ; = ; f ( == M) return - M; else return N; backup 14 lgorthmc challenges n substrng search rute-force s not always good enough. Theoretcal challenge. Lnear-tme guarantee. Practcal challenge. vod backup n text stream. fundamental algorthmc problem often no room or tme to save text rute force Knuth-Morrs-Pratt oyer-moore Rabn-Karp SUSTRING SRH Now s the tme for all people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for many good people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for a lot of good people to come to the ad of ther party. Now s the tme for all of the good people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for each good person to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for all good Republcans to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for many or all good people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for all good Democrats to come to the ad of ther party. Now s the tme for all people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for many good people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for a lot of good people to come to the ad of ther party. Now s the tme for all of the good people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther attack at dawn party. Now s the tme for each person to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for all good Republcans to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for many or all good people to come to the ad of ther party. Now s the tme for all good people to come to the ad of ther party. Now s the tme for all good Democrats to come to the ad of ther party. 15

5 Knuth-Morrs-Pratt substrng search Intuton. Suppose we are searchng n text for pattern. Suppose we match 5 chars n pattern, wth msmatch on th char. We know prevous chars n text are. Don't need to back up text ponter text after msmatch on sxth char brute-force backs pattern up to try ths and ths and ths and ths and ths but no backup s needed assumng, alphabet Knuth-Morrs-Pratt algorthm. lever method to always avod backup. () Determnstc fnte state automaton (DF) DF s abstract strng-searchng machne. Fnte number of states (ncludng start and halt). xactly one transton for each char n alphabet. ccept f sequence of transtons leads to halt state. nternal representaton pat.chart() graphcal representaton, ,, If n state readng char c: f s halt and accept else move to state dfa[c][] DF smulaton DF smulaton pat.chart() pat.chart(),, , ,, 19, 2

6 DF smulaton DF smulaton pat.chart() pat.chart(),, , ,, 21, 22 DF smulaton DF smulaton pat.chart() pat.chart(),, , ,, 23, 24

7 DF smulaton DF smulaton pat.chart() pat.chart(),, , ,, 25, 2 DF smulaton DF smulaton pat.chart() pat.chart(),, , ,, 27, 28

8 DF smulaton DF smulaton pat.chart() pat.chart(),, , ,, 29, 3 DF smulaton DF smulaton pat.chart() pat.chart(),, , , substrng found, 31, 32

9 Interpretaton of Knuth-Morrs-Pratt DF Q. What s nterpretaton of DF state after readng n []?. State = number of characters n pattern that have been matched. x. DF s n state 3 after readng n [..]., 7 8 suffx of text[..] pat prefx of pat[] , length of longest prefx of pat[] that s a suffx of [..] Knuth-Morrs-Pratt substrng search: Java mplementaton Key dfferences from brute-force mplementaton. Need to precompute dfa[][] from pattern. Text ponter never decrements. publc nt search(strng ) nt,, N =.length(); for ( =, = ; < N && < M; ++) = dfa[.chart()][]; f ( == M) return - M; else return N; Runnng tme. Smulate DF on text: at most N character accesses. uld DF: how to do effcently? [warnng: trcky algorthm ahead] no backup, Knuth-Morrs-Pratt substrng search: Java mplementaton Key dfferences from brute-force mplementaton. Need to precompute dfa[][] from pattern. Text ponter never decrements. ould use nput stream. Knuth-Morrs-Pratt constructon Include one state for each character n pattern (plus accept state). publc nt search(in n) nt, ; for ( =, = ; n.smpty() && < M; ++) = dfa[n.readhar()][]; f ( == M) return - M; else return NOT_FOUND; no backup pat.chart() onstructng the DF for KMP substrng search for , ,, 35 3

10 Knuth-Morrs-Pratt constructon Match transton. If n state and next char c == pat.chart(), go to +1. frst characters of pattern next char matches now frst +1 characters of have already been matched pattern have been matched Knuth-Morrs-Pratt constructon Msmatch transton: back up f c = pat.chart(). pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for, Knuth-Morrs-Pratt constructon Msmatch transton: back up f c = pat.chart(). Knuth-Morrs-Pratt constructon Msmatch transton: back up f c = pat.chart(). pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, , 39 4

11 Knuth-Morrs-Pratt constructon Msmatch transton: back up f c = pat.chart(). Knuth-Morrs-Pratt constructon Msmatch transton: back up f c = pat.chart(). pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, , , 41, 42 Knuth-Morrs-Pratt constructon Knuth-Morrs-Pratt constructon Msmatch transton: back up f c = pat.chart(). pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, , ,, 43, 44

12 How to buld DF from pattern? Include one state for each character n pattern (plus accept state). How to buld DF from pattern? Match transton. If n state and next char c == pat.chart(), go to +1. frst characters of pattern have already been matched next char matches now frst +1 characters of pattern have been matched pat.chart() dfa[][] pat.chart() dfa[][] How to buld DF from pattern? How to buld DF from pattern? Msmatch transton. If n state and next char c = pat.chart(), then the last -1 characters of nput are pat[1..-1], followed by c. To compute dfa[c][]: Smulate pat[1..-1] on DF and take transton c. Runnng tme. Seems to requre steps. stll under constructon () Msmatch transton. If n state and next char c = pat.chart(), then the last -1 characters of nput are pat[1..-1], followed by c. state X To compute dfa[c][]: Smulate pat[1..-1] on DF and take transton c. Runnng tme. Takes only constant tme f we mantan state X. x. dfa[''][5] = 1; dfa[''][5] = 4 smulate ; take transton '' = dfa[''][3] smulate ; take transton '' = dfa[''][3] pat.chart() x. dfa[''][5] = 1; dfa[''][5] = 4; from state X, take transton '' = dfa[''][x] from state X, take transton '' = dfa[''][x] X'= from state X, take transton '' = dfa[''][x], smulaton of, X , ,, 47, 48

13 Knuth-Morrs-Pratt constructon (n lnear tme) Include one state for each character n pattern (plus accept state). Knuth-Morrs-Pratt constructon (n lnear tme) Match transton. For each state, dfa[pat.chart()][] = +1. frst characters of pattern now frst +1 characters of have already been matched pattern have been matched pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For state and char c = pat.chart(), set dfa[c][] =. Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For each state and char c = pat.chart(), set dfa[c][] = dfa[c][x]; then update X = dfa[pat.chart()][x]. X = smulaton of empty strng pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, X 51 52

14 Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For each state and char c = pat.chart(), set dfa[c][] = dfa[c][x]; then update X = dfa[pat.chart()][x]. Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For each state and char c = pat.chart(), set dfa[c][] = dfa[c][x]; then update X = dfa[pat.chart()][x]. X = smulaton of X = smulaton of pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, , X X, Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For each state and char c = pat.chart(), set dfa[c][] = dfa[c][x]; then update X = dfa[pat.chart()][x]. Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For each state and char c = pat.chart(), set dfa[c][] = dfa[c][x]; then update X = dfa[pat.chart()][x]. X = smulaton of X = smulaton of pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, X, X,, 55, 5

15 Knuth-Morrs-Pratt constructon (n lnear tme) Knuth-Morrs-Pratt constructon (n lnear tme) Msmatch transton. For each state and char c = pat.chart(), set dfa[c][] = dfa[c][x]; then update X = dfa[pat.chart()][x]. X = smulaton of pat.chart() pat.chart() onstructng the DF for KMP substrng search for onstructng the DF for KMP substrng search for,, X , ,, 57, 58 onstructng the DF for KMP substrng search: Java mplementaton For each state : opy dfa[][x] to dfa[][] for msmatch case. Set dfa[pat.chart()][] to +1 for match case. Update X. publc KMP(Strng pat) ths.pat = pat; M = pat.length(); dfa = new nt[r][m]; dfa[pat.chart()][] = 1; for (nt X =, = 1; < M; ++) for (nt c = ; c < R; c++) dfa[c][] = dfa[c][x]; copy msmatch cases dfa[pat.chart()][] = +1; set match case X = dfa[pat.chart()][x]; update restart state Runnng tme. M character accesses (but space proportonal to R M). KMP substrng search analyss Proposton. KMP substrng search accesses no more than M + N chars to search for a pattern of length M n a text of length N. Pf. ach pattern char accessed once when constructng the DF; each text char accessed once (n the worst case) when smulatng the DF. Proposton. KMP constructs dfa[][] n tme and space proportonal to R M. Larger alphabets. Improved verson of KMP constructs nfa[] n tme and space proportonal to M KMP NF for 59

16 Knuth-Morrs-Pratt: bref hstory Independently dscovered by two theoretcans and a hacker. - Knuth: nspred by esoterc theorem, dscovered lnear-tme algorthm - Pratt: made runnng tme ndependent of alphabet sze - Morrs: bult a text edtor for the D 4 computer Theory meets practce. rute force Knuth-Morrs-Pratt oyer-moore Rabn-Karp SUSTRING SRH SIM J. OMPUT. Vol., No. 2, June 1977 FST PTTRN MTHING IN STRINGS* DONLD. KNUTHf, JMS H. MORRIS, JR.:l: ND VUGHN R. PRTT bstract. n algorthm s presented whch fnds all occurrences of one. gven strng wthn another, n runnng tme proportonal to the sum of the lengths of the strngs. The constant of proportonalty s low enough to make ths algorthm of practcal use, and the procedure can also be extended to deal wth some more general pattern-matchng problems. theoretcal applcaton of the algorthm shows that the set of concatenatons of even palndromes,.e., the language can*, can be recognzed n lnear tme. Other algorthms whch run even faster on the average are also consdered. Don Knuth Jm Morrs Vaughan Pratt 1 oyer-moore: msmatched character heurstc Intuton. Scan characters n pattern from rght to left. an skp as many as M text chars when fndng one not n the pattern. oyer-moore: msmatched character heurstc Q. How much to skp? ase 1. Msmatch character not n pattern. before text F I N D I N H Y S T K N D L I N 5 N D L pattern 5 5 N D L 11 4 N D L 15 N D L pat after T L N D L return = 15 Msmatched character heurstc for rght-to-left (oyer-moore) substrng search pat T L N D L msmatch character 'T' not n pattern: ncrement one character beyond 'T' 3 4

17 oyer-moore: msmatched character heurstc Q. How much to skp? ase 2a. Msmatch character n pattern. oyer-moore: msmatched character heurstc Q. How much to skp? ase 2b. Msmatch character n pattern (but heurstc no help). before before pat N L N D L pat L N D L after algned wth rghtmost? pat N L N D L pat L N D L msmatch character 'N' n pattern: algn text 'N' wth rghtmost pattern 'N' msmatch character '' n pattern: algn text '' wth rghtmost pattern ''? 5 oyer-moore: msmatched character heurstc Q. How much to skp? ase 2b. Msmatch character n pattern (but heurstc no help). oyer-moore: msmatched character heurstc Q. How much to skp?. Precompute ndex of rghtmost occurrence of character c n pattern (-1 f character not n pattern). before pat after pat L N D L L N D L rght = new nt[r]; for (nt c = ; c < R; c++) rght[c] = -1; for (nt = ; < M; ++) rght[pat.chart()] = ; N D L c rght[c] D L M N oyer-moore skp table computaton msmatch character '' n pattern: ncrement by 1 7 8

18 oyer-moore: Java mplementaton oyer-moore: analyss publc nt search(strng ) nt N =.length(); nt M = pat.length(); nt skp; for (nt = ; <= N-M; += skp) skp = ; for (nt = M-1; >= ; --) f (pat.chart() =.chart(+)) skp = Math.max(1, - rght[.chart(+)]); break; n case other term s nonpostve f (skp == ) return ; return N; compute skp value match Property. Substrng search wth the oyer-moore msmatched character heurstc takes about ~ N / M character compares to search for a pattern of length M n a text of length N. sublnear Worst-case. an be as bad as ~ M N. skp pat oyer-moore-horspool substrng search (worst case) oyer-moore varant. an mprove worst case to ~ 3 N by addng a KMP-lke rule to guard aganst repettve patterns. 9 7 rute force Knuth-Morrs-Pratt oyer-moore Rabn-Karp SUSTRING SRH Rabn-Karp fngerprnt search asc dea = modular hashng. ompute a hash of pattern characters to M - 1. For each, compute a hash of text characters to M If pattern hash = text substrng hash, check for a match. pat.chart() % 997 = 13.chart() % 997 = % 997 = % 997 = % 997 = % 997 = % 997 = 929 match return = % 997 = 13 ass for Rabn-Karp substrng search 72

19 ffcently computng the hash functon Modular hash functon. Usng the notaton t for.chart(), we wsh to compute x = t R M-1 + t+1 R M t+m-1 R (mod Q) Intuton. M-dgt, base-r nteger, modulo Q. Horner's method. Lnear-tme method to evaluate degree-m polynomal. ffcently computng the hash functon hallenge. How to effcently compute x+1 gven that we know x. x = t R M 1 + t+1 R M t+m 1 R x+1 = t+1 R M 1 + t+2 R M t+m R Key property. an update hash functon n constant tme x+1 = ( x t R M 1 ) R + t +M current value subtract leadng dgt multply by radx add new tralng dgt (can precompute R M 2 ) pat.chart() % 997 = % 997 = (2*1 + ) % 997 = % 997 = (2*1 + 5) % 997 = % 997 = (25*1 + 3) % 997 = 59 R % 997 = (59*1 + 5) % 997 = 13 Q // ompute hash for M-dgt key prvate long hash(strng key, nt M) long h = ; for (nt = ; < M; ++) h = (R * h + key.chart()) % Q; return h; current value text new value current value subtract leadng dgt * 1 multply by radx add new tralng dgt new value Rabn-Karp substrng search example Rabn-Karp: Java mplementaton % 997 = 3 Q % 997 = (3*1 + 1) % 997 = % 997 = (31*1 + 4) % 997 = % 997 = (314*1 + 1) % 997 = % 997 = (15*1 + 5) % 997 = 58 RM R % 997 = ((58 + 3*(997-3))*1 + 9) % 997 = % 997 = ((21 + 1*(997-3))*1 + 2) % 997 = % 997 = (( *(997-3))*1 + ) % 997 = % 997 = (( *(997-3))*1 + 5) % 997 = 442 match % 997 = (( *(997-3))*1 + 3) % 997 = return -M+1 = % 997 = (( *(997-3))*1 + 5) % 997 = 13 publc class RabnKarp prvate long pathash; prvate nt M; prvate long Q; prvate nt R; prvate long RM; publc RabnKarp(Strng pat) M = pat.length(); R = 25; Q = longrandomprme(); RM = 1; for (nt = 1; <= M-1; ++) RM = (R * RM) % Q; pathash = hash(pat, M); // pattern hash value // pattern length // modulus // radx // R^(M-1) % Q prvate long hash(strng key, nt M) /* as before */ publc nt search(strng ) /* see next slde */ a large prme (but avod overflow) precompute R M 1 (mod Q) 75 7

20 Rabn-Karp: Java mplementaton (contnued) Monte arlo verson. Return match f hash match. publc nt search(strng ) check for hash collson nt N =.length(); usng rollng hash functon nt Hash = hash(, M); f (pathash == Hash) return ; for (nt = M; < N; ++) Hash = (Hash + Q - RM*.chart(-M) % Q) % Q; Hash = (Hash*R +.chart()) % Q; f (pathash == Hash) return - M + 1; return N; Las Vegas verson. heck for substrng match f hash match; contnue search f false collson. Rabn-Karp analyss Theory. If Q s a suffcently large random prme (about M N 2 ), then the probablty of a false collson s about 1 / N. Practce. hoose Q to be a large prme (but not so large as to cause overflow). Under reasonable assumptons, probablty of a collson s about 1 / Q. Monte arlo verson. lways runs n lnear tme. xtremely lkely to return correct answer (but not always). Las Vegas verson. lways returns correct answer. xtremely lkely to run n lnear tme (but worst case s M N) Rabn-Karp fngerprnt search dvantages. xtends to 2d patterns. xtends to fndng multple patterns. Dsadvantages. rthmetc ops slower than char compares. Las Vegas verson requres backup. Poor worst-case guarantee. Substrng search cost summary ost of searchng for an M-character pattern n an N-character text. algorthm verson operaton count guarantee typcal backup n nput? correct? brute force M N 1.1 N yes yes 1 Knuth-Morrs-Pratt full DF (lgorthm 5. ) msmatch transtons only extra space 2 N 1.1 N no yes MR 3 N 1.1 N no yes M full algorthm 3 N N / M yes yes R oyer-moore msmatched char heurstc only (lgorthm 5.7 ) M N N / M yes yes R Rabn-Karp Monte arlo (lgorthm 5.8 ) 7 N 7 N no yes 1 Las Vegas 7 N 7 N no yes yes 1 probablstc guarantee, wth unform hash functon 79 8