5.3 Substring Search

Transcription

1 5.3 Substring Search brute force Knuth-Morris-Pratt Boyer-Moore Rabin-Karp lgorithms, 4 th Edition Robert Sedgewick and Kevin Wayne opyright pril 5, :29:31 PM

2 Substring search Goal. Find pattern of length M in a text of length N. typically N >> M pattern N E E D L E text I N H Y S T K N E E D L E I N match Substring search omputer forensics. Search memory or disk for signatures, e.g., all URLs or RS keys that the user has entered. 2

3 pplications Parsers. Spam filters. Digital libraries. Screen scrapers. Word processors. Web search engines. Electronic surveillance. Natural language processing. omputational molecular biology. FBIs Digital ollection System Feature detection in digitized images.... 3

4 pplication: spam filtering Identify patterns indicative of spam. PROFITS L0SE WE1GHT herbal Viagra There is no catch. L0W M0RTGGE RTES This is a one-time mailing. This message is sent in compliance with spam regulations. 4

5 pplication: electronic surveillance Need to monitor all internet traffic. (security) No way! (privacy) Well, we re mainly interested in TTK T DWN OK. Build a machine that just looks for that. TTK T DWN substring search machine found 5

6 pplication: screen scraping Goal. Extract relevant data from web page. Ex. Find string delimited by <b> and </b> after first occurrence of pattern Last Trade:. <tr> <td class= "yfnc_tablehead1" width= "48%"> Last Trade: </td> <td class= "yfnc_tabledata1"> <big><b>452.92</b></big> </td></tr> <td class= "yfnc_tablehead1" width= "48%"> Trade Time: </td> <td class= "yfnc_tabledata1">... 6

7 Screen scraping: Java implementation Java library. The indexof() method in Java's string library returns the index of the first occurrence of a given string, starting at a given offset. public class StockQuote { public static void main(string[] args) { String name = " In in = new In(name + args[0]); String text = in.readll(); int start = text.indexof("last Trade:", 0); int from = text.indexof("<b>", start); int to = text.indexof("</b>", from); String price = text.substring(from + 3, to); StdOut.println(price); } } % java StockQuote goog % java StockQuote msft

8 brute force Knuth-Morris-Pratt Boyer-Moore Rabin-Karp 8

9 Brute-force substring search heck for pattern starting at each text position. i j i+j txt B D B R B R pat B R entries in red are mismatches B R B R entries in gray are for reference only B R entries in black match the text B R B R return i when j is M match 9

10 Brute-force substring search: Java implementation heck for pattern starting at each text position. i = 4, j = B D B R D R public static int search(string pat, String txt) { int M = pat.length(); int N = txt.length(); for (int i = 0; i <= N - M; i++) { int j; for (j = 0; j < M; j++) if (txt.chart(i+j)!= pat.chart(j)) } break; if (j == M) return i; } return N; not found index in text where pattern starts 10

11 Brute-force substring search: worst case Brute-force algorithm can be slow if text and pattern are repetitive. i j i+j txt B B pat B B B B B Brute-force substring search (worst case) Worst case. ~ M N char compares. 11

12 Backup In typical applications, we want to avoid backup in text stream. Treat input as stream of data. bstract model: standard input. TTK T DWN substring search machine found Brute-force algorithm needs backup for every mismatch. matched chars mismatch B B backup B B shift pattern right one position pproach 1. Maintain buffer of size M (build backup into standard input). pproach 2. Stay tuned. 12

13 Brute-force substring search: alternate implementation Same sequence of char compares as previous implementation. i points to end of sequence of already-matched chars in text. j stores number of already-matched chars (end of sequence in pattern). i = 6, j = B D B R D R public static int search(string pat, String txt) { int i, N = txt.length(); int j, M = pat.length(); for (i = 0, j = 0; i < N && j < M; i++) { if (txt.chart(i) == pat.chart(j)) j++; else { i -= j; j = 0; } } if (j == M) return i - M; else return N; } backup 13

14 lgorithmic challenges in substring search Brute-force is often not good enough. Theoretical challenge. Linear-time guarantee. fundamental algorithmic problem Practical challenge. void backup in text stream. often no room or time to save text Now is the time for all people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for many good people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for a lot of good people to come to the aid of their party. Now is the time for all of the good people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for each good person to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for all good Republicans to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for many or all good people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for all good Democrats to come to the aid of their party. Now is the time for all people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for many good people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for a lot of good people to come to the aid of their party. Now is the time for all of the good people to come to the aid of their party. Now is the time for all good people to come to the aid of their attack at dawn party. Now is the time for each person to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for all good Republicans to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for many or all good people to come to the aid of their party. Now is the time for all good people to come to the aid of their party. Now is the time for all good Democrats to come to the aid of their party. 14

15 brute force Knuth-Morris-Pratt Boyer-Moore Rabin-Karp 15

16 Knuth-Morris-Pratt substring search Intuition. Suppose we are searching in text for pattern B. Suppose we match 5 chars in pattern, with mismatch on 6 th char. We know previous 6 chars in text are BB. Don't need to back up text pointer! assuming {, B } alphabet text after mismatch on sixth char brute-force backs up to try this B B B and this and this B and this i B and this pattern B B B but no backup is needed B Text pointer backup in substring searching Knuth-Moris-Pratt algorithm. lever method to always avoid backup. (!) 16

17 Deterministic finite state automaton (DF) DF is abstract string-searching machine. Finite number of states (including start and halt). Exactly one transition for each char in alphabet. ccept if sequence of transitions leads to halt state. internal representation j pat.chart(j) dfa[][j] B B B If in state j reading char c: if j is 6 halt and accept else move to state dfa[c][j] graphical representation B, B 0 1 B 2 3 B B, B, 17

18 KMP substring search: trace read this char in this state go to this state B B B B match: set j to dfa[txt.chart(i)][j] = dfa[pat.chart(j)][j] = j B B B B B B B B B B B B B B B B B mismatch: set j to dfa[txt.chart(i)][j] implies pattern shift to align pat.chart(j) with txt.chart(i+1) B B B B found return i - M = 9 B B B B B B B B B B B B j pat.chart(j) dfa[][j] B i txt.chart(i) j B B Trace of KMP substring search (DF simulation) for B B 18

19 Interpretation of Knuth-Morris-Pratt DF Q. What is interpretation of DF state after reading in txt[i]?. State = number of characters in pattern that have been matched. (length of longest prefix of pat[] that is a suffix of txt[0..i]) Ex. DF is in state 3 after reading in character txt[6]. txt B B B pat B B suffix of text[0..6] prefix of pat[] B, B 0 1 B 2 3 B B, B, 19

20 KMP search: Java implementation Key differences from brute-force implementation. Text pointer i never decrements. Need to precompute dfa[][] from pattern. public int search(string txt) { int i, j, N = txt.length(); for (i = 0, j = 0; i < N && j < M; i++) j = dfa[txt.chart(i)][j]; if (j == M) return i - M; else return N; } no backup Running time. Simulate DF on text: at most N character accesses. Build DF: how to do efficiently? [warning: tricky algorithm ahead] 20

21 KMP search: Java implementation Key differences from brute-force implementation. Text pointer i never decrements. Need to precompute dfa[][] from pattern. ould use input stream. public int search(in in) { int i, j; for (i = 0, j = 0;!in.isEmpty() && j < M; i++) j = dfa[in.readhar()][j]; if (j == M) return i - M; else return NOT_FOUND; } no backup B, B 0 1 B 2 3 B B, B, 21

22 Building DF from pattern: easy case Match transition. then go to state j+1. If in state j and next char c == pat.chart(j), now first j+1 characters of pattern have been matched first j characters of pattern have already been matched next char matches j pat.chart(j) B B B, B 0 1 B 2 3 B B, B, 22

23 Knuth-Morris-Pratt construction Include one state for each character in pattern (plus accept state). j pat.chart(j) dfa[][j] B B B onstructing the DF for KMP substring search for B B 23

24 Knuth-Morris-Pratt construction Match transition. For each state j, dfa[pat.chart(j)][j] = j+1. first j characters of pattern have already been matched now first j+1 characters of pattern have been matched j pat.chart(j) dfa[][j] B B B B 0 1 B onstructing the DF for KMP substring search for B B 24

25 Building DF from pattern: tricky case Mismatch transition. Suppose DF is in state j and next char is c!= pat.chart(j)... B B... B B B B B Brute-force solution: Back up j-1 chars and restart. match mismatch case 1 mismatch case 2 Key idea 1. The last j characters of input are known to be pat[1..j-1], so use the DF itself to find what state (X) it would be in if restarted after backup Key idea 2. Maintain the value of X while constructing DF. Ex. B, X j B 0 1 B 2 3 B B, B, pat[1..j-1] = B B X = 3 dfa[''][5] = dfa[''][x] = 1 dfa['b'][5] = dfa['b'][x] = 4 dfa[''][5] = 6 new X = dfa[''][x] = 0 25

26 Knuth-Morris-Pratt construction j pat.chart(j) dfa[][j] B B B B, j B 0 1 B onstructing the DF for KMP substring search for B B 26

27 Knuth-Morris-Pratt construction Mismatch transition. For each state j and char c!= pat.chart(j), dfa[c][j] = dfa[c][x]; then update X = dfa[pat.chart(j)][x]. X = simulation of empty string j pat.chart(j) dfa[][j] B B B j = 1 pat[1..j-1] = '' X = 0 dfa[''][1] = dfa[''][x] = 1 dfa['b'][1] = 2 dfa[''][1] = dfa[''][x] = 0 new X = dfa['b'][x] = 0 B, j B 0 1 B X onstructing the DF for KMP substring search for B B 27

28 Knuth-Morris-Pratt construction Mismatch transition. For each state j and char c!= pat.chart(j), dfa[c][j] = dfa[c][x]; then update X = dfa[pat.chart(j)][x]. X = simulation of B j pat.chart(j) dfa[][j] B B B j = 2 pat[1..j-1] = 'B' X = 0 dfa[''][2] = 3 dfa['b'][2] = dfa['b'][x] = 0 dfa[''][2] = dfa[''][x] = 0 new X = dfa[''][x] = 1 B, j X B 0 1 B B, onstructing the DF for KMP substring search for B B 28

29 Knuth-Morris-Pratt construction Mismatch transition. For each state j and char c!= pat.chart(j), dfa[c][j] = dfa[c][x]; then update X = dfa[pat.chart(j)][x]. X = simulation of B j pat.chart(j) dfa[][j] B B B j = 3 pat[1..j-1] = 'B' X = 1 dfa[''][3] = dfa[''][x] = 1 dfa['b'][3] = 4 dfa[''][3] = dfa[''][x] = 0 new X = dfa['b'][x] = 2 B, B 0 1 B B, X j onstructing the DF for KMP substring search for B B 29

30 Knuth-Morris-Pratt construction Mismatch transition. For each state j and char c!= pat.chart(j), dfa[c][j] = dfa[c][x]; then update X = dfa[pat.chart(j)][x]. X = simulation of BB j pat.chart(j) dfa[][j] B B B j = 4 pat[1..j-1] = 'BB' X = 2 dfa[''][4] = 5 dfa['b'][4] = dfa['b'][x] = 0 dfa[''][4] = dfa[''][x] = 0 new X = dfa['b'][x] = 3 B, X B 0 1 B B, B, j 30

31 Knuth-Morris-Pratt construction Mismatch transition. For each state j and char c!= pat.chart(j), dfa[c][j] = dfa[c][x]; then update X = dfa[pat.chart(j)][x]. X = simulation of BB j pat.chart(j) dfa[][j] B B B j = 5 pat[1..j-1] = 'BB' X = 3 dfa[''][5] = dfa[''][x] = 1 dfa['b'][5] = dfa['b'][x] = 4 dfa[''][5] = 6 new X = dfa[''][x] = 0 B, X j B 0 1 B 2 3 B B, B, 31

32 onstructing the DF for KMP substring search: Java implementation For each state j: opy dfa[][x] to dfa[][j] for mismatch case. Set dfa[pat.chart(j)][j] to j+1 for match case. Update X. public KMP(String pat) { this.pat = pat; M = pat.length(); dfa = new int[r][m]; dfa[pat.chart(0)][0] = 1; for (int X = 0, j = 1; j < M; j++) { for (int c = 0; c < R; c++) dfa[c][j] = dfa[c][x]; dfa[pat.chart(j)][j] = j+1; X = dfa[pat.chart(j)][x]; } } copy mismatch cases set match case update restart state Running time. M character accesses (but space proportional to R M). 32

33 KMP substring search analysis Proposition. KMP substring search accesses no more than M + N chars to search for a pattern of length M in a text of length N. Pf. Each pattern char accessed once when constructing the DF; each text char accessed once (in the worst case) when simulating the DF. Proposition. KMP constructs dfa[][] in time and space proportional to R M. Larger alphabets. Improved version of KMP constructs nfa[] in time and space proportional to M. 0 1 B 2 3 B

34 Knuth-Morris-Pratt: brief history Independently discovered by two theoreticians and a hacker. - Knuth: inspired by esoteric theorem, discovered linear-time algorithm - Pratt: made running time independent of alphabet size - Morris: built a text editor for the D 6400 computer Theory meets practice. SIM J. OMPUT. Vol. 6, No. 2, June 1977 FST PTTERN MTHING IN STRINGS* DONLD E. KNUTHf, JMES H. MORRIS, JR.:l: ND VUGHN R. PRTT bstract. n algorithm is presented which finds all occurrences of one. given string within another, in running time proportional to the sum of the lengths of the strings. The constant of proportionality is low enough to make this algorithm of practical use, and the procedure can also be extended to deal with some more general pattern-matching problems. theoretical application of the algorithm shows that the set of concatenations of even palindromes, i.e., the language {can}*, can be recognized in linear time. Other algorithms which run even faster on the average are also considered. Don Knuth Jim Morris Vaughan Pratt 34

35 Knuth-Morris-Pratt application string s is a cyclic rotation of t if s and t have the same length and s is a suffix of t followed by a prefix of t. yes ROTTEDSTRING STRINGROTTED yes BBBBBBB BBBBBBB no ROTTEDSTRING GNIRTSDETTOR Problem. Given two strings s and t, design a linear-time algorithm that determines if s is a cyclic rotation of t. Solution. heck that s and t are the same length. Search for s in t + t using KMP. 35

36 brute force Knuth-Morris-Pratt Boyer-Moore Rabin-Karp Robert Boyer J. Strother Moore 36

37 Boyer-Moore: mismatched character heuristic Intuition. Scan characters in pattern from right to left. an skip M text chars when finding one not in the pattern. i text j F I N D I N H Y S T K N E E D L E I N 0 5 N E E D L E pattern 5 5 N E E D L E 11 4 N E E D L E 15 0 N E E D L E return i = 15 Mismatched character heuristic for right-to-left (Boyer-Moore) substring search 37

38 Boyer-Moore: mismatched character heuristic Q. How much to skip?. ompute right[c] = rightmost occurrence of character c in pat. right = new int[r]; for (int c = 0; c < R; c++) right[c] = -1; for (int j = 0; j < M; j++) right[pat.chart(j)] = j; N E E D L E c right[c] B D E L M N Boyer-Moore skip table computation 38

39 Boyer-Moore: mismatched character heuristic Q. How much to skip?. ompute right[c] = rightmost occurrence of character c in pat. basic idea i i+j N L E N E E D L E j could do better with increment i by j - right[ N ] i KMP-like table to line up text with N in pattern N L E N E E D L E reset j to M-1 j Mismatched character heuristic (mismatch in pattern) 39

40 Boyer-Moore: mismatched character heuristic Q. How much to skip?. ompute right[c] = rightmost occurrence of character c in pat. i i+j T L E..... N E E D L E j could do better with i KMP-like table increment i by j T L E..... N E E D L E reset j to M-1 Mismatched character heuristic (mismatch not in pattern) j haracter not in pattern? Set right[c] to

41 Boyer-Moore: mismatched character heuristic Q. How much to skip?. ompute right[c] = rightmost occurrence of character c in pat. heuristic is no help i i+j E L E N E E D L E lining up text with rightmost E would shift pattern left E L E N E E D L E so increment i by 1 N E E D L E reset j to M-1 Mismatched character heuristic (mismatch in pattern) Heuristic no help? Increment i and reset j to M-1 i j j could do better with KMP-like table E L E

42 Boyer-Moore: Java implementation public int search(string txt) { int N = txt.length(); int M = pat.length(); int skip; for (int i = 0; i <= N-M; i += skip) { skip = 0; for (int j = M-1; j >= 0; j--) { if (pat.chart(j)!= txt.chart(i+j)) { skip = Math.max(1, j - right[txt.chart(i+j)]); break; } } if (skip == 0) return i; } return N; } compute skip value match 42

43 Boyer-Moore: analysis Property. Substring search with the Boyer-Moore mismatched character heuristic takes about ~ N / M character compares to search for a pattern of length M in a text of length N. sublinear Worst-case. an be as bad as ~ M N. i skip txt B B B B B B B B B B 0 0 B B B B pat 1 1 B B B B 2 1 B B B B 3 1 B B B B 4 1 B B B B 5 1 B B B B Boyer-Moore-Horspool substring search (worst case) Boyer-Moore variant. an improve worst case to ~ 3 N by adding a KMP-like rule to guard against repetitive patterns. 43

44 brute force Knuth-Morris-Pratt Boyer-Moore Rabin-Karp Michael Rabin, Turing ward '76 and Dick Karp, Turing ward '85 44

45 Rabin-Karp fingerprint search Basic idea = modular hashing. ompute a hash of pattern characters 0 to M - 1. For each i, compute a hash of text characters i to M + i - 1. If pattern hash = text substring hash, check for a match. pat.chart(i) i % 997 = 613 txt.chart(i) i % 997 = % 997 = % 997 = % 997 = % 997 = % 997 = 929 match 6 return i = % 997 = 613 Basis for Rabin-Karp substring search 45

46 Efficiently computing the hash function Modular hash function. Using the notation t i for txt.chart(i), we wish to compute xi = ti R M-1 + ti+1 R M ti+m-1 R 0 (mod Q) Intuition. M-digit, base-r integer, modulo Q. Horner's method. Linear-time method to evaluate degree-m polynomial. pat.chart() i % 997 = 2 R Q % 997 = (2*10 + 6) % 997 = % 997 = (26*10 + 5) % 997 = % 997 = (265*10 + 3) % 997 = % 997 = (659*10 + 5) % 997 = 613 // ompute hash for M-digit key private long hash(string key, int M) { long h = 0; for (int j = 0; j < M; j++) h = (R * h + key.chart(j)) % Q; return h; } omputing the hash value for the pattern with Horner s method 46

47 Efficiently computing the hash function hallenge. How to efficiently compute x i+1 given that we know x i. xi = ti R M 1 + ti+1 R M ti+m 1 R 0 xi+1 = ti+1 R M 1 + ti+2 R M ti+m R 0 Key property. an update hash function in constant time! xi+1 = xi R t i R M + t i +M shift left subtract leftmost digit add new rightmost digit i current value new value text * current value subtract leading digit multiply by radix add new trailing digit new value 47

48 Rabin-Karp substring search example i % 997 = 3 Q % 997 = (3*10 + 1) % 997 = % 997 = (31*10 + 4) % 997 = % 997 = (314*10 + 1) % 997 = % 997 = (150*10 + 5) % 997 = 508 RM R % 997 = (( *(997-30))*10 + 9) % 997 = % 997 = (( *(997-30))*10 + 2) % 997 = % 997 = (( *(997-30))*10 + 6) % 997 = % 997 = (( *(997-30))*10 + 5) % 997 = 442 match % 997 = (( *(997-30))*10 + 3) % 997 = % 997 = (( *(997-30))*10 + 5) % 997 = 613 return i-m+1 = 6 Rabin-Karp substring search example 48

49 Rabin-Karp: Java implementation public class RabinKarp { private long pathash; private int M; private long Q; private int R; private long RM; // pattern hash value // pattern length // modulus // radix // R^(M-1) % Q public RabinKarp(String pat) { M = pat.length(); R = 256; Q = longrandomprime(); a large prime (but not so large to cause long overflow) } RM = 1; for (int i = 1; i <= M-1; i++) RM = (R * RM) % Q; pathash = hash(pat, M); precompute R M - 1 (mod Q) private long hash(string key, int M) { /* as before */ } } public int search(string txt) { /* see next slide */ } 49

50 Rabin-Karp: Java implementation (continued) Monte arlo version. Return match if hash match. public int search(string txt) check for hash collision { using rolling hash function int N = txt.length(); int txthash = hash(txt, M); if (pathash == txthash) return 0; for (int i = M; i < N; i++) { txthash = (txthash + Q - RM*txt.chart(i-M) % Q) % Q; txthash = (txthash*r + txt.chart(i)) % Q; if (pathash == txthash) return i - M + 1; } return N; } Las Vegas version. heck for substring match if hash match; continue search if false collision. 50

51 Rabin-Karp analysis Theory. If Q is a sufficiently large random prime (about M N 2 ), then the probability of a false collision is about 1 / N. Practice. hoose Q to be a large prime (but not so large as to cause overflow). Under reasonable assumptions, probability of a collision is about 1 / Q. Monte arlo version. lways runs in linear time. Extremely likely to return correct answer (but not always!). Las Vegas version. lways returns correct answer. Extremely likely to run in linear time (but worst case is M N). 51

52 Rabin-Karp fingerprint search dvantages. Extends to 2d patterns. Extends to finding multiple patterns. Disadvantages. rithmetic ops slower than char compares. Poor worst-case guarantee. Requires backup. Q. How would you extend Rabin-Karp to efficiently search for any one of P possible patterns in a text of length N? 52

53 Substring search cost summary ost of searching for an M-character pattern in an N-character text. algorithm version operation count guarantee typical backup in input? correct? extra space brute force M N 1.1 N yes yes 1 Knuth-Morris-Pratt full DF (lgorithm 5.6 ) mismatch transitions only 2 N 1.1 N no yes MR 3 N 1.1 N no yes M full algorithm 3 N N / M yes yes R Boyer-Moore mismatched char heuristic only (lgorithm 5.7 ) M N N / M yes yes R Rabin-Karp Monte arlo (lgorithm 5.8 ) 7 N 7 N no yes 1 Las Vegas 7 N 7 N no yes yes 1 probabilisitic guarantee, with uniform hash function 53