ALG 4.1 Randomized Pattern Matching:

Transcription

1 Algorithms Professor John Reif Generalized Pattern Matching ALG 4.1 Randomized Pattern Matching: Reading Selection: CLR: Chapter 12 Handout: R. Karp and M. Rabin, "Efficient Randomized Pattern-Matching" input index set R for each r e R strings X(r), Y(r) problem find r e R s.t. X(r) = Y(r) e {0,1} m Examples: (1) ID String Pattern Matching input pattern X = X 1... X Œ m {0,1} m text Y = Y 1... Y n Œ {0,1} n index set R = {1,2,..., n-m+1} "r ŒR X(r) = X Y( r ) = Y r Y r+1... Y r+(m-1) 1 2

2 Pattern Matching by Fingerprinting (2)2D Array Matching input pattern s s binary array X = (X ij ), m = s 2 text b b binary array Y = (Y ij ), n = b 2 index set R = {< i,j > s i,j b} X(< i,j >) = string of rows of X Y(< i.j >) = string of rows in s s block of Y with (i,j) in lower right postion. ( note Karp & Pratt reverse n,m) S is a finite set "p es F p ( ) is function {0,1} m Æ small range D p F p (X) is "fingerprint" for string X idea compare X(r) = Y(r) only if fingerprints agree: F p (X(r)) = F p (Y(r)) Algorithm p random element of S 3 for each r e R in order do begin compute a p (r) = F p (X(r)) compute b p (r) = F p (Y(r)) if a p (r) = b p (r) then if X(r) = Y(r) then output "Match at r" end fingerprint fn F p : {0,1} m Æ D p 4

3 Requirements (1) small domain D p (2) small probability of false match F p (X(r)) = F p (Y(r)) but X(r) π Y( r ) ) (3) fingerprints F p (X(r)), F p (Y(r)) are easily updatable from previous r Examples of fingerprints: represent binary string X = X 1... X m by integer m H( x ) = Â i=1 X i 2 m-i modular fingerprint F p (x) = res (H(x), p) modular fingerprint F p (X) = res (H(x), p) H( X) = p Î p - H( X) note F p (x) H(X) mod p (A)integer modular fns. (B) unimodular matrices (C) irreducible polynomial modular fns. 5 6

4 Facts about Prime Numbers let P(k) = number of primes k Define S = {p p is prime and p M} where M is a (suf. large) integer k FACT 1 If k 29 and a 2, then a has P(k) distinct prime divisors idea choose random p e S fi must prove F p (X) = res (H(X), p) is good fingerprint proof follows from Rosser & Schoenfeld bound: P p > e k k2 p prime p k 1 FACT 2 for all k 17 k ln k P(k) k ln k (prime number theorem) 7 8

5 Suppose Randomized Pattern Match Algorithm is executed, with fingerprint F p (X) = res(h(x), p) wi th s et S = {p p prime M} where M = mn 2 and mn 29 proof A false match occurs only if $ r e R X( r ) π Y(r) and P (H(x(r)) - H(Y(r))) iff p L where L = P H(X(r)) - H(Y(r)) X( r) π Y( r) Theorem for each r e R, p r obability of false match F p (X(r)) = F p (Y(r)) but X(r) π Y( r ) is n But L 2 mn so by Fact 1, L has at most P(mn) prime divisors Since p is chosen at and only Prob(false match) by Fact 2. random from P(M) primes, P(mn) give false matches, P(mn) P(M) n 9 10

6 Updating Modular Fingerprints pattern text X = X 1... X m Y = Y 1... Y n Theorem Total Exp. Time for finding a match is O(n) X( r ) = X, Y( r ) = Y r Y r+1... Y r+m-1 Si nce update formula: Y(r+1) = (Y(r) - 2 m-1 Y r ) 2 + Y r+m a p (r+1) = (a p (r) + a p (r) + x Y r + Y r+m ) mod p where x = - 2 m mod p proof Expected Time is: O( n ) + nm Prob(false match) O( n) + nm O( n O(n) 1) gives F p (Y(r+1)) = a p (r+1) from F p (Y(r)) = a p (r) in O(1) arithmetic steps using O(log n) bit integers on range [O, mn 2-1] 11 12

7 2D Randomized Pattern Matching input pattern X = (X ij ) is s s boul. array text Y = (Y ij ) is b b boul. array text window Y(< i,j >) = concatination of rows of s s subarray of Y with < i,j > in lower right corner b s i,j s Index i Update fingerprint a p (< i+1, j >) = a p (< i,j >) + ( a (< i,j >) - l Y + p i-s+1,j Y i+1,j ) mod p where Index j Update a p (< i,j+1 >) = where l = 2 s mod p - a p (< i,j >) + 2 Y i,j Y 2,j Ys,j Y i,1 2 + Y 2, Ys, s(s-1) q = - 2 mod p 2 s s s q - mod p b 13 14

8 Unimodular Matrices as Fingerprints Definition homomorphism k from {0,1} * into unimodular matrices with k(e ) = ( ) (, k(0) = 10 11) (, k(1) = 11 01) k(x * Y) = k(x) k(y) concatenation matrix multiplication Suppose the unimodular fingerprint fn F p is used with random p e S = {p p M is prime} where M = mn 2. Theorem The Random Pattern Matching Algorithm has probability of false match. n Fact 1 ' If X e {0,1} m, then each entry of k(x) is F m = mth Fibonacci (where F 0 = F 1 = 1, and F m = F m-1 + F m-2 for m 2) Fact 2' log F m ~.694m 15 16

9 proof A false match occurs if $i,j e {1,2} F p (X(r)) i,j = F p (Y(r)) i,j but k(x(r)) i,j π k(y(r)) i,j iff p L' where L' = P k(x(r)) i,j - k(y(r)) i,j r e R i,j e {1,2} But L' (F m ) 4n 2 È 4n log Fm By fact 1, the number of primes that divide L' is at most P ( È 4n log Fm ) Updating Using Unimodular Matrices ID string matching: a p (r+1) = K p (Y r ) -1 a p (r) H(Y r+m ) a p (r) = F p (Y(r)) where K p (0) -1 = ( 1 0 p-1 1) and K p (1) -1 = ( 1 p-1 0 1) (can also extend to 2D string matching) mod p The probability of a false match is P(È 4n log Fm ) P(M) n since by Fact 2' log(f ) m ~.694m 17 18

10 (2) S = {p p M and p is M-fat} p is M-fat if p M and p has prime divisor > M Simplifications of Modular Fingerprinting S = {p p M and p prime or pseudoprime} p is pseudoprime if 2 p-1 1 mod p but p not prime -c(log m log log m)2 # pseudoprimes is Me 1 fact S ~ M(ln2 + 0(1)) A false match occurs when p L where L = P(H(X(r)) - H(Y(r))) X(r) π Y(r) nm Bound L < 2 Let N be the number of M-fat integers dividing L Then N N ( M) M ln 2 2 <L < 2 nm so n 4 (log n) 2 The prob of false match is: N <.5 if M = n 4 S (ln n)

11 Fingerprinting by Random Polynomials (3) S = {p p M} with some M idea use new p when get false match expected time cn (.5 + (.5) 2 + (.5) ) O( n) Gal oi s Fi e l d GF( 2 k ) = {b 1... b k b i,... b k e{0,1}} Z 2 [t] = pol y nomials of form p(t) = t k + a k-1 t k a o where a k-1, a k-2,..., a e 0 {0,1} p(x) irreducible if can't be factored Lemma If k is prime, the number of irreducible polynomials of degree k in Z 2 [t] is (2 k - 2) k Fi nger p r int fn F p (x) = x 1 t m x m mod p (t) (residue comp can be done efficiently) 21 22

12 Open Problems Theorem If use random Fingerprint fn with degree k > log (nm e -1 ), then prob of false match is < e. proof use usual argument and above Lemma (1) Are there deterministic methods for Fingerprinting? (2) What are optimal trade offs for prob of error and size of S for randomized Fingerprinting? 23 24

13 Application of Fingerprinting to Computer Security for fixed random p, idea store F p (F 1 ),..., F p (F k ) fingerprints of files F 1,..., F k Security: only operator knows p, so if ' any file F i modi fi ed, to F i then with high likelihood F p (F i ' ) π F p (F i ) fi can build a secure operating system from this idea! 25