Approximate Matching

Transcription

1 Approximate Matching Ben Langmead Department of Computer Science lease sign guestbook ( to tell me briefly how you are using the slides. For original Keynote files, me

2 Approximate matching Strings Online algorithms Naive exact matching Offline algorithms Index-assisted Boyer-Moore k-mer index We have focused on exact matching in reality, we have to deal with differences

3 Read CTCAAACTCCTGACCTTTGGTGATCCACCCGCCTNGGCCTTC Reference GATCACAGGTCTATCACCCTATTAACCACTCACGGGAGCTCTCCATGCATTTGGTATTTT CGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATGTC GCAGTATCTGTCTTTGATTCCTGCCTCATCCTATTATTTATCGCACCTACGTTCAATATT ACAGGCGAACATACTTACTAAAGTGTGTTAATTAATTAATGCTTGTAGGACATAATAATA ACAATTGAATGTCTGCACAGCCACTTTCCACACAGACATCATAACAAAAAATTTCCACCA AACCCCCCCTCCCCCGCTTCTGGCCACAGCACTTAAACACATCTCTGCCAAACCCCAAAA ACAAAGAACCCTAACACCAGCCTAACCAGATTTCAAATTTTATCTTTTGGCGGTATGCAC TTTTAACAGTCACCCCCCAACTAACACATTATTTTCCCCTCCCACTCCCATACTACTAAT CTCATCAATACAACCCCCGCCCATCCTACCCAGCACACACACACCGCTGCTAACCCCATA CCCCGAACCAACCAAACCCCAAAGACACCCCCCACAGTTTATGTAGCTTACCTCCTCAAA GCAATACACTGACCCGCTCAAACTCCTGGATTTTGGATCCACCCAGCGCCTTGGCCTAAA CTAGCCTTTCTATTAGCTCTTAGTAAGATTACACATGCAAGCATCCCCGTTCCAGTGAGT TCACCCTCTAAATCACCACGATCAAAAGGAACAAGCATCAAGCACGCAGCAATGCAGCTC AAAACGCTTAGCCTAGCCACACCCCCACGGGAAACAGCAGTGATTAACCTTTAGCAATAA ACGAAAGTTTAACTAAGCTATACTAACCCCAGGGTTGGTCAATTTCGTGCCAGCCACCGC GGTCACACGATTAACCCAAGTCAATAGAAGCCGGCGTAAAGAGTGTTTTAGATCACCCCC TCCCCAATAAAGCTAAAACTCACCTGAGTTGTAAAAAACTCCAGTTGACACAAAATAGAC TACGAAAGTGGCTTTAACATATCTGAACACACAATAGCTAAGACCCAAACTGGGATTAGA TACCCCACTATGCTTAGCCCTAAACCTCAACAGTTAAATCAACAAAACTGCTCGCCAGAA CACTACGAGCCACAGCTTAAAACTCAAAGGACCTGGCGGTGCTTCATATCCCTCTAGAGG AGCCTGTTCTGTAATCGATAAACCCCGATCAACCTCACCACCTCTTGCTCAGCCTATATA CCGCCATCTTCAGCAAACCCTGATGAAGGCTACAAAGTAAGCGCAAGTACCCACGTAAAG ACGTTAGGTCAAGGTGTAGCCCATGAGGTGGCAAGAAATGGGCTACATTTTCTACCCCAG AAAACTACGATAGCCCTTATGAAACTTAAGGGTCGAAGGTGGATTTAGCAGTAAACTAAG AGTAGAGTGCTTAGTTGAACAGGGCCCTGAAGCGCGTACACACCGCCCGTCACCCTCCTC AAGTATACTTCAAAGGACATTTAACTAAAACCCCTACGCATTTATATAGAGGAGACAAGT CGTAACCTCAAACTCCTGCCTTTGGTGATCCACCCGCCTTGGCCTACCTGCATAATGAAG AAGCACCCAACTTACACTTAGGAGATTTCAACTTAACTTGACCGCTCTGAGCTAAACCTA GCCCCAAACCCACTCCACCTTACTACCAGACAACCTTAGCCAAACCATTTACCCAAATAA AGTATAGGCGATAGAAATTGAAACCTGGCGCAATAGATATAGTACCGCAAGGGAAAGATG AAAAATTATAACCAAGCATAATATAGCAAGGACTAACCCCTATACCTTCTGCATAATGAA TTAACTAGAAATAACTTTGCAAGGAGAGCCAAAGCTAAGACCCCCGAAACCAGACGAGCT ACCTAAGAACAGCTAAAAGAGCACACCCGTCTATGTAGCAAAATAGTGGGAAGATTTATA GGTAGAGGCGACAAACCTACCGAGCCTGGTGATAGCTGGTTGTCCAAGATAGAATCTTAG TTCAACTTTAAATTTGCCCACAGAACCCTCTAAATCCCCTTGTAAATTTAACTGTTAGTC CAAAGAGGAACAGCTCTTTGGACACTAGGAAAAAACCTTGTAGAGAGAGTAAAAAATTTA ACACCCATAGTAGGCCTAAAAGCAGCCACCAATTAAGAAAGCGTTCAAGCTCAACACCCA CTACCTAAAAAATCCCAAACATATAACTGAACTCCTCACACCCAATTGGACCAATCTATC ACCCTATAGAAGAACTAATGTTAGTATAAGTAACATGAAAACATTCTCCTCCGCATAAGC CTGCGTCAGATTAAAACACTGAACTGACAATTAACAGCCCAATATCTACAATCAACCAAC AAGTCATTATTACCCTCACTGTCAACCCAACACAGGCATGCTCATAAGGAAAGGTTAAAA AAAGTAAAAGGAACTCGGCAAATCTTACCCCGCCTGTTTACCAAAAACATCACCTCTAGC ATCACCAGTATTAGAGGCACCGCCTGCCCAGTGACACATGTTTAACGGCCGCGGTACCCT Sequence differences occur because of Sequencing error 2. Genetic variation

4 Approximate matching T: GGAAAAAGAGGTAGCGGCGTTTAACAGTAG : GTAACGGCG Mismatch (Substitution)

5 Approximate matching T: GGAAAAAGAGGTAGC-GCGTTTAACAGTAG GTAGCGGCG : Insertion

6 Approximate matching T: GGAAAAAGAGGTAGCGGCGTTTAACAGTAG GT-GCGGCG : Deletion

7 Hamming distance For X & Y where X = Y, hamming distance = minimum # substitutions needed to turn one into the other X: Y: GAGGTAGCGGCGTT GGTAG GTT GTGGTAACGGGGTT GGTAA GTT Hamming distance = 3

8 Edit distance (AKA Levenshtein distance) For X & Y, edit distance = minimum # edits (substitutions, insertions, deletions) needed to turn one into the other X: Y: X: Y: TGGCCGCGCAAAAACAGC CAGC TGACCGCGCAAAACAGC CCGCGCAAAA-CAGC GCGTATGCGGCTAACGC TATGCGGCTA-ACGC GCTATGCGGCTATACGC GC- ACGC Edit distance = 2 Edit distance = 2

9 Approximate matching Like exact matching, but pattern may be within a certain distance (usually Hamming or edit) of T. Each such place is an approximate match. Allowing edits is more challenging than just allowing mismatches We ll return to edits

10 Approximate matching def naive(p, t): occurrences = [] for i in range(len(t) - len(p) + 1): # loop over alignments match = True for j in range(len(p)): # loop over characters if t[i+j]!= p[j]: # compare characters match = False break if match: occurrences.append(i) return occurrences # mismatch; reject alignment # all chars matched; record

11 Approximate matching def naive_approx_hamming(p, t, maxdistance): occurrences = [] for i in range(len(t) - len(p) + 1): # loop over alignments nmm = 0 for j in range(len(p)): # loop over characters if t[i+j]!= p[j]: # compare characters nmm += 1 # mismatch if nmm > maxdistance: break # exceeded max hamming dist if nmm <= maxdistance: occurrences.append(i) # approximate match return occurrences

12 Approximate matching Wanted: way to apply exact matching algorithms to approximate matching problems

13 Approximate matching u v If occurs in T with 1 edit, then u or v appears with no edits u v u v : T: CGC GCA CGA GCA : AATTGG T: AATTAG

14 Approximate matching... p 1 p 2 p 3 p 4 p k+1 If occurs in T with up to k edits

15 Approximate matching... p 1 p 2 p 3 p 4 p k+1 If occurs in T with up to k edits, then at least one of p 1, p 2,..., p k+1 must appear with 0 edits

16 Approximate matching XXX X 5 partitions 4 edits (X) XX X X X X X X

17 Approximate matching igeonhole principle: k+1 pigeons, k holes. At least one has >1 pigeon!

18 Approximate matching We have k pigeons, k+1 holes, at least one......is empty

19 igeonhole principle p 1 p 2 p 3 p 4 p 5 Exact matching T What algorithm can we use? Any exact matching algorithm If we have a k-mer index, we can use that Naive exact matching Boyer-Moore

20 igeonhole principle p 4 match T

21 igeonhole principle p 1 p 2 p 3 p 4 p 5 match verify verify verify verify T For Hamming distance, verification is essentially just the inner loop of naive_approx_hamming from before

22 igeonhole principle... p 1 p 2 p 3 p 4 p k+1 Advantages Reuse favorite exact matching algos; fast and easy Flexible; works for Hamming and edit distance* Disadvantages Large k yields small partitions matching many times by chance; lots of verification work k+1 exact matching problems, one per partition * we don t know how to do edit distance verification yet

23 Implementation of pigeonhole principle with Boyer-Moore as exact matching algorithm: Boyer-Moore, exact Boyer-Moore, 1 mismatch with pigeonhole Boyer-Moore, 2 mismatches with pigeonhole # character comparisons wall clock time # matches # character comparisons wall clock time # matches # character comparisons wall clock time # matches : tomorrow T: Shakespeare s complete works 786 K 1.91s M 7.73 s M s 382 : 50 nt string from Alu repeat* T: Human reference (hg19) chromosome M s M 209 s 1, M 328 s 2,798 * GCGCGGTGGCTCACGCCTGTAATCCCAGCACTTTGGGAGGCCGAGGCGGG

24 Generalizing pigeonhole, part 1 If occurs in T with up to k edits, then at least one of p 1, p 2,..., p k+1 must appear with 0 edits... p 1 p 2 p 3 p 4 p k+1 But doesn't have to be at least one of what would we have to change for at least two of? If occurs in T with up to k edits, then at least two of must appear with 0 edits

25 Generalizing pigeonhole, part 1 If occurs in T with up to k edits, then at least one of p 1, p 2,..., p k+1 must appear with 0 edits... p 1 p 2 p 3 p 4 p k+1 But doesn't have to be at least one of what would we have to change for at least two of? If occurs in T with up to k edits, then at least two of p 1, p 2,..., p k+2 must appear with 0 edits

26 Generalizing pigeonhole, part 2 If occurs in T with up to k edits, then at least one of p 1, p 2,..., p k+1 must appear with 0 edits... p 1 p 2 p 3 p 4 p k+1 Let p1, p2,..., pj be a partitioning of. If occurs with up to k edits, then at least one of p1, p2,..., pj must occur with floor(k / j) edits.???... p 1 p 2 p 3 p 4 p j

27 Generalizing pigeonhole, part 2 If occurs in T with up to k edits, then at least one of p 1, p 2,..., p k+1 must appear with 0 edits... p 1 p 2 p 3 p 4 p k+1 Let p1, p2,..., pj be a partitioning of. If occurs with up to k edits, then at least one of p1, p2,..., pj must occur with floor(k / j) edits.... p 1 p 2 p 3 p 4 p j

28 Generalizing pigeonhole, part 2 At least one of p1, p2,..., p5 occurs with... X X X X k = 4 edits 0 edits X X X X X k = 5 edits 1 edits X X X X X X X X X k = 9 edits 1 edits X X X X X X X X X X k = 10 edits 2 edits etc

29 Generalizing pigeonhole, part 2 Let j = k + 1 Specific General igeonhole principle If occurs in T with up to k edits, then at least one of p 1, p 2,..., p j must appear with floor(k / j) edits igeonhole principle with j = k + 1 If occurs in T with up to k edits, then at least one of p 1, p 2,..., p k+1 must appear with 0 edits Why? Smallest value s.t. floor(k / j) = 0 Why make floor(k / j) = 0? So we can use exact matching Why is smaller j good? Yields fewer, longer partitions Why are long partitions good? Makes exact-matching filter more specific, minimizing # candidates

30 A different principle We partitioned into non-overlapping substrings p1 p2 p3 p4... pk+1 Now consider overlapping substrings 3 6 n-l n-l 1 4 n-l-1

31 Approximate string matching: more principles q x x x x... n n - q + 1 of these Say substrings are length q. There are n - q + 1 such substrings. 1 edit to changes at most q substrings kq is worst case; could be < kq Minimum # of length-q substrings unedited after k edits? n - q kq q-gram lemma: if occurs in T with up to k edits, alignment must contain t exact matches of length q, where t n - q kq

32 Approximate string matching: more principles If occurs in T with up to k edits, alignment contains an exact match of length q, where q floor(n / (k + 1)) Obtained by solving for q: n - q kq 1 Exact matching filter: find matches of length floor(n / (k + 1)) between T and any substring of. Check vicinity for full match.

33 Approximate matching principles Non-overlapping substrings igeonhole principle q-gram lemma Overlapping substrings Specific General p1, p2,..., pj is a partitioning of. If occurs with k edits, at least one partition matches with floor(k / j) edits. p1, p2,..., pk+1 is a partitioning of. If occurrs in T with k edits, at least one partition matches exactly. If occurs with k edits, alignment contains t exact matches of length q, where t n - q kq igeonhole principle with j = k + 1 q-gram lemma with t = 1 If occurs with k edits, alignment contains an exact match of length q where q floor(n / (k + 1))....

34 Sensitivity Sensitivity = fraction of true approximate matches discovered by the algorithm Lossless algorithm finds all of them, lossy algorithm doesn t necessarily We ve seen lossless algorithms. Most everyday tools are lossy. Lossy algorithms are usually much speedier & still acceptably sensitive... Example lossy algorithm: pick q > floor(n / (k + 1))