What s Behind BLAST. Gene Myers, Director MPI for Cell Biology and Genetics Dresden, DE

Transcription

1 Wht s Behind BLAST Gene Myers, Director MPI for Cell Biology nd Genetics Dresden, DE

2 Approximte String Serch Given string A of length n, query Q of length p n, n lignment scoring function δ, nd threshold d: Find ll sustrings of A, sy M, s.t. δ(q,m) d? δ here = Simple Levenstein (unit cost mismtch, insert, & delete)...xxxxxxxxcgt-gcttcxxxxxxxx...! tgtggc-ttc A 3-mtch (solute) A 25%-mtch (reltive)

3 Edit Grphs Dynmic Progrmming Mtrix

4 0ttcggtgt Alignments A 0 cgtgctt N Corresponds to pth in the edit grph of the two sequences. B cgtg-ctt tgtggc-tt M

5 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions

6 The Beginning Workshop for Algorithms in Moleculr Genetics Mrch 26-28, 1988 S. Altschul W. Fitch Z. Glil W. God T. Hunkpillr S. Krlin G. Lndu E. Lnder D. Lipmn J. Misel H. Mrtinez C. Snders T. Smith R. Stden J. Turner M. Zuker A. Mukherjee M. Wtermn D. Snkoff P. Sellers E. Ukkonen W. Miller G. Myers

7 Glil s 2 Questions Workshop for Algorithms in Moleculr Genetics Mrch 26-28, 1988 Zvi gve tlk out suffix trees: Q1: Cn one get rid of the nnoying dependence on lphet size Σ?! Mner & Myers, Suffix Arrys 1990 Q2: Cn one use n index to get fster pproximte serch?

8 Suffix Arrys Given suject string of size n over lphet of size Σ, uild n index tht determines if query string of length p occurs in the suject efficiently Suffix tree: Index is O(nΣ) spce, then O(p) time Index is O(n) spce, then O(plogΣ) time Glil: Remove nnoying dependence on Σ. Mner & Myers: Suffix rry: Index is O(n) spce, O(p + logn/logσ) time. Glil sys we misunderstood his chllenge, sigh. But suffix trees enled Burroughs-Wheeler Trnsform tht re sprse index commonly in use tody for NGS.

9 A Simple Index Φ( ccgt ) = = 283 (10) [0, Σ k -1] for ny fixed k-mer size. 0 1 Idx Pos Scn 1: Count how ig ech set Occ(c) will e (in Idx[c+1]), then set Idx[c] += Idx[c-1] to point to proper Pos index.! Scn 2: Fill in ech set using Idx[c] s finger to plce the next position, then redjust indices (Idx[c] = Idx[c-1]). c c+1 Occurences of k-mers with code c, Occ(c) = { p : Φ( A[p..p+k-1] ) = c } = { Pos[j] : j [Idx[c],Idx[c+1]-1] } Σ k -1 Σ k n-1 n Performnce: O(n+ Σ k ) time nd exctly n+ Σ k integers. If choose k ~ log Σ n then O(n). O(p+h) expected-time to find ny string of length p with h hits.

10 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions

11 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend

12 APM Filters A filter is n lgorithm tht elimintes lot of tht which isn t desired. 100% Sp Exct 100% Sn < 100% Sn < 100% Sp Filter Heuristic Filter Exct If fst & specific then cn improve speed of n exct lgorithm. Approximte mtch filter ides:! Look for exct mtches to k-mers of the query (in n index) ( Person & Lipmn FASTA, Chng & Lwler, O(dn/lg p) )! Insted look for k-mers tht re smll distnce wy, e.g. 1 or 2 diff s, from k-mer of the query, i.e. the neighorhood N d (w) = { v : v nd w re d differences prt } N d (k) ( d k )(2Σ) d

13 The Power of Neighorhoods Consider looking for 9%-mtch of 40 symols ( 3 differences or 3-mtch): If divide query into 4 10-mers then t lest one must mtch exctly:! Get hit every Σ 10 / 4 symols (e.g for DNA) If divide into 2 20-mers then t lest one of the N 1 strings must mtch exctly:! Get hit every Σ 20 / 2N 1 (20) symols (e.g / = for DNA)! 10,000 times more specific! (ut 80x more lookups)

14 Seed & Extend The seed mtches (either exct or from neighorhood) re in effect defining res within the edit grph of Q vs A where the lignment of n ε-mtch could e: A 2 4 s1 Q s2 s3 s4 Q = s1s2s3s4

15 s1 Seed & Extend The seed mtches (either exct or from neighorhood) re in effect defining res within the edit grph of Q vs A where the lignment of n ε-mtch could e: A 2 4 Q s2 s3 s4 Q = s1s2s3s4 ±d/4

16 s1 Seed & Extend The seed mtches (either exct or from neighorhood) re in effect defining res within the edit grph of Q vs A where the lignment of n ε-mtch could e: A 2 4 Q s2 s3 s4 Spend O(pdh + pz) time where! h(k) = the numer of seed k-hits vs. z(k) = neighorhood size k-words! Both z nd h re functions of k nd the optiml k is slightly igger thn logσ n ±d

17 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek

18 The 1st Converstion X

19 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born

20 Blst = Seed & Extend Seeds re neighorhoods of ll k-mers of query under weighted Levenstein (e.g. PAM120) Find seeds with deterministic finite utomton ccepting ll neighorhood words ( O(n)) Extend is just weighted Hmming ut stop when score drops too much A heuristic lst ws inspired y slm = suliner pproximte mtch

21 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born!

22 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm!

23 The Splitting Lemm Lemm: If w ε-mtches v then either () w0 hs n ε-mtch to prefix (cll it v0) of v, or () w1 hs n ε-mtch to suffix (cll it v1) of v. Proof: w0 w k errors v w1 k = ε w k/2 errors? w0 v k/2 errors? w1 By Pigeon Hole Principle k/2 errors v1

24 w0 w1 k/2 errors w v1 w0 w1 w10 w11 k/4? v10 k/4? w0 w w1 w10 w11 w100 w101 k/8 v101

25 The Splitting Lemm w w0 Let w ε = w w β = w β [1.. wβ /2] if = 0 w β [ wβ /2+1.. wβ ] if = 1 e.g. α= w1 w10 w100 w101 k/8 v101 w11 Lemm: If w ε-mtches v then α s.t. prefixes β of α, (1) w β hs n ε-mtch to sustring (cll it vβ) of v, nd (2) vβ0 is prefix of vβ (if β0 is prefix of α), nd (3) vβ1 is suffix of vβ (if β1 is prefix of α).

26 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm!

27 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm Fll 89: Seed & Extend y Douling

28 Use logσ n s the seed size! Douling Extension Lemm: Any ε-mtch of Q hs n ε-mtch to t lest one seed segment of size logσ n Use the splitting lemm to split Q to seeds of size logσ n, nd insted of extending ll t once, extend y douling using the splitting lemm. Time for ech extension telescopes hyper-geometriclly nd so is dominted y the first term: O(p/logΣn h logσn εlogσn) = O(dhlogΣn)

29 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm Fll 89: Seed & Extend y Douling

30 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm Fll 89: Seed & Extend y Douling Spr 90: Generting Condensed Neighorhoods

31 Generting (Condensed) Neighorhoods N d (w) = { v : v nd w re d differences prt nd v is not proper prefix of nother word in N d (w) } N 1 () = {,,,,,,,,,,,,,, } It suffices to find the words in the condensed neighorhood. But how do you do tht efficiently, including finding them in the index? Compute rows of dynmic progrmming mtrix s one trverses the trie of ll strings over Σ

32 Condensed Neighorhoods v? w Done: 1 in the right corner

33 Condensed Neighorhoods v w Only need ±1 nd!

34 Condensed Neighorhoods _ v w _

35 _ Condensed Neighorhoods _1 _ _ If ll entries re d then wsting time on D.P. _ _ _ _ _ _

36 _ Condensed Neighorhoods _1 _ _ If ll entries re d then wsting time on D.P. _ _ _ _ _ _

37 Condensed Neighorhoods _ _ _ _

38 Condensed Neighorhoods Use KMP on reverse of w to efficiently discover these. _ _ A shorter suffix of w tht is prefix of the extension is lso possile

39 Condensed Neighorhoods _ _ _ _ Lemm: Neighorhoods nd their hits in A cn e generted in O(zd+h) time where z = Nd(w)

40 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm Fll 89: Seed & Extend y Douling Spr 90: Generting Condensed Neighorhoods

41 The Story Mrch 88: The Lister Hill Meeting & Glil s 2 questions June 88: Seed & Extend My 89: The TRW Chip & The Cigrette Brek Fll 89: Blst is Born Fll 89: The Splitting Lemm Fll 89: Seed & Extend y Douling Spr 90: Generting Condensed Neighorhoods Fll 90: Finle: Complexity

42 Complexity How ig is N d (k)? Developed recurrence for non-redundnt edit scripts: () DI = S () DS = SD (c) IS = SI (d) ID = Φ Lemm: d-1 S(k,d) = S(k-1,d) + (Σ-1)S(k-1,d-1) + (Σ-1) Σ Σ j S(k-1,d-1) d-2 j=0 + (Σ-1) 2 Σ j d-1 Σ S(k-2,d-2-j) + Σ S(k-2-j,d-1-j) j=0 j=0 N d (k) S(k,d) + d Σ j=1 Σ j S(k-1,d-j)

43 Complexity So how ig is it? Lemm: Nε(k) α(ε) k where α(ε) = Σ pow(ε) nd pow(ε) = logσ ( c(ε)+1 c(ε) 1 nd c(ε) = ε -1 + (1 + ε -2 ).5 ) + ε logσ c(ε) + ε Also Pr(w in Nε(k)) = O( 1 / β(ε) k ) where β(ε) = Σ 1-pow(ε)

44 pow(ε)

45 So how ig is it? Lemm: Complexity Nε(k) = O( α k ) where α = Σ pow(ε) Pr(w in Nε(k)) = O( 1 / β k ) where β = Σ 1-pow(ε) Strts t 1 (ε=0) nd grows Flex fctor Strts t Σ (ε=0) nd shrinks Effective lphet size And when k = logσ n? Nε(k) = O(n pow(ε) ) nd Pr(w in Nε(k)) = O(n pow(ε)-1 )

46 The Result Theorem: Given () A is effectively Bernouilli, () simple O(n) spce, precomputed index of A, nd (c) there re h d-mtches of query Q to A then they cn e found in O(d n pow(ε) log n + pd h) expected-time.

47 To my knowledge no one hs improved on this in the lst 20 yers!?! Algorithmic 12, 4-5 (1994), ! (sumitted 1991! )! A recent retrospective:! Computtionl Biology 19, (Springer-Verlg 2013), 3-15.