Exact Matching. Exact Matching Algorithms 5/19/2015. Exact Matching Problem: search pattern P in text T (P,T are strings)

Transcription

1 Exct Mtching Exct Mtching Problem: Exct Mtching Algorithms serch pttern P in text T (P,T re strings) Knuth Morris Prtt preprocess pttern P Aho Corsick preprocess pttern P of severl strings P = { P 1,, P r } Suffix Trees preprocess text T or severl texts, dtbse of texts 1

2 Exct Mtching: Preprocessing Pttern(s) Knuth-Morris-Prtt (KMP) Aho-Corsick Serching Pttern P in Text T Nive: T: neeneeneeneeeneneeeneeneneeneneen P: neeneeen neeneen neeneen neeneen Better: T: neeneeneeneeeneneeeneeneneeneneen P: neeneeen neeneeen neeneeen 2

3 Pttern P ϵ {,b, } * : KMP Preprocessed P exmple Automton: b b b Tble: b b b filure links (suffix = prefix): top: s utomton bottom: s tble how to determine? how to use? P: P: P: P: P: P: 6 b b b b b b 3 7 b b b b b b 4 8 b b b b b... 3 KMP computing filure links filure link ~ new best mtch (fter mismtch) òr 0 Pttern P: k-1 b b k Flink[1] = 0; for k from 2 to PtLen do fil = Flink[k-1] while ( fil>0 nd P[fil] P[k-1] ) do fil = Flink[fil]; od Flink[k] = fil+1; od 3

4 KMP computing filure links Flink[1] = 0; for k from 2 to PtLen do fil = Flink[k-1] while ( fil>0 nd P[fil] P[k-1] ) do fil = Flink[fil]; od Flink[k] = fil+1; od Tble: b b b fil: 0 0->F->Flink[2]=0+1 1->T->0->F->Flink[3]=1 1->F->Flink[4]= prefixes vi filure links Pttern P: P 1 P r-1 P k-r+1 P k-1 Flink[k]=r r k P r P 1 P r-1 = P k-r+1 P k-1 mximl r<k prefix ll such vlues r: suffix r 4 r 3 r 2 r 1 k P 1 P r2-1 = P k-r2 +1 P k-1 = P r1 -r 2 +1 P r1-1 Flink[r 1 ]=r 2 4

5 other methods Boyer-Moore T = mrktkoopmn P = schoenveter schoe work bckwrds Krp-Rbin fingerprint i-1 i i+n-1 i+n p 1 p n hsh-vlue i B n-1 + i+1 B n-2 + i+n-1 B 0 i+1 B n i+n-1 B 1 + i+n B 0 exct mtching with set of ptterns P = { P 1,, P r } ll occurrences in text T totl length m length n AHO CORASICK generlizes KMP filure links: longest suffix tht is prefix (perhps in nother string) > ssume no subwords within P 5

6 keyword tree - trie edges ~ letters t t o p o e t t e r r s y i c e n h c o { potto, poetry, pottery, science, school } o l y e leves ~ keywords filure links o t t o p t t o h e t o o t t h e r e { potto, tttoo, theter, other } r potto other potto tttoo filure links into other brnches! => into other ptterns 6

7 Algorithm for dding the filure links: follow the links existing filure links new edge with lbel : follow existing filure links strting t the prent node of the edge until n outgoing edge with lbel is found Adding filure links o t t o p t t t o o h o e t t h e r e { potto, tttoo, theter, other } r potto other t heter ttoo bredth first (level-by-level) 7

8 filure links o t t o p t t o h e t o o t h t e r e { potto, tttoo, theter, other } r shortcuts root to child of root (see for exmple po ) [single letter] Preprocess text T: Suffix Trees 8

9 trie vs. suffix tree Text T: bb Suffixes: bb b b b b b Text string T (= bb ) + ll its suffixes b b b b b b b b trie suffix tree From: Trie vs. Suffix Tree Trie: b b b b b Suffix Tree: b b b b b Given text T: Trie(T) = O( T ) 2 qudrtic bd exmple: T = n b n Trie(T) like DFA for the suffixes of T minimize DFA directed cyclic word grph DFA = Deterministic Finite Automton (lso used by Ukkonen) only brnching nodes nd leves represented edges lbeled by substrings of T correspondence of leves nd suffixes T leves, hence < T internl nodes Tree(T) = O( T + size(edge lbels)) liner 9

10 nitty Order of insertion is position of suffix in Suffix Tree of text T: nitty itty tty ty y ritty itty tty ty y nitty 2 8 ε itty ty ε 3 9 t y y ε ritty 4 10 ε Text T nd its suffixes: nitty nitty itty tty ty y ritty itty tty ty y nitty ε itty t 3-3 ty 4-5 y ε ritty 7-11 y ε ε implementtion: refer to positions of the substrings in T 10

11 liner time construction Text T nd its suffixes: nitty itty tty ty y ritty itty tty ty y Weiner (1973) lgorithm of the yer McCreight (1976) on-line lgorithm (Ukkonen 1992) Filure link to mximl prefix of suffix in T online construction of suffix trie for T = bb suffix links b b bb bb b b εb ε b b b b b b ε ε b b b b b next symbol = b from here b lredy exists 11

12 Suffix Tree ppliction: full text index T P pos pos P in T P is prefix of suffix of T P subtree under P ~ loctions of P pos pos exmple: find itt in nitty nitty itty tty ty y ritty itty tty ty y nitty 2 8 ε itty ty ε ritty t y y ε ε positions 12

13 Suffix Tree ppliction: longest common substring T P pples plte T pos pos P Construct generlized suffix tree Contining suffixes of both T nd T. Mrk T nd T suffixes: pos pos ppliction: counting motifs nitty itty tty ty y ritty itty tty ty y nitty 2 8 ε itty ty 2 2 ε 11 y t 4 y 2 ε ritty 7 2 ε The suffix tree now contins counts for the number of suffixes in sub-tree 13

14 Suffix Trees Experiments: motifs, repets in DNA s reported by Ukkonen humn chromosome 3 the first bses 31 min cpu time (8 processors, 4 GB) humn genome: 3x10 9 bses suffix tree for Humn Genome is fesible longest repet? Occurrences t: , r Length: 2559 ttgggtctgtgcccgtgcggtttgttcttgttccgtgcctgtggtgtgctgccccttctcgtctttgcgtt ggtttctccgtgcttccctcccccctccccccccccccgtccccggtgtgtgtgttccccttcctgtgtcctgtgttctc ttgttcttcccccttggtggctgcggtgtttggttttttgtccttgcggtttgctggtgtggtttccgcttctcct tccctcggctgctctcttttttttggctgctgtttcctggtgtttgtgcccttttcttcccgtctcccttgttg gctctgggttggttccgtctttgctttgtgtgtgccgctctcgtgtgctgtgtcttttgcgctgtttttcc tttgggtttcccgttgggtggctgggtctggttttctgttctgtccctgggtccccctgcttccctggt tgctgtttcgtcccgccgttccttttctccctcctctccgccctgttgtttcctgcttttttgtcgccttctctg gtgtggtggttctcttgtggttttgtttgctttctctgtggccgtgtgtggcttttttctgtgttttttggctgcttgtctt cttttgggtgtctgttcttccttcgccccttttgtggggttgtttgtttttttcttgttttgttgggttcttgtgttctgggttt gccctttgtcgtggtggttgcttttctcccttctgtggttgcctgttcctctgtggtggtttcttctgctgtgcggctct ttgtttttgtccctttgtcttttggcttttgttgcctgcttttggtgttttgctggtccttgccctgccttgtcctgtg gtttgcctggttttcttctgggttttttggttttggtctctgtgtcttttcctcttgtttttggtgtttttggtg ttttggtgtttttttttttggtgtttttttggtgtgggggtccgtttcgctttctcttggctg ccgttttccctgcccttttttgggtcctttccccttgcttgtttttgtcggtttgtcgtcgtgttgtgttgcgg ctttttctggggctctgttctgttccttggtctttctctgttttggtccgtcctgctgttttggttctgtgccttgtgttgttt ggtcggtgcgtgtggttccgctttgttcttttggcttggttgcttggctgtgggctcttttttggttccttgctttgt gttttttccttctgtggttcttggtgcttgtggggtggcttgtcttttccctgggcgttggccttttcc tttgtcttcctccctggcgtgtctgttcttcctttgtttgttcctctttttttcttggcgtggtttgtgttctccttggggt ccttcctcccttgtgttggttcctggtttttttctctttggcttgtgtggggttcctctgtttgctctctgtttgtctg ttttggtgttgtgcttgtgtttttgccttgttttgttcctggctttgctggttgctttcgcttgggttttgggctg gcgtggggttttctgttctctgtctctgccgggctttgcttcctcttttcctttgtcccgtttttccctctc ctgcctgttgccctggccgcttccccttgttgtgggtggtggggggctccctgtcttgtgccgttttcggg tgcttccgtttttgtccttcgttgtttggctgtgggtttgtctgtgctcttttttttggtctccctctcctttt ttgggtttttgctgggttcttgttttgtcggccttttctgctcttttggttctgtggtttctgtctttggttctgtttt tgctgggtcgtttttgttttcgttgttgccgccttgctcccgggtggccccttgtctggtggtgctttttgtgt gctgctggttcggtttgccgtttttttgggtttctgctcgtgttctcggtttggtctttctctttttttgttgtgtctctgt cggctttggttcggtgtgctggcctcttggttgg 14

15 ten occurrences? ttttttttttttttggcgggtctcgctctgtcgcccggctgggtgcgtg gcgggtctcggctcctgcgctccgcctcccgggttccgccttct cctgcctcgcctcccgtgctgggctcggcgcccgccctcg cccggctttttttgttttttgtggcggggtttcccgttttgccgg gtggtctcgtctcctgcctcgtgtccgcccgcctcggcctcccg tgctgggttcggcgt Length: 277 Occurrences t: , , , , , In the reversed complement t: , , , Suffix Tree Remrks suffix tree efficient (liner) storge, but constnt ±40, lrge, overhed suffix rry hs constnt ±5 overhed hence more prcticl but hs its own complictions nïve n log(n) lgorithm is in some cses not too bd (next slide) 15

16 Suffix Arry nitty itty tty ty y ritty itty tty ty y itty itty nitty ritty tty tty ty ty y y lexicogrphic order of the suffixes sources Dn Gusfield Algorithms on Strings, Trees, nd Sequences Computer Science nd Computtionl Biology lists mny pplictions for suffix trees (nd extended implementtion detils) slides on suffix-trees bsed on/copied from Esko Ukkonen, Univ Helsinki (Erice School, 30 Oct 2005) 16

17 Next Genertion Sequencing Bsed on Lecture Notes by R. Shmir [7] E.M. Bkker Overview Introduction Next Genertion Technologies The Mpping Problem The MAQ Algorithm The Bowtie Algorithm Burrows-Wheeler Trnsform Sequence Assembly Problem De Bruijn Grphs Other Assembly Algorithms 17

18 Introduction 1953 Wtson nd Crick: the structure of the DNA molecule DNA crrier of the genetic informtion, the chllenge of reding the DNA sequence becme centrl to biologicl reserch. methods for DNA sequencing were extremely inefficient, lborious nd costly Holley: relibly sequencing the yest gene for trna Al required the equivlent of full yer's work per person per bse pir (bp) sequenced (1bp/person yer) Two clssicl methods for sequencing DNA frgments by Snger nd Gilbert. Snger sequencing (sketch):. The polymerse extends the lbeled primer, rndomly incorporting either: norml dctp bse, or modified ddctp bse At every position where ddctp is inserted, polymeriztion termintes! => popultion of frgments, where the length of ech frgment is function of the reltive distnce from the modified bse to the primer. b. electrophoretic seprtion of the products of ech of the four rection tubes (ddg, dda, ddt, nd ddc), run in individul lnes. The bnds on the gel represent the respective frgments (Lbelled Strnds) shown to the right => the complement of the originl templte (bottom to top) 18

19 Introduction 1980 In the 1980s methods were ugmented by prtil utomtion the cloning method, which llowed fst nd exponentil repliction of DNA frgment Strt of the humn genome project: sequencing efficiency reched 200,000 bp/person-yer End of the humn genome project: 50,000,000 bp/person-yer. (Totl cost summed to $3 billion.) Note: Moore s Lw doubling trnsistor count every 2 yers Here: doubling of #bse pirs/person-yer every 1.5 yers Introduction Recently ( /2012): new sequencing technologies next genertion sequencing (NGS) or deep sequencing relible sequencing of 100x10 9 bp/person-yer. NGS now llows compiling the full DNA sequence of person for ~ $10,000 within 3-5 yers cost is expected to drop to ~$1000. (2013) full humne genome scn with coverge of 3 for round 2000 euro. 19

20 From: E.C. Hyden, The 1000$ Genome. Nture, Mrch Next Genertion Sequencing Technologies Next Genertion Sequencing technology of Illumin, one of the leding compnies in the field: DNA is replicted => millions of copies. All DNA copies re rndomly shredded into vrious frgments using restriction enzymes or mechnicl mens. The frgments form the input for the sequencing phse. Hundreds of copies of ech frgment re generted in one spot (cluster) on the surfce of huge mtrix. Initilly it is not known which frgment sits where. Note: There re quite few other vilble technologies vilble nd currently under development. 20

21 Next Genertion Sequencing technology of Illumin (continued): A DNA repliction enzyme is dded to the mtrix long with 4 slightly modified nucleotides. The 4 nucleotides re slightly modified chemiclly so tht: ech would emit unique color when excited by lser, ech one would terminte the repliction. Hence, the growing complementry DNA strnd on ech frgment in ech cluster is extended by one nucleotide t time. Next Genertion Sequencing technology of Illumin (continued): A lser is used to obtin red of the ctul nucleotide tht is dded in ech cluster. (The multiple copies in cluster provide n mplified signl.) The solution is wshed wy together with the chemicl modifiction on the lst nucleotide which prevented further elongtion nd emitted the unique signl. Millions of frgments re sequenced efficiently nd in prllel. The resulting frgment sequences re clled reds. Reds form the input for further sequence mpping nd ssembly. 21

22 Next Genertion Sequencing Technologies (1) A modified nucleotide is dded to the complementry DNA strnd by DNA polymerse enzyme. (2) A lser is used to obtin red of the nucleotide just dded. (3) The full sequence of frgment thus determined through successive itertions of the process. (4) A visuliztion of the mtrix where frgment clusters re ttched to flow cells. The Short Red Mpping Problem The Mpping Problem INPUT: QUESTION: m reds S 1,, S m of length l nd n pproximte reference genome R. Wht re the positions x 1,, x m long R where ech red S 1,, S m mtches, respectively? For exmple: fter sequencing the genome of person we wnt to mp it to n existing sequence of the humn genome. The new smple will not be 100% identicl to the reference genome: nturl vrition in the popultion some reds my lign to their position in the reference with mismtches or gps sequencing errors repetitive regions in the genome mke it difficult to decide where to mp the red Humns re diploid orgnisms different lleles on the mternl nd pternl chromosomes two slightly different reds mpping to the sme loction (some perhps with mismtches) 22

23 Possible Solutions for the Mpping Problem The problem of ligning short red to long genome sequence is exctly the problem of locl lignment. But in cse of the humn genome: the number of reds m is usully the length of red l is bp the length of the genome R is 3x 10 9 bp (or twice, for the diploid genome). Solutions for the Mpping Problem Nive lgorithm for ech S i scn the reference string R mtch the red t ech position p nd pick the best mtch Time complexity: O(m l R ) for exct or inexct mtching. m number of reds, l red length Considering the prmeters for our problem, this is clerly imprcticl. Less nive solution use the Knuth-Morris-Prtt lgorithm to mtch ech S i to R Time complexity: O(m (l+ R )) = O(ml + m R ) for exct mtching. A substntil improvement but still not enough 23

24 The Mpping Problem: Suffix Trees 1. Build suffix tree for R. 2. For ech S i (i in {1,,m})find mtches by trversing the tree from the root. Time complexity: O(ml+ R ), ssuming the tree is built using Ukkonen's liner-time lgorithm. Notes: Time complexity is prcticl. we only need to build the tree for the reference genome once. The tree cn be sved nd used repetedly for mpping new sequences, until n updted version of the genome is issued. The Mpping Problem: Suffix Trees Spce complexity now becomes the obstcle: The lefs of the suffix tree lso hold the indices where the suffixes begin => the tree requires O( R log R ) bits just for coding of the indices. The constnts re lrge due to the dditionl lyers of informtion required for the tree (such s suffix links, etc.). The originl humn genome reference sequence demnds just R log(#symbols) bits => we cn store the humn genome using ~750MB But! ~64GB for the Suffix Tree (2015: doble)! Finl problem: suffix trees llow only for exct mtching. 24

25 The Mpping Problem: Hshing Preprocess the reference genome R into hsh tble H. The keys of the hsh re ll the substrings of length l in R: Tble H contins: the position p in R where the substring ends. Mtching: given S i the lgorithm returns H(S i ). Time complexity: O(m l+l R ) Note: The spce complexity is O(l R + R log R ) since we must lso hold the binry representtion of ech substring's position. (Still too high.) Pcking the substrings into bit-vectors representing ech nucleotide s 2-bit code reduces the spce complexity by fctor of 4. Prcticl solution: Prtition the genome into severl chunks, ech of the size of the cche memory, nd run the lgorithm on ech chunk in turn. Agin, only exct mtching llowed. The Mpping Problem: The MAQ Algorithm The MAQ (Mpping nd Alignment with Qulities) lgorithm by Li, Run nd Durbin, 2008 [5] An efficient solution to the short red mpping problem Allows for inexct mtching, up to predetermined mximum number of mismtches. Memory efficient. Mesures for bse nd red qulity, nd for identifying sequence vrints. 25

26 The MAQ Algorithm A key insight: If red in its correct position to the reference genome hs one mismtch, then prtitioning tht red into two mkes sure one prt is still exctly mtched. Hence, if we perform n exct mtch twice using only subset of the red bses ( templte), one of the subsets is enough to find the mtch. For 2 mismtches, 6 templtes: some re noncontiguous ech templte covers hlf the red Ech templte hs prtner templte tht complements it to form the full red If the red hs no more thn two mismtches, t lest one templte will not be ffected by ny mismtch. For 3-mismtches, 20 templtes gurntee t lest one fully mtched templte. Etc. The MAQ Algorithm: Templtes A nd B re reds, where purple boxes indicte mismtches with respect to the reference. The numbered templtes hve blue boxes in the positions they cover. 1 mismtch: compring red A through templte 1 => no full mtch templte 2 fully mtched 2 mismtches: red B is fully mtched with templte 6 Any combintion of up to two mismtched positions will be voided by t lest one of the 6 templtes. Note: 6 templtes gurntee 57% of ll 3 mismtches will lso hve t lest one fully mtching templte. 26

27 The MAQ Algorithm The lgorithm: Generte the number of templtes required to gurntee t lest one full mtch for the desired mximl number of mismtches, nd For ech red: mtch the red ginst the templtes use the exct mtching templte s seed for extending cross the full red. Identifying the exct mtches is done by hshing the red templtes into templte-specific hsh tbles, nd scnning the reference genome R ginst ech tble. Note: It is not necessry to generte templtes nd hsh tbles for the full red, since the initil seed will undergo extension, e.g. using the Smith- Wtermn lgorithm. Therefore the lgorithm initilly processes only the first 28 bses of ech red. (The first bses re the most ccurte bses of the reds.) The MAQ Algorithm Algorithm (for the cse of up to two mismtches): 1. Index the first 28 bses of ech red with complementry templtes 1, nd 2, thereby generting hsh tbles H1 nd H2, respectively. 2. Scn the reference genome R: for ech position nd ech orienttion, query 28-bp window through templtes 1, 2 ginst the pproprite tbles H1, H2, respectively. If hit, extend nd score the complete red bsed on mismtches. For ech red, keep the two best scoring hits nd the number of mismtches therein. 3. Repet steps 1+2 with complementry templtes 3, 4, then 5, 6. Remrk: The reson for indexing ginst pir of complementry templtes ech time hs to do with feture of the lgorithm regrding pired-end reds (see originl pper for detils). 27

28 The MAQ Algorithm Complexity Time Complexity: O(m l) for generting the hsh tbles in Step 1, nd O(l R ) for scnning the genome in Step 2. Repeting Steps 1+2 three times in this implementtion hs no effect on the symptotic complexity. Spce Complexity: O(m l) for holding the hsh tbles in Step 1, nd O(m l+ R ) totl spce for Step 2, but only O(m l) spce in min memory t ny one time. The MAQ Algorithm Complexity Note tht, not the full red length l is used, but window of length l' <= l (28-bp in the implementtion presented bove). The size of ech key in templte hsh tble is only l' /2, since ech templte covers hlf the window. => Time nd spce complexity of step 1 is: O(2 m l' /2) = O(ml' ). This reduction in spce mkes running the lgorithm on PC fesible. The time nd spce required for extending mtch in Step 2 using the Smith-Wtermn lgorithm, where p the probbility of hit (using the l length window): The time complexity : O(l' R + p R l 2 ). The spce complexity: O(ml' + l 2 ). Note: The vlue of p is smll, nd decreses drsticlly the longer l' is, since most l' -long windows in the reference genome will likely not cpture the exct coordintes of true red. 28

29 Red Mpping Qulities The MAQ lgorithm provides mesure of the confidence level for the mpping of ech red, denoted by QS nd defined s: QS = -10 log 10 (Pr{red S is wrongly mpped}) For exmple: QS = 30 if the probbility of incorrect mpping of red S is 1/1000. This confidence mesure is clled phred-scled qulity, following the scling scheme originlly introduced by Phil Green nd collegues[3] for the humn genome project. The Bowtie Algorithm Another wy to mp reds to reference genome is given by the Bowtie lgorithm, presented in 2009 by Lngmed et l.[1]. It solves the mpping problem using spce-efficient indexing scheme. The indexing scheme used is clled the Burrows- Wheeler trnsform[2] nd ws originlly developed for dt compression purposes. 29

30 The Burrows-Wheeler Trnsform Applying the Burrows-Wheeler trnsform BW(T) to the text T = "the next text tht i index.": 1. First, we generte ll cyclic shifts of T. 2. Next, we sort these shifts lexicogrphiclly. define the chrcter '.' s the minimum nd we ssume tht it ppers exctly once, s the lst symbol in the text. followed lexicogrphiclly by ' (spce) followed by the English letters ccording to their nturl ordering. Cll the resulting mtrix M. The trnsform BW(T) is defined s the sequence of the lst chrcters in the rows of M. Note tht, the lst column is permuttion of ll chrcters in the text since ech chrcter ppers in the lst position in exctly one cyclic shift. Burrows-Wheeler Trnsform Some of the cyclic shifts of T sorted lexicogrphiclly nd indexed by the lst chrcter. 30

31 Burrows-Wheeler Trnsform Storing BW(T) requires the sme spce s the size of the text T since it is simply permuttion of T. In the cse of the humn genome, ech chrcter cn be represented by 2 bits => storing the permuttion requires 2 times 3x10 9 bits (insted of ~30 times 3x10 9 for storing ll indices of T). Burrows-Wheeler Trnsform The following holds for BW(T): 1. The number of occurrences in T of the chrcter c = the number of occurrences of c in BW(T) (= permuttion of the T). 2. The first column of the mtrix M cn be obtined by sorting BW(T) lexicogrphiclly. 3. The number of occurrences of the substring 'xt' in T: BW(T) is the lst column of the lexicogrphicl sorting of the shifts. The chrcter t the lst position of row ppers in the text T immeditely prior to the first chrcter in the sme row (ech row is cyclicl shift). => consider the intervl of 't' in the first column, nd check whether ny of these rows hve n 'x t the lst position. 31

32 Sorted BW(T) BW(T) Recovering the first column (left) by sorting the lst column. Burrows-Wheeler Trnsform Given BW(T) lso the second column cn be derived: 'xt' ppers twice in the text, nd we see tht 3 rows strt with n 'x'. Two of those must be followed by 't, where the lexicogrphicl sorting determines which 'x'. The third 'x' is followed by '.' (see first row) => '.' must follow the first 'x' in the first column since '.' is smller lexicogrphiclly thn 't'. The second nd third occurrences of 'x' in the first column re therefore followed by 't'. Note: We cn use the sme process to recover the chrcters t the second column for ech intervl, nd then the third, etc. t text tht i index. the nex t tht i index. the next tex text tht i index.the next tht i index. the next text the next text tht i index. 32

33 F L Lst-first mpping: Ech 't' chrcter in L is linked to its position in F nd no crossed links re possible. The j-th occurrence of chrcter X in L corresponds to the sme text chrcter s the j-th occurrence of X in F. Burrows-Wheeler Trnsform The previous two centrl properties of the BW-trnsform re cptured in the Lemm by Ferrgin nd Mnzini[4]: Lemm 12.1 (Lst-First Mpping): Let M be the mtrix whose rows re ll cyclicl shifts of T sorted lexicogrphiclly, nd let L(i) be the chrcter t the lst column of row i nd F(i) be the first chrcter in tht row. Then: 1. In row i of M, L(i) precedes F(i) in the originl text: T = L(i) F(i) 2. The j-th occurrence of chrcter X in L corresponds to the sme text chrcter s the j-th occurrence of X in F. 33

34 F L Lst-first mpping: j-th occurrence of chrcter 't' chrcter in L is linked to its j-th position in F nd no crossed links re possible. Burrows-Wheeler Trnsform Proof: 1. Follows directly from the fct tht ech row in M is cyclicl shift. 2. Let X j denote the j-th occurrence of chrcter X in L, nd let α be the chrcter following X j in the text nd β the chrcter following X j+1. Then, since X j ppers bove X j+1 in L, α ppers t the beginning of row bove the row tht strts with β. The rows re lexicogrphiclly ordered, hence α must be equl or lexicogrphiclly smller thn β. Now clerly X α lexicogrphiclly X β holds. Hence, s the rows re lexicogrphiclly ordered, if chrcter X j ppers in F it is followed by α, nd thus will be bove X j+1 which is followed by β. Thus proofing the Lemm. 34

35 Reconstructing the Text Algorithm UNPERMUTE for reconstructing text T from its Burrows-Wheeler trnsform BW(T) utilizing Lemm 12.1 [4]: ssume the ctul text T is of length u ppend unique $ chrcter (=. ) t the end, which is the smllest lexicogrphiclly UNPERMUTE[BW(T)] 1. Compute the rry C[1,, Σ ] : where C(c) is the number of chrcters {$, 1,,c-1} in T, i.e., the number of chrcters tht re lexicogrphiclly smller thn c 2. Construct the lst-first mpping LF, trcing every chrcter in L to its corresponding position in F: LF[i] = C(L[i]) + r(l[i], i) + 1, where r(c, i) is the number of occurrences of chrcter c in the prefix L[1, i -1] 3. Reconstruct T bckwrds: s = 1, T(u) = L[1]; for i = u 1,, 1 do s = LF[s]; T[i] = L[s]; od; Notice, tht C(e) + 1 = 9 is the position of the first occurrence of e' in F. C(e) = 8, s there re 8 chrcters in T tht re smller thn e. C F L Lst-first mpping: Ech 't' chrcter in L is linked to its position in F nd no crossed links re possible. 35

36 Exmple T = ccg$ (u = 6) is trnsformed to BW(T) = gc$c, nd we now wish to reconstruct T from BW(T) using UNPERMUTE: 1. Compute rry C. For exmple: C(c) = 4 since there re 4 occurrences of chrcters smller thn 'c' in T ('$' nd 3 occurrences of ''). Now C(c) + 1 = 5 is the position of the first occurrence of 'c' in F. 2. Perform the LF mpping. For exmple, LF[c 2 ] = C(c) + r(c,7) + 1 = 6, nd indeed the second occurrence of 'c' in F is t F[6]. 3. Determine the lst chrcter in T: T(6) = L(1) = 'g'. 4. Iterte bckwrds over ll positions using the LF mpping. For exmple, to recover the chrcter T(5), we use the LF mpping to trce L(1) to F(7), nd then T(5) = L(7) = 'c'. Remrk We do not ctully hold F in memory, we only keep the rry C defined bove, of size Σ, which we cn esily obtined from L. r(c,7) = # of occurences of c in length 7-1 prefix, used s n offset to obtin the right c. Determine its loction in F nd find its corresponding predecessor in L using C(.) nd r(.,.) i.e., find corresponding g using the lst-first mpping First Lst Exmple of running UNPERMUTE to recover the originl text. Source: [1]. 36

37 Exct Mtching EXACTMATCH exct mtching of query string P to T, given BW(T)[4] is similr to UNPERMUTE, we use the sme C nd r(c, i) denote by sp the position of the first row in the intervl of rows in M we re currently considering denote by ep the position of the first row beyond this intervl of rows => the intervl of rows is defined by the rows sp,, ep - 1. EXACTMATCH[P[1,,p], BW(T)] 1. c = P[p]; sp = C[c] + 1; ep = C[c+1] + 1; i = p - 1; 2. while sp < ep nd i >= 1 c = P[i]; sp = C[c] + r(c, sp) + 1; ep = C[c] + r(c, ep) + 1; i = i - 1; 3. if (sp == ep) return "no mtch"; else return sp, ep; P = c Exmple of running EXACTMATCH to find query string in the text. Source: [1]. 37

38 Inexct Mtching Sketch Ech chrcter in red hs numeric qulity vlue, with lower vlues indicting higher likelihood of sequencing error. Similr to EXACTMATCH, clculting mtrix intervls for successively longer query suffixes. If the rnge becomes empty ( suffix does not occur in the text), then the lgorithm selects n lredy-mtched query position nd substitute different bse there, introducing mismtch into the lignment. The EXACTMATCH serch resumes from just fter the substituted position. The lgorithm selects only those substitutions tht re consistent with the lignment policy nd yield modified suffix tht occurs t lest once in the text. If there re multiple cndidte substitution positions, then the lgorithm greedily selects position with mximl qulity vlue. The Assembly Problem How to ssemble n unknown genome bsed on mny highly overlpping short reds from it? Problem Sequence ssembly INPUT: m l-long reds S 1,, S m. QUESTION: Wht is the sequence of the full genome? The crucil difference between the problems of mpping nd ssembly is tht now we do not hve reference genome, nd we must ssemble the full sequence directly from the reds. 38

39 De Bruijn Grphs Definition A k-dimensionl de Bruijn grph of n symbols is directed grph representing overlps between sequences of symbols. It hs n k vertices, consisting of ll possible k-tuples of the given symbols. The sme symbol my pper multiple times in tuple. If we hve the set of symbols A = { 1,, n } then the set of vertices is: V = { ( 1,, 1, 1 ), ( 1,, 1, 2 ),, ( 1,, 1, n ), ( 1,, 2, 1 ),, ( n,, n, n )} If vertice w cn be expressed by shifting ll symbols of nother vertex v by one plce to the left nd dding new symbol t the end, then v hs directed edge to w. Thus the set of directed edges E is: E = {( (v 1, v 2,, v k ), (w 1,w 2,, w k ) ) v 2 = w 1, v 3 = w 2,, v k = w k-1, nd w k new} A portion of 4-dimensionl de Bruijn grph. The vertex CAAA hs directed edge to vertex AAAC, becuse if we shift the lbel CAAA to the left nd dd the symbol C we get the lbel AAAC. In this wy the word CAAAC defines directed edge 39

40 De Bruijn Grphs Given red, every contiguous (k+1)-long word in it corresponds to n edge in the de Bruijn grph. Form subgrph G of the full de Bruijn grph by introducing only the edges tht correspond to (k+1)-long words in some red. A pth in this grph defines potentil subsequence in the genome. Hence, we cn convert red to its corresponding pth in the constructed subgrph G Using the grph from the previous slide, we construct the pth corresponding to CCAACAAAAC: shift the sequence one position to the left ech time, nd mrking the vertex with the lbel of the 4 first nucleotides. 40

41 De Bruijn Grphs Form pths for ll the reds Identify common vertices on different pths Merge different red-pths through these common vertices of the pths into one long sequence The two reds in () re converted to pths in the grph The common vertex TGAG is identified Combine these two pths into (b). 41

42 De Bruijn Grphs Velvet (2008) by Zerbino nd Birney[6], is n lgorithm which uses de Bruijn grphs to ssemble reds; with the following difficulties: repets will show up s cycles in the merged pth tht we form. We do not know how mny times ech cycle must be trversed in order to form the full sequence of the genome. If we hve two cycles strting t the sme vertex, we cnnot tell which one to trverse first. Velvet ttempts to ddress this issue by utilizing the extr informtion we hve in the cse of pired-ends sequencing (both ends of DNA frgment re sequenced in Red 1 nd Red 2, respectively. The distnce between ech pired red is known). Other Assembly Algorithms HMM bsed Mjority bsed Etc. 42

43 Bibliogrphy [1] Ben Lngmed, Cole Trpnell, Mihi Pop nd Steven L. Slzberg. Ultrfst nd memory-effcient lignment of short DNA sequences to the humn genome. Genome Biology, [2] Michel Burrows nd Dvid Wheeler. A block sorting lossless dt compression lgorithm. Technicl Report 124, Digitl Equipment Corportion, [3] Brent Ewing nd Phil Green. Bse-clling of utomted sequencer trces using phred. II. Error probbilities. Genome Reserch, [4] Pulo Ferrgin nd Giovni Mnzini. Opportunistic dt structures with pplictions. FOCS '00 Proceedings of the 41st Annul Symposium on Foundtions of Computer Science, [5] Heng Li, Jue Run nd Richrd Durbin. Mpping short DNA sequencing reds nd clling vrints using mpping qulity scores. Genome Reserch, [6] Dniel R. Zerbino nd Ewn Birney. Velvet: lgorithms for de novo short red ssembly using de Bruijn grphs. Genome Reserch, [7] Ron Shmir, Computtionl Genomics Fll Semester, 2010 Lecture 12: Algorithms for Next Genertion Sequencing Dt, Jnury 6, 2011 Scribe: Ant Gluzmn nd Ern Mick Appendix Some bsic terminology nd techniques 43

44 DNA Repliction DNA Polymerse is fmily of enzymes crrying out DNA repliction. A Primer is short frgment of DNA or RNA tht pirs with templte DNA. Polymerse Chin Rection Mixture of templtes nd primers is heted such tht templtes seprte in strnds Uses pir of primers spnning trget region in templte DNA Polymerizes prtner strnds in ech direction from the primers using thermo stble DNA polymerse repet BWT Exct Mtching Exmple Use T = ccg while serching for P = 'c': 1. First, we initilize sp nd ep to define the intervl of rows beginning with the lst chrcter in P, which is 'c': sp = C(c) + 1 = 5. ep = C(g) + 1 = Next, we consider the preceding chrcter in P, nmely ''. We redefine the intervl s the rows tht begin with 'c' utilizing the LF mpping. Specificlly: sp = C() + r(, 5) + 1 = = 3. ep = C() + r(, 7) + 1 = = Now, we consider the next preceding chrcter, nmely ''. We now redefine the intervl s the rows tht begin with 'c'. Specificlly: sp = C() + r(, 3) + 1 = = 2. ep = C() + r(, 5) + 1 = = Hving thus covered ll positions of P, we return the finl intervl clculted (whose size equls the number of occurrences of P in T). 44

45 BWT Exct Mtching EXACTMATCH returns the indices of rows in M tht begin with the query it does not provide the offset of ech mtch in T. If we kept the position in T corresponding to the strt of ech row we would wste lot of spce. we cn mrk only some rows with pre-clculted offsets. Then, if the row where EXACTMATCH found the query hs this offset, we cn return it immeditely. Or use LF mpping to find row tht hs preclculted offset, nd dd the number of times the procedure is pplied to obtin the position in which we re interested. => time-spce trde-of ssocited with this process. DNA Repliction Enzyme 45

46 Primers 46

47 DNA Restriction DNA restriction enzymes Recognize specific nucleotide sequence (4 8 bp long, often plindromic) Mke double-strnded DNA cut For exmple HeIII recognizes the following sequences nd cuts between djcent G nd C giving blunt end: 5 GGCC3 3 CCGG5 Also restriction endonuclese (restriction enzymes) tht cut with n offset nd thus producing sticky ends re possible. Restriction Enzymes 47

48 Red Mpping Qulities The MAQ lgorithm provides mesure of the confidence level for the mpping of ech red, denoted by QS nd defined s: QS = -10 log 10 (Pr{red S is wrongly mpped}) For exmple: QS = 30 if the probbility of incorrect mpping of red S is 1/1000. This confidence mesure is clled phred-scled qulity, following the scling scheme originlly introduced by Phil Green nd collegues[3] for the humn genome project. QS Confidence Levels A simplified sttisticl model for inferring QS: Assume we know the reds re supposed to mtch the reference precisely. So ny mismtches must be the result of sequencing errors. Assuming tht sequencing errors long red re independent, we cn define: the probbility p(z x, u) of obtining the red z given tht its position in reference x is u s the product of the error probbilities of the observed mismtches in the lignment of the red t tht position. These bse error probbilities re empiriclly determined during the sequencing phse bsed on the chrcteristics of the emitted signl t ech step. 48

49 QS Exmple If red z ligned t position u yields 2 mismtches tht hve phred-scled bse qulity of 10 nd 20, then: p(z x, u) = 10 -(20+10)/10 = Note tht we will be ble to clculte the posterior probbility p s (u x, z) of position u being the correct mpping for n observed red z using Byes' theorem (see text). Being ble to clculte these posterior probbility, we cn now give our confidence mesure s: Q s (u x, z) = -10 log 10 (1 - p s (u x, z)) Note: MAQ uses more sophisticted sttisticl model to del with the issues we neglected in this simplifiction, such s true mismtches tht re not the result of sequencing errors (SNPs = single nucleotide polymorphisms). 49