Exact String Matching and searching for SNPs (2) CMSC423

Transcription

1 Exact String Matching and searching fr SNPs (2) CMSC423

2 The prblem Given: 100 s f millins f shrt reads: bp reads A lng reference genme (~3Bbp fr human) D: Find high scring scring (fifng) alignments fr each read What we knw: Dynamic prgramming slukn fr fifng alignment: 1e8 * 1e9 * 1e2 perakns, 1e9 * 1e2 memry 2

3 Strategies What if we nly allw a small number f subsktukns? Let s first try t find exact matches and wrk frm thse (the d+1 trick in the midterm) We are aligning t the same reference 100 s f millins f Kme Is there preprcessing we can d t amrkze Kme? Genmes are repekkve Can we search fr matches in the genme in a smart way? Can we cmpress the genme, and search ver the cmpressed representakn? 3

4 Suffix Tree T: abaaba$ Cllapse nn-branching ndes - #ndes O( T ) Memry requirement is nt O( T ) - In the wrst case, space required fr edge labels is O( T )

5 Suffix Tree T: abaaba$ Cllapse nn-branching ndes - #ndes O( T ) Label edges with substring [start,end] - O(1) per edge Memry nw O( T ) Cnstructin algrithm O( T ) (see Gusfield)

6 Recap Structure Prcessing Time Memry Search Suffix Trie O( T ) O( T 2 ) O( P ) Suffix Tree O( T ) O( T )* O( P ) Suffix Array O( T ) O( T ) (but much smaller than Suffix Tree) O( P lg2 T ) *In best implementatins abut 20 bytes per character (as ppsed t 4 bytes fr suffix array)

7

8 Suffix Arrays Even thugh Suffix Trees are O(n) space, the cnstant hidden by the big-oh ntatin is smewhat big : 20 bytes / character in gd implementatins. If yu have a 10Gb genme, 20 bytes / character = 200Gb t stre yur suffix tree. Linear but large. Suffix arrays are a mre efficient way t stre the suffixes that can d mst f what suffix trees can d, but just a bit slwer. Slight space vs. time tradeff.

9 Example Suffix Array s = attcatg$ Idea: lexicgraphically srt all the suffixes. Stre the starting indices f the suffixes in an array attcatg$ ttcatg$ tcatg$ catg$ atg$ tg$ g$ $ srt the suffixes alphabetically the indices just cme alng fr the ride $ atg$ attcatg$ catg$ g$ tcatg$ tg$ ttcatg$ index f suffix suffix f s

10 Example Suffix Array s = attcatg$ Idea: lexicgraphically srt all the suffixes. Stre the starting indices f the suffixes in an array attcatg$ ttcatg$ tcatg$ catg$ atg$ tg$ g$ $ srt the suffixes alphabetically the indices just cme alng fr the ride index f suffix suffix f s

11

12 Search via Suffix Arrays s = cattcat$ $ at$ attcat$ cat$ cattcat$ t$ tcat$ ttcat$ Des string at ccur in s? Binary search t find at. What abut tt?

13 Cunting via Suffix Arrays s = cattcat$ $ at$ attcat$ cat$ cattcat$ t$ tcat$ ttcat$ Hw many times des at ccur in the string? All the suffixes that start with at will be next t each ther in the array. Find ne suffix that starts with at (using binary search). Then cunt the neighbring sequences that start with at.

14 Cnstructing Suffix Arrays Easy O(n 2 lg n) algrithm: srt the n suffixes, which takes O(n lg n) cmparisns, where each cmparisn takes O(n). There are several direct O(n) algrithms fr cnstructing suffix arrays that use very little space. The Skew Algrithm is ne that is based n divide-and-cnquer. An simple O(n) algrithm: build the suffix tree, and explit the relatinship between suffix trees and suffix arrays (next slide)

15 Relatinship between Suffix Arrays and Suffix Trees T: abaaba$ $ 5 a$ 2 aaba$ 3 aba$ 0 abaaba$ 4 ba$ 1 baaa$ Build suffix trees with edge labels srted lexicgraphically Order f leaves: 6,5,2,3,0,4,1

16 Recap Structure Prcessing Time Memry Search Suffix Trie O( T ) O( T 2 ) O( P ) Suffix Tree O( T ) O( T )* O( P ) Suffix Array O( T ) O( T ) (but much smaller than Suffix Tree) O( P lg2 T ) *In best implementatins abut 20 bytes per character (as ppsed t 4 bytes fr suffix array)

17 Burrws-Wheeler Transfrm Text transfrm that is useful fr cmpressin & search. banana banana$ anana$b nana$ba ana$ban na$bana a$banan $banana srt $banana a$banan ana$ban anana$b banana$ nana$ba na$bana BWT(banana) = annb$aa Tends t put runs f the same character tgether. Makes cmpressin wrk well. bzip is based n this.

18 Anther Example appellee$ appellee$ ppellee$a pellee$ap ellee$app llee$appe lee$appel ee$appell e$appelle $appellee srt $appellee appellee$ e$appelle ee$appell ellee$app lee$appel llee$appe pellee$ap ppellee$a BWT(appellee$) = e$elplepa Desn t always imprve the cmpressibility...

19

20 BWT e $ Recvering the string srt BWT first clumn $a first 2 clumns e $a $ap first 3 clumns $appellee appellee$ e$appelle ee$appell ellee$app lee$appel llee$appe pellee$ap ppellee$a $ e l p l e a e e e l l srt these 2 clumns ap e$ ee el le ll prepend BWT clumn $ e l p l e ap e$ ee el le ll Srt these 3 clumns app e$a ee$ ell lee lle p p pe p pe pel a p pp a pp ppe

21 Inverse BWT def inversebwt(s): B = [s1,s2,s3,...,sn] fr i = 1..n: srt B prepend si t B[i] return rw f B that ends with $

22 Anther BWT Example dgwd$ $dgwd gwd$d d$dgw gwd$d dgwd$ wd$dg srt gwd$d last clumn d$dgw d$dgw BWT(dgwd$) = d$dgw gwd$d d$dwg d$dgw d$dgw $dgwd wd$dg

23 Anther BWT Example d$dwg d $ d w g $ d d g w $d d$ d gw d g w $d d$ d gw d g w d $ d w g $d d$d dg gw d$ gw d w d $ d w g $d d$d dg gw d$ gw d w $dg d$d dgw gw d$d gw d$ wd $dg d$d dgw gw d$d gw d$ wd d $ d w g $dgw d$dg dgw gwd d$d gw d$d wd$ $dgw d$dg dgw gwd d$d gw d$d wd$ d $ d w g $dgw d$dgw dgw gwd$ d$dg gwd d$d wd$d d $ d w g $dgw d$dgw dgw gwd$ d$dg gwd d$d wd$d $dgw d$dgw dgwd gwd$d d$dgw gwd$ d$dg wd$d $dgw d$dgw dgwd gwd$d d$dgw gwd$ d$dg wd$d d $ d w g $dgwd d$dgw dgwd$ gwd$d d$dgw gwd$d d$dgw wd$dg Prepend Srt Prepend Srt Prepend Srt Prepend Srt Prepend Srt Prepend Srt Prepend Srt

24 Searching with BWT: LF Mapping BWT(unabashable) $unabashable abashable$un able$unabash ashable$unab bashable$una ble$unabasha e$unabashabl hable$unabas le$unabashab nabashable$u shable$unaba unabashable$ LF Mapping $ a b e h l n s u # f times letter appears befre this psitin in the last clumn. LF Prperty: The i th ccurrence f a letter X in the last clumn crrespnds t the i th ccurrence f X in the first clumn.

25

26 BWT Search BWTSearch(aba) Start frm the end f the pattern Step 1: Find the range f a s in the first clumn Step 2: Lk at the same range in the last clumn. Step 3: b is the next pattern character. Set B = the LF mapping entry fr b in the first rw f the range. Set E = the LF mapping entry fr b in the last + 1 rw f the range. Step 4: Find the range fr b in the first rw, and use B and E t find the right subrange within the b range. BWT(unabashable) $unabashable abashable$un able$unabash ashable$unab bashable$una ble$unabasha e$unabashabl hable$unabas le$unabashab nabashable$u shable$unaba unabashable$ LF Mapping $ a b e h l n s u

27 BWT Searching Example 2 a $bananna a$banann ananna$b anna$ban bananna$ na$banan nanna$ba nna$bana a $bananna a$banann ananna$b anna$ban bananna$ na$banan nanna$ba nna$bana (B,E) = 1, 2 $ a b n $ a b n $ a b n (B,E) = 0, 2 n $bananna a$banann ananna$b anna$ban bananna$ na$banan nanna$ba nna$bana a $bananna a$banann ananna$b anna$ban bananna$ na$banan nanna$ba nna$bana (B,E) = 0, 1 $ a b n pattern = bana n $bananna a$banann ananna$b anna$ban bananna$ na$banan nanna$ba nna$bana b $bananna a$banann ananna$b anna$ban bananna$ na$banan nanna$ba nna$bana $ a b n $ a b n

28 BWT Searching Ntes Dn t have t stre the LF mapping. A mre cmplex algrithm (later slides) lets yu cmpute it in O(1) time in cmpressed data n the fly with sme extra strage. T find the range in the first clumn crrespnding t a character: Pre-cmpute array C[c] = # f ccurrences in the string f characters lexicgraphically < c. Then start f the a range, fr example, is: C[ a ] + 1. Running time: O( pattern ) Finding the range in the first clumn takes O(1) time using the C array. Updating the range takes O(1) time using the LF mapping.

29 Relatinship Between BWT and Suffix Arrays s = appellee$ $appellee appellee$ e$appelle ee$appell ellee$app lee$appel llee$appe pellee$ap ppellee$a $ appellee$ e$ ee$ ellee$ lee$ llee$ pellee$ ppellee$ These are still in srted rder because $ cmes befre everything else subtract 1 s[9-1] = e s[1-1] = $ s[8-1] = e s[7-1] = l s[4-1] = p s[6-1] = l s[5-1] = e s[3-1] = p s[2-1] = a BWT matrix The suffixes are btained by deleting everything after the $ Suffix array (start psitin fr the suffixes) Suffix psitin - 1 = the psitin f the last character f the BWT matrix ($ is a special case)

30 Relatinship Between BWT and Suffix Trees Remember: Suffix Array = suffix numbers btained by traversing the leaf ndes f the (rdered) Suffix Tree frm left t right. Suffix Tree Suffix Array BWT. Ordered suffix tree fr previus example: = {$,e,l,p} s = appellee$ $ 9 llee$ 4 e$ 7 $ e l ee$ lee$ 6 5 p ellee$ 3 pellee$ 3

31 Cmputing BWT in O(n) time Easy O(n 2 lg n)-time algrithm t cmpute the BWT (create and srt the BWT matrix explicitly). Several direct O(n)-time algrithms fr BWT. These are space efficient. Als can use suffix arrays r trees: Cmpute the suffix array, use crrespndence between suffix array and BWT t utput the BWT. O(n)-time and O(n)-space, but the cnstants are large.

32 Recap BWT useful fr searching and cmpressin. BWT is invertible: given the BWT f a string, the string can be recnstructed! BWT is cmputable in O(n) time. Clse relatinships between Suffix Trees, Suffix Arrays, and BWT: Suffix array = rder f the suffix numbers f the suffix tree, traversed left t right BWT = letters at psitins given by the suffix array entries - 1 Even after cmpressin, can search string quickly.

33 Recap Structure Prcessing Time Memry Search Suffix Trie O( T ) O( T 2 ) O( P ) Suffix Tree O( T ) O( T )* O( P ) Suffix Array O( T ) O( T ) (but much smaller than Suffix Tree) O( P lg2 T ) BWT O( T ) O( T )** O( P ) *In best implementatins abut 20 bytes per character (as ppsed t 4 bytes fr suffix array) **Cmpressed! Fr human genme ~2GB

34 Mve-T-Frnt Cding T encde a letter, use its index in the current list, and then mve it t the frnt f the list. List with all letters frm the allwed alphabet $dgw d$gw d$gw $dgw $dgw $dgw d$gw d$dwg wd$g = MTF(d$dwg) Benefits: Runs f the same letter will lead t runs f 0s. Cmmn letters get small numbers, while rare letters get big numbers.

35 Cmpressing BWT Strings Lts f pssible cmpressin schemes will benefit frm preprcessing with BWT (since it tends t grup runs f the same letters tgether). One gd scheme prpsed by Ferragina & Manzini: replace runs f 0s with the cunt f 0s PrefixCde(rle(MTF(BWT(S)))) Huffman cde that uses mre bits fr rare symbls

36 Pseudcde fr CuntingOccurrences in BWT w/ stred LF mapping C[c] = index int first clumn functin Cunt(Sbwt, P): where the c s begin. c = P[p], i = p sp = C[c] + 1; ep = C[c+1] while (sp ep) and (i 2) d c = P[i-1] sp = C[c] + Occ(c, sp-1) + 1 ep = C[c] + Occ(c, ep) i = i - 1 if ep < sp then return nt fund else return ep - sp + 1 Occ(c, p) = # f f c in the first p characters f BWT(S), aka the LF mapping.

37 Cmputing Occ in Cmpressed String Break BWT(S) int blcks f length L (we will decide n a value fr L later): BT1 BT2 BT3... BWT(S) PrefixCde(rle(MTF(BWT(BT2)))) Occ(c, p) = # f c up thru p BZ1 BZ2 BZ3... Assumes every run f 0s is cntained in a blck [just fr ease f explanatin]. We will stre sme extra inf fr each blck (and sme grups f blcks) t cmpute Occ(c, p) quickly.

38 Extra Inf t Cmpute Occ blck BZ1 BZ2 BZ3... L L 2 L 2 superblck superblck: stre -lng array giving # f ccurrences f each character up thru and including this superblck blck: stre -lng array giving # f ccurrences f each character up thru and including this blck since the end f the last super blck.

39 Extra Inf t Cmpute Occ u = cmpressed length Chse L = O(lg u) u/l blcks, each array is lg L lng u L lg L = u lg u lg lg u ttal space. blck: stre -lng array giving # f ccurrences f each character up thru and including this blck since the end f the last super blck. blck BZ1 BZ2 BZ3... L L 2 L 2 superblck superblck: stre -lng array giving # f ccurrences f each character up thru and including this superblck

40 Extra Inf t Cmpute Occ u = cmpressed length Chse L = O(lg u) u/l blcks, each array is lg L lng u L lg L = u lg u lg lg u ttal space. blck: stre -lng array giving # f ccurrences f each character up thru and including this blck since the end f the last super blck. blck BZ1 BZ2 BZ3... L L 2 L 2 superblck superblck: stre -lng array giving # f ccurrences f each character up thru and including this superblck u/l 2 superblcks, each array is lg u lng u ttal space. (lg u) 2 lg u = u lg u

41 u = cmpressed length Chse L = O(lg u) Extra Inf t Cmpute Occ blck BZ1 BZ2 BZ3... L L 2 L 2 superblck Occ(c, p) = # f c up thru p: sum value at last superblck, value at end f previus blck, but then need t handle this blck. Stre an array: M[c, k, BZi, MTFi] = # f ccurrences f c thrugh the kth letter f a blck f type (BZi, MTFi). Size: O( L2 L ) = O(L2 L ) = O(u c lg u) fr c < 1 (since the string is cmpressed)