11/3/13. Indexing techniques. Short-read mapping software. Indexing a text (a genome, etc) Some terminologies. Hashing

Transcription

1 I9 Introdution to Bioinformtis, 0 Indeing tehniques Yuzhen Ye (yye@indin.edu) Shool of Informtis & Computing, IUB Contents We hve seen indeing tehnique used in BLAST Applitions tht rely on n effiient indeing tehnique Compring short reds ginst referene genomes (e.g., RNA-Seq dt nlysis) Compring lrge genomes (e.g., MUMer) Indeing tehniques Hsh tble Suffi tree & suffi rry BWT Short-red mpping softwre Softwre Tehnique Developer Elnd Hshing reds Illumni SOAP Hshing refs BGI Mq Hshing reds Snger (Li, Heng) Bowtie & Bowtie BWT Slzberg/UMD BWA BWT Snger (Li, Heng) SOAP BWT & hshing BGI Indeing tet ( genome, et) Emple : we wnt to inde genome suh tht we n look up ny k-mer long the genome in O() time (without snning the whole genome). Emple : we wnt to inde protein dtbse suh tht we n look up ll the proteins ontining word (k-mer) in onstnt time. All need to use n effiient indeing tehnique netgenertionsequening.html (under Alignment tegory) Hshing Hshing is n indeing tehnique tht enble fst serh by omputing inde diretly bsed on the key inde key ASTTSA ASTTSS ASTTST vlues protein protein9 protein90 Some terminologies The proess of finding reord using some omputtion to mp its key vlue to position in the rry is lled hshing. The funtion tht mps key vlues to positions is lled hsh funtion (h). The rry tht holds the hsh tble is lled the hsh tble (HT). A position in the hsh tble is lso known s slot. Hsh funtion key inde

2 Hsh funtions nd ollisions The funtion tht mps key vlues to positions is lled hsh funtion (h). Typilly there re mny more vlues in the key rnge thn there re slots in the hsh tble. Given hsh funtion h nd two keys k nd k, if h(k )=h(k )=β, we sy tht k nd k hve ollision t slot β under hsh funtion h. Perfet hshing is system in whih reords re hshed suh tht there re no ollisions (e.g., indeing k-mers when k is smll). An idel hsh funtion stores the tul reords in the olletion suh tht eh slot in the hsh tble hs equl probbility of being filled; but lustering of reords hppens (mny reords hsh to only few of the slots) A simple hsh funtion for integers A funtion used to hsh integers to tble of slots int h(int ) { return ( % ) } The vlue returned by this hsh funtion depends solely on the lest signifint four bits of the key. These bits re likely to be poorly distributed (s n emple, high perentge of the keys might be even numbers, so the low order bit is zero), so the result my lso be poorly distributed. A simple hsh funtion for k-mers k= inde key The strings re DNA sequenes. 0 A int h(string, int k): T bint = { A :0, T :, C :, G :} C dd = id = 0 for i in k to : b = [i] id = id + bseint[b] dd dd = dd return id G How mny slots in the tble for k =? k = 0? h( CA, ) = 8 k= inde key 0 AA AT AC AG TA TT TC TG CA CT CC CG GA GT GC GG Collision resolution While the gol of hsh funtion is to minimize ollisions, some ollisions re unvoidble in prtie. Hshing implementtions must inlude some form of ollision resolution poliy. Two lss of ollision resolution tehniques: Open hshing (seprte hining) ollisions re stored outside the tble Closed hshing ollisions result in storing one of the reords t nother slot in the tble. Open hshing The simplest form of open hshing defines eh slot in the hsh tble to be the hed of linked list. All reords tht hsh to prtiulr slot re pled on tht slot s linked list. Reords within slot s list n be ordered in severl wys: by insertion order, by key vlue order, or by frequeny-of-ess order. The verge ost for hshing should be Θ(); however, if lustering of reords eists, then the ost to ess reord n be muh higher beuse mny elements on the linked list must be serhed. Closed hshing Closed hshing stores ll reords diretly in the hsh tble. A ollision resolution poliy mush be built to determine whih slot to use when ollision is deteted. The sme poliy must be followed during serh s during insertion. Some ommon losed hshing Buket hshing --- overflow goes to n overflow buket

3 Suffi tree In CS, suffi tree is ompressed trie ontining ll the suffies of the given tet s their keys nd positions in the tet s their vlues. Suffi tree llows one to find, etremely effiiently, ll distint subsequenes in given sequene. There re effiient lgorithms to onstrut suffi trees given by Weiner (9) nd MCreight (9) (in liner time) For the tsk of ompring two DNA sequenes, suffi trees llow one to quikly find ll subsequenes shred by the two inputs. The genome lignment is then built upon this informtion. Suffi tree of short sequene Lef is unique suffi An internl node is repeted sequene in the originl string ATCGTA# # A# TA# GTA# CGTA# TCGTA# ATCGTA# ATCGAT$ $ T$ AT$ GAT$ CGAT$ 0 TCGAT$ 9 ATCGAT$ 8 Mthing two sequenes 0 A CG T G $ # TA# AT$ AT$ TA# A# CG T$ T 0 CG AT$ TA# 9 AT$ TA# 8 ATCG is the longest ommon substring Every unique mthing sequene is represented by n internl node with etly two hild nodes, suh tht the hild nodes re lef nodes from different genomes Applied in MUMer MUMer method MUMer ombines suffi trees, the longest inresing subsequene (LIS) nd SW lignment Miml Unique Mth (MUM) Identifition - Identify the longest strings in Genome tht hve one identil mth in Genome Nïve method: O(N ) Using suffi tree: O(N) Ordered MUM Seletion - Identify the longest set of MUMs suh tht they our in order in eh of the genomes (using vrition of the well-known lgorithm to find the LIS of sequene of integers) Proessing Non-mthed Regions - Clssify nonmthed regions s either insertions, SNPs or highly polymorphi regions A toy emple of string (pttern) mthing T = b suffies ={b, b, b,,, } Pttern P : Pttern P : b b Preproess tet T, not pttern P b b Suffi tree for string mthing Preproess tet T, not pttern P O(m) preproess time (m: the length of the tet) O(n+k) serh time (n: the length of the pttern) k is number of ourrenes of P in T Mth pttern P ginst tree strting t root until Cse, P is ompletely mthed Every lef below this mth point is the strting lotion of P in T Cse : No mth is possible P does not our in T

4 Suffi rry Suffi rry is spe-effiient dt struture, whih is more ompt thn suffi tree The suffi rry is bsilly sorted rry position of ll the suffies of tet. The strt positions re sorted in leiogrphil (lphbetil) order A suffi rry for tet of length n n be built in O (n logn ) time, Serhing the tet for pttern of length m n be done in O (m log n ) time by binry serh; redued to O(m + logn) if using LCP (longest ommon prefi) A B R A C A D A B R A # # 0 A# ABRA# 0 ABRACADABRA# ACADABRA# ADABRA# 8 BRA# BRACADABRA# CADABRA# DABRA# 9 RA# RACADABRA# Serh in suffi rry Binry serh in suffi rry: in eh of the O(logn) omprisons, the input pttern P is ompred to the urrent entry of the suffi rry, whih mens full string omprison of up to m hrters (the whole pttern). So the ompleity is O(mlogn). The ompleity is redued to O(m + logn) when LCP is used. Longest ommon prefi (LCP) LCP LCP[i] is the length of the longest ommon prefi between the suffies strting from SA[i ] nd SA[i]. It keeps trk of the length of the longest ommon prefi mong two onseutive suffies of S when rrnged in leiogrphi order Liner-time longest-ommon-prefi omputtion in suffi rrys nd its pplitions. Ksi et l, 00 How LCP helps? Assume t one step of the binry serh, [L,...,R] is the rnge of the suffi rry with entrl point M; P is ompred to suffi SA[M]. Assume P nd the orresponding suffi shre the first k hrters, nd P is leiogrphilly lrger thn suffi SA[M], so, in the net step, [M,...,R] is onsidered nd new entrl point M needs to be determined: M... M'... R. We know lp(p,m)==k; lso LCP-LR is preomputed suh tht O()-lookup gives lp(m, M ), the longest ommon prefi of M nd M. Now there re three possibilities: Cse : k < lp(m,m'), i.e. P hs fewer prefi hrters in ommon with M thn M hs in ommon with M'. This mens the (k+)-th hrter of M' is the sme s tht of M, nd sine P is leiogrphilly lrger thn M, it must be leiogrphilly lrger thn M', too. So we ontinue in the right hlf [M',...,R]. Cse : k > lp(m,m'), i.e. P hs more prefi hrters in ommon with M thn M hs in ommon with M'. Consequently, if we were to ompre P to M', the ommon prefi would be smller thn k, nd M' would be leiogrphilly lrger thn P, so, without tully mking the omprison, we ontinue in the left hlf [M,...,M ]. Cse : k == lp(m,m'). So M nd M' re both identil with P in the first k hrters. To deide whether we ontinue in the left or right hlf, it suffies to ompre P to M' strting from the (k+)-th hrter. The onsequene is tht no hrter of P is ompred to ny hrter of the tet more thn one. The totl number of hrter omprisons is bounded by m, so the totl ompleity is indeed O(m+logn). Burrows-Wheeler Trnsform Burrow M & Wheeler D (99) Reversible permuttion of tet to llow better ompression (e.g. bzip) Algorithms eist to perform fst serh on BWtrnsformed dt Burrows-Wheeler Trnsform (BWT) $g g$ g$ 0 g$ g$ g$ g$ g$ 0 sorted in leiogrphil order Suffi rry $g g$ g$ g$ g$ g$ g$ Burrows-Wheeler Mtri (BWM) g$

5 Why Burrows-Wheeler? BWT very ompt (ws developed for ompression purpose) Approimtely ½ byte per bse (by ontrst, n integer number my tke bytes) As lrge s the originl tet, plus few etrs (the FM indies) Cn fit onto stndrd omputer with GB of memory (for indeing humn genome) Liner-time serh lgorithm Proportionl to the length of query for et mthes Mississippi emple s i s s i p p i $ m i s s s i p p i $ m i s s i Sorted leiogrphilly F s i s s i p p i $ m i s s s i p p i $ m i s s i C tble: C[] # of tet hrters whih re lphbetilly smller thn F s i s s i p p i $ m i s s s i p p i $ m i s s i FM indies $ i m p s 0 8 $ i m p s SA intervls?? C[ m ] = ( i + $ ) O funtion O(,q): # of ourrenes of hrter in the prefi [,q] O( s,0) = ( s in ipssm$piss ) F Lst to front mpping Suffi-id prior to the first T bwt [i] O(t [i], i)th t [i] LF() = lst to front mpping: The hrter [i] is loted in the first olumn F t position LF[i]; i.e., [i] =F[LF[i]] LF(i)=C[t [i]] + O(t [i], i) e.g., LF(0)=C[ s ] + O( s, 0) = 8 + = s i s s i p p i $ m i s Both t [0] nd F[] s s i p p i $ m i s s i orrespond to the first s in mississippi Only t is stored; F is shown for demonstrtion purpose Tet: g$ C tble $ g 0 O tble # $ g Reversible trnsform (using LF mpping) to reover tet LF(0)=C[ g ]+O( g,0) = + = LF()=C[ ]+O(,) = + = the lst hrter in the originl tet

6 Serhing in BWT-ompressed tet: ounting the ourrenes Algorithm for ounting the number of ourrenes of P[; p] in T[; u]; bkwrd serh lgorithm At the i-th phse, the prmeter sp points to the first row of M prefied by P[i; p] nd the prmeter ep points to the lst row of M prefied by P[i; p]. Ferrgin nd Mnzini showed tht it is possible to ompute O(,, k) in onstnt time. O(,, k) sometimes is shown s O(, k) : # of ourrenes of hrter in the prefi to k. Serhing in BWT-ompressed tet: determining the ourrenes (lotions) We know how to ount the ourrenes, then for s = sp, sp +,, ep, we need to find the position pos(s) in T of the suffi whih prefies the sth row M[s]; two methods were proposed. The first one is simple, whih relies on subset of the indies in tht re ssoited with position in suffi rry. If [j] hs position ssoited with it, lote(j) is trivil. If it's not ssoited, the string is followed with LF(i) (lst front mpping) until n ssoited inde is found. Lote n be implemented to find o ourrenes of pttern P[..p] in tet T[..u] in O(p + o log ε u) time. The seond method is fster nd relies on the very speil properties of the the string Tbw nd on different ompression lgorithm. Ferrgin nd Mnzini, 000 Opportunisti Dt Strutures with Applitions. Et mth: A simple emple Ref: Et mth (nother emple) BWT(gggt) = tg$gg Serh for pttern: g g g g g $gggt $gggt $gggt $gggt t$ggg t$ggg t$ggg t$ggg gt$gg gt$gg gt$gg gt$gg ggt$g ggt$g ggt$g ggt$g gggt$ gggt$ gggt$ gggt$ gt$gg gt$gg gt$gg gt$gg ggt$g ggt$g ggt$g ggt$g t$ggg t$ggg t$ggg t$ggg gt$gg gt$gg gt$gg gt$gg ggt$g ggt$g ggt$g ggt$g gggt$ gggt$ gggt$ gggt$ t$ggg t$ggg t$ggg t$ggg Test with your own seq nd pttern t: Inet mth For the et mthing problem, we only need to hek one SA intervl; for the inet mthing problem, there my be mny. Min dvntge of BWT ginst suffi rry BWT needs less memory thn suffi rry For humn genome m = * 0 9 (m is the length of the genome): Suffi rry: mlog (m) bits = m bytes = GB BWT: m/ bytes plus etrs = - GB m/ bytes to store BWT ( bits per hr) Suffi rry nd ourrene ounts rry tke lot more bits In prtie, SA nd OCC only prtilly stored, most elements re omputed on demnd (tkes time!) Trdeoff between time nd spe

7 Short-red mpping softwre Softwre Tehnique Developer Elnd Hshing reds Illumni SOAP Hshing refs BGI Mq Hshing reds Snger (Li, Heng) Bowtie & Bowtie BWT Slzberg/UMD BWA BWT Snger (Li, Heng) SOAP BWT & hshing BGI netgenertionsequening.html (under Alignment tegory)