1 APL13: Suffix Arrays: more space reduction

Transcription

1 1 APL13: Suffix Arrys: more spce reduction In Section??, we sw tht when lphbet size is included in the time nd spce bounds, the suffix tree for string of length m either requires Θ(m Σ ) spce or the minimum of O(m logm) nd O(m log Σ ) time. Similrly, serching for pttern P of length n using suffix tree cn be done in O(n) time only if Θ(m Σ ) spce is used for the tree, or if we ssume tht up to Σ chrcter comprisons cost only constnt time. Otherwise, the serch tkes the minimum of O(n log m) nd O(n log Σ ) comprisons. For these resons, suffix tree my require too much spce to be prcticl in some pplictions. Hence more spce efficient pproch is desired tht still retins most of the dvntges of serching with suffix tree. In the context of the substring problem (see Section??) where fixed string T will be serched mny times, the key issues re the time needed for the serch, nd the spce used by the fixed dt structure representing T. The spce used during the preprocessing of T is of less concern, lthough it should still be resonble. Mnber nd Myers [1] proposed new dt structure, clled suffix rry, tht is very spce efficient nd yet cn be used to solve the exct mtching problem or the substring problem lmost s efficiently s with suffix tree. Suffix rrys re likely to be n importnt contribution to certin string problems in computtionl moleculr biology, where the lphbet cn be lrge (we will discuss some of the resons for lrge lphbets below). Interestingly, lthough the more forml notion of suffix rry nd the bsic lgorithms for building nd using it were developed in [1], mny of the ides were nticipted in the biologicl literture by Mrtinez [2]. After defining suffix rrys we show how to convert suffix tree to suffix rry in liner time. It is importnt to be cler on the setting of the problem. String T will be held fixed for long time, while P will vry. Therefore the gol is to find spce efficient representtion for T ( suffix rry) tht will be held fixed nd tht fcilittes serch problems in T. But the mount of spce used during the construction of tht representtion is not so criticl. In the exercises we consider more spce efficient wy to build the representtion itself. Definition: Given n m chrcter string T,suffix rry for T, clled Pos, is n rry of the integers in the rnge 1 to m, specifying the lexicogrphic order of the m suffixes of string T. Tht is, the suffix strting t position Pos(1) of T is the lexiclly smllest suffix, nd in generl suffix Pos(i) of T is lexiclly smller thn suffix Pos(i+1). As usul, we will ffix terminl symbol $ to the end of S, but now interpret it to be lexiclly less thn ny other chrcter in the lphbet. This is in contrst to its interprettion in the previous section. As n exmple of suffix rry, if T is mississippi, then the suffix rry Pos is 11, 8, 5, 2, 1, 10, 9, 7, 4, 6, 3. Figure 1 lists the eleven suffixes in lexicogrphic order. Notice tht the suffix rry holds only integers nd hence contins no informtion bout the lphbet used in string T. Therefore, the spce required by suffix rrys 1

2 11: i 8: ippi 5: issippi 2: ississippi 1: mississippi 10: pi 9: ppi 7: sippi 4: sisippi 6: ssippi 3: ssissippi Figure 1: The eleven suffixes of mississippi listed in lexicogrphic order. The strting positions of those suffixes define the suffix rry Pos. is modest for string of length m, the rry cn be stored in exctly m computer words, ssuming word size of t lest log m bits. When ugmented with n dditionl 2m vlues (clled Lcp vlues nd defined lter), the suffix rry cn be used to find ll the occurrences in T of pttern P in O(n + log 2 m) single-chrcter comprison nd bookkeeping opertions. Moreover, this bound is independent of the lphbet size. Since for most problems of interest log 2 m is O(n), the substring problem is solved by using suffix rrys s efficiently s by using suffix trees. 1.1 Suffix tree to suffix rry in liner time We ssume tht sufficient spce is vilble to build suffix tree for T (this is done once during preprocessing phse), but tht the suffix tree cnnot be kept intct to be used in the (mny) subsequent serches for ptterns in T. Insted we convert the suffix tree to the more spce efficient suffix rry. Exercises??,?? nd?? develop more spce efficient (but slower) method for building suffix rry. A suffix rry for T cn be obtined from the suffix tree T for T by performing lexicl depth-first trversl of T. Once the suffix rry is built, the suffix tree is discrded. Definition Define n edge (v, u) to be lexiclly less thn n edge (v, w) if nd only if the first chrcter on the (v, u) edge is lexiclly less thn the first chrcter on (v, w). (In this ppliction, the end of string chrcter $ is lexiclly less thn ny other chrcter.) Since no two edges out of v hve lbels beginning with the sme chrcter, there is strict lexicl ordering of the edges out of v. This ordering implies tht the pth 2

3 from the root of T following the lexiclly smllest edge out of ech encountered node leds to lef of T representing the lexiclly smllest suffix of T. More generlly, depth-first trversl of T which trverses the edges out of ech node v in their lexicl order will encounter the leves of T in the lexicl order of the suffixes they represent. Suffix rry Pos is therefore just the ordered list of suffix numbers encountered t the leves of T during the lexicl depth- first-serch. The suffix tree for T is constructed in liner time, nd the trversl lso tkes only liner time, so we hve the following Theorem 1.1 The suffix rry Pos for string T of length m cn be constructed in O(m) time. For exmple, the suffix tree for T = trtr is shown in Figure 2. The lexicl depth-first trversl visits the nodes in the order 5,2,6,3,4,1, defining the vlues of rry Pos. r $ 1 t t r $ 4 r t r r $ t 6 $ r 5 Figure 2: The lexicl depth-first trversl of the suffix tree visits the leves in order 5,2,6,3,4,1. As n implementtion detil, if the brnches out of ech node of the tree re orgnized in sorted linked list (s discussed in Section??, pge??), then the overhed to do lexicl depth-first serch is the sme s for ny depth-first serch. Every time the serch must choose n edge out of node v to trverse, it simply picks the next edge on v s linked list. 1.2 How to serch for pttern using suffix rry The suffix rry for string T llows very simple lgorithm to find ll occurrences of ny pttern P in T. The key is tht if P occurs in T then ll the loctions of those 3 $ $ 2 3

4 occurrences will be grouped consecutively in Pos. For exmple, P = issi occurs in mississippi strting t loctions 2 nd 5, which re indeed djcent in Pos (see Figure 1). So to serch for occurrences of P in T simply do binry serch over the suffix rry. In more detil, suppose tht P is lexiclly less thn the suffix in the middle position of Pos (i.e., suffix Pos( m/2 )). In tht cse, the first plce in Pos tht contins position where P occurs in T must be in the first hlf of Pos. Similrly, if P is lexiclly greter thn suffix Pos( m/2 ), then the plces where P occurs in T must be in the second hlf of Pos. Using binry serch, one cn therefore find the smllest index i in Pos (if ny) such tht P exctly mtches the first n chrcters of suffix Pos(i). Similrly, one cn find the lrgest index i with tht property. Then pttern P occurs in T strting t every loction given by Pos(i) through Pos(i ). The lexicl comprison of P to ny suffix tkes time proportionl to the length of the common prefix of those two strings. Tht prefix hs length t most n, hence Theorem 1.2 By using binry serch on rry Pos, ll the occurrences of P in T cn be found in O(n log m) time. Of course, the true behvior of the lgorithm depends on how mny long prefixes of P occur in T. If very few long prefixes of P occur in T then it will rrely hppen tht specific lexicl comprison ctully tkes Θ(n) time nd generlly the O(n logm) bound is quite pessimistic. In rndom strings (even on lrge lphbets) this method should run in O(n + log m) expected time. In cses where mny long prefixes of P do occur in T, then the method cn be improved with the following two tricks. 1.3 A simple ccelernt As the binry serch proceeds, let L nd R denote the left nd right boundries of the current serch intervl. At the strt, L equls 1 nd R equls m. Then in ech itertion of the binry serch, query is mde t loction M = (R + L)/2 of Pos. The serch lgorithm keeps trck of the longest prefixes of Pos(L) nd Pos(R) tht mtch prefix of P. Let l nd r denote those two prefix lengths respectively, nd let mlr = min(l, r). The vlue mlr cn be used to ccelerte the lexicl comprison of P nd suffix Pos(M). Since rry Pos gives the lexicl ordering of the suffixes of T, if i is ny index between L nd R, the first mlr chrcters of suffix Pos(i) must be the sme s the first mlr chrcters of suffix Pos(L), nd hence of P. Therefore, the lexicl comprison of P nd suffix Pos(M) cn begin from position mlr+1 of the two strings, rther thn strting from the first position. Mintining mlr during the binry serch dds little dditionl overhed to the lgorithm but voids mny redundnt comprisons. At the strt of the serch, when L = 1 nd R = m, explicitly compre P to suffix Pos(1) nd suffix Pos(m) to find l nd r nd mlr. However, the worst cse time for this revised method is still O(n log m). Myers nd Mnber report tht the use of mlr lone llows the serch to 4

5 run s fst in prctice s the O(n+log m) worst cse method tht we first dvertised. Still, if only becuse of its elegnce, we present the full method tht gurntees tht better worst cse bound. 1.4 A super-ccelernt Cll n exmintion of chrcter in P redundnt if tht chrcter hs been exmined before. The gol of the ccelertion is to reduce the number of redundnt chrcter exmintions to t most one per itertion of the binry serch hence O(log m) in ll. The desired time bound, O(n+log m), follows immeditely. The use of mlr lone does not chieve this gol. Since mlr is the minimum of l nd r, whenever l r, ll chrcters in P from mlr + 1 to the mximum of l nd r will hve lredy been exmined. So ny comprisons of those chrcters will be redundnt. Wht is needed is wy to begin comprisons t the mximum of l nd r. Definition Lcp(i,j) is the length of the longest common prefix of the suffixes specified in positions i nd j of Pos. Tht is, Lcp(i,j) is the length of the longest prefix common to suffix Pos(i) nd suffix Pos(j). The term Lcp stnds for longest common prefix. For exmple, when T = mississippi, suffix Pos(3) is issippi, suffix Pos(4) is ississippi, nd so Lcp(3,4) is four (see Figure 1). To speed up the serch, the lgorithm uses Lcp(L,M) nd Lcp(M,R) for ech triple (L, M, R) tht rises during the execution of the binry serch. For now, we ssume tht these vlues cn be obtined in constnt time when needed, nd show how they help the serch. Lter we will show how to compute the prticulr Lcp vlues needed by the binry serch during the preprocessing of T. How to use Lcp vlues Simplest cse: In ny itertion of the binry serch, if l = r, then compre P to suffix Pos(M) strting from position mlr + 1 = l + 1 = r + 1, s before. Generl cse: When l r, let us ssume without loss of generlity tht l > r. Then there re three subcses: If Lcp(L,M) >l, then the common prefix of suffix Pos(L) nd suffix Pos(M) is longer thn the common prefix of P nd Pos(L). Therefore P grees with suffix Pos(M) up through chrcter l. In other words, chrcters l +1 of suffix Pos(L) nd suffix Pos(M) re identicl nd lexiclly less thn chrcter l + 1 of P (the lst fct follows since P is lexiclly greter thn suffix Pos(L)). Hence ll (if ny) strting loctions of P in T must occur to the right of position M in Pos. So in ny itertion of the binry serch where this cse occurs, no exmintions 5

6 of P re needed; L just gets chnged to M, nd l nd r remin unchnged. (See Figure 3). If Lcp(L,M) <l, then the common prefix of suffix Pos(L) nd Pos(M) is smller thn the common prefix of suffix Pos(L) nd P. Therefore P grees with suffix Pos(M) up through chrcter Lcp(L,M). The Lcp(L,M)+1 chrcters of P nd suffix Pos(L) re identicl nd lexiclly less thn chrcter Lcp(L,M)+1 of suffix Pos(M). Hence ll (if ny) strting loctions of P in T must occur to the left of position M in Pos. So in ny itertion of the binry serch where this cse occurs, no exmintions of P re needed; r is chnged to Lcp(L,M), l remins unchnged, nd R is chnged to M. If Lcp(L,M) = l, then P grees with suffix Pos(M) up to chrcter l. The lgorithm then lexiclly compres P to suffix Pos(M) strting from position l + 1. In the usul mnner, the outcome of tht lexicl comprison determines which of L or R chnge, long with the corresponding chnge of l or r. lcp(l,m) n g g z m f f y e d e d l e d x d r c c c c b b b b P L M R Figure 3: Subcse 1 of the super-ccelernt. Pttern P is bcdemn shown verticlly running upwrds from the first chrcter. The suffixes Pos(L), Pos(M), Pos(R) re lso shown verticlly. In this cse, Lcp(L,M) > 0, nd l > r. Any strting loction of P in T must occur in Pos to the right of M, since P grees with suffix Pos(M) only up to chrcter l. Theorem 1.3 Using the Lcp vlues, the serch lgorithm does t most O(n + log m) comprisons, nd runs in tht time. Proof First, by simple cse nlysis it is esy to verify tht neither l nor r ever decreses during the binry serch. Also, every itertion of the binry serch either termintes the serch, or exmines no chrcters of P, or ends fter the first mismtch occurs in tht itertion. In the two cses (l = r or Lcp(L,M) = l > r) where the lgorithm exmines chrcter during the itertion, the comprisons strt with chrcter mx(l,r) of P. 6

7 Suppose there re k chrcters of P exmined in tht itertion. Then there re k 1 mtches during the itertion, nd t the end of the itertion mx(l,r) increses by k 1 (either l or r is chnged to tht vlue). Hence t the strt of ny itertion, chrcter mx(l,r) of P my hve lredy been exmined, but the next chrcter in P hs not been. Tht mens t most one redundnt comprison per itertion is done. So no more thn log 2 m redundnt comprisons re done overll. There re t most n non-redundnt comprisons of chrcters of P, giving totl bound of n + log m comprisons. All the other work in the lgorithm cn clerly be done in time proportionl to these comprisons. 1.5 How to obtin the Lcp vlues The originl method of Mnber nd Myers hs been improved nd notes on it will be distributed. 1.6 Where do lrge lphbet problems rise? A lrge prt of the motivtion for suffix rrys comes from problems tht rise in using suffix trees when the underlying lphbet is lrge. So it is nturl to sk where lrge lphbets occur. First, there re nturl lnguges, such s Chinese, with lrge lphbets (using some computer representtion of the Chinese pictogrms.) But most lrge lphbets of interest to us rise becuse the string contins numbers, ech of which is treted s chrcter. One simple exmple is string tht comes from picture where ech chrcter in the string gives the color or grey level of pixel. String nd substring mtching problems where the lphbet contins numbers, nd where P nd T re lrge, lso rise in computtionl problems in moleculr biology. One exmple is the mp mtching problem. A restriction enzyme mp for single enzyme specifies the loctions in DNA string where copies of certin substring ( restriction enzyme recognition site) occurs. Ech such site my be seprted from the next one by mny thousnds of bses. Hence, the restriction enzyme mp for tht single enzyme is represented s string consisting of sequence of integers specifying the distnces between successive enzyme sites. Considered s string, ech integer is chrcter of (huge) underlying lphbet. More generlly, mp my disply the sites of mny different ptterns of interest (whether or not they re restriction enzyme sites), so the string (mp) consists of chrcters from finite lphbet (representing the known ptterns of interest) lternting with integers giving the distnces between such sites. The lphbet is huge becuse the rnge of integers is huge, nd since distnces re often known with high precision, rounding is not used. Moreover, the vriety of known ptterns of interest is itself lrge (see [3] for librry of severl hundred known, significnt ptterns). It often hppens tht DNA substring is obtined nd studied without knowing 7

8 where tht DNA is locted in the genome or whether tht substring hs been previously reserched. If both the new nd the previously studied DNA re fully sequenced nd put in dtbse, then the issue of previous work or loctions would be solved by exct string mtching. But most DNA substrings tht re studied re not fully sequenced mps re esier nd cheper to obtin thn sequences. So the following mtching problem on mps rises nd trnsltes to n mtching problem on strings with lrge lphbets: Given n estblished (restriction enzyme) mp for lrge DNA string, nd mp from smller string, determine if the smller string is substring of the lrger one. Since ech mp is represented s n lternting string of chrcters nd integers, the underlying lphbet is huge. This provides one motivtion for using suffix rrys for mtching or substring serching in plce of suffix trees. Of course, the problems become more difficult in the presence of errors, when the integers in the strings my not be exct, or when sites re missing or spuriously dded. Tht problem, clled mp lignment, is discussed in Section??. References [1] U. Mnber nd G. Myers. Suffix rrys: new method for on-line serch. SIAM J. on Computing, pges , [2] H. Mrtinez. An efficient method for finding repets in moleculr sequences. Nucl. Acids Res., 11: , [3] E. Trifonov nd V. Brendel. Gnomic: A dictionry of genetic codes. VCH Press, Deerfield, Florid,