Sequence Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fischer) January 21,

Transcription

1 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, Suffix Trees and Suffix Arrays This leture is based on the following soures, whih are all reommended reading: D. Gusfield, Algorithms on Strings, Trees and Sequenes, Cambridge UP, M. Crohemore, W. Rytter, Jewels of Stringology, World Sientifi, M. Crohemore, C. Hanart, T. Leroq, Algorithms on Strings, Cambridge UP, J. Kärkkäinen et al, Simple Linear Work Suffix Array Constrution, Journal of the ACM (JACM) Volume 53 Issue 6, November 2006 Pages S. Kurtz, Foundations of Sequene Analysis, Universität Bielefeld, Introdution Problem Assume we are given a long text T and many short queries Q 1,..., Q k. For eah query sequene Q i, find all its ourrenes in T. We would like to have a data-struture that allows us to solve this problem effiiently. Important appliations are in the omparison of genomes (in programs suh as MUMmer that omputes alignments between genomi sequenes, based on maximal unique mathes ) and in the analysis of repeats (in programs suh as REPuter). For example, the text T might be a genome and the queries might be short words suh as transription fator binding sites. Text T = tttttttgagaggagttgttgtgaggtggagtgagt gggggattggtatgaagtgtgggttaga tttgtagtaagtagtgggataagggga ggtaattttttgtatttttagtagagaggggtttagtttta gggatggttgatttgatgtgatggtggt aaagtgtgggattaagggt Query Q = tttta Find (all) ourrenes of the query Q in the text T 9.2 Basi definitions Definition (Prefix, suffix, fator) We use Σ to denote an alphabet and Σ for the set of all strings over Σ. We use ɛ to denote the empty string and Σ + = Σ \ {ɛ}. Let T = t 1 t 2... t n be a text. A prefix of T is a substring of T beginning at the first position in T. A suffix of T is a substring T i...n = t i t i+1... t n of T ending at the last position in T. A fator (or infix) of T is any substring T i...j = t i t i+1... t j of T.

2 12 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, 2015 In bioinformatis, the alphabet Σ is usually of size 4 or 20. In other appliations, the alphabet an be muh larger, for example, in the analysis of web-surfing patterns, Σ onsists of the set of all links ontained in a olletion of web sites. 9.3 The role of suffixes Consider the text abab It has the following suffixes: abab, bab, ab, b and. Queries are prefixes of suffixes: To determine whether a given query Q is ontained in the text, we ould simply hek whether Q is the prefix of one of the suffixes. 9.4 Sharing prefixes E.g., the query ab is the prefix of both abab and ab. To speed up the searh for all suffixes that have the query as a prefix, we use a tree struture to share ommon prefixes between the suffixes. (a) The suffixes abab and ab both share the prefix ab. (b) The suffixes bab and b both share the prefix b. () The suffix doesn t share a prefix. a b (a) b a b b a (b) Here is an example of how the idea of searhing the prefix of every suffix an be optimized by sharing prefixes whenever possible: () b a b abab ab bab b a b b a A suffix tree for abab is obtained by sharing prefixes whenever possible. The leaves are annotated by the positions of the orresponding suffixes in the text. 9.5 Σ + -tree Definition (Σ + -tree) A ompat Σ + -tree is a rooted tree S = (V, E) with edge labels from Σ + that fulfills the following two onstraints: For eah node v V, the labels of all outgoing edges eah starts with a different letter.

3 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, Apart from the root, all nodes have out-degree 1. You an think of a ompat Σ + -tree as a tree that looks like a trie data struture, but with degreeone-paths ontrated into a single edge. A leaf is a node with no hildren and an edge leading to a leaf is alled a leaf edge. A node with at least two hildren is alled a branhing node. Definition (Notations in Σ + -trees) Let S = (V, E) be a ompat Σ + -tree. Then v denotes the onatenation of all path labels from the root of S to node v. Then d(v) denotes the string-depth v of v. A string α ours in S, or S displays α, iff there exists a node v and a (not neessarily nonempty) string β with v = αβ. Then words(s) denotes the set of all all strings that our in S. If v = α for some node v and string α, then we also use α to denote v. root p For example u = pqr: The root is alled ɛ. q r u 9.6 Suffix tree Definition (Suffix tree) Let fator(t ) denote the set of all fators of T. The suffix tree of T is a ompat Σ + -tree S with words(s) = fator(t ). It is onvenient to ensure that eah suffix ends at a leaf of S. This an be aomplished by always assuming that the text T ends on a sentinel letter $ / Σ that is lexiographially smaller than all other letters. Then there is a one-to-one between suffixes of T $ and the leaves of S and we an label eah leaf v by the start index l(v) = i of the orresponding suffix T i...n, i.e. we set l(v) = i v = T i...n.

4 14 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, 2015 This explains the name suffix tree. A suffix tree S built for T without a sentinel $ is alled an impliit suffix tree. (Note that we did not define suffix trees in terms of suffixes, but rather in terms of fators.) We an use S to find ourrenes of a query Q Σ as follows. Starting at the root, attempt to math the letters of Q against the labels of the edges of S, along a path into the tree. If at some point the path annot be extended, then Q does not our in T. Otherwise, Q ours in T. In this ase, the mathing proedure ends at some loation p in S, either at a node or in an edge. To identify all ourrenes of Q in T, we traverse the subtree below p and report all leaf-labels. The deision query ( does Q our in T? ) takes O(m) time, with m the length of Q. The reporting query ( report all ourrenes of Q in T ) takes O(m + k) time, where k denotes the number of ourrenes of Q in T. The orretness of the two query algorithms follows from the fat that eah fator of T is a prefix of some suffix of T. For an alphabet of onstant size, finding the appropriate outgoing edge an be done in onstant time. An important implementation detail is that the edge labels in a suffix tree an be represented by a

5 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, pair (i, j), 1 i j n suh that T i...j equals the orresponding edge label. Hene, eah edge label uses only a onstant amount of memory. From this implementation detail and the fat that S ontains exatly n leaves and hene less than n branhing nodes, we get the following result: Theorem (Linear spae) A suffix tree requires only O(n) spae. 9.7 Suffix Array We will now introdue two arrays that are losely related to the suffix tree, the suffix array A and the LCP array. Definition (Suffix array) The suffix array A of T is a permutation of {1, 2,..., n} suh that A[i] is the i-th smallest suffix in lexiographi order: T A[i 1]...n < T A[i]...n for all 1 < i n. Let S be the assoiated suffix tree. If we do a traversal through S, visiting hildren in lexiographi order of the labels of the edges leading to them, then all leaves will be visited in the order given in the suffix array A. This links the onepts of suffix array A and suffix tree S. 9.8 LCP Array The LCP array is based on the suffix array: Definition (LCP array) The LCP array of T is defined as follows: set LCP[1] = 0, and for all i > 1, let LCP[i] equal the length of the longest ommon prefix (lp) of T A[i]...n and T A[i 1]...n. To relate the LCP array to the suffix tree S, we need to define the onept of lowest ommon anestors: Definition (Lowest ommon anestor) Given a tree S = (V, E) and two nodes v, w V, the lowest ommon anestor of v and w is the deepest node in S that is an anestor of both v and w, denoted by LCA(v, w).

6 16 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, 2015 Lemma (LCA and LCP) The string-depth of the lowest ommon anestor of the two leaves with labels A[i] and A[i 1], respetively, is given by the orresponding entry LCP[i] of the LCP array. In other words, for all i > 1 we have: LCP[i] = d(lca(t A[i]...n, T A[i 1]...n )). i A[i] suffix LCP[i] 0 9 $ ACCTTCCT$ CCT$ CCTTTCCT$ CT$ CTTCCT$ T$ TCCT$ TTCT$ Suffix Tree from Suffix- and LCP Array Assume we are given T, A, and LCP, and we wish to onstrut S, the suffix tree of T. We will show how to do this in O(n) time. (Later, we will also see how to onstrut A and LCP only from T in linear time. In total, this will give us an O(n)-time onstrution algorithm for suffix trees.) The idea of the algorithm is to insert the suffixes into S in the order of the suffix array: T A[1]...n, T A[2]...n,..., T A[n]...n. To this end, let S i denote the partial suffix tree for 0 i n, i.e. S i is the ompat Σ + -tree with In the end, we will have S = S n. words(s i ) = {T A[k]...j : 1 k i, A[k] j n}. We start with S 0, the tree onsisting only of the root (and displaying only ɛ). In step i + 1, we limb up the rightmost path of S i (i.e., the path from the leaf labeled A[i] to the root) until we meet the deepest node v with d(v) LCP[i + 1]. If d(v) = LCP[i + 1], we simply insert a new leaf x to S i as a hild of v, and label (v, x) by T A[i+1]+LCP[i+1]...n. Leaf x is labeled by A[i + 1]. This gives us S i+1. Otherwise (i.e., d(v) < LCP[i + 1]), let w be the hild of v on S i s rightmost path. In order to obtain S i+1, we split the edge (v, w) as follows: Delete (v, w). Add a new node y and a new edge (v, y), whih gets labeled by T A[i]+d(v)...A[i]+LCP[i+1] 1. Add (y, w) and label it by T A[i]+LCP[i+1]...A[i]+d(w) 1. Add a new leaf x (labeled A[i + 1]) and an edge (y, x). Label (y, x) by T A[i+1]+LCP[i+1]...n.

7 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, Corretness of Algorithm The orretness of this algorithm follows from results stated above. What is the exeution time of this algorithm? Although limbing up the rightmost path ould take O(n) time in a single step, the running time an be bounded by O(n) in total: Eah node traversed in step i (apart from the last) is removed from the rightmost path and will not be traversed again for all subsequent steps j > i. Hene, at most 2n nodes are traversed in total. Theorem (Suffix tree from arrays) We an onstrut T s suffix tree in linear time from T s suffix- and LCP array. This suffix tree onstrution algorithm is good in terms of memory loality; there is a high orrelation between how lose two piees of data are in the tree, and how lose they are in physial memory. Hene, an average query will produe only few ahe misses and leads to a good performane in pratie Linear-Time Constrution of Suffix Arrays For an easier presentation, we assume in this setion that T is indexed from 0 to n 1, T = t 0 t 1... t n 1. Also, we will assume that the end of T is padded by a suffiient number of $ s. We use T i to denote the suffix that starts at index i T = y a b b a d a b b a d o We want to obtain the following in linear time: A = (1, 6, 4, 9, 3, 8, 2, 7, 5, 10, 11, 0) We will desribe the DC3 algorithm 1, whih operates in three steps: (0) Extrat a sample that ontain 2/3 of all suffixes. (1) Reursively sort the extrated suffixes. (2) Use these to sort the other 1/3 of suffixes. (3) Merge the two sorted lists of suffixes to obtain the final suffix array. Step 0: Extrat sample. For k = 0, 1, 2 define B k = {i [0, n] i mod 3 = k}. Let C = B 1 B 2 be the set of sample positions and T C the set of sample suffixes. B 1 = {1, 4, 7, 10}, B 2 = {2, 5, 8, 11} and C = {1, 2, 4, 5, 7, 8, 10, 11}. Let max B k denote the maximum element in B k, in this example max B 1 = 10 and max B 2 = 11. Step 1: Sort sample suffixes. For k = 1, 2 onstrut the strings 1 J. Kärkkäinen et al 2006 R k = [t k t k+1 t k+2 ][t k+3 t k+4 t k+5 ]... [t max Bk t max bk +1t max Bk +2],

8 18 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, 2015 whose letters are triples [t i t i+1 t i+2 ]. Note that the last letter of R k is always unique beause it ends on $. Let R = R 1 R 2 be the onatenation of R 1 and R 2. Then the (nonempty) suffixes of R orrespond to the set T C of sample suffixes: [t i t i+1 t i+2 ][t i+3 t i+4 t i+5 ]... orresponds to T i. This orrespondene is order-preserving, so by sorting the suffixes of R we get the order of the sample suffixes T C. R = [abb][ada][bba][do$][bba][dab][bad][o$$]. To sort the suffixes of R, first radix sort the letters (whih are triples) of R and then rename them with their ranks to obtain a new string R. i = R = [abb] [ada] [bba] [do$] [bba] [dab] [bad] [o$$] R = If all the haraters are different, then the lexiographi ordering of the haraters determines the order of the suffixes. Otherwise, we reursively all Algorithm DC3 to sort the suffixes of R. A R = (0, 1, 6, 4, 2, 5, 3, 7). Now assign a rank to eah sample suffix based on A R. i = Index in T = R = rank= This sorts the set of suffixes in T C. T 1 < T 4 < T 8 < T 2 < T 7 < T 5 < T 10 < T 11. For i C let rank(t i ) denote the rank of suffix T i in the sample set T C, obtained from the rank of the orresponding suffix in R. Additionally, define rank(t n+1 ) = rank(t n+2 ) = 0. For i B 0, the value of rank(t i ) is undefined ( ). i rank(t i ) Step 2: Sort nonsample suffixes. Represent eah nonsample suffix T i T B0 as the pair (t i, rank(t i+1 )). Note that rank(t i+1 ) is defined for all i B 0. For all i, j B 0 we have: T i T j (t i, rank(t i+1 )) (t j, rank(t j+1 )). Using this and applying radix sort to the set of all pairs (t i, rank(t i+1 )), we sort all nonsample suffixes. T 12 < T 6 < T 9 < T 3 < T 0 beause ($, 0) < (a, 5) < (a, 7) < (b, 2) < (y, 1). Step 3: Merge. The two sorted sets of suffixes are merged in the straightforward way, omparing any sample suffix T i T C with a non-sample suffix T j T B0 as follows: if i B 1 then: else (i B 2 ): T i T j (t i, rank(t i+1 )) (t j, rank(t j+1 )), T i T j (t i, t i+1, rank(t i+2 )) (t j, t j+1, rank(t j+2 )).

9 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, T 1 < T 6 beause (a, 4) < (a, 5), and T 3 < T 8 beause (b, a, 6) < (b, a, 7). The time omplexity is given by the following result: Theorem (Linear time) The time omplexity of Algorithm DC3 is O(n). Proof Exluding the reursive all, everything an learly be done in linear time. The reursion is on a string of length 2n/3. Thus the time is given by the reurrene T (n) = T (2n/3) + O(n), whose solution is T (n) = O(n) Linear-Time Constrution of LCP Arrays It remains to be shown how the LCP array an be onstruted in O(n) time. Here, we assume that we are given T, A, and A 1, the latter being the inverse suffix array that maps the i-th suffix to its rank in the lexiographial ordering of all suffixes. We will onstrut the LCP in the original order of the suffixes, using A 1 to index the appropriate element of LCP. This is a good order in whih to proeed beause the following result states that in eah step the value an only derease by at most one: Lemma For all 1 i < n: LCP[A 1 [i + 1]] LCP[A 1 [i]] 1. Proof Let T j be the lexiographi predeessor of T i, i.e., T j < T i and there are no other suffixes between them in the lexiographial order. Then T j is diret predeessor of T i in A and so If h = 0, then the laim is trivially true. LCP [A 1 [i]] = lp(t i, T j ) =: h. If h > 0, then for some letter we have T i = T i+1 and T j = T j+1. Then T j+1 < T i+1 and lp(t i+1, T j+1 ) = h 1. Let T k be the lexiographial predeessor of T i+1. Then either T j+1 = T k or T j+1 < T k < T i+1. In either ase, LCP [A 1 [i + 1]] = lp(t i+1, T k ) lp(t i+1, T j+1 ) = h 1, based on the following argument: If T j+1 = T K, then equality holds in the equation above. Otherwise, T j+1 must ome before T k. Beause T k is the diret predeessor of T i+1, it follows that in the orresponding suffix tree we have that the leaf representing T k is immediately followed by the leaf representing T k+1, and both leaf edges have the same parent. Beause T j+1 is a predeessor of T k, it follows that T j+1 attahes diretly, or indiretly, to a node on the path from the root to T k. Thus the longest ommon prefix of T j+1 and T i+1 annot exeed lp(t k, T i+1 ), as seen here: lp(t k$,t i+1 )## # T j+1# T T i+1# k$ This gives rise to the following elegant algorithm to onstrut the LCP:

10 20 Sequene Analysis, WS 14/15, D. Huson & R. Neher (this part by D. Huson & J. Fisher) January 21, 2015 Algorithm (LCP from suffix array) begin 0 LCP[0] = 0 1 h 0 / length of lp for previous suffix / 2 for i = 1,..., n do 3 k A 1 [i] 4 j A[k 1] / T j is lexiographial predeessor of T i / 5 while t i+h = t j+h do 6 h h LCP[k] h 8 h max{0, h 1} end Theorem (Linear time onstrution of LCP) The LCP array for a text of length n an be onstruted in O(n) time. Proof This follows from the presented algorithm: The for loop (line 2) is exeuted n times. Note that h annot grow larger than n, is deremented at most n times, and is inremented in eah round of the while loop (line 5). It follows that line (6) is exeuted at most 2n times Summary and outlook The LCP array and suffix array A an be onstruted in linear time for any given sequene T. We an onstrut the suffix tree S from LCP and A in linear time. In total, this provides a linear time onstrution of S from T. Suffix array and LCP an be used together to replae a suffix tree, using less spae (not shown here).