Preview: Text Indexing

Transcription

1 Simon Gog - Simon Gog: KIT University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

2 Text Indexing Motivation Problems Given a text T of length n over an alphabet Σ of size σ and a pattern P of length m. Typical questions we want to answer efficiently: Does P occur in T? (Existence query) How often does P occur in T? (Count query) Where does P occur in T? (Locate query) Two scenarios: Scan the whole text T for each query (time complexity O(n + m))). Build an index for T once. Use index to answer the query (time complexity not linear dependent on n or even independent!). Simon Gog:

3 Text Indexing Motivation Index solution Suffix Array (SA) was already explained in this lecture. Existence/count time: O(m log n) Locate time: O(m log n + occ), where occ is the number of occurrences. Drawbacks SA is larger than T : n log n compared to n log σ. For σ = 256 and n = 2 32 we get factor 4. SA requires T to answer queries. Other classical indexes: Suffix trees (ST), String B-Tree (SBT), and uncompressed positional Inverted Indexes (PII). 2 Simon Gog:

4 Text Indexing Suffix Array i SA[i] T [SA[i]..n ] 8 $ 7 a$ 2 abarbara$ 3 7 abrabarbara$ 4 abracadabrabarbara$ 5 3 acadabrabarbara$ 6 5 adabrabarbara$ 7 5 ara$ 8 2 arbara$ 9 4 bara$ barbara$ 8 brabarbara$ 2 bracadabrabarbara$ 3 4 cadabrabarbara$ 4 6 dabrabarbara$ 5 6 ra$ 6 9 rabarbara$ 7 2 racadabrabarbara$ 8 3 rbara$ 3 Simon Gog: T [SA[i]..n ] < T [SA[i + ]..n ] SA[i] contains the starting position of the ith lex. smallest suffix of T. Matching algorithm: binary search (forward, left-to-right)

5 Compressed Text Indexes The FM-Index Ferragina and Manzini [2] Index based on Burrows-Wheeler-Transform (BWT) Matching algorithm works backwards (right-to-left) Existence and count queries in time O(m log σ) BWT BWT [i] = T [SA[i] mod n] uncompressed size: n log σ bits compressed size: nh k (T ) bits (+information for contexts of length k) 4 Simon Gog:

6 i SA[i] BWT T [SA[i]..n ] 8 a $ 7 r a$ 2 r abarbara$ 3 7 d abrabarbara$ 4 $ abracadabrabarbara$ 5 3 r acadabrabarbara$ 6 5 c adabrabarbara$ 7 5 b ara$ 8 2 b arbara$ 9 4 r bara$ a barbara$ 8 a brabarbara$ 2 a bracadabrabarbara$ 3 4 a cadabrabarbara$ 4 6 a dabrabarbara$ 5 6 a ra$ 6 9 b rabarbara$ 7 2 b racadabrabarbara$ 8 3 a rbara$ BWT [i] = T [SA[i] ], for SA[i] > BWT [i] = T [n ], for SA[i] = I.e. BWT [i] is the character preceding suffix SA[i] 5 Simon Gog:

7 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ Array C contains for each c Σ the position of the first suffix in SA which starts with c: $ a b c d r r Operation rank(i, X, BWT ) returns how often character X Σ occurs in the prefix BWT [..i ]. Example: search for P = bar. 6 Simon Gog:

8 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = C[r]+rank(sp, r, BWT ) ep = C[r]+rank(ep +, r, BWT ) 7 Simon Gog:

9 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = 5+rank(, r, BWT ) ep = 5+rank(9, r, BWT ) 7 Simon Gog:

10 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = 5+ ep = 5+rank(9, r, BWT ) 7 Simon Gog:

11 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Initial interval: [sp, ep ] = [..n ] Determine interval for r: sp = 5+ = 5 ep = 5+4 = 8 7 Simon Gog:

12 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = C[a]+rank(sp, a, BWT ) ep 2 = C[a]+rank(ep +, a, BWT ) 8 Simon Gog:

13 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +rank(5, a, BWT ) ep 2 = +rank(ep, a, BWT ) 8 Simon Gog:

14 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +rank(5, a, BWT ) ep 2 = +rank(ep, a, BWT ) 8 Simon Gog:

15 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +6 ep 2 = +rank(9, a, BWT ) 8 Simon Gog:

16 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp, ep ] = [5..8] Determine interval for ar: sp 2 = +6 = 7 ep 2 = +8 = 8 8 Simon Gog:

17 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = C[b]+rank(sp 2, b, BWT ) ep 3 = C[b]+rank(ep 2 +, b, BWT ) 9 Simon Gog:

18 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+rank(7, b, BWT ) ep 3 = 9+rank(ep, b, BWT ) 9 Simon Gog:

19 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+rank(7, b, BWT ) ep 3 = 9+rank(ep, b, BWT ) 9 Simon Gog:

20 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+ ep 3 = 9+rank(9, b, BWT ) 9 Simon Gog:

21 i BWT T [SA[i]..n ] a $ r a$ 2 r abarbara$ 3 d abrabarbara$ 4 $ abracadabrabarbara$ 5 r acadabrabarbara$ 6 c adabrabarbara$ 7 b ara$ 8 b arbara$ 9 r bara$ a barbara$ a brabarbara$ 2 a bracadabrabarbara$ 3 a cadabrabarbara$ 4 a dabrabarbara$ 5 a ra$ 6 b rabarbara$ 7 b racadabrabarbara$ 8 a rbara$ C $ a b c d r Search backwards for bar. Interval: [sp 2, ep 2 ] = [7..8] Determine interval for bar: sp 3 = 9+ = 9 ep 3 = 9+2 = 9 Simon Gog:

22 Summary Only C and a data structure R supporting the rank operation on BWT are required for existence and count queries. Space: σ log n bits for C + space for R Time: O(m t rank ), where t rank is time for one rank operation. Independent from n? Next: How to implement rank? Rank operation Constant time and o(n) extra space solution on bitvectors (Jacobson [989]) Solution on general sequences: Wavelet Tree (Grossi et al. [23]) Simon Gog:

23 Summary Only C and a data structure R supporting the rank operation on BWT are required for existence and count queries. Space: σ log n bits for C + space for R Time: O(m t rank ), where t rank is time for one rank operation. Independent from n? If t rank is independent from n Next: How to implement rank? Rank operation Constant time and o(n) extra space solution on bitvectors (Jacobson [989]) Solution on general sequences: Wavelet Tree (Grossi et al. [23]) Simon Gog:

24 Wavelet Tree Example: Calculate Rank arrd$rcbbraaaaaabba a$bbaaaaaabba rrdrcr a$aaaaaaa bbbb dc rrrr a = $ aaaaaaaa c d rank(, a, WT ) = rank(rank(rank(,, b ɛ ) = 5,, b ) = 3,, b ) = 2 Simon Gog:

25 Wavelet Tree Example: Calculate Rank arrd$rcbbraaaaaabba a$bbaaaaaabba rrdrcr a$aaaaaaa bbbb dc rrrr a = $ aaaaaaaa c d rank(, a, WT ) = rank(rank(rank(,, b ɛ ) = 5,, b ) = 3,, b ) = 2 Simon Gog:

26 Wavelet Tree Example: Calculate Rank arrd$rcbbraaaaaabba a$bbaaaaaabba rrdrcr a$aaaaaaa bbbb dc rrrr a = $ aaaaaaaa c d rank(, a, WT ) = rank(rank(rank(,, b ɛ ) = 5,, b ) = 3,, b ) = 2 Simon Gog:

27 Wavelet Tree Example: Calculate Rank arrd$rcbbraaaaaabba a$bbaaaaaabba rrdrcr a$aaaaaaa bbbb dc rrrr a = $ aaaaaaaa c d rank(, a, WT ) = rank(rank(rank(,, b ɛ ) = 5,, b ) = 3,, b ) = 2 Simon Gog:

28 Compressed Text Indexing meets Algorithm Engineering State-of-the-art Recent FM-Indexes are as small as the output of state-of-the-art compressors (like gzip,xz) while matching takes microseconds per character. This is the result of theoretical and practical improvements: Shape of the WT (balanced, Huffman, Hu-Tucker,...) Bitvector representation (uncompressed/compressed) Hardware (popcount instruction, page size) Different sampling strategies for SA values (for locate queries)... 2 Simon Gog:

29 Compressed Text Indexing Combining a H -compressed bitvector with a Huffman shaped wavelet tree results in H k (T ) bits of space. 2 MB test instance WT-HUFF WT-HUFFcompr Time Space Time Space (µs) (%) (µs) (%) DBLP.XML DNA ENGLISH PROTEINS SOURCES Simon Gog:

30 Compressed Text Indexing Our Toolbox for Compact/Succinct Data Structures Succinct Data Structure Library (SDSL) A C++ template library for compact/succinct structures Parametrizable structures Bitvectors Compressed Integer Vectors Rank/Select Structures Wavelet Trees/Wavelet Matrices Compressed Suffix Arrays/Trees Search Engines Available at 4 Simon Gog:

31 Lecture Text Indexing Content Theory: Classical indexes (Suffix Arrays/Suffix Trees/Inverted Indexes) Building blocks for compact/succinct structures Compressed Bitvector Rank Structures Select Structures Range-Min-Max-Tree Compressed indexes FM-Indexes/Compressed Suffix Arrays Versions for highly-repetitive text Compressed Suffix Trees Search Engines Practice Use SDSL to implement and analyze structures. Design a code search engine. 5 Simon Gog:

32 Bibliography Paolo Ferragina and Giovanni Manzini. Opportunistic data structures with applications. In Proceedings of the 4st Annual Symposium on Foundations of Computer Science, (FOCS 2), pages , 2. Roberto Grossi, Ankur Gupta, and Jeffrey Scott Vitter. High-order entropy-compressed text indexes. In Proceedings of the 4th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 23), pages 84 85, 23. Guy Jacobson. Space-efficient static trees and graphs. In FOCS, pages , Simon Gog:

33 H k of selected Pizza&Chili 2MB test cases H k (T ) contexts/ T in percent k DBLP.XML DNA ENGLISH PROTEINS Simon Gog: