Dictionary Matching in Elastic-Degenerate Texts with Applications in Searching VCF Files On-line

Transcription

1 Dictionary Matching in Elastic-Degenerate Texts with Applications in Searching VF Files On-line MatBio 18 Solon P. Pissis and Ahmad Retha King s ollege London 02-Aug-2018 Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 1 / Files 20 O

2 ontents Pan-Genomes and data structures. Elastic Degenerate Strings and other definitions. EDSM vs MEDSM searches. Multi-EDSM search example. Experiments. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 2 / Files 20 O

3 Introduction A pan-genome is a set of genomes from the same species or closely-related species. Various data structures to store pan-genomes have been suggested: De Bruijn graphs [PanGenome onsortium, Brief. Bioinform, 2016] Variation graphs [J. Siren, ALENEX, 2017] Bloom Filter Trie [Holly et al, Algorithms Mol Biol, 2016] Suffix Tree-based structures [Baier et al, Bioinformatics, 2016] Elastic Degenerate Strings [Huang et al, Bioinformatics, 2013] All but the last are Off-line solutions. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 3 / Files 20 O

4 Elastic Degenerate Strings onsider the following Multiple Sequence Alignment of three closely-related sequences: ATGAAGGGTA--TTTTA ATGAAGGGTATATTTTA ATGATGG----TTTTA ED string compacted represention of length n and total size N: T = ATGA A G T GG TA TATA ε TTTTA Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 4 / Files 20 O

5 Definitions An Elastic Degenerate (ED) string consists of deterministic and non-deterministic/degenerate positions or segments. Fixed Sized Alphabet Σ = O(1) and Σ = A,, G, T. Plus ε, representing deleted segment. An ED string has length n positions, with each character in deterministic segments and each degenerate position counted as 1 position. Total size of an ED string N, with ε = 1, is defined by: n 1 N = i=0 X[i] 1 j=0 X[i][j]. Problem EDSM - We report ending position <i> for every match of pattern p in the ED string, where 0 i < n. Problem MEDSM - We report <i, j> for any ending position i we find any pattern j in set P, where 0 j < P. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 5 / Files 20 O

6 Variant all Format 1000 Human Genomes Project, Phase 3, 2504 individuals many SNPs and other variants. VF files record the position of variants of every sample relative to the reference genome. #HROM POS ID REF ALT HG00096 HG00097 HG rs T When the variants are applied for these samples, reference: TTTGTTAT, they produce: allele 0 allele 1 HG00096 TTTGTTAT TTTGTTATT HG00097 TTTGTTATT TTTGTTATT HG00143 TTTGTTAT TTTGTTAT Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 6 / Files 20 O

7 EDSM Example Problem EDSM: Let T be an ED string for which we want to find p = AAA. We find a match ending at position i = 2: A T = A AA ε A And another match at i = 4: T = A A AA ε A Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 7 / Files 20 O

8 MEDSM Example Problem MEDSM: Let T be an ED string for which we want to find P = AA, AA. T = Our matches are: A A AA ε Position i Pattern j A Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 8 / Files 20 O

9 Algorithm Multi-EDSM Three/Four steps at each position T [i]: 1 Memorize positions of prefixes. 2 heck if possible to extend pattern prefixes. 3 heck if suffix ends here and report. 4 If T [i][j] m min, do full pattern matching, and report if found. Pre-processing by storing the letter positions of P = AA, AA in vector I: Letter Bitvector I A I I G I T Time O(M + σ M/w ) and space O(σ M/w ). Where w is the computer word size - usually 64. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 9 / Files 20 O

10 Algorithm Multi-EDSM continued... In pre-processing we also construct a Suffix Tree of P = AA#AA$, ST P in time O(M). (Some nodes ommitted) A 7 $ 0 A#AA$ #AA$ 2 A 5A$ #AA$ 1 6$ A$ 4 Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 10 / Files 20 O

11 Algorithm Multi-EDSM continued... The tree will be used answer OccVec queries given a string α, return a bitvector marking starting positions where α occurs in P. Lemma Given an integer parameter τ, where 1 τ M/w, a data structure of size O( M/τ M/w ) can be constructed in time and space O( M/τ M/w ) answering OccVec queries in time O( α + τ). But how? Lemma Given a k-ary tree, where k represents edge cardinality and k 3, its equivalent binary tree T P can be generated in time and space O(M). Lemma We partition T P with cardinality k 3 into n/τ disjoint connected subgraphs (micro trees). Each micro tree contains up to τ nodes and upper and lower boundary nodes connecting them with other micro trees. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 11 / Files 20 O

12 Algorithm Multi-EDSM continued... By setting τ = M w, any OccVec query requires at most time O( α + τ). v u Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 12 / Files 20 O

13 Algorithm Multi-EDSM continued... At each boundary node we store a bitvector b marking all positions of the suffix. When we query OccVec(α) we spell down the tree down to a node v: If v is a boundary node of some micro tree, we simply return a pointer to b v ; this takes constant time. If v is not a boundary node, we can find the bottom boundary node u (if it exists) of the micro tree. We create a new empty bitvector b u and if node u exists, we set b u = b u. Then we traverse the micro tree rooted at v, obtaining the suffix number of each leaf node and setting the corresponding bits on in b u. We then return a pointer to the updated b u; this takes time O(τ). Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 13 / Files 20 O

14 MEDSM Example Back to our problem of finding P = AA, AA in T : Letter Bitvector I A I T = A A AA ε A At position 1, B = , we mark the position of prefixes of P. At position 2, we are able to extend the prefix of P 0 from B and report a match. At position 2, we do full pattern matching and report P 1. We finish at position 2 with prefix B = Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 14 / Files 20 O

15 MEDSM Example ontinued Still looking for P = AA, AA in T : T = A A AA ε A Remember B = from previous segment (2). At position 3 we run B = OccVec() to get B = Note that positions are pre-shifted right and boundary 1s are excluded. We bitwise-and B with B. B = & to mark valid infixes and shift it by α. We have matched A so far. The presence of ε means we simply do a bitwise-or (set union) of previously matched prefixes with prefixes found in the current position. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 15 / Files 20 O

16 MEDSM Example ontinued Still looking for P = AA, AA in T : T = A A AA ε A At position 4, when we try to extend the prefix on the first word using OccVec() but it fails on the first letter. Recall that B = because we memorised positions from segments 2 and 3. So then we try the next word and we are able to extend the prefix and complete the pattern, reporting a match for P 1. We do these three/four steps for every segment. Searching takes time O(N M/w ) and requires O( M/w ) extra space. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 16 / Files 20 O

17 Experiments: Time Performance DNA; randomly-generated synthetic ED texts; 10% of degenerate positions. 7 Log processing time (s) Log processing time (s) Total pattern length (M) (a) Processing time with increasing patterns total length on a fixed ED text of length n = (N = ) Total text size (N) (b) Processing time with increasing ED total text size on a fixed set of patterns of total length M = Figure: Time performance of Multi-EDSM. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 17 / Files 20 O

18 Experiments: Multi-EDSM vs EDSM-BV DNA; randomly-generated synthetic ED texts; 10% of degenerate positions Multi-EDSM EDSM-BV Time (s) No. of patterns Figure: Elapsed-time comparison of Multi-EDSM and EDSM-BV with an ED text of total size N = and sets of randomly-generated patterns of length 40 each. Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 18 / Files 20 O

19 Experiments: MAW Validation We designed a three stage pipeline for determining the validity of Minimal Absent Words (MAWs) discovered in the human genome. A Minimal Absent Word (MAW) y of a text x is a word whose proper factors occur in x but y itself is absent from x. We searched a list of patterns (max-length 12) with combined length M = We discovered 73% of MAWs were invalid, leaving only of potential MAWs remaining. We checked the work of Silva et al [Bioinform, 2015], who had identified 3 MAWs present in Ebola virus genomes not present in the Human reference genome... but it is present in Human PanGenome: id sequence position variant id sample id ethnicity RAW1 TTTGGAT 6: rs NA18606 Han hinese RAW2 TAGTATG 1: rs HG02146 Peruvian RAW3 TAGGAAA 15: rs HG03598 Bengali Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 19 / Files 20 O

20 onclusion Multi-EDSM is an algorithm to solve the Multiple Elastic Degenerate String Matching (MEDSM) problem On-line. Preprocessing time and space (with τ = M/w ) is O(M). Multi-EDSM solves the EDSM problem with time O(N m/w ) and space O(m) whereas EDSM-BV requires space O(m m/w ). Search time is O(N M/w ). Multi-EDSM is a robust and fast algorithm. It is a useful tool for searching VF data on-line, especially for partially indexed data. Simplified implementation of Multi-EDSM: Other tools: Solon P. Pissis and Ahmad Retha (King s ollege Dictionary London) Matching in Elastic-Degenerate Texts with Applications 02-Aug-2018 in Searching VF 20 / Files 20 O