Fast Hash-Based Algorithms for Analyzing Tens of Thousands of Evolutionary Trees

Transcription

1 Fast Hash-Based Algorithms for Analyzing Tens of Thousands of Evolutionary Trees Tiffani L. Williams Department of Computer Science & Engineering Texas A&M University

2 What is an Evolutionary (or Phylogenetic) Tree? Evolutionary relationships between organisms (taxa) are depicted in a family tree structure called a phylogenetic tree. Popular techniques often return tens to hundreds of thousands of phylogenetic trees that represent equally-plausible hypotheses for how the taxa evolved from a common ancestor. (a) rooted tree (b) unrooted tree

3 Where do large tree collections come from? 33,306 trees on 567 taxa of flowering plants (U. of Florida) 90,000 trees on 264 taxa of fish (Texas A&M) 150,000 trees on 525 taxa of insects (Texas A&M) We need computational approaches for analyzing these large tree collections especially as the size of phylogenetic studies continue to increase.

4 Our Hash-Based Algorithms for Analyzing Large-Scale Tree Collections Let t represent the number of phylogenetic trees of interest. HashRF: computes a t t Robinson-Fould matrix MrsRF: multi-core version (using MapReduce) of HashRF HashCS: computes a majority or strict consensus tree TreeZip: compresses t trees into a smaller representation (c) MDS (d) Heatmap

5 Why Do We Need to Compress Trees? Large collections of trees can be expensive to store and transfer, and they are only continuing to growing in size. Ideally, we should be able to share tree collections quickly with little or no cost. The most convenient way to share trees seems to be via . Most of the tree collections we have received have been ed. However, large collections either had to be broken into smaller pieces and sent to us or hand-delivered to our lab. When compressed with our approach, these large collections can now be sent via .

6 Our Solution: TreeZip

7 The Newick File Format The Newick file format is the most widely used format to store a phylogenetic tree in a file. Topology of an phylogenetic tree is uniquely defined by its set of bipartitions. TreeZip represents these bipartitions internally as bitstrings. In Newick format, the phylogenetic tree is represented using a notation based on balanced parentheses.

8 Compression: Extracting Bipartitions from Trees (e) Newick file (f) Hash table Figure: During compression, TreeZip parses the input Newick file, extracting all bipartitions and inserting them into a hash table.

9 Compression: Generating.trz file (a) Hash table (b) Shared table Figure: Once the hash table is populated, contents are loaded into a ragged array structure called the shared table.

10 Compression: Generating.trz file (a) Shared table (b) Compact table Figure: Using the contents of the shared table, lines are compacted according to which is shorter: the original line or its complement.

11 Compression: Generating.trz file (a) Compact table (b) TreeZip (.trz) file Figure: Using this compact representation, the contents of the shared table are run length encoded and included as the heart of the.trz file.

12 Decompression (a) Bipartition Collector (b) Newick File Figure: In decompression, the bipartition data stored in the.trz file are extracted and used to rebuild the Newick file.

13 Experimental Methodology: Biological Tree Collections Datasets Taxa Trees File size (MB) Bipartitions 1 mammals 16 8, freshwater , ,168 3 angiosperms , ,444 4 fish , ,115 5 insects ,

14 Experimental Methodology: Platform and Performance Metric System: 2.5Ghz Intel Core 2 quad-core machine with 4GB of RAM running Ubuntu Linux Performance measure: Space savings S is the reduction in size relative to the uncompressed size. S = 1 Our implementation of TreeZip can be found at compressed file original file.

15 TreeZip Results

16 TreeZip Running Time: Compression + Decompression 7zip TreeZip TreeZip+7zip Total Time (s) fish angiosperms freshwater mammals Data Set insects

17 Using Different, but Equivalent Newick Strings There are O(2 n 1 ) newick strings for a tree with n taxa. The designers of TASPI note that their algorithm is affected by the ordering of the taxa in the Newick string. This will result in a larger compressed file. TreeZip, however, is not effected by the ordering of taxa in a Newick string.

18 TreeZip Results: Using Different, but Equivalent Newick Strings (Collections 1 5) 7zip TreeZip TreeZip+7zip Different Newick File/Original Newick File fish angiosperms freshwater mammals insects Data Set

19 Conclusions & Future Work Compression algorithms such as TreeZip will become critical tools for helping biologists manage their rapidly expanding phylogenetic tree collections. TreeZip allows large phylogenetic tree collections to be easily exchanged with others, which is essential for successful scientific collaboration. When compressed with TreeZip+7zip, our largest dataset (434 MB) is compressed to such a small size (32 KB), allowing it to be easily sent over ! In the future, TreeZip will be optimized for speed and will support branch lengths.

20 Thanks for Listening! A big thank you to: Ph.D. students: Suzanne Matthews and Seung-Jin Sul. Additional input from Grant Brammer, Charles Lively, Ralph Crosby and Brian Davis. Funding for this project was supported by NSF under grants DEB and IIS

21 Backup Slides Backup Slides

22 How Encoding Works

23 Decompression: Extracting Bipartitions from.trz file (a) TreeZip file (.trz) (b) Bipartition Collector Figure: In decompression, the bipartition data stored in the.trz file are loaded into the bipartition collector.

24 Decompression: Rebuilding Trees (a) Bipartition Collector (b) Newick File Figure: The bipartition collector is then used to rebuild the phylogenetic trees and output the corresponding Newick file.

25 TreeZip and TASPI Results (Collections 6 14): Full Results 10 gzip bz2 7zip TreeZip TreeZip+gzip TreeZip+bz2 TreeZip+7zip TASPI TASPI+bz2 Compression Ratio (%) lipsc439 john921 eern476 aster328 will2000 three567 rbcl500 ocho854 mari2594 Data Set TreeZip achieves a better (lower) compression ratio than TASPI on all these sets.

26 TreeZip Results (Collections 1 5): Full Results 100 gzip bz2 7zip TreeZip TreeZip+gzip TreeZip+bz2 TreeZip+7zip Compression Ratio (%) fish angiosperms freshwater mammals Data Set insects

27 TreeZip Results: Using Different, but Equivalent Newick Strings (Collections 1 5): Full Results gzip bz2 7zip TreeZip TreeZip+gzip TreeZip+bz2 TreeZip+7zip Different Newick File/Original Newick File fish angiosperms freshwater mammals insects Data Set

28 TreeZip Running Time: Compression + Decompression: Full Results gzip bz2 7zip TreeZip TreeZip+gzip TreeZip+bz2 TreeZip+7zip Total Time (s) fish angiosperms freshwater mammals Data Set insects

29 TreeZip Running Time: Compression (Full Results) gzip bz2 7zip TreeZip TreeZip+gzip TreeZip+bz2 TreeZip+7zip Compression Time (s) fish angiosperms freshwater mammals Data Set insects

30 TreeZip Running Time: Decompression (Full Results) gzip bz2 7zip TreeZip TreeZip+gzip TreeZip+bz2 TreeZip+7zip Decompression Time (s) fish angiosperms freshwater mammals Data Set insects