Cloud-scale RNA-sequencing differential expression analysis with Myrna

Transcription

1 Cloud-scale RNA-sequencing differential expression analysis with Myrna Jeff Leek Johns Hopkins Bloomberg School of Public Health e: t: myrna:

2 Acknowledgements Kasper Hansen Ben Langmead

3 RNA-Sequencing Experiments Sample A TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B

4 Big Picture Ideas A Problem: The growth in data from sequencing is outstripping the growth in processing speed/storage of individual computers A solution: Take advantage of economies of scale and rent scalable computing resources for sequencing analysis Today: Myrna Tri-mode software for RNA-sequencing analysis

5 Intimidating trends GA II 1.6 billion bp per day (old) GA IIx 5 billion bp per day (current) HiSeq billion bp per day (for release later in 2010) Images from:

6 Computational throughput Moore s Law: The number of transistors that can be placed inexpensively on an integrated circuit doubles approximately every two years.

7 Intimidating Trends Stein Genome Biology (2010) doi: /gb

8 Intimidating trends Kahvejian et al. Nature Biotechnology (2008) doi: /nbt1494

9 Difficulties With Storage/Support

10 Throughput growth gap > 4-5x per year 2x per 2 years

11 Throughput growth gap = Idle

12 Throughput growth gap = Faster algorithms

13 Throughput growth gap

14 Throughput growth gap

15 Cloud computing Rent; don t buy. : Cloud vendor : Electric Company

16 Renting Computing Time Easier access to ever larger economies of scale Columbia river for cheap hydroelectric power & cooling

17 Cloud computing Pros & Cons Why? Cost? Handles demand that grows, shrinks dramatically No hardware maintenance No alternative? Why not? Cost? Harder to program Less user-friendly Data movement is inconvenient & can outpace network Privacy (e.g. IRB concerns)

18 Amazon Web Services

19 Cloud computing 1.7 GB RAM, 1 32-bit virtual processor core clocked at ~1.2Ghz, ~160 GB local storage 70 GB RAM, 8 64-bit virtual processor cores clocked at ~4.0 Ghz each, ~1.5 TB local storage 7 GB RAM, 8 64-bit virtual processor cores clocked at ~2.6 Ghz each, ~1.5 TB local storage

20 RNA-Sequencing Experiments Sample A TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B

21 Sample A Simple Goal: Determine If Gene 1 Is Differentially Expressed TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B

22 RNA-Sequencing Experiments Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC

23 RNA-Sequencing Experiments Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TGTCGCAGTATCTGTC TATGTCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATGTCGCA AGCACCCTATATCGCA AGCACCCTATGTCGCA GAGCACCCTATGTCGC CCGGAGCACCCTATAT TCTCTCCCANNAGAGC CCGGAGCACCCTATAT TCTCTCCCAGGAGAGC GCCGGAGCACCCTATG TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B TGTCGCAGTATCTGTC AGCACCCTATGTCGCA GTCGCAGTANCTGTCT GCCGGAGCACCCTATG GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC

24 RNA-Sequencing Experiments Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TGTCGCAGTATCTGTC TATGTCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATGTCGCA AGCACCCTATATCGCA AGCACCCTATGTCGCA GAGCACCCTATGTCGC CCGGAGCACCCTATAT TCTCTCCCANNAGAGC CCGGAGCACCCTATAT TCTCTCCCAGGAGAGC GCCGGAGCACCCTATG TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B TGTCGCAGTATCTGTC AGCACCCTATGTCGCA GTCGCAGTANCTGTCT GCCGGAGCACCCTATG GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC

25 RNA-Sequencing Experiments Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TGTCGCAGTATCTGTC TATGTCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATGTCGCA AGCACCCTATATCGCA AGCACCCTATGTCGCA GAGCACCCTATGTCGC CCGGAGCACCCTATAT TCTCTCCCANNAGAGC CCGGAGCACCCTATAT TCTCTCCCAGGAGAGC GCCGGAGCACCCTATG TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B TGTCGCAGTATCTGTC AGCACCCTATGTCGCA GTCGCAGTANCTGTCT GCCGGAGCACCCTATG GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC

26 RNA-Sequencing Experiments Sample A GTCGCAGTANCTGTCT GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TGTCGCAGTATCTGTC TATGTCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG TATATCGCAGTATCTG CCCTATATCGCAGTAT AGCACCCTATGTCGCA AGCACCCTATATCGCA AGCACCCTATGTCGCA GAGCACCCTATGTCGC CCGGAGCACCCTATAT TCTCTCCCANNAGAGC CCGGAGCACCCTATAT TCTCTCCCAGGAGAGC GCCGGAGCACCCTATG Gene 1 differentially expressed?: YES p-value: TGGTATTTTCGTCTGGGGGGTATGCACGCGATAGCATTGCGAGACGCTGGAGCCGGAGCACCCTATTTGATTCCTGCCTCATCCTATTATTTATCGCACCTAC Sample B TGTCGCAGTATCTGTC AGCACCCTATGTCGCA GTCGCAGTANCTGTCT GCCGGAGCACCCTATG GGATCTGCGATATACC GGATCT-CGATATACC AATCTGATCTTATTTT AATCTGATCTTATTTT ATATATATATATATAT ATATATATATATATAT TCTCTCCCANNAGAGC TCTCTCCCAGGAGAGC

27 The Myrna Pipeline

28 Bet-hedging architecture Cloud driver script Hadoop driver script Singleton driver script Wrapper Wrapper Wrapper bowtie Wrapper Hadoop bowtie Wrapper Hadoop bowtie Wrapper Perl, fork, sort soapsnp soapsnp soapsnp Postprocess Postprocess Postprocess Cloud mode Hadoop mode Single-computer mode

29 Pickrell Study 69 lymphoblastoid Hapmap cell lines Sequenced in two labs Argonne and Yale Total of 1.1 Billion 35 bp reads

30 Runtime/Costs On EC2 Myrna Runtime, Cost for 1.1 billion reads from Pickrell et al study EC2 Nodes 1 master, 1 master, 1 master, 10 workers 20 workers 40 workers Worker CPU cores Wall clock time 4h:20m 2h:32m 1h:38m Cluster setup 4m 4m 3m Align 2h:56m 1h:31m 54m Overlap 52m 31m 16m Normalize 6m 7m 6m Statistics 9m 6m 6m Summarize & Postprocess 13m 14m 13m Approximate cost (N. Virginia / Elsewhere) $44.00 / $49.50 $50.40 / $56.70 $65.60 / $73.80 Table 1. Timing and cost for a Myrna experiment with 1.1 billion 35 bp unpaired reads from the Pickrell et al study as input. Costs are approximate and based on the pricing as of this writing, that is, $0.68 per extra-large high-cpu EC2 node per hour in the Northern Virginia zone and $0.78 in other zones, plus a $0.12 per-node-per-hour surcharge for Elastic MapReduce in all zones. Times can vary subject to, for example, congestion and Internet traffic conditions.

31 Compute Time Nearly Linear in CPU Cores

32 Three Myrna Vignettes 1. Comparing models for differential expression 2. Biological variation in RNA-seq & microarrays 3. Identifying batch effects in sequencing

33 Vignette 1 Comparing RNA-Seq DE Models 69 lymphoblastoid Hapmap cell lines Sequenced in two labs Argonne and Yale Total of 1.1 Billion 35 bp reads Randomly assigned samples to 2-groups

34 Gene by Gene Statistical Model g(e[ f (c ij ) y j ]) = b i0 + η i log(q j ) + b i1 y j

35 Gene by Gene Statistical Model Normalized Counts For Gene i, Sample j Normalization Constant For Sample j g(e[ f (c ij ) y j ]) = b i0 + η i log(q j ) + b i1 y j Link Function Group Indicator Parameter We Test

36 Different Models = Different Results Poisson Model Gaussian Model Left Column η i = 1 Bullard et al. (2010) BMC Bioinformatics doi: / Right Column η i = estimated Langmead et al. (2010) Genome Biology doi: /gb r83 Permutation Model

37 P-values By Log-Count Poisson Model Gaussian Model Left Column η i = 1 Bullard et al. (2010) BMC Bioinformatics doi: / Right Column η i = estimated Langmead et al. (2010) Genome Biology doi: /gb r83 Permutation Model

38 Vignette 2 - Biological Variability in Sequencing Stranger et al. (2007) vs. Montgomery et al. (2010) Choy et al. (2008) vs. Pickrell et al. (2010)

39 RNA-Sequencing Studies With Small n

40 Vignette 3 Batch Effects

41 Batch Effect in 1,000 Genomes Input: 2 billion reads from the thousand genomes project Blue: 3 sds below the mean Orange: 3 sds above the mean Horizontal lines delimit process dates Human chromosome 16

42 Batch Effects Are Strong

43 Big Picture Ideas A Problem: The growth in data from sequencing is outstripping the growth in processing speed/storage of individual computers A solution: Take advantage of economies of scale and rent scalable computing resources for sequencing analysis Today: Myrna Tri-mode software for RNA-sequencing analysis

44 Acknowledgments Myrna Ben Langmead Kasper Hansen Biological Variability Rafael Irizarry Kasper Hansen Zhijin Wu Batch Effects Robert Scharpf Hector Corrada-Bravo David Simcha Benjamin Langmead W. Evan Johnson Donald Geman Keith Baggerly Rafael A. Irizarry

45 References + Further Information Leek Group Twitter Feed: Myrna Website: Langmead B, Hansen KD, Leek JT (2010), "Cloud-scale RNA-sequencing differential expression analysis with Myrna." Genome Biology 11:R83 Leek JT, Scharpf RB, Corrada Bravo H, Simcha D, Langmead B, Johnson WE, Geman D, Baggerly K, Irizarry RA (2010), "Tackling the widespread and critical impact of batch effects in high-throughput data." Nature Reviews Genetics, 11: Hansen KD, Wu Z, Irizarry RA, Leek JT (Submitted), Sequencing technology does not eliminate biological variability.