docRNA spike-in controls & analysis methods for
download 
Report Transcription
RNA spike-in controls & analysis methods for
RNA spike-in controls & analysis methods for trustworthy genome-scale measurements Sarah A. Munro, Ph.D. Genome-Scale Measurements Group ABRF Meeting March 29, 2015 Overview • External RNA Controls Consortium (ERCC) RNA spike-in controls • ‘erccdashboard’ analysis tool • ERCC 2.0: Building an updated suite of RNA controls Overview • External RNA Controls Consortium (ERCC) RNA spike-in controls • ‘erccdashboard’ analysis tool • ERCC 2.0: Building an updated suite of RNA controls How can we have trustworthy gene expression results? • We’re simultaneously measuring thousands of RNA molecules in gene expression experiments • But are we getting it right? External RNA Controls Consortium (ERCC) initiated by industry, hosted by NIST • Initiated by Janet Warrington, VP Clinical Genomics at Affymetrix • Open to all interested parties • Voluntary • More than 90 participants – Industry, Academia, Government – All major microarray technology developers – Other gene expression assay developers Spikeins ERCC control sequences are in NIST Standard Reference Material 2374 • DNA sequence library • 96 unique control sequences in DNA plasmids • Controls intended to mimic mammalian mRNA • In vitro transcription to make RNA controls NIST SRM 2374 and related data files are available directly from NIST @ http://tinyurl.com/erccsrm Making ERCC ratio mixtures with true positive and true negative ratios NIST Plasmid DNA Library RNA transcripts … in vitro transcription Pooling Mixtures with known abundance ratios Using ERCC ratio mixtures Treated (n>3) Control (n>3) Using ERCC ratio mixtures Treated (n>3) Control (n>3) Using ERCC ratio mixtures Treated (n>3) Control (n>3) Using ERCC ratio mixtures Treated (n>3) Control (n>3) Measurement process Expression Measures Statistical Analysis Multiple steps Many people & labs Takes days to weeks Example gene expression data Treated Control Are the RNA molecule ratios statistically different across the samples? Treated Control Evaluate technical performance with ERCC true positive and true negative ratios Treated Control Overview • External RNA Controls Consortium (ERCC) RNA spike-in controls • ‘erccdashboard’ analysis tool • ERCC 2.0: Building an updated suite of RNA controls Use erccdashboard to produce standard performance metrics for any experiment • R package is available from: – Bioconductor – NIST GitHub Site • Open source and open access for use in – Other analysis tools and pipelines – Commercial software Gauge technical performance with 4 erccdashboard figures • Developed as part of SEQC study, with ABRF partners • Technology-independent ratio performance measures • Assessed differences in performance across – Experiments – Laboratories – Measurement processes Munro, S. A. et al. Nature Communications 5:5125 doi: 10.1038/ncomms6125 (2014). Ambion ERCC Ratio Mixtures 23 Controls per Subpool Design abundance spans 220 range within each Subpool Spike-in design for SEQC RNA Sequencing Experiments Samples replicates for sequencing Rat Experiment Treated and Control Rat RNA Biological Replicates Interlaboratory Experiment Human Reference RNA Samples Technical Replicates What is the dynamic range of my experiment? Interlaboratory Experiment Log2 Normalized ERCC Counts Log2 Normalized ERCC Counts Rat Experiment Log2 ERCC Spike Amount (attomol nt µg-1 total RNA) Log2 ERCC Spike Amount (attomol nt µg-1 total RNA) What is the dynamic range of my experiment? Typical Sequencing ~40 million sequence reads per replicate Log2 ERCC Spike Amount (attomol nt µg-1 total RNA) Interlaboratory Experiment Log2 Normalized ERCC Counts Log2 Normalized ERCC Counts Rat Experiment Deep Sequencing ~260 million sequence reads per replicate Log2 ERCC Spike Amount (attomol nt µg-1 total RNA) What was the diagnostic power? Interlaboratory Experiment True Positive Rate True Positive Rate Rat Experiment False Positive Rate False Positive Rate What was the diagnostic power? Area Under the Curve (AUC) depends on the number of controls detected! False Positive Rate Interlaboratory Experiment True Positive Rate True Positive Rate Rat Experiment False Positive Rate AUC is a reasonable summary statistic… But we’d like to evaluate our diagnostic performance as a function of abundance… Log2 Normalized Ratio of Counts Rat Experiment MA Plot Log2 Normalized Average Counts LODR: Limit of Detection of Ratios Rat Experiment Reference RNA • Model P-values as a function of average signal DE Test P-values • Find P-value threshold based on chosen false discovery rate • Here FDR = 0.1 • Default is FDR = 0.05 Average Counts • Estimate LODR from intersection of model confidence interval upper bound and P-value threshold LODR: Limit of Detection of Ratios Rat Experiment Reference RNA LODR provides DE Test P-values • Specified confidence in the differentially expressed transcripts above LODR (90% chance of <10% FDR) Average Counts • Guidance for experimental design increase signal for transcripts above LODR estimate 4:1 LODR Log2 Ratio of Normalized Counts Rat Experiment MA Plot Log2 Normalized Average Counts 4:1 LODR Log2 Ratio of Normalized Counts Rat Experiment ** MA Plot * Log2 Normalized Average Counts 4:1 LODR ** * Log2 Ratio of Normalized Counts Rat Experiment MA Plot Log2 Normalized Average Counts Increased sequencing depth shifts endogenous transcript ratio measurements above LODR What are the LODR estimates for my experiment? Interlaboratory Experiment DE Test P-values DE Test P-values Rat Experiment Average Counts Average Counts How do the endogenous samples relate to LODR? Interlaboratory Experiment Log2 Ratio of Normalized Counts Log2 Ratio of Normalized Counts Rat Experiment 4:1 LODR Log2 Normalized Average Counts 4:1 LODR Log2 Normalized Average Counts How much technical variability & bias is there? Rat Experiment Interlaboratory Experiment Log2 Ratio of Normalized Counts Log2 Ratio of Normalized Counts Decreased Variability Significant Ratio Bias mRNA Fraction Differences Between Samples Contributes to Bias in ERCC Ratios Spike-in Spike-in Total RNA mRNA mRNA mRNA enrichment rRNA Sample 1 Sample 2 Sample 1 The RNA fractions are exaggerated for illustration purposes Sample 2 Dynamic Range AUC • Variability • Bias • LODR & Sample Transcripts LODR Diagnostic performance Limit of Detection of Ratios EVALUATE REPRODUCIBILITY ACROSS LABORATORIES Good Performance Poor Performance Interlaboratory Analysis Using erccdashboard performance metrics Lab 1-6 Illumina + poly-A selection (Illumina kit) Lab 7-9 Life Tech + poly-A selection (Life Tech kit) Lab 10-12 Illumina + ribosomal RNA depletion • Diagnostic performance was consistent within and amongst measurement processes • Lab 7 was an outlier for diagnostic performance LODR (Average Counts) Consistent LODR across 11 of 12 Labs • LODR agreement with AUC Laboratory Ratio bias is highly variable amongst experiments Ratio bias (rm) can be attributed to mRNA fraction difference between samples: Rs = nominal subpool ratio (E1/E2)s = empirical ratio • Shippy et al. 2006 mRNA fraction Difference Log(rm) • Large standard errors indicate that mRNA fraction isn’t the only factor contributing to ERCC ratio bias – mRNA enrichment protocol is a factor… Laboratory Protocol-dependent bias from poly-A selection affects ERCC controls due to short poly-A tails Lab 1-6 ILM Poly-A Lab 7-9 LIF Poly- Lab 10-12 ILM Ribo mRNA enrichment protocol biases vary across individual ERCCs but are consistent for a protocol mRNA enrichment protocol biases vary across individual ERCCs but are consistent for a protocol Results of the erccdashboard Publication • Ratio performance measures for any technology platform and any experiment – Diagnostic Power – Novel LODR metric – Technical Variability & Bias • Comparison across experiments • Quantification of mRNA fraction differences between samples • Show protocol-dependent bias Overview • External RNA Controls Consortium (ERCC) RNA spike-in controls • ‘erccdashboard’ analysis tool • ERCC 2.0: Building an updated suite of RNA controls ERCC 2.0: A New Suite of RNA Controls • Approached by industry and academia to build new RNA controls • NIST-hosted open, public ERCC 2.0 workshop – Workshop report and presentations available: slideshare.net/ERCC-Workshop • All interested parties are welcome to participate – Sequence contributions – Interlaboratory analysis • New and Improved mRNA Mimics • Transcript Isoforms • miRNA New and Improved mRNA Mimics • Additional controls • Expand distributions of RNA control properties – Length (> 2kb) – GC content – Poly-A tail length Transcript Isoform Controls • Transcript Design – Non-cognate Spike-in RNA Variants (SIRVs) developed by Lexogen – Cognate sequence selection in progress • Schizosaccharomyces pombe • Mixture design – Dynamic Range • 24 – Design Ratios • < 2:1 Lukas Paul, Lexogen Small and miRNA Controls • Needed for validation of clinical applications – Early Detection Research Network – Tgen • Other applications relevant to bacterial RNA-Seq • Non-cognate miRNA controls • Include some pre-miRNA • Direct RNA control synthesis by Agilent – no need for DNA templates Karol Thompson, FDA Recap • External RNA Controls Consortium (ERCC) RNA spike-in controls • ‘erccdashboard’ analysis tool • ERCC 2.0: Building an updated suite of RNA controls Acknowledgements • All External RNA Controls Consortium participants • NIST – – – – – – – – Marc Salit Steve Lund P. Scott Pine Justin Zook David Duewer Jerod Parsons Jennifer McDaniel Margaret Klein • Empa – Matthias Roesslein • SEQC study participants • Co-authors on erccdashboard manuscript: S. P. Lund, P. S. Pine, H. Binder, D. Clevert, A. Conesa, J. Dopazo, M. Fasold, S. Hochreiter, H. Hong, N. Jafari, D. P. Kreil, P. P. Łabaj, S. Li, Y. Liao, S. M. Lin, J. Meehan, C. E. Mason, J. Santoyo-Lopez, R. A. Setterquist, L. Shi, W. Shi, G. K. Smyth, N. Stralis-Pavese, Z. Su, W. Tong, C. Wang, J. Wang, J. Xu, Z. Ye, Y. Yang, Y. Yu, & M. Salit For more information contact: [email protected]
