= Prob( gene i is selected in a typical lab) (1)

Transcription

1 Supplementary Document This supplementary document is for: Reproducibility Probability Score: Incorporating Measurement Variability across Laboratories for Gene Selection (006). Lin G, He X, Ji H, Shi L, Davis R and Zhong S, Nature Biotechnology. Definition of the RPS The RPS (Reproducibility Probability Score) as developed to identify differentially expressed genes. In many aspects, the RPS behaves lie the False Discovery Rate (FDR) or the p-value in such a ay that it is correlated ith ho strong the evidence for differential expression is in the current study. Hoever, the RPS taes into consideration the variability of the expression measurement across different laboratories. The value of the RPS for a gene ill be adversely affected if the expression measurement for this gene is highly variable (not consistent) across laboratories. The exact lab-to-lab variability for a given gene is not non, and may vary from experiment to experiment. The RPS procedure estimates the gene specific measurement variability across laboratories from the data in a given experiment together ith the reference data from the MAQC proect. The computation of RPS does not require the biological samples in this given experiment to be the same as those in the reference data. The RPS for gene i is defined as the probability that gene i ill be selected if the experiment is repeated in a typical laboratory: RPSi = Prob( gene i is selected in a typical lab) () Here, a typical laboratory is a hypothetical laboratory that is equally qualified as the current laboratory here the actual experiment is conducted. A laboratory is qualified if it follos the same protocol to study the same biological materials as the original laboratory. The RPS is a mathematical evaluation of the reproducibility of a gene selection result across qualified laboratories. For simplicity of annotation, hereafter e ill suppress the subscript i. To compute

2 the RPS for a gene, the user needs to choose a gene selection procedure as ell as the threshold accompanying that procedure. For example, if the user chooses to select genes ith FDR<0. and FC (fold-change)>3, then RPS = E { I[ FDR < 0. AND FC > 3]}, () here denotes a typical laboratory, E {} i is the expectation over, I[] i is an indicator function taing values 0 or, and FDR and FC are the FDR and the FC computed from the data in laboratory. Usually a microarray experiment is performed only in a single laboratory. We use =0 (laboratory 0) to denote the laboratory here the real microarray data are generated. For all other laboratories, >0, the data are generally unavailable. It is orth noticing that hen there is a perfect correlation of the inter-laboratory measurements, Equation () reduces to: RPS = I[ FDR0 < 0. AND FC0 > 3] (3), hich is identical to applying the user chosen gene selection procedure and threshold to the data from the real laboratory (laboratory 0). We use Θ to denote the metric(s) of user chosen gene selection procedure and θ to denote the threshold(s) for Θ. For example, Equation () can be reritten as RPS = E I FDR > AND FC > {[ 0. 3]} = E {[ I Θ > θ ]} (4), FDR 0. here Θ= and θ = (5). FC 3 Equation (4) is equivalent to Equation (), but easier to use in practice. Θ> θ in Equation 4 broadly represents any gene selection procedures, and users can choose to plug in FDR, MAANOVA or any other procedure. Unless the context specifies otherise, throughout the paper e plug in FDR > 0. AND FC > 3. We ill compare some gene selection procedures in the Results section. Throughout this document, e use FC>λ to denote {FC> λ and FC< / λ, for any λ >}.

3 To compute the RPS, e use simulation to mimic the gene expression measurements across laboratories. Recall that laboratory 0 denotes the lab in hich the actual microarray data is generated. Suppose that to biological samples, A and B, are studied at this laboratory. For each sample, e utilize to sets of data to simulate the expression measurements in other laboratories. These to sets of data are: ) the actual expression data of this gene in laboratory 0 for samples A and B. ) ρ : the inter-laboratory correlation of expression measurements for this gene. The correlation ρ is not sample dependent, i.e. ρ is gene specific but not sample specific inter-laboratory correlation of the expression measurements. We estimate ρ from a mixed-effects model using reference data as described in the Methods section. The simulation procedure is also given in the Methods section. With the simulated data, the user chosen gene selection procedure Θ> θ can be applied to select genes in every simulated laboratory. From Equation (4), the RPS for a gene can be simply estimated by the average number of times that this gene satisfies the gene selection criteria over all simulated laboratories (Figure S). Statistical procedure for computing RPS Step: modeling inter-laboratory correlation from reference data. This procedure estimates a sample independent correlation of microarray measurements beteen any to typical laboratories. To achieve this estimation, e developed a mixed-effects statistical model []. Let Y ir denote the r-th replicate of the expression measurement for gene in laboratory i on sample. Before fitting the mixed-effects model, Y ir should be properly preprocessed. In our preprocessing, e log transformed the normalized expression data. For

4 simplicity, e suppress gene index and use Y hereafter. For each gene, e fit a mixed-effects model: Y = μ + L + S + ε (6), ir i ir ir here L i is the effect of the i th laboratory, and i =,,, n L. The notion of laboratory here includes other unspecified factors that change from laboratory to laboratory. For example, every laboratory in the MAQC proect had a different experimenter. The L is treated as a random effect. S is the effect of the i th sample, and =,,, n. The S is treated as a random effect as ell. S μ is the offset constant. The folloing assumptions are made: L ~ N(0, σ ) =,,,, (7) i L S ~ N(0, σ ) =,,,, (8) ir ~ N(0, ε ) s ε σ i =,,, n, =,,, (9) L, S and i n L L n S ε ir are independent for any i and. n S Based on Model (6), the correlation ρ for gene beteen any to labs can be ritten as: cov( Y, Y ) σ ρ = = var( Y )*var( ) σ + σ + σ r r s r Yr L s ε (0) The correlation ρ beteen laboratories is gene-specific. I.e. for every gene, there is a corresponding correlation ρ. simplicity of notation. is suppressed in Equation (0) for ρ represents the correlation beteen expression measurements in any to typical labs on a hypothetical sample. Here e made an assumption that the reference data comes from random biological samples. In the RPS softare pacage, e pre-computed and stored the ρ s for the five commercial microarray platforms that participated in the MAQC study. RPS softare also provides a function to compute ρ s for other reference data.

5 Step: Simulation of ne data. Let denote the ne laboratory oring on the ne samples A and B. We ill simulate the gene expression measurements in K hypothetical labs for the same samples A and B. To do the simulation, e model the expression measurements of a gene on the same sample in any to labs ith a bivariate normal distribution: Y u ~ N, Y u ρσ 0 r σ ρσ σ r (), here Y 0 r and Y r are the microarray measurements of sample from laboratory (real) and L (imaginary), respectively. r is the index for replicates. L0 ρ is the same as in Equation 0. This model entails the folloing conditional distribution: Y ( Y = y )~ N( u, σ ) r 0r 0r *0 *0 (), here and are the conditional mean and conditional variance, u *0 σ *0 respectively. Moreover, u = u+ y u) (3) *0 ρ( 0 r The observed average expression among replicates in is a natural estimate of the u and the observed sample variance in follos that: is an estimate of σ. And it σ ( ρ ) σ *0 = (4) The ρ derived from Equation (0) is used as the estimate of ρ in (4). Let R denote the number of replicates in laboratory L0 on sample, i.e r =,,, R. For each sample, e first dra a replicate from the real lab. For each dran replicate e then dra a corresponding replicate in the imaginary laboratory L using the conditional distribution (). This step is repeated for every sample and for K times to simulate the data from K imaginary

6 labs. The simulated data depends on the gene specific cross laboratory correlation ρ learnt from the reference data. It also depends on the mean and variance of the replicates in the real laboratory. Step 3: Assigning reproducibility probability score. Recall that the metric RPS (Reproducibility Probability Score) for gene defined as: RPS = Prob( gene is selected in a typical lab) is This is equivalent to RPS = Prob(gene being selected by comparing the same samples from a randomly chosen lab, given the user chosen gene selection criteria ) And therefore RPS K I( gene is selected in Lab L ) (5), K + = 0 here I () i is an indicator function. is the real laboratory. L (=,,K) are K simulated imaginary laboratories. Here e use empirical frequency to approximate the RPS. For example, if the gene selection criteria is simply selecting any genes ith p-values smaller or equal to, then formula (5) P c becomes: RPS K + K = 0 I( p P ), here is the p-value calculated from the c p th simulated laboratory, and L 0. p 0 is the p-value calculated from the real laboratory The statistical procedure above assigns a RPS to each gene. Genes ith higher RPSs are more liely to be reproduced by the same study in other labs. Reference:. Pinheiro, J.B., DM, Mixed-Effects Models in S and S-PLUS. 000: Springer.

7 Figure S: Flo chart for the computation of the RPS for every probe(set). The MAQC data and the Ne experiment data represent data from completely different laboratories and different samples. For each gene: MAQC data Model signal (Y), labs (L i ) and samples (S ) Y = μ + L + S + ε ir i ir Compute inter-lab correlation ρ σ ( σ + σ + ) = s L s ρ σ ε Ne experiment data Simulate hypothetical lab data for each sample K times RPS = E{ prob( gene i is selected in a typical lab)} K I( gene i is selected in simulated Lab L ) + K = 0