Supplementary Materials for Robust Estimation of the False Discovery Rate

Transcription

1 Sulementary Materials for Robust Estimation of the False Discovery Rate Stan Pounds and Cheng Cheng This sulemental contains roofs regarding theoretical roerties of the roosed method (Section S1), rovides figures for the examle alications (Section S), uses some simulation results to illustrate theoretical roerties (Section S3) and lots summarizing the results of many simuations (Section S4). S1 Proofs of Theoretical Results S1.1 Notation and Definitions We use the notation from the manuscrit and also introduce some additional notation. Throughout sections S1. through S1.4, we let i =1,...,mindex the hyothesis tests and P i reresent the random variable corresonding to the -value for test i. Let F i (α) =Pr(P i α) and f i (α) be the corresonding robability distribution function or robability mass function. Additionally, we let H A reresent the set of all i such that the alternative hyothesis is true and let H 0 reresent the set of all i such that the null hyothesis is true. Thus, we say i H 0 if the null hyothesis of test i is true. Additionally, let P be the average of P 1,...,P m.for each i, define A i = min(p i, 1 P i ) and G i (α) = Pr(A i α). Finally, let I( ) be the indicator function, so that I( ) = 1 if the enclosed statement is true, and I( ) = 0 if the enclosed statement is false. Now, in terms of these definitions, we have that π = 1 m m I(i H 0 ) (1) i=1 1

2 and υ(α) = 1 m m I(i H 0 )F i (α). () i=1 S1. Results for Two-Sided Tests Throughout this section, we assume that all -values are based on two-sided tests. We now rove a trivial but imortant result. Theorem 1 If E(P i ) 1 for all i H 0, then E( P ) π. Proof: E( P )= 1 m E(P i ) m i=1 ( ) = 1 E(P i )+ E(P i ) m i H 0 i H A 1 E(P i ) m i H m i H 0 π. (3) Theorem 1 is now used to rove the following theorem regarding the conservativeness of υ(α) as defined for two-sided tests. Theorem If Pr(P i α) α for all α and all i H 0 and Pr( P > 1 ) = 0, then E( υ(α)) υ(α). Proof: First, note that Pr(P i α) α imlies that E(P i ) 1. Therefore, E( υ(α)) = E( πα) = αe( π) { ( = α E P P 1 ) ( Pr P 1 ) ( +E 1 P > 1 ) ( Pr P > 1 )} αe( P ) πα. (4)

3 Additionally, observe that ( ) 1 m υ(α) =E I(P i α)i(i H 0 ) m i=1 = 1 E(I(P i α)) m i H 0 = 1 Pr(P i α) m i H 0 1 α = πα. m i H 0 Combining (4) and (5), we have E ( υ(α)) υ(α). Together, Theorems 1 and show that our roosed estimator υ(α) (for two-sided tests) is conservative so long as Pr( P > 1 ) = 0, which should be the case if π is sufficiently less than 1 and there is reasonable ower for each i H A. Otherwise, it is clear from (3) that E( π) 1, and thus the roosed υ(α) will not dramatically underestimate υ(α). The following theorem roves a useful relationshi between υ(α) and the ower of each test i H A. Theorem 3 Assume that E(P i )= 1 for all i H 0 and Pr ( P > 1 ) =0. Additionally, assume that there exists some i such that i H A. Then E( π) π as Pr(P i ɛ) 1 for any fixed ɛ>0 and each i H A. That is, the bias of π aroaches 0 as the owers of the tests of false null hyotheses aroach 1 for any fixed level. Proof: First note that the definition of π from the manuscrit imlies that ( E( π) =E P P 1 ) ( Pr P 1 ) ( +E 1 P > 1 ) ( Pr P > 1 ) = m E(P i ) m i=1 ( ) = E(P i )+ E(P i ) m i H 0 i H A = π + E(P i ). i H A If Pr(P i α) 0 for each i H A, then for each i H A it follows that E(P i ) 0. Thus, the second term in (6) aroaches 0. 3 (5) (6)

4 Therefore, under the conditions of Theorem 3, the bias of our roosed estimator υ(α) aroaches 0 as the statistical ower of each test i H 0 aroaches 1 for an arbitrarily small level. S1.3 Results for One-Sided Tests with Continuous -values Throughout this section, we assume that each test i is one-sided and yields a continuously distributed -value. First, we note that the definition of A i imlies that ( x ) ( G i (x) =F i +1 F i 1 x ). (7) Additionally, using integration by arts, we note that E(P i )= 1 0 xf i (x)dx =1 1 0 F i (x)dx (8) for all i. Clearly, an analogous relationshi exists between E(A i ) and G i (x) for all i. Therefore, (7) imlies that 1 ( E(A i )= [F i x + 1 ) ] F i (x) dx. (9) 0 These observations allow us to rove the following result. ( Theorem 4 For each i H 0, assume that F i (x) F 1 ) i x for 0 <x 1 ( and F i(x) F 1 ) i x for 1 x 1. Then υ(α) E(Ā)α for each fixed α 1. Proof: Using (9) and the assumtions, we have that 1 E(A i )= ( 1 =4F i ) [ ( x ) ( F i +1 F i 1 x )] dx [ ( )( 1 F i x + 1 ) ( ) ] 1 F i x dx ) dx (10) = F i ( 1 4

5 for each i H 0. Therefore, for α 1,wehave E(Ā)α = 1 (E(A i )α)+ (E(A i )α) m m i H 0 i H A 1 ( ) 1 F i α m i H 0 1 F i (α) m i H 0 = υ(α). (11) Therefore, using Theorem 4 and arguments analogous to those of section S1.1, we can rove bias and ower roerties analogous to those shown in section S1.. S1.4 Results for One-Sided Tests with Discrete -values Throughout this section, we assume that all tests are one-sided and yield discretely distributed -values. Let u = {u 1,...,u d } reresent the set of d uniue ossible -values. Let λ be the largest member of u that is less than or eual to 1. Now, we rove the following result. Theorem 5 For each i H 0, assume that F i (α) F i(λ) α for all α 1 and that F λ i(α) F i ( (λ) λ α 1 ) for all α 1 λ. Then 8E(Ā)α υ(α) for all α 1. 5

6 Proof: By (9) we have that 1 E(A i )= 0 { λ = 0 { λ 0 { λ 0 [F i ( x + 1 ) ] F i (x) dx ( [F i x + 1 ) ] 1 ( F i (x) dx + [F i x + 1 ) ] F i (x) dx λ [ ] } F i (λ) F i(λ) λ [ ( ) 1 F i λ { λ = F i (λ)dx + 0 F i(λ) 1 { = λf i (λ)+ F i(λ) λ [ ( 1 = F i (λ) λ + λ) λ λ ] 1 [ x dx + F i (x + 1 λ ) F i(x) 1 [ Fi (λ) ] x dx + F i (λ) λ xdx ) ( 1 4 λ ]. λ λ 0 λ F i(λ) λ λ dx ] } x F i(λ) dx. F i (λ) λ xdx ( 1 λ 1 λ ) F i (λ)dx } F i (λ) Therefore, E(A i ) λ + ( F i (λ) (13) 1 λ) λ and thus E(A i) λ + ( α 1 λ) λ F i(λ) α λ F i(α) (14) λ for all α 1. The denominator of the left-most term of (14) achieves a minimum value of 1 8 at λ = 1. Therefore, for each i H 4 0, F i (α) 8E(A i ) for all α 1. Using Theorem 5 and arguments analogous to those of section S1., we can rove bias and ower roerties analogous to those of section S1.. Additionally, we note that (14) suggests ower can be imroved without comromising conservativeness, if λ is known. Therefore, in ractice, one can let λ be the largest observed -value that is less than or eual to 1. Then by substituting λ into (14), one can obtain the constant k( λ) = 1 λ + ( 1 λ) 6 λ } } (1) (15)

7 so that E(k( λ))α υ(α). By definition, the largest -value less than or eual to 1 must be less than or eual to λ. Because (15) has a uniue minimum at λ = 1, one can use (15) to 4 imrove ower so long as λ

8 S Figures for Examles In this section, we dislay figures for the two examle alications: the randomization test of Gadburdy et al. (003) and the robe set filtering of Pounds and Cheng (005a). Scaled Mass St0 Proosed Local FDR Estimate St0 Proosed value -value Figure S.1: Results for the -values reorted by Gadbury et al. (003). The left anel shows the -value df estimates obtained by,, and against a roerly scaled histogram of the -values. The values of π comuted or used by, St0, and the roosed method are shown by oints to the right of the histogram. The right anel shows the values of the lfdr estimates t as a function of the -value cut-off. 8

9 Density /St0 Proosed Local FDR Estimate /St0 Proosed value -value Figure S.: Results for the ooled -values used for robe-set filtering by Pounds and Cheng (005). The left anel shows the -value df estimates obtained by and against a histogram of the -values. The left anel also shows the values of π comuted or used by, St0, and the roosed method are shown as oints above =0.50. The right anel shows t(α) from each method. In this examle, = t for all and all methods. 9

10 S3 Simulation of -values for One-sided Tests In this section, we examine the distribution of the -values one-sided Wilcoxon rank-sum tests (H 0 : δ = 0 vs. H A : δ>0) comuted from resamling-based simulation from the data set with π =0.5 and δ =1.0. We consider samle sizes of n =3, 5, 10. Figure S.3 shows the results. We now elaborate on how these lots show the difficulties that FDR methods can encounter when alied to -values from one-sided tests and how our roosed method overcomes those difficulties. Figure S.3 clearly shows that the -values behave as described in section.4 of the manuscrit. The the distribution of -values from tests with true nulls roughly follows the line y = x, which would corresond to uniformity. The -values from tests with true tested alternatives (i.e. δ > 0) are stochastically less than uniform (distribution is above the line y = x). The -values from tests with true untested alternatives (i.e. δ < 0) are stochastically greater than uniform (distribution is below the line y = x). The grahs also show the roblems that one-sided testing can introduce in FDR estimation and/or control. For δ<0, we observe that it is often the case that F(α) increases at a rate faster than α for some α> 1 α. This causes to decrease for some large α. In F(α) α some cases, such as in Pounds and Cheng (005a), it is ossible that can be smaller for F(α) α 1 than for α 0. Such a case clearly introduces downward bias into euation 3 of the manuscrit. In all anels of Figure S.3 with δ =0orδ<0, the simulation estimate of the average -value CDF falls below the diagonal line for all α< 1. This suggests that the ratio πα F(α) is conservative for all α< 1. The roosed method counts all -values greater than or eual to 1 as if they come from tests with a true null hyothesis; therefore, the roosed method does not lose this conservativeness for α 1. 10

11 delta = 0 delta < 0 delta > 0 delta = 0 delta < 0 delta > 0 delta = 0 delta < 0 delta > 0 Figure S.3: Null Distribution of P-values from Resamling-Based Simulation of the Data Set with π =0.5 and δ = 1. In each anel above, we lot the -value F(α) distribution (averaged across res) against α for the subsets of tests. The anels on the first, middle, and bottom row deict results for samle sizes n =3,n = 5, and n = 10, resectively. The left, middle, and right columns of anels corresond to -values from tests with δ = 0, δ < 0, and δ>0, resectively. In this examle, all -values were comuted for a test of the form H 0 : δ = 0 vs. H 0 : δ>0. 11

12 S4 Simulation Results In this section, we dislay lots for all simulation results. As mentioned in the manuscrit, we erformed resamling-based simulations for all combinations of three samle sizes (n = 3, 5, 10), number of sides of the tests (one-sided or two-sided) for each of five data sets. Therefore, a total of 30 simulations were erformed. We show the results for each of those simulations below. Each figure includes 6 anels, each lotting the simulation results for a method resented in the manuscrit: Benjamini and Hochberg (1995), ; Storey (00), St0; Pounds and Morris (003), ; Pounds and Cheng (004), ; and Cheng et al. (004),. In each lot, the thick solid line reresents the simulation estimate of the FDR incurred by using the -value threshold; the thin solid line reresents the average value of at for the indicated method; and the dashed lines reresent the value of at for each relication. In all simulations, the rank-sum test was used to comute -values. Therefore, the cations of the figures only indicate the sidedness of the tests (one-sided or two-sided), the er-grou samle size (n =3, 5, or 10), the roortion of robe sets that are not differentially exressed (π =0.5, 0.9, or 1.0) in the data set, and the effect size of differentially exressed robe sets relative to the standard deviation (δ = 0.5 or 1.0) in data sets with π

13 Proosed St0 Figure S.4: Simulation Results with One-sided Tests, n = 3, and π =

14 Proosed St0 Figure S.5: Simulation Results with One-sided Tests, n = 5, and π =

15 Proosed St0 Figure S.6: Simulation Results with One-sided Tests, n = 10, and π =

16 Proosed St0 Figure S.7: Simulation Results with Two-sided Tests, n = 3, and π =1.0 16

17 Proosed St0 Figure S.8: Simulation Results with Two-sided Tests, n = 5, and π =

18 Proosed St0 Figure S.9: Simulation Results with Two-sided Tests, n = 10, and π =

19 Proosed St0 Figure S.10: Simulation Results with One-sided Tests, n =3,π =0.9 and δ =

20 Proosed St0 Figure S.11: Simulation Results with One-sided Tests, n =5,π =0.9, and δ =0.5. 0

21 Proosed St0 Figure S.1: Simulation Results with One-sided Tests, n = 10, π =0.9, and δ =0.5. 1

22 Proosed St0 Figure S.13: Simulation Results with Two-sided Tests, n =3,π =0.9, and δ =0.5.

23 Proosed St0 Figure S.14: Simulation Results with Two-sided Tests, n =5,π =0.9, and δ =0.5. 3

24 Proosed St0 Figure S.15: Simulation Results with Two-sided Tests, n = 10, π =0.9, and δ =0.5. 4

25 Proosed St0 Figure S.16: Simulation Results with One-sided Tests, n =3,π =0.9, and δ =1.0. 5

26 Proosed St0 Figure S.17: Simulation Results with One-Sided Tests, n = 5, π = 0.9, and δ =

27 Proosed St0 Figure S.18: Simulation Results with One-Sided Tests, n = 10, π =0.9, and δ =1.0. 7

28 Proosed St0 Figure S.19: Simulation Results with Two-Sided Tests, n =3,π =0.9, and δ =1.0. 8

29 Proosed St0 Figure S.0: Simulation Results with Two-Sided Tests, n =5,π =0.9, and δ =1.0. 9

30 Proosed St0 Figure S.1: Simulation Results with Two-Sided Tests, n = 10, π =0.9, and δ =

31 Proosed St0 Figure S.: Simulation Results with One-Sided Tests, n = 3, π = 0.5, and δ =

32 Proosed St0 Figure S.3: Simulation Results with One-Sided Tests, n = 5, π = 0.5, and δ =

33 Proosed St0 Figure S.4: Simulation Results with One-Sided Tests, n = 10, π =0.5, and δ =

34 Proosed St0 Figure S.5: Simulation Results with Two-Sided Tests, n =3,π =0.5, and δ =

35 Proosed St0 Figure S.6: Simulation Results with Two-Sided Tests, n =5,π =0.5, and δ =

36 Proosed St0 Figure S.7: Simulation Results with Two-Sided Tests, n = 10, π =0.5, and δ =

37 Proosed St0 Figure S.8: Simulation Results with One-Sided Tests, n = 3, π = 0.5, and δ =

38 Proosed St0 Figure S.9: Simulation Results with One-Sided Tests, n = 5, π = 0.5, and δ =

39 Proosed St0 Figure S.30: Simulation Results with One-Sided Tests, n = 10, π =0.5, and δ =

40 Proosed St0 Figure S.31: Simulation Results with Two-Sided Tests, n =3,π =0.5, and δ =

41 Proosed St0 Figure S.3: Simulation Results with Two-Sided Tests, n =5,π =0.5, and δ =

42 Proosed St0 Figure S.33: Simulation Results with Two-Sided Tests, n = 10, π =0.5, and δ =1.0. 4