Resampling-Based Control of the FDR

Transcription

1 Resampling-Based Control of the FDR Joseph P. Romano 1 Azeem S. Shaikh 2 and Michael Wolf 3 1 Departments of Economics and Statistics Stanford University 2 Department of Economics University of Chicago 3 Department of Economics University of Zurich

2 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

3 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

4 General Set-Up & Notation Data X =(X 1,...,X n ) from distribution P Interest in parameter vector θ(p)=θ =(θ 1,...,θ s ) The individual hypotheses concern the elements θ i, for i=1,...,s, and can be (all) one-sided or (all) two-sided One-sided hypotheses: Two-sided hypotheses: H i : θ i θ 0,i vs. H i : θ i > θ 0,i H i : θ i = θ 0,i vs. H i : θ i θ 0,i

5 General Set-Up & Notation Data X =(X 1,...,X n ) from distribution P Interest in parameter vector θ(p)=θ =(θ 1,...,θ s ) The individual hypotheses concern the elements θ i, for i=1,...,s, and can be (all) one-sided or (all) two-sided One-sided hypotheses: Two-sided hypotheses: H i : θ i θ 0,i vs. H i : θ i > θ 0,i H i : θ i = θ 0,i vs. H i : θ i θ 0,i Test statistic T n,i =(ˆθ n,i θ 0,i )/ ˆσ n,i or T n,i = ˆθ n,i θ 0,i / ˆσ n,i ˆσ n,i is a standard error for ˆθ n,i or ˆσ n,i 1/ n ˆp n,i is an individual p-value

6 The False Discovery Rate Consider s individual tests H i vs. H i. False discovery proportion F = # false rejections; R = # total rejections False discovery rate FDR P = E P (FDP) FDP= F R 1{R>0}= F max{r, 1} Goal: (strong) asymptotic control of the FDR at level α: lim supfdr P α n for all P

7 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

8 Benjamini and Hochberg (1995) Stepup method: Ordered p-values: ˆp n,(1) ˆp n,(2)... ˆp n,(s) Let j = max { j : ˆp n,(j) α j }, where αj = jα/s Reject H (1),...,H (j ) Comments: Original proof assumes independence of the p-values Validity has been extended to certain dependence types (Benjamini and Yekutieli, 2001)

9 Modifications of BH (1995) Storey, Taylor and Siegmund (2004): Under sufficient conditions for BH (1995): FDR P s 0 s α where s 0 = I(P) =#{true hypotheses} Instead of α j = jα/s use α j = jα/ŝ 0 with ŝ 0 = #{ˆp n,i > λ}+1 1 λ for some 0<λ < 1 Proof assumes the ˆp n,i to be almost independent

10 Modifications of BH (1995) Bejamini, Krieger and Yekutieli (2006): Step 1: Apply the BH (1995) procedure at nominal level α = α/(1+α) Let r denote the number of rejected hypotheses (a) If r=0, reject nothing, and stop (b) If r=s, reject everything, and stop (c) Otherwise, continue Step 2: Apply the BH (1995) procedure at nominal level α Instead of α j = jα/s use α j = jα/ŝ 0 with ŝ 0 = s r Proof assumes independence of the ˆp n,i, but simulations show robustness against various dependence structures

11 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

12 Basic Idea (Troendle, 2000) For any stepdown procedure with critical values c 1,...,c s : [ ] F 1 FDR P = E P = max{r, 1} 1 r s r E P[F R=r]P{R=r} with P{R=r}=P{T n,(s) c s,...,t n,(s r+1) c s r+1,t n,(s r) < c s r }

13 Basic Idea (Troendle, 2000) For any stepdown procedure with critical values c 1,...,c s : [ ] F 1 FDR P = E P = max{r, 1} 1 r s r E P[F R=r]P{R=r} with P{R=r}=P{T n,(s) c s,...,t n,(s r+1) c s r+1,t n,(s r) < c s r } If all false hypotheses are rejected with p. 1, then with p. 1: FDR P = s s 0 +1 r s r s+s 0 r P{T n,s0 :s 0 c s0,...,t n,s r+1:s0 c s r+1,t n,s r:s0 < c s r } (1) Here T n,r:t is the rth largest of the test statistics T n,1,...,t n,t, and we assume w.l.o.g. that I(P)={1,...,s 0 }.

14 Basic Idea (continued) Goal: Bound (1) above by α for any P, at least asymptotically In particular, this must be ensured for any 1 s 0 s. First, consider any P such that s 0 = 1: Then (1) reduces to 1 s P{T n,1:1 c 1 } And so c 1 = inf{x R: 1 s P{T n,1:1 x} α}

15 Basic Idea (continued) Goal: Bound (1) above by α for any P, at least asymptotically In particular, this must be ensured for any 1 s 0 s. First, consider any P such that s 0 = 1: Then (1) reduces to 1 s P{T n,1:1 c 1 } And so c 1 = inf{x R: 1 s P{T n,1:1 x} α} Next, consider any P such that s 0 = 2. Then (1) reduces to: 1 s 1 P{T n,2:2 c 2,T n,1:2 < c 1 }+ 2 s P{T n,2:2 c 2,T n,1:2 c 1 } And so c 2 is the smallest x R for which 1 s 1 P{T n,2:2 x,t n,1:2 < c 1 }+ 2 s P{T n,2:2 x,t n,1:2 c 1 } α

16 Basic Idea (continued) Goal: Bound (1) above by α for any P, at least asymptotically In particular, this must be ensured for any 1 s 0 s. First, consider any P such that s 0 = 1: Then (1) reduces to 1 s P{T n,1:1 c 1 } And so c 1 = inf{x R: 1 s P{T n,1:1 x} α} Next, consider any P such that s 0 = 2. Then (1) reduces to: 1 s 1 P{T n,2:2 c 2,T n,1:2 < c 1 }+ 2 s P{T n,2:2 c 2,T n,1:2 c 1 } And so c 2 is the smallest x R for which 1 s 1 P{T n,2:2 x,t n,1:2 < c 1 }+ 2 s P{T n,2:2 x,t n,1:2 c 1 } α And so forth...

17 Estimation of the c i Since P is unknown, so are the ideal critical values c i. We sugggest a bootstrap method to estimate the c i : ˆP n is an unrestricted estimate of P with θ i ( ˆP n )= ˆθ n,i X is generated from ˆP n and the T n,i are computed from X, but centered at ˆθ n,i rather than at θ 0,i E.g., for one-sided testing: T n,i =(ˆθ n,i ˆθ n,i )/ ˆσ n,i

18 Estimation of the c i Since P is unknown, so are the ideal critical values c i. We sugggest a bootstrap method to estimate the c i : ˆP n is an unrestricted estimate of P with θ i ( ˆP n )= ˆθ n,i X is generated from ˆP n and the T n,i are computed from X, but centered at ˆθ n,i rather than at θ 0,i E.g., for one-sided testing: T n,i =(ˆθ n,i ˆθ n,i )/ ˆσ n,i Important detail: The ordering of the original T n,i determines the true null hypotheses in the bootstrap world The permutation{k 1,...,k s } of {1,...,s} is defined such that T n,k1 = T n,(1),...,t n,ks = T n,(s) Then T n,r:t is the rth smallest of the statistics T n,k 1,...,T n,k t

19 Estimation of the c i (continued) Start with c 1 : ĉ 1 = inf{x R: 1 s ˆP n {T n,1:1 x} α}

20 Estimation of the c i (continued) Start with c 1 : ĉ 1 = inf{x R: 1 s ˆP n {T n,1:1 x} α} Then move on to c 2 : ĉ 2 is the smallest x R for which 1 s 1 ˆP n {T n,2:2 x,t n,1:2 < ĉ 1 }+ 2 s ˆP n {T n,2:2 x,t n,1:2 ĉ 1 } α

21 Estimation of the c i (continued) Start with c 1 : ĉ 1 = inf{x R: 1 s ˆP n {T n,1:1 x} α} Then move on to c 2 : ĉ 2 is the smallest x R for which 1 s 1 ˆP n {T n,2:2 x,t n,1:2 < ĉ 1 }+ 2 s ˆP n {T n,2:2 x,t n,1:2 ĉ 1 } α And so forth... Unlike Troendle (2000), monotonicity ĉ i+1 ĉ i is not enforced.

22 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

23 Some Theory Assumptions (1) The sampling distribution of n( ˆθ n θ) under P converges to a limit distribution with continuous marginals (2) The bootstrap consistently estimates this limit distribution (3) n ˆσ n,i and n ˆσ n,i converge to the same constant in probability (for i=1,...,s) (4) The limiting joint distribution corresponding to the true test statistics is exchangeable

24 Some Theory Assumptions (1) The sampling distribution of n( ˆθ n θ) under P converges to a limit distribution with continuous marginals (2) The bootstrap consistently estimates this limit distribution (3) n ˆσ n,i and n ˆσ n,i converge to the same constant in probability (for i=1,...,s) (4) The limiting joint distribution corresponding to the true test statistics is exchangeable Theorem (i) Any false H i will be rejected with p. 1 as n (ii) The method asymptotically controls the FDR at level α

25 Some Practice Assumption (4) is somewhat restrictive (though less restrictive than an assumption of independence) But simulations indicate that the method appears robust to different limiting variances of the true test statistics different limiting correlations of the true test statistics

26 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

27 Set-Up I Data generating process and testing problem: I.i.d. random vectors from N(θ,Σ), with n=100 θ i = 0 or θ i = 0.2, with s=50 Σ has constant correlation ρ H i : θ i 0 vs. H i : θ i > 0 T n,i is the usual t-statistic Methods considered: (BH) Benjamini and Hochberg (1995) (STS) Storey et al. (2004) with λ = 0.5 (BKY) Benjamini et al. (2006) (Boot) Bootstrap method Criteria: Empirical FDR (nominal level α = 10%) Average number of true rejections

28 Results I ρ = 0 ρ = 0.9 BH STS BKY Boot BH STS BKY Boot All θ i = 0 Control Rejected Ten θ i = 0.2 Control Rejected Twenty five θ i = 0.2 Control Rejected All θ i = 0.2 Control Rejected

29 Set-Up II Data generating process and testing problem: I.i.d. random vectors from N(θ,Σ), with n=100 Three θ i = 0 and one θ i = 0.2 or 20, with s=4 Σ is a random correlation matrix: take 1,000 draws H i : θ i 0 vs. H i : θ i > 0 T n,i is the usual t-statistic Methods considered: (BH) Benjamini and Hochberg (1995) (STS) Storey et al. (2004) with λ = 0.5 (BKY) Benjamini et al. (2006) (Boot) Bootstrap method Criteria: Boxplot of empirical FDRs (nominal level α = 10%)

30 Results II Realized FDRs: one theta = BH STS BKY Boot Realized FDRs: one theta = BH STS BKY Boot

31 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

32 Conclusions Methodology: Bootstrap method implicitly accounts for the dependence structure of the test statistics Extended the approach of Troendle (2000) to non-normal data

33 Conclusions Methodology: Bootstrap method implicitly accounts for the dependence structure of the test statistics Extended the approach of Troendle (2000) to non-normal data Advantages: Appears more powerful than current competitors At least compared to those, that are also robust against various dependence structures

34 Conclusions Methodology: Bootstrap method implicitly accounts for the dependence structure of the test statistics Extended the approach of Troendle (2000) to non-normal data Advantages: Appears more powerful than current competitors At least compared to those, that are also robust against various dependence structures Disadvantage: Computationally more expensive than methods based on the individual p-values Should be considered negligible this day and age

35 Outline 1 Problem Formulation 2 Existing Methods 3 New Method 4 Theory & Practice 5 Simulations 6 Conclusions 7 References

36 References Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1): Benjamini, Y., Krieger, A. M., and Yekutieli, D. (2006). Adaptive linear step-up procedures that control the false discovery rate. Biometrika, 93(3): Benjamini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4): Storey, J. D., Taylor, J. E., and Siegmund, D. (2004). Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society, Series B, 66(1): Troendle, J. F. (2000). Stepwise normal theory test procedures controlling the false discovery rate. Journal of Statistical Planning and Inference, 84(1):