Sankhyā : The Indian Journal of Statistics 2008, Volume 70-A, Part 2, pp. 135-168 c 2008, Indian Statistical Institute On Methods Controlling the False Discovery Rate 1 Sanat K. Sarkar Temple University, Philadelphia, PA 19122, USA Abstract A great upsurge in research has taken place during the last decade or so in the area of multiple testing due to its increased relevance in modern scientific investigations. A major portion of this research has been devoted to developing methods that control error rates more meaningful as well as powerful than traditional ones in the context of these investigations. The error rates measuring false discovery proportion, particulary the false discovery rate FDR), and procedures controlling them have received the most attention. This article relates specifically to the FDR and is written with two basic objectives: i) To put together some important developments in theory and methodology of the FDR with finite number of hypotheses and report them with newer formulas, proofs and insights, and ii) to develop an error rate extending the FDR to one that directly incorporates dependence among the test statistics and methods that control it. AMS 2000) subject classification. Primary 62H35; Secondary 62G20, 62F40. Keywords and phrases. Adpative BH methods, BH method, BY method, false non-discovery rate, pair-wise FDR, positive dependence, multivariate totally positive of order two, stepwise multiple testing methods 1 Introduction In the past decade, there has been renewed interest in multiple testing due to its increased relevance as a statistical tool to analyze data from modern scientific investigations, such as genomics and brain imaging. As these experiments typically involve a large number of null hypotheses to test, making some of the more traditional multiple testing methods, like those designed to control the familywise error rate FWER), not as powerful as one would like, the research in modern multiple testing has been focused 1 The research is supported by the NSF grant DMS-0603868.
136 Sanat K. Sarkar to a large extent on developing more meaningful and powerful notions of error rate than the FWER and methods that control them. The false discovery rate FDR) introduced by Benjamini and Hochberg 1995) is one of these error rates that has been the most popular so far. Consider testing n null hypotheses H 1,...,H n simultaneously against certain alternatives using their respective p-values P 1,...,P n. A multiple testing of these hypotheses is typically carried out using a stepwise or singlestep method. Let P 1) P n) be the ordered versions of these p-values, with H 1),...,H n) being their corresponding null hypotheses. Then, given a non-decreasing set of critical constants 0 < t 1 t n < 1, a stepup method rejects the set {H i), i i SU } and accepts the rest, where i SU = max{1 i n : P i) t i }, if the maximum exists, otherwise accepts all the null hypotheses. A stepdown method, on the other hand, rejects the set of null hypotheses {H i), i i SD } and accepts the rest, where i SD = max{1 i n : P j) t j j i}, if the maximum exists, otherwise accepts all the null hypotheses. A stepup-stepdown method of order k, which generalizes both stepup and stepdown methods, rejects the set {H i), i i SUD } and accepts the rest, where i SUD = max{k i n : P j) t j j i} if P k) t k, and = max{1 i < k : P i) t i } if P k) t k, provided the maximum exists when P k) t k, otherwise, accepts all the null hypotheses. A stepup-stepdown method reduces to a stepdown method when k = 1 and a stepup method when k = n. When the constants are same in any of these stepwise procedures, it reduces to what is defined as a single-step method. We will, however, concentrate mainly on stepup or stepdown methods in this article as they are most commonly considered in the literature. Readers are referred to Hochberg and Tamhane 1987), Tamhane, Liu and Dunnett 1998) and Sarkar 2002) for descriptions of theses methods. Let R n denote the total number of rejections and V n denote the number of those that are false, the type I errors, while testing n null hypotheses using a multiple testing method. Then, the FDR of this method is defined by FDR n = E FDP n ), where FDP n = V n R n 1 1.1) is the false discovery proportion. Several methods controlling the FDR have been proposed in the literature. Among these, the methods of Benjamini and Hochberg 1995), the BH method, and of Benjamini and Yekutieli 2001), the BY method, have received the most attention. The BH method is the stepup method with the critical values t i = iα/n providing a control of the
Controlling the False discovery rate 137 FDR at α under independence or positive dependence of the p-values. More specifically, with the number of true null hypotheses n 0, its FDR is n 0 α/n under independence and less than or equal to n 0 α/n under the type of positive dependence for which the conditional distribution of the p-values given any null p-value is stochastically increasing with respect to the conditioning p-value. The BY method is the stepup method with the critical values t i = iα/n n j=1 1 j controlling the FDR at α under any form of dependence of the p-values. As noted in Sarkar 2002, 2008b), the stepup-stepdown method of order 1 k < n), in particular the stepdown method, with the same critical values as those of the BH method also controls the FDR at n 0 α/n, of course conservatively, under independence or the same kind of positive dependence of the p-values, and there is an alternative to the BY method that also controls the FDR under any form of dependence of the p-values. Storey 2002) advocated controlling the FDR through a conservatively biased point estimate of it given a rejection region and made a connection between his approach and the BH method of controlling the FDR considering a mixture model with independent p-values. This opened up the door for developing methods other than the BH method. A variety of such methods have been developed [Benjamini, Krieger and Yekutieli 2006), Blanchard and Roquain 2008), Gavrilov, Benjamini and Sarkar 2009), Storey, Taylor and Siegmund 2004) and Sarkar 2008b)]. These are generally referred to as adaptive BH methods, being devised to improve the FDR control of the BH method by incorporating into it an estimate of n 0 based on the same set of p-values, and have been shown to control the FDR only under the independence of the p-values, at least in the setting of finite number of hypotheses. We revisit the aforementioned FDR methods in this paper and offer alternative proofs of their FDR controls under various dependence situations, often covering both stepup and stepdown methods in a unified way, based on formulas we present in this article for the FDR of stepup and stepdown methods that are different from those available in the literature, for instance, in Benjamini and Yekutieli 2001) and Sarkar 2002, 2006). Our proofs offer newer, interesting insights. For instance, our proofs of the FDR controls of some of the adaptive BH procedures reveal connections between their FDR and the FNR, the false non-discovery rate, of the same or related procedures. The FNR is an analog of the FDR defined in terms of the type II errors among all accepted null hypotheses and was introduced by Genovese and Wasserman 2002) and independently by Sarkar 2004) who calls it the false negatives rate). More formally, with A n and T n denoting the total number of accepted and falsely accepted null hypotheses, respectively, it is
138 Sanat K. Sarkar defined as FNR n = E FNP n ), with FNP n = T n A n 1 1.2) being the false non-discovery proportion. The aforementioned connections are being made using formulas for the FNR of stepup and stepdown methods we derive here analogously to the FDR formulas and, again, they are different from those given before in Sarkar 2004, 2006). Although the BH is the most preferred method of controlling the FDR due mainly to its ability to maintain its control over the FDR even when the p-values are dependent, albeit positively, a disturbing aspect of it is that the FDR control becomes conservative, resulting in loss of power, with increasing dependence among the p-values. Of course, one of the the main reasons behind it is that, like many other multiple testing methods, this method is constructed using the marginal, not the joint, distributions of the p-values. So, methods with improved FDR control have been developed utilizing dependence structure of the p-values [Yekutieli and Benjamini 1999), Troendle 2000), Dudoit and van der Laan 2008) and Romano, Shaikh and Wolf 2008)], but these require finding joint distributions of the p-values of all dimensions, either approximately using re-sampling based methods or exactly under known distributional settings, which often involve extensive computations. Moreover, the methods due to Troendle 2000) and Romano, Shaikh and Wolf 2008) are also asymptotic in the sense that the sample size corresponding to each p-value has to be really large. We propose a different approach in this article to improving the control of false discoveries offered by the BH method when there is dependence among the p-values. It is built upon the idea of extending the usual definition of the FDR to one that inherently takes into account dependence among the p-values in a certain way, thereby making it more robust in the sense of maintaining a tighter, conservative control over false discoveries, thus improving the power, under increasing dependence among the p-values. When the p-values are positively dependent, the chance is higher for a pair of hypotheses to be rejected at the same time than when these p-values are independent. Thus, a p-value might get rejected simply because it is pooled toward rejection by another one that has also been rejected. So, by controlling the false discoveries in pairs, one can directly take into account the pairwise dependence of the p-values. With that rationale, we consider controlling the following pairwise version of the FDR:
Controlling the False discovery rate 139 Definition 1.1 Pairwise False Discovery Rate). [ Pairwise- FDR = E V n V n 1) R n R n 1) 2 ]. 1.3) We develop a method of controlling the Pairwise-FDR by providing a natural generalization of the BH method in terms of bivariate distributions of the null p-values. With independent p-values, this method at level α 2 is approximately for large n) the squared-fdr of the BH method at level α, and thus, at the same level, it is more powerful than the BH method. With positively dependent p-values, as our numerical study indicates, this power improvement continues to hold as long as the dependence is moderately high. We organize the paper as follows. In the next section, we give our new formulas and related results for the FDR and FNR of both stepup in particular, single-step) and stepdown methods. Based on these formulas and a few supporting lemmas, we offer in Section 4 our alternative proofs of the FDR control of different methods that we present in Section 3. Section 5 is devoted to the development of our method controlling the Pairwise-FDR, along with the results of a numerical study investigating its power performance. The paper concludes with some final remarks made in Section 6 and an appendix where a required inequality related to a notion of positive dependence assumed in this paper is presented. 2 Formulas for the FDR and FNR of Stepup and Stepdown Methods For any multiple testing method, the FDR is the expectation of the following: FDP n = 1 r I H i is rejected, R n = r), 2.1) where I 0 is the set of indices of true null hypotheses. In this section, we first consider a stepup method and give an identity for its FDR providing an explicit, convenient representation of the FDR in terms of its critical values. For a stepdown method, we obtain an upper bound for the FDR with a similar representation in terms of its critical values. Then, we give analogous results for the FNR of stepup and stepdown methods starting with the the original formula for the FNR of any multiple testing method,
140 Sanat K. Sarkar which is the expectation of the following: FNP n = i I 1 a=1 1 a I H i is accepted, A n = a), 2.2) with I 1 being the set of indices of false null hypotheses. We will use the notation P i) 1) P i) n 1) components of the set {P 1,...,P n } \ {P i }. to denote the ordered Theorem 2.1. For a stepup method of testing the n null hypotheses H 1,...,H n using the critical constants t 1 t n, the FDR is given by FDR SU = I E P i t R i) SU,n 1 t 2,...,t n)+1 R i) SU,n 1 t 2,...,t n ) + 1 ) where R i) SU,n 1 t 2,...,t n ) is the number of rejections in testing the n 1 null hypotheses other than H i using a stepup method based on their p-values and the critical constants t 2 t n. Proof. Since for the steup method, {R n = r} { P r) t r,p r+1) > t r+1,...,p n) > t n } and H i is rejected given R n = r if and only if P i t r, we have from 2.1) FDP n = = 1 r I ) P i t r,p r) t r,p r+1) > t r+1,...,p n) > t n 1 ) r I P i t r,p i) r 1) t r,p i) r) > t r+1,...,p i) n 1) > t n = 1 ) r I P i t r,r i) n 1 = r 1 = I ) P i t i) R n 1 +1 R i) n 1 + 1, 2.3) with R i) n 1 R i) SU,n 1 t 2,...,t n ). The theorem then follows by taking expectation of 2.3).
Controlling the False discovery rate 141 Remark 2.1. The above theorem says that the FDR of a stepup method with critical values t 1 t n is same as that of the procedure in which each H i is rejected if and only if P i t i) R n 1 +1, with R i) n 1 being the number of rejections while testing the n 1 null hypotheses other than H i using the stepup method based on their p-values and the critical constants t 2 t n. Interestingly, as shown in the following theorem, a similar result holds for a stepdown method in that the FDR of a stepdown method with critical values t 1 t n is less than or equal to that of the procedure in which each H i is rejected if and only if P i t R i) n 1 +1, where R i) n 1 is the number of rejections while testing the n 1 null hypotheses other than H i using the stepdown method based on their p-values and the critical constants t 1 t n 1. Theorem 2.2. For a stepdown method of testing the n null hypotheses H 1,...,H n using the critical constants t 1 t n, the FDR satisfies the following inequality: FDR SD I E P i t R i) SD,n 1 t 1,...,t n 1 )+1 R i) SD,n 1 t 1,...,t n 1 ) + 1 ), where R i) SD,n 1 t 1,...,t n 1 ) is the number of rejections in testing the n 1 null hypotheses other than H i using s stepdown method based on their p- values and the critical constants t 1 t n 1. Proof. We follow the line of arguments used by Gavrilov, Benjamini and Sarkar 2009) with independent p-values for a somewhat related result. FDP n = = 1 r I ) P i t r,p 1) t 1,...,P r) t r,p r+1) > t r+1 1 r I t r 1 < P i t r,p 1) t 1,...,P r) t r ) 2.4)
142 Sanat K. Sarkar r=2 1 rr 1) I ) P i t r 1,P 1) t 1,...,P r) t r with t 0 = 0) 1 ) r I t r 1 < P i t r,p i) 1) t 1,...,P i) r 1) t r 1 = r=2 1 ) rr 1) I P i t r 1,P i) 1) t 1,...,P i) r 1) t r 1 1 ) r I P i t r,p i) 1) t 1,...,P i) r 1) t r 1,P i) r) > t r with P i) n) 1) = 1 ) r I P i t r,r i) n 1 = r 1 ) = I P i t i) R n 1 +1 R i) n 1 + 1, 2.5) with R i) following: n 1 R i) SD,n 1 t 1,...,t n 1 ). The inequality in 2.5) follows from the { P1) t 1,...,P n) t n,t j 1 < P i t j } { P i) 1) t 1, }...,P i) j 1) t j 1,P i) j) t j+1,...,p i) n 1) t n,t j 1 < P i t j { } P i) 1) t 1,...,P i) n 1) t n 1,t j 1 < P i t j, 2.6) j = 1,...,n, for any increasing set of constants t 0 = 0 t 1... t n 1. The theorem follows by taking expectation of 2.5). Results analogous to Theorems 2.1 and 2.2 can be obtained for the FNR.
Controlling the False discovery rate 143 Theorem 2.3. For a stepup method of testing the n null hypotheses H 1,...,H n using the critical constants t 1 t n, the FNR is FNR SU I E i I 1 P i > t R i) SU,n 1 t 2,...,t n)+1 n R i) SU,n 1 t 2,...,t n ) ). Proof. Let us express the FNR in 2.2) for the stepup test in terms of Q i = 1 P i, i = 1,...,n. With Q 1) Q n), the ordered versions of the Q i s, and t n i+1 = 1 t i, i = 1,...,n, we get with Q i) n) 1), FNP n = i I 1 a=1 1 a I ) Q i t a,q 1) t 1,...,Q a) t a,q a+1) > t a+1 1 ) a I Q i t a,q i) 1) t 1,...,Q i) a 1) t a 1,Q i) a) > t a i I 1 a=1 ) = I Q i t i) A n 1 +1 i I 1 A i) n 1 + 1, 2.7) by making similar arguments as used in deriving { the inequality in 2.5) while proving Theorem 2.2, where A i) n 1 =max 1 j n 1 : Q i) t j) j }. The theorem then follows by taking expectation of 2.7) and noting that A i) n 1 = n 1 R i) SU,n 1 t 2,...,t n ). Theorem 2.4. For a stepdown method of testing the n null hypotheses H 1,...,H n using the critical constants t 1 t n, the FNR is given by FNR SD = I E i I 1 P i > t R i) SD,n 1 t 1,...,t n 1 )+1 n R i) SD,n 1 t 1,...,t n 1 ) ).
144 Sanat K. Sarkar Proof. FNP n = n 1 1 { } Pi > t r+1,p n r 1) t 1,...,P r) t r,p r+1) > t r+1 i I 1 r=0 = n 1 1 { } P i > t r+1,p i) n r 1) t 1,...,P i) r) t r,p i) r+1) > t r+1 i I 1 r=0 = n 1 i I 1 = i I 1 I with R i) identity. 1 n r P i > t i) R n 1 +1 r=0 n 1 R i) n R i) n 1 { P i > t r+1,r i) n 1 = r } ), n 1,SD t 1,...,t n 1 ), the expectation of which is the desired By taking t i = t, i = 1,...,n, in Theorem 2.1, we get the following: Corollary 2.1. For a single-step method of testing the n null hypotheses H 1,...,H n using the critical constant t, the FDR is given by FDR = [ ] I P i t) E R i) n 1 t) + 1, 2.8) where R i) n 1 t) is the number of rejections in testing the n 1 null hypotheses other than H i using a single-step method based on their p-values and the critical constant t. The following corollary is obtained from Theorem 2.4. Corollary 2.2. For a single-step method of testing the n null hypotheses H 1,...,H n using the critical constant t, the FNR is given by FNR = [ ] I P i > t) E i I 1 n R i) n 1 t). 2.9) Remark 2.2. If the summation in Theorem 2.1 or Corollary 2.1) is extended over all i from 1 to n, it reduces to the probability of at least one
Controlling the False discovery rate 145 rejection, PrR n > 0), for the corresponding stepup or single-step) method. This can be seen easily from the original expression for the FDP n in 2.1) that equals IR n > 0) when the first summation is taken over all i from 1 to n. Similarly, the summation in Theorem 2.4 or Corollary 2.2) will reduce to the probability of at least one acceptance, Pr A n > 0), for the corresponding stepdown or single-step) method once it is extended over all i from 1 to n. Remark 2.3. Let us go back to the FNR s of the stepup and stpdown methods with the critical constants t 1... t n in Theorem 2.3 and 2.4. Since { Pi > t r+1,p r) t r,p r+1) > t r+1,...,p n) > t n } { } P i > t n,p r) t r,p r+1) > t r+1,...,p n) > t n { } P i > t n,p i) r) t r,p i) r+1) > t r+1,...,p i) n 1) > t n 1 [see 2.6)], we notice that FNR SU = PrA n > 0) n 1 1 n r Pr { P i > t r+1,p r) t r, r=0 P r+1) > t r+1,...p n) > t n } PrA n > 0) n 1 1 { n r Pr P i > t n,p i) r) t r, r=0 } P i) r+1) > t r+1,...,p i) n 1) > t n 1 1 E I P i > t n ), n R i) SU,n 1 t 1,...,t n 1 ) where R i) n 1 R i) n 1,SU t 1,...,t n 1 ). Similarly, from Theorem 2.4, we see that FNR SD = PrA n > 0) I E P i > t R i) SD,n 1 t 1,...,t n 1 )+1 n R i) SD,n 1 t 1,...,t n 1 ) ) 2.10) 1 E I P i > t n ) n R i) SD,n 1 t. 2.11) 1,...,t n 1 ) These will be useful later in Section 4.
146 Sanat K. Sarkar 3 Methods Controlling the FDR In this section, we consider different FDR controlling methods, some already proposed in the literature, but revisited with different proofs, and some new, under various dependence conditions on the p-values. We present here only the theorems stating the FDR controls of these methods and related discussions and give their proofs in the next section. 3.1.The BH method. Theorem 3.1. Let the p-values be independent with each null p-value being distributed as U0,1). Then, with the critical values t i = iα/n, i = 1,...,n, the FDR is exactly n 0 α/n for the stepup method and is less than or equal to n 0 α/n for the stepdown method. Remark 3.1. The above critical values are same as those in the Simes global test for testing the intersection of the null hypotheses [Simes 1986), Sarkar and Chang 1997), Sarkar 1998, 2008c)]. Benjamini and Hochebrg 1995) originally showed a conservative control of the FDR of the stepup method, the BH method, at n 0 α/n under independence, while the exact control was proved later in Finner and Roters 2001), Benjamini and Yekutieli 2001), Sarkar 2002) and Storey, Taylor and Siegmund 2004). The stepdown analog of the BH procedure was first considered in Sarkar 2002). The alternative proofs of these results presented in this paper in a unified way under independence based on the formulas in the above section appear quite straightforward compared to those known in the literature. If each null P i is assumed to be stochastically larger than U0,1), that is, PrP i u) u, u 0,1), in the above result, the FDR will continue to be less than or equal to n 0 α/n for the stepup or stepdown method. The following is the result on the FDR control of the BH method and its stepdown analogue under dependence. Theorem 3.2. The FDR of the stepup or stepdown method with the critical values t i = iα/n, i = 1,...,n, is less than or equal to n 0 α/n when the p-values are positively dependent in the following sense: E {ψ P 1,...,P n ) P i u} is non-decreasing in u for each i I 0, 3.1) for any coordinatewise) non-decreasing function ψ.
Controlling the False discovery rate 147 Remark 3.2. The above positive dependence condition is slightly more relaxed than the following condition used originally by Benjamini and Yekutieli 2001) to prove Theorem 3.2 for the steup method, the BH method: E {ψ P 1,...,P n ) P i = u} is non-decreasing in u for each i I 0, 3.2) for any coordinatewise) non-decreasing function ψ, which they called the PRDS positive regression dependence on subset) condition, although it was introduced before as the positive dependence through stochastic ordering PDS) condition by Block, Savits and Shaked 1985). Sarkar 2002) extended the proof of Benjamini and Yekutieli 2001) to a general stepup-stepdown method. Finner, Dickhaus and Roters 2008) noted that the weaker condition 3.1) than the original PRDS condition is sufficient for the BH method to control the FDR, while Sarkar and Guo 2009 b) used this condition in the context of a generalized FDR. Again, we present in this article alternative, unified proof of the FDR control of both the BH method and its stepdown analogue under positive dependence. Theorem 3.2 continues to hold if PrU u) u, u 0,1), for each null p-value. 3.2. Adaptive BH methods. When Benjamini and Hochberg 1995) introduced the notion of FDR and their method of controlling it, their approach was to find a rejection region based on the p-values with its FDR controlled at the desired, fixed level. Storey 2002) took a different approach. With the rejection region considered fixed, he proposed studying its FDR using a conservatively biased point estimate of it. There is a connection between these two approaches, as Storey 2002) pointed out, in the sense that they yield the same method when the rejection region is determined from the p- values through a certain estimate of the FDR for a single-step test developed under the following model: Definition 3.1 Mixture Model). With H i = 0 or 1 indicating that H i is true or false, respectively, let P i,h i ), i = 1,...,n, be independently and identically distributed with PrP i u H i ) = 1 H i )u + H i F 1 u), u 0,1), for some continuous cdf F 1 u), and PrH i = 0) = π 0 = 1 PrH i = 1). More specifically, for the following class of conservatively biased point estimates of the FDR when using a single-step test rejecting an H i if P i t, FDR n,λ t) = nˆπ 0λ)t, λ [0,1), 3.3) R n t) 1
148 Sanat K. Sarkar where ˆπ 0 λ) = n R nλ) n1 λ) and R n λ) = rejecting H 1),...,H ˆlλ)), with { } ˆlλ) = max 1 j n : FDR n,λ P j) ) α IP i λ), 3.4) i=1 3.5) providing the stepup rejection threshold, is equivalent to the BH method at level α when λ = 0. Remark 3.3. Notice that rejecting H 1),...,H ˆl λ)), with { } ˆl λ) = max 1 j n : FDR n,λ P i) ) α i = 1,...,j, 3.6) providing the stepdown rejection threshold, is equivalent to the stepdown version of the BH method considered in Sarkar 2002) when λ = 0. Storey s idea of thresholding the p-values through an estimate of the FDR, as in 3.5) or 3.6), opens up the possibility of developing methods other than the BH method or its stepdown analog that can potentially control the FDR. The following is a list of those considered in the literature, each corresponding to an estimated FDR, ˆn 0 t FDR n t) = R n t) 1, 3.7) of the type 3.3) for the single step test with a particular estimate ˆn 0 of n 0 = nπ 0. These are: 1. Storey, Taylor and Siegmund 2004)) The method using the stepup rejection threshold based on FDR n t) with for any fixed λ t 0,1). ˆn 0 = n R nλ) + 1 1 λ, 2. Benjamini, Krieger and Yekutieli 2006)) The method using the stepup rejection threshold based on FDR n t) with ˆn 0 = n k + 1 1 P k),
Controlling the False discovery rate 149 for any fixed 1 k n. 3. Benjamini, Krieger and Yekutieli 2006)) The method using the stepup rejection threshold based on FDR n,λ t) with ˆn 0 = n R SU,nλ 1,...,λ n ) + 1 1 λ n, where λ i = iλ n /n, i = 1,...,n, and λ n = α/1 + α). 4. Gavrilov, Benjamini and Sarkar 2009)) The method using the stepdown rejection threshold based on FDR n t) with ˆn 0 = n R nt) + 1 1 t. 5. Blanchard and Roquain 2008)) The method using the rejection threshold that is the minimum of λ and the stepup rejection threshold based on FDR n t) with ˆn 0 = n R nt) + 1 1 λ, for any fixed λ t 0,1). The above methods are all referred to as adaptive BH methods, designed to improve the FDR control of the BH method by suitably incorporating into it an estimate of n 0 obtained from the same set of p-values. Each of the first three of these methods is a stepup method with the critical values of the form t i = iα/ˆn 0 P), for some estimate ˆn 0 P) of n 0 based on P = P 1,...,P n ). The fourth one is a stedwon method with the critical values t i = iα/[n i1 α) + 1], i = 1,...,n, and the fifth one is again a stepup method with the critical values t i = min {iα1 λ)/n i + 1),λ}, i = 1,...,n, for a fixed λ 0,1). Of course, the first and third methods are slight variations of their original ones in Storey, Taylor and Siegmund 2004) and Benjamini, Krieger and
150 Sanat K. Sarkar Yekutieli 2006), respectively. While Storey, Taylor and Siegmund 2004) considered truncating above each t i at λ, Benjamini, Krieger and Yekutieli 2006) used the ˆn 0 without the +1 in the numerator and restricted the t i s to 0 or 1 according as R SU,n λ 1,...,λ n ) is 0 or n. Also, the fourth method is a special case of the multiple-stage BH method proposed in Benjamini, Krieger and Yekutieli 2006), without any proof of its FDR control, and has been referred to as a multiple-stage stepdown method there, though it is actually an adaptive stepdown analog of the BH method, as seen here and also discussed in Sarkar and Heller 2008). An asymptotic derivation of it is given in Finner, Dickhaus and Roters 2008b). Unfortunately, all these methods have been shown to control the FDR at α only under the independence of the p-values. In this article, we will consider several other FDR controlling adaptive methods for independent p-values. More specifically, we have the following theorem establishing the FDR control of a larger class of adaptive methods than the first three in the above list. Theorem 3.3. Consider a stepup or stepdown method with the critical values t i = iα/ˆn 0 P), i = 1,...,n, where ˆn 0 P) > 0 is non-decreasing in each P i. The FDR of this procedure is less than or equal to α when the p-values are independent with each null p-value being distributed as U0, 1) if E 1 ˆn i I0 0 P i),0 ) 1, 3.8) where ˆn 0 P i),0 ) represents ˆn 0 as a function of P i) = {P 1,...,P n }\{P i } when P i = 0. The following proposition presents a variety of choice for ˆn 0 P) in Theorem 3.3. Proposition 3.1. The condition 3.8) in Theorem 3.3 is satisfied by ˆn 0 P) = n R nλ 1,...,λ n ) + 1 1 λ n, 3.9) with R n λ 1,...,λ n ) denoting the number of rejections in a stepup or stepdown method for any fixed 0 λ 1 λ n 1.
Controlling the False discovery rate 151 Proof. Since ˆn 0 P i),0 ) = [n R i) n 1 λ 2,...,λ n )]/1 λ n ), we see from Remark 2.3 that E [ 1 λ n n R i) n 1 λ 2,...,λ n ) ] [ ] 1 FNRλ 2,...,λ n,λ n ) 1, 3.10) where FNRλ 2,...,λ n,λ n ) is the FNR of the stepup or stepdown method based on the critical values λ 2,...,λ n,λ n. Remark 3.4. Clearly, using an ˆn 0 P) of the type in Proposition 3.1, we now have a much larger class of adaptive BH or adaptive stepdown analogue of BH methods containing the first and third methods in the list. The ˆn 0 P) in the second method also satisfies the condition 3.8), which can be checked as follows using arguments simpler than that made in Benjamini, Krieger and Yekutieli 2006). [ ] Since for this ˆn 0, ˆn 0 P i),0) = n k + 1)/ 1 P i) k 1), we need to just show that 1 E k P i) k) ) 1, 3.11) for any fixed 1 k n 1. When 1 k n 0 1, P i) k) P k):n0 1, the kth ordered among the n 0 1 null p-values, for every i I 0, implying that 1 ) E P i) k k) n 0 k E ) n 0 k P k):n0 1 = 1, k n 0 that is, the inequality 3.11) holds. When n 0 k n 1, this inequality is obviously true since P i) k) 1 for every i I 0. The first and third adaptive methods continue to control the FDR if PrU u) u,u 0,1), for each null p-value. The following theorem covers the fourth and fifth methods in the above list. Theorem 3.4. Consider the stepdown method with the critical values t i s satisfying t i /1 t i ) iα/n i + 1), i = 1,...,n, or the stepup method with the critical values t i s satisfying t i = min {iα1 λ)/n i + 1),λ}, i = 1,...,n, for a fixed λ 0,1). They both have the FDR less than or equal to α when the p-values are independent with each null p-value being distributed as U0,1).
152 Sanat K. Sarkar Remark 3.5. The proof presented here for the FDR control of the stepdown method in Theorem 3.4, which is slightly different from its original proof given in Gavrilov, Benjamini and Sarkar 2009), reveals an interesting connection between its FDR and FNR. More specifically, it is proved that for this procedure FDR α1 FNR). Our proof for the stepup method in this theorem is also different from that given in Blanchard and Roquain 2008); it is presented by connecting the FDR to the FNR of a related method. Connecting the FDR of an adaptive method to the FNR of the same or a related method has been the main theme in our proofs for the FDR control of almost all of such methods considered in this article. The second method in Theorem 3.4 will continue to be a valid FDR controlling procedure if PrU u) u,u 0,1), for each null p-value. Remark 3.6. Before we conclude this subsection, we would like to make some remarks about the connection between the BH method and finding a stepup rejection threshold based on an estimated FDR of a single step test under the mixture model. We argue that this connection can be made much more directly in terms of the FDR of a stepup method, rather than the FDR of a single-step method. The FDR of the stepup method with the critical values t 1 t n under the mixture model, as seen from Theorem 2.1, is given by t i) R SU,n 1 FDR SU = π 0 E t 2,...,t n)+1 i=1 R i) SU,n 1 t, 3.12) 2,...,t n ) + 1 implying that it can be unbiasedly estimated with a known π 0 by π 0 i=1 t R i) SU,n 1 t 2,...,t n)+1 R i) SU,n 1 t 2,...,t n ) + 1. 3.13) When π 0 is unknown, a suitable estimate of it, ˆπ 0, can be plugged into this estimate. Similarly, for the stepdown method with same set critical values, the FDR can be estimated, but conservatively, using π 0 i=1 t R i) SD,n 1 t 1,...,t n 1 )+1 R i) SD,n 1 t 1,...,t n 1 ) + 1. 3.14) when π 0 is known, or using a modification of it with a ˆπ 0 plugged into it when this π 0 is unknown. With the following class of conservatively biased
Controlling the False discovery rate 153 estimates of the FDR of the stepup method: FDR SU,λ t 1,...,t n ) = ˆπ 0 λ) i=1 t i) R SU,n 1 t 2,...,t n)+1 R i) SU,n 1 t, λ [0,1), 3.15) 2,...,t n ) + 1 it is now quite clear that FDR SU,λ=0 t 1,...,t n ) = α when t i = iα/n, i = 1,...,n, that is, when we have the BH method. In other words, controlling the estimated FDR of a stepup method with λ = 0) is directly connected to controlling the FDR using the BH method. A similar statement can be made in terms of a stepdown method. More interestingly, for the single step test, our estimate of the FDR, ˆπ 0 λ)t n i=1 {1/R i) n 1 t) + 1)}, which as shown in Sarkar and Liu 2008) simplifies to n + 1)ˆπ 0 λ)t {R n t) 1 3.16) n } + 1, is actually better than Storey s original estimate 3.3) in the sense of being less conservative. 3.3. FDR methods under arbitrary dependence. Finally, we consider FDR controlling methods under arbitrary dependence of the p-values. The following theorem puts the method of Benjamini and Yekutieli 2001), the only one commonly preferred when no specific dependence assumption is made about the p-values, in the context of a more general result. Theorem 3.5. The FDR of the stepup or stepdown test with any set of critical constants 0 = t 0 t 1 t n satisfying n i=1 t i t i 1 )/i α/n is less than or equal to n 0 α/n. Remark 3.7. The following are two of several different stepup or stepdown procedures controlling the FDR under arbitrary dependence that can be determined from the above theorem i) t i = iα/n n j=1 1 j Benjamini and Yekutieli 2001)) and ii) t i = ii + 1)α/2n 2 Sarkar 2008b)). The inequality in Theorem 3.5, a proof of which will be presented in the next section, was noted first by Sarkar 2008b) and later independently by Guo and Rao 2008) for the stepup method. In fact, for the stepdwon method, Guo and Rao 2008) gave the following alternative, but stronger, result:
154 Sanat K. Sarkar Result. The FDR of the stepdown test with any set of critical constants 0 = t 0 t 1 t n satisfying [ { n n0 +1 max n 0 1 n 0 n i=1 t i t i 1 i + i=n n 0 +2 }] n n 0 )t i t i 1 ) α 3.17) ii 1) is less than or equal to α. 4 Proofs of the Theorems in Section 3 First, we have some useful lemmas. Lemma 4.1. Let U U0,1). Then, for any positive valued non-decreasing function g ) and constant t, E {gu)iugu) t)} t. The lemma is easy to prove. Lemma 4.2. Consider two random variables U and R, where U U0,1) and R is discrete defined on {1,...,n}. Then, i) given an increasing set of constants 0 = t0) t1) tn), [ ] IU tr)) E R tr) tr 1), r and ii) given an increasing set of constants t1) tn) such that tr)/r is non-increasing in r = 1,...,n, { } IU tr)) E t1), R if U and R are positively dependent in the sense that PrR r U u) is non-decreasing in u.
Proof. Controlling the False discovery rate 155 I U tr)) R 1 = IU tr),r = r) r = n 1 1 r IU tr),r r) [ IU tr)) IU tr)) + I R r) r r=2 [ ] Itr 1) U < tr)) I R r) r I tr 1) < U tr)). r 1 IU tr),r r + 1) r ] IU tr 1)) r 1 The first part of the lemma then follows by taking expectations of both sides. The second part is proved from the following two inequalities: { } I U tr)) tr) E = Pr {R = r U tr)} R r t1) Pr {R = r U tr)}, 4.1) and Pr {R = r U tr)} = Pr {R r U tr)} n 1 Pr {R r + 1 U tr)} Pr {R r U tr)} n 1 Pr {R r + 1 U tr + 1)} = 1. 4.2)
156 Sanat K. Sarkar Remark 4.1. Both Lemmas 4.1 and 4.2 ii) continue to hold if PrU u) u, u 0,1). Proof of Theorem 3.1. A proof is immediate once we see from Theorems 2.1 and 2.2 that given the critical values t 1 t n the FDR is equal for a stepup test and less than or equal for a stepdown test to the following: E [ t R i) n 1 t 2,...,t n)+1 R i) n 1 t 2,...,t n ) + 1 ]. 4.3) Proof of Theorem 3.2. Consider the random variables U and R in Lemma 4.2 to be respectively the P i and R i) SU,n 1 t 2,...,t n ) + 1 or R i) SD,n 1 t 1,...,t n 1 ) + 1) in the expression for the FDR in Theorem 2.1 or Theorem 2.2). Since R i) SU,n 1 t 2,...,t n ) or R i) SD,n 1 t 1,...,t n 1 ) is nonincreasing in each of the p-values on which they are defined, in either case, IR r) is a non-decreasing function of each of its arguments. So, under the assumed positive dependence condition, E {I R r) U u}, the conditional probability Pr R r U u), is a non-decreasing function of u; that is, the condition in Lemma 4.2 ii) holds. Thus, the FDR of the stepup or stepdown method with t i = iα/n satisfies the inequality FDR t 1 = n 0 α/n, which proves the theorem. Proof of Theorem 3.3. Our proof is somewhat along the line of Benjmaini, Krieger and Yekutieli 2006); see also Blanchard and Roquain 2008). Consider the expression for the FDR in Theorem 2.1 or 2.2. Notice that both t i, i = 1,...,n, and R i) SU,n 1 t 2,...,t n ) or R i) SU,n 1 t 1,...,t n 1 )) in those theorems are now dependent on P. Expressing them more explicitly in terms of the P i s, we see that the FDR of this procedure is less than or equal to [ I Pi φ P i) ),P i t1 P i) ))],P i φ ) P i), 4.4),P i E where φ P i) ) i),p i = R SU,n 1 t 2,...,t n ) + 1 or R i) SD,n 1 t 1,...,t n 1 ) + 1). The number of rejections in a stepwise method, as a function of the p-values and critical constants, is non-decreasing in each of its critical constants for fixed p-values and is non-increasing in each p-value for fixed critical constants. So, for fixed λ i s, each t i as well as φ are non-increasing in each
Controlling the False discovery rate 157 p-value when the other p-values are held fixed, which implies that we can let P i go to 0 in t 1 to obtain FDR E [ I Pi φ P i) ),P i t1 P i),0 )) ] φ ) P i). 4.5),P i Applying Lemma 4.1 to the conditional expectation given P i) and observing that t 1 P i),0 ) = α/ˆn 0 P i),0 ), we get FDR α [ ] 1 E ˆn P i),0 ) α. 4.6) 0 Thus, the theorem is proved. Proof of Theorem 3.4. Consider the stepdown method first. As the p-values are independent, we see from Theorems 2.2 and 2.4 and Remark 2.2 that the FDR of this method is less than or equal to α 1 t i) R SD,n 1 E t 1,...,t n 1 )+1 n R i) SD,n 1 t 1,...,t n 1 ) ) = α Pr P i > t i) R SD,n 1 E t 1,...,t n 1 )+1 n R i) SD,n 1 t 1,...,t n 1 ) = α PrA n > 0) Pr E i I 1 P i > t R i) SD,n 1 t 1,...,t n 1 )+1 n R i) SD,n 1 t 1,...,t n 1 ) ) α 1 FNR SD ), 4.7) proving the FDR control of the stepdown method.
158 Sanat K. Sarkar For the stepup method, we see from Theorem 2.1 that the FDR of this method is less than or equal to α E 1 λ n R i) SU,n 1 t 2,...,t n ) α E 1 t n n R i) SU,n 1 t 2,...,t n ) [ ] α 1 FNR SU t 2,...,t n,t n ), 4.8) where FNR SU t 2,...,t n,t n ) is the FNR of the stepup method with the critical values t 2,...,t n,t n ; see Remark 2.3. Thus, the FDR is controlled for the stepup method. Proof of Theorem 3.5. The result follows by considering the random variables U and R in Lemma 4.2 i) to be respectively the P i and R i) SU,n 1 t 2,...,t n ) + 1 or R i) SD,n 1 t 1,...,t n 1 ) + 1) in the expression for the FDR in Theorem 2.1 or Theorem 2.2), as in the proof of Theorem 3.2, and noting that this FDR is less than or equal to n n 0 i=1 t i t i 1 )/i. 5 The Pairwise-FDR We present in this section our results on the Pairwise-FDR. First we have the following theorem giving a general formula for the Pairwise-FDR of a stepup method in terms of its critical values: Theorem 5.1. For a stepup method of testing the n null hypotheses H 1,...,H n using the critical constants t 1 t n, the Pairwise-FDR is given by Pairwise-FDR = i j I 0 E ) P i,p j t i, j) R SU,n 2 +2 I ) ) R i, j) SU,n 2 + 1 R i, j) SU,n 2 + 2, 5.1) where R i, j) SU,n 2 R i, j) SU,n 2 t 3,...,t n ) is the number of rejections in testing the n 2 null hypotheses other than H i and H j using s stepup method based on their p-values and the critical constants t 3 t n.
Controlling the False discovery rate 159 Proof. The theorem can be proved in the same way as Theorem 2.1. The pairwise false discovery proportion is Pairwise-FDP V n V n 1) = R n R n 1) 2 = 1 rr 1) I H i,h j are rejected, R = r) i j I 0 r=2 = i j I 0 r=2 = i j, I 0 r=2 1 rr 1) I ) P i,p j t r,p r) t r,p r+1) > t r+1,...,p n) > t n 1 ) rr 1) I P i,p j t r,r i, j) SU,n 2 = r 2, 5.2) the expectation of which is the right-hand side in 5.1). 5.1. A stepup method of controlling the Pairwise-FDR. Based on the above formula, we now develop our stepup method of controlling the Pairwise- FDR at an appropriate level. To that end, we have the following: Theorem 5.2. Assume that the p-values are positively dependent in the following sense: for each i,j I 0 E {ψ P 1,...,P n ) P i u,p j v} is non-decreasing in u,v).5.3) for any coordinatewise) non-decreasing function ψ, and that the null p- values have a common pairwise joint distribution Hu,v) = Pr P i u,p j v), i,j I 0. 5.4) Then, for the stepup method with the critical values t i s satisfying Ht i,t i ) = ii 1)α nn 1), for i = 2,...,n, 5.5) the Pairwise-FDR is less than or equal to n 0 n 0 1)α/nn 1). Proof. From 5.2), we get
160 Sanat K. Sarkar Pairwise-FDR = i j I 0 r=2 = Ht 2,t 2 ) 2 Ht r,t r ) ) rr 1) Pr R i, j) SU,n 2 = r 2 max{p i,p j } t r i j I 0 r=2 ) Pr R i, j) SU,n 2 = r 2 max{p i,p j } t r n 0n 0 1)α. 5.6) nn 1) The last inequality in 5.6) follows from the fact that r=2 ) Pr R i, j) SU,n 2 = r 2 max{p i,p j } t r 1, for each i,j I 0, 5.7) which can be proved as in 4.2). Remark 5.1. Among the distributions of the p-values, or the test statistics from which they are generated, satisfying the conditions in the above theorem are those that satisfy the multivariate totally positive of order two MTP 2 ) condition of Karlin and Rinott 1980) and are exchangeable under null hypotheses. The MTP 2 property provides the required positive dependence condition 5.3), while the exchangeability under null hypotheses ensures a common bivariate distribution of the null p-values. For example, the p-values obtained from the multivariate normal with a common variance and a nonnegative common correlation, or, in general, certain mixtures of independent distributions, commonly considered in multiple testing, satisfy the conditions in the theorem. The definition of MTP 2 and several important related results have been recalled from Karlin and Rinott 1980) in Sarkar 2008a), based on which we show in the Appendix why the condition 5.3) follows from the MTP 2 property. Remark 5.2. Even though the first critical value t 1 is arbitrary in any stepup method controlling the pairwise-fdr, it would be counterintuitive to equate it to 0, which would allow accepting a null hypothesis with an extremely small p-value, or to 1, which would violate the monotonicity of the critical values. The best option is to set it equal to t 2. Thus, more specifically, the stepup method in Theorem 5.2 defined with the critical
values t i s satisfying: Controlling the False discovery rate 161 Ft i,t i ) = = 2α nn 1) ii 1)α nn 1) for i = 1 for i = 2,...,n, 5.8) is our proposed level α Pairwise-FDR method under the assumptions stated in the theorem. When the p-values are independent with each null P i distributed as U0,1), the Pairwise-FDR in Theorem 5.1 reduces to Pairwise-FDR = i j I 0 E t 2 R i, j) SU,n 2 +2 R i, j) SU,n 2 + 1 ) R i, j) SU,n 2 + 2 ) which is exactly equal to n 0 n 0 1)α/nn 1) with the t i s in 5.8), that is, with the t i s given by 2α t i = for i = 1, nn 1) ii 1)α = for i = 2,...,n. 5.9) nn 1) Notice that n 0 n 0 1)α/nn 1) is approximately for large n) the square of n 0 α/n, the FDR of the BH method at level α, indicating that between the Pairwise-FDR and BH methods, both at the same level, the Pairwise-FDR method is more powerful when the p-values are independent and expected to be so when the dependence among the p-values is moderately positive. To see it more clearly, we performed a numerical study comparing the average power, the expected proportion of false null hypotheses that are correctly rejected. This is discussed in the following sub-section. 5.2. A numerical study. We ran a simulation study in which we i) generated a sample of n = 1000 dependent normal random variates Nµ i,1), i = 1,...,1000, with a common correlation ρ, and by setting n 0 of the 1000 µ i s at 0 each and the rest n 1 ) at 2 each, applied both the BH FDR and the proposed Pairwise-FDR method in Theorem 5.2 for testing the null hypothesis µ i = 0 against the alternative µ i > 0 simultaneously for all i = 1,...,1000, with α = 0.05, ii) repeated step i) 1000 times to simulate,
162 Sanat K. Sarkar the average power of the BH and Pairwise-FDR methods. Figure 1 compares the simulated average powers for different values of ρ and n 1 the number of false null hypotheses). As seen from this figure, the Pairwise-FDR seems to be a more powerful approach to controlling the false discoveries than the FDR as long as there is a moderately high positive dependence among the p-values. 6 Concluding Remarks One of our goals in this paper is to provide an account of the development of some important results in the area of FDR with newer proofs and insights. Our proofs for the FDR control of some of the adaptive BH methods highlight the importance of FNR, despite the lack of attention it has received in the literature. Although our primary focus has been on construction of FDR controlling procedures, particulary those stemming from the BH method, we should point out that a number of other important results enriching the theory of FDR have been developed. Stepdown FDR controlling procedures other than those discussed here have been proposed in the literature, notable among these are the one in Benjamini and Liu 1999) proposed for the first time under independence, later extended to positive dependence by Sarkar 2002), Guo and Rao s 2008) procedure constructed under arbitrary dependence and Romano and Shaikh s 2006) procedure developed under a weak form dependence. Methods improving the BH method by incorporating prior information about or importance of each hypothesis through appropriately weighting the p-values have been developed in Benjamini and Hochberg 1997), Benjamini and Heller 2007) and Genovese, Roeder and Wasserman 2006). Asymptotic for large number of hypotheses) results on properties of the BH method and FDR control of related methods are available in Farcomeni 2007), Ferreira and Zwinderman 2006), Finner, Dickhaus and Roters 2008, 2009), Genovese and Wasserman 2002, 2004), and Storey, Taylor and Siegmund 2004). BH type methods to control false discoveries in the contexts of multiple confidence intervals and directional decisions have been given in Benjamini and Yekutieli 2005). The BH method has been extended by Yekutieli 2008) from testing a single family to testing multiple hierarchical families of hypotheses. The other goal of this paper, to define a measure of false discoveries directly incorporating into it a notion of dependence among the p-values and
Controlling the False discovery rate 163 ρ = 0 ρ = 0.2 power 0.8 0.6 0.4 0.2 0.0 0 200 400 600 800 1000 n 1 ρ = 0.4 power 0.8 0.6 0.4 0.2 0.0 0 200 400 600 800 1000 n 1 ρ = 0.6 power 0.6 0.4 0.2 0.0 0 200 400 600 800 1000 n 1 ρ = 0.8 power 0.6 0.5 0.4 0.3 0.2 0.1 0.0 0 200 400 600 800 1000 n 1 ρ = 1 power 0.5 0.4 0.3 0.2 0.1 0.0 0 200 400 600 800 1000 n 1 power 0.6 0.4 0.2 0.0 0 200 400 600 800 1000 n 1 Figure 1: Comparison of simulated average power of the Pairwise-FDR solid line) and BH dashed line) methods with α = 0.05.
164 Sanat K. Sarkar then developing a method to control it, touches upon an open and challenging issue in large-scale multiple testing - to accommodate correlations among the p-values to construct valid multiple testing procedures with improved efficiency. We have made an attempt to address this issue, which is of course an initial attempt needing to be fine-tuned and further investigated. Our present idea of controlling the expected proportion of falsely rejected pairs among all pairs of rejected null hypotheses, which appears to be a natural extension of the FDR directly involving the pairwise dependence among the p-values, came from Guo and Sarkar 2008), Sarkar 2007, 2008a) and Sarkar and Guo 2009a, b) where the k-fdr, a different type of generalized FDR, has been considered and methods controlling the k-fwer, a generalized form of the FWER, have been constructed parallel to Guo and Romano 2007), Korn, Troendle, McShane and Simon 2004), Romano and Shaikh 2006), Romano and Wolf 2007) and Dudoit and van der Laan 2008). Unfortunately, however, our idea requires the pairwise joint distributions of the null p-values to be common with a known form, which, one might argue, is often not practical. So, it would be interesting to investigate if a method controlling the Pairwise-FDR can be developed by relaxing these requirements, at least asymptotically for large number of hypotheses), and/or appropriately estimating the pairwise null distributions of the p-values. 7 Appendix We will show here using simple arguments why the positive dependence condition 5.3) follows from the MTP 2 property by referring to the results related to this property given in Sarkar 2008a), who recalled them from Karlin and Rinott 1980), although a much more general, sophisticated result than this proof is available in the literature. Let fx,y) be the probability density of P i,p j ). Then, for any two fixed pairs u,v) and u,v ), with u u, v v, we have E { ψp 1,...,P n ) P i u,p j v } E {ψp 1,...,P n ) P i u,p j v} = E {gp i,p j )hp i,p j )}, 7.1) where gx,y) = E {ψp 1,...,P n ) P i = x,p j = y}, hx,y) = 1 Pr P i u,p j v ) Ix u,y v), 7.2) Pr P i u,p j v)
Controlling the False discovery rate 165 and the expectation E is taken with respect to the probability density f x,y) = fx,y)ix u,y v ) PrP i u,p j v. 7.3) ) We will now show that the expectation in 7.1) is greater than or equal to 0 using the results on MTP 2 stated in Sarkar 2008a). The probability density fx,y) is MTP 2, in fact, now it is TP 2, being a bivariate marginal density of an MTP 2 multivariate distribution, and Ix u,y v )/PrP i u,p j v ) is trivially TP 2 in x,y). Hence, the probability density f x,y), being a product of two TP 2 functions, is TP 2, implying that the corresponding random variables are positively associated, that is, any two co-monotone functions of them are positively correlated. Going back to the expectation E in 7.1), we notice that gx,y) is non-decreasing in x,y), as a consequence of the positive regression dependence condition enjoyed by the MTP 2 property of the p-values, and hx,y) is also non-decreasing in x,y), as Ix u,y v) is non-increasing in x, y). Hence, this expectation is greater than or equal to E {gp i,p j )} E {hp i,p j )}, which is equal to 0, as E {hx,y)} = 1 Pr P i u,p j v ) Pr P i u,p j v) This proves the required inequality. PrP i u,p j v) PrP i u,p j v ) = 0. 7.4) Acknowledgements. I am thankful to Fang Liu who did the numerical calculations, and to Wenge Guo and an anonymous referee who provided useful comments that led to an improved presentation of the earlier version of the paper. References Benjamini, Y. and Heller, R. 2007). False discovery rates for spatial signals. J. Amer. Satist. Assoc. 102, 1272 1281. Benjamini, Y. and Hochberg, Y. 1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57, 289 300. Benjamini, Y. and Hochberg, Y. 1997). Multiple hypotheses testing with weights. Scand. J. Statist. 24, 407-418. Benjamini, Y., Krieger, A.M. and Yekutieli, D. 2006). Adaptive linear step-up false discovery rate controlling procedures. Biometrika 93, 491 507. Benjamini, Y. and Liu, W. 1990). A step-down multiple hypotheses testing procedures that controls the false discovery rate under independence. J. Statist. Plann. Inf. 82, 163 170.