On adaptive procedures controlling the familywise error rate

Transcription

1 , pp. 3 On adaptive procedures controlling the familywise error rate By SANAT K. SARKAR Temple University, Philadelphia, PA 922, USA sanat@temple.edu Summary This paper considers the problem of developing data-adaptive FWR methods for multiple testing providing control of the FWR with finite number of tests under positive dependence of the underlying p-values. Guo 2009 has recently given an adaptive Bonferroni method controlling the FWR with finite number of tests, and an adaptive Holm method controlling the FWR with infinitely large number of tests. We give a wider a class of FWR controlling adaptive Bonferroni methods containing that of Guo 2009, and adjust his adaptive Holm method to provide a control of the FWR with finite number of tests. We propose the steup analog of the present adaptive Holm method as an adaptive Hochberg method and prove that it controls the FWR. In addition, we introduce an FWR controlling adaptive Hommel method. Some key words: Familywise error rate; Multiple testing; Stepwise procedure.. Introduction When testing a family of null hypotheses simultaneously using their respective p- values, the familywise error rate FWR is frequently chosen as an overall measure of Type I error to control, especially when this family is not very large. The FWR is the probability of at least one false rejection. Several FWR controlling procedures are available in the literature [see, for example, Hochberg and Tamhane 987, and Hsu 996], among which the Bonferroni, Holm 987, Hochberg 988 and Hommel 988 procedures are commonly used in practice. Conceptually, these procedures can be made less conservative, and hence more powerful, by extracting information about the number of true null hypotheses from the available data and suitably incorporating that into the procedures. Hochberg and Benjamini 990 first realized this and presented dataadaptive versions of Bonferroni, Holm and Hochberg procedures utilizing an estimate of the number of true null hypotheses developed by them more formally than what was initially presented in Schweder and Spjφtvoll 982. However, whether or not these adaptive procedures ultimately control the FWR has not yet been mathematically established. Recently, Guo 2009 offers a partial answer to the open problem. He considers the aforementioned adaptive Bonferroni and Holm procedures, modifies them slightly by replacing the estimate of the number of true null hypotheses by the simpler estimate that Storey et al considered in the context of false discovery rate FDR, and proves that, when the p-values are independent or exhibit certain types of dependence,

2 2 this version of adaptive Bonferroni procedure controls the FWR for finite number of tests, while this adaptive Holm procedure controls the FWR approximately for infinitely large number of tests. He does not, however, consider the adaptive Hochberg procedure. The motivation of this paper lies in Guo We revisit the open problem in Hochberg and Benjamini 990 to investigate if this can be answered for adaptive versions of other FWR procedures or different adaptive versions of the same FWR procedures. While making this investigation, we are able to strengthen the work of Guo 2009 with finite number of tests. Assuming a slightly more general distributional form for the p-values than what he has considered, we first show that there is a larger class of adaptive Bonferroni procedures that can also control the FWR, and then adjust his adaptive Holm procedure providing a control over the FWR with finite number of tests. In addition, we address the adaptive Hochberg procedure in Hochberg and Benjamini 990. We consider the stepup analog of our adaptive Holm procedure, and propose that as our version of adaptive Hochberg procedure and prove that it controls the FWR. We introduce an adaptive Hommel procedure, for the first time, providing an ultimate control of its FWR. Importantly, we offer proofs of our results that seem less involved than those in Guo 2009 for similar results. The paper is organized as follows. We set the stage for our main results in the next section before presenting them in Section 3, and make some concluding remarks in Section 4. Proofs of some of the supporting results needed to establish our main results are deferred to Appendix. 2. Background and Motivation Consider testing n null hypotheses H,..., H n simultaneously using their p-values P,..., P n, respectively. A simultaneous testing is typically carried out using a stepwise stepup or stepdown or single-step method. Let P P n be the ordered versions of these p-values, with H,..., H n being their corresponding null hypotheses. Then, given a non-decreasing set of critical constants 0 < c c n <, to be determined subject to a control of the chosen error rate, which is the FWR in this paper, a stepdown method with these critical constants rejects the set of null hypotheses H i, i i SD, where i SD = max i n : P j c j j i, if the maximum exists, otherwise accepts all the null hypotheses. A stepup method rejects the set of null hypotheses H i, i i SU, where i SU = max i n : P i c i, if the maximum exists, otherwise accepts all the null hypotheses. Stepwise procedure with a common critical constant is referred to as a single-step method. Let V denote the number of falsely rejected null hypotheses. So, FWR = prv. We consider a model for the p-values in this paper to be described more formally in the next section in which the truth or falsity of each null hypothesis is assumed to be a random phenomenon. The rest of the discussions in this section relates to the FWR based on the distribution of the p-values conditionally given a set of true and false null hypotheses. Suppose n 0 of these null hypotheses are true. Since FWR = 0 if n 0 = 0, we assume without any loss of generality that n 0. For notational convenience, we denote the p-values corresponding to the true null hypotheses by P i, i =,..., n 0, and their ordered versions by P P n0.

3 For a single-step method with critical constant c, FWR = pr P c, and for a stepwise method with critical constants 0 < c c n <, the FWR satisfies the inequality given in the following: Lemma. For stepdown method: FWR pr P c n n0 +, n0 For stepup method: FWR pr Pi c n n0 +i. 2 We assume that pr P i u u, where u 0,, for each i =,..., n 0. Since pr P c n 0 pr P i c, due to the Bonferroni inequality, the single-step method with c = /n provides a control of the FWR at, irrespective of what n 0 is. This is the Bonferroni method. The inequalities in Lemma, to be proved using arguments different from what is seen in the literature for example, Lehmann and Romano 2995 and Romano and Shaikh 2006, provide the first steps towards developing Holm s 979 stepdown and Hochberg s 988 stepup methods. For instance, for a stepdown method, we have n 0 FWR pr Pi c n n0 + n 0 c n n0 +, which is less than or equal to, for any n 0, if c i = /n i +, i =,..., n. This is the Holm method. For the stepup method with these same critical values, c i = /n i +, i =,..., n, we have n0 n0 FWR pr Pi c n0 i+i pr Pi i/n 0, which is less than or equal to under independence or some type positive dependence among the p-values, because of the Simes inequality [Simes 986, Sarkar 998, and Sarkar and Chang 997]. This is the Hochberg method. The Hommel 988 method is neither a stepdown nor a stepup method, rather it is a two-stage method that first finds ĵ = min i : n i + P k for all i k n, k i + and then rejects all H i with P i /n ĵ +, provided the above minimum exists, otherwise it rejects all hypotheses. The fact that it controls the FWR under the same type of dependence condition on the p-values as required for the Hochberg method follows from its being a closed testing method [Markus and Peritz 976] based on Simes global test [Simes 986, Sarkar 998, and Sarkar and Chang 997]. To see how these FWR methods, except Hommel s, can potentially be improved by suitably extracting information about n 0 from the available data, as discussed in Hochberg and Benjamini 990, first consider the Bonferroni method. If n 0 were known, the single-step method with c = /n 0, instead of /n, would be the most ideal, in the sense of being the least conservative, single-step method controlling the FWR at, of course when no particular dependence structure among the p-values is assumed. So, when n 0 is unknown, one can consider using an appropriate estimate n 0 of it that satisfies / n 0 /n 0 and replacing n 0 in c = /n 0 by n 0. This can potentially yield 3

4 4 a more powerful FWR controlling method than the original Bonferroni method. An adaptive Bonferroni method is such a potentially improved version of the Bonferroni method when the estimate n 0 is constructed from the available p-values. Hochberg and Benjamini 990 first proposed an adaptive Bonferroni method based on an estimate of n 0 developed along the line of Schweder and Spjφtvoll 982, but its FWR control has not been theoretically established. Guo 2009 modifies it slightly using the following class of estimates of n 0 n 0 = n Rλ +, λ 0,, where Rλ = IP i λ, 3 λ the same one Storey et al considered, to define his adaptive versions of the Bonferroni method, and proves that these adaptive methods can control the FWR under certain types of dependent p-values. In the next section, we will show that this result can be extended considering a larger class of estimates of n 0 and assuming a slightly more general distributional form for the p-values. In case of the Holm method, an ideal, more powerful version of it if n 0 were known would be, as seen from the inequality, the stepdown method with the critical values c i = / min n 0, n i +, i =,..., n. Thus, when n 0 is unknown, the stepdown method with the random critical values ĉ i = / min n 0, n i +, i =,..., n, based on a suitable estimate n 0 of n 0, like that in 3, obtained from the available p-values is an appropriate adaptive version of the Holm method. Similarly, an appropriate adaptive Hochberg method is the stepup method with the critical values ĉ i = / min n 0, n i +, i =,..., n, for some estimate n 0 of n 0. With regard to the Hommel method, the idea of constructing a data-adaptive version of it has not been considered before, although one might argue that it is already an adaptive method, more specifically, a two-stage adaptive Bonferroni method, since n ĵ + represents the maximum size of the subset of null hypotheses declared non-significant by Simes global test and thus provides an estimate of n 0. Nevertheless, we will show that the Hommel method itself can be adapted to the data through an estimate of n 0 as in 3. This is done by describing the Hommel method in the framework of a stepwise method with an increasing sequence of functions of the p-values treated as the p-values and the same critical constants as in the original Holm or Hochberg method, and then developing its adaptive version analogously to that for adaptive Holm or Hochberg method. Remark. It is important to point out that the inequalities in Lemma hold even when the c i s are random. 3. Main Results We present in this section our main results of this paper. These results are about newer adaptive versions of standard FWR controlling procedures with ultimate control of the FWR under the following model for the p-values: Definition Conditional independence model. Let H i = 0 or according to whether it is true or false. ach H i Bernoulli π 0, where π 0 = prh i = 0. Conditionally given H i, i =,..., n, P i, i =,..., n, are independent with prp i u H i = 0 u, u 0,.

5 Remark 2. This model is less restrictive than what Guo 2009 has considered. The model in Guo 2009 implies but not implied by conditional independence. Moreover, we assume each null p-value to be stochastically larger than, instead of equal to, U0,. 3. Adaptive Bonferroni methods An adaptive Bonferroni method, as discussed above, is one that rejects each H i if P i /n π 0, for some suitable estimate π 0 of π 0. We will introduce a class of such adaptive Bonferroni methods, each corresponding to one of a class of estimates of π 0 satisfying certain conditions. Let us denote the vector P,..., P n by P, and by P i, 0 when P i = 0. We consider the class of estimates π 0 P of π 0 that satisfy the following: Property. π 0 P is non-decreasing in each P i and IHi = 0 n π 0 P i., 0 5 This class contains the estimates in 3 as well as others, like those given in the following examples. xample. Consider the estimate π 0 = n k + n[ P k ], for any fixed k n. It satisfies Property, since it is clearly non-decreasing in each P i, and, as explained below, it satisfies the other desired condition also. Let P i k be the k th ordered among all the p-values except P i with P i 0 = 0. Then, for this estimate, n IHi = 0 π 0 P i =, 0 n k + IH i = 0 P i k, 4 for 2 k n, and is equal to π 0 and hence,, for k =. Assume 2 k n. Since P i k is non-decreasing in each P i and the conditional distribution of the p-values given H i, i =,..., n, are independent with each null p-value having a stochastically larger then U0, distribution, the expectation in 4, conditional on H i, i =,..., n, is less than or equal to the conditional expectation where the conditional distribution of each null p-value is distributed as U0,. As shown in Benjamini et al and Sarkar 2009, n k + P i k H,..., H n n IH i = 0, assuming of course that n IH i = 0 0 otherwise, the expectation in 4 would be zero, which implies that the expectation in 4 is less than or equal to. xample 2. Let R n λ,..., λ n be the number of rejections observed while testing the null hypotheses using a stepwise stepup or stepdown method with any set of critical

6 6 constants 0 λ λ n. Consider the estimate π 0 = n R nλ,..., λ n + n λ n It satisfies Property. It is non-decreasing, as R n λ,..., λ n is non-increasing, in each P i. It also satisfies the other condition, that is, for this estimate, the following IHi = 0 λ n IH i = 0 n π 0 P i =, 0 n R i n λ, 2,..., λ n where R i n λ 2,..., λ n is the number of rejections in the stepwise test based on all the p-values except P i and the critical values λ 2 λ n, is less than or equal to. This can be seen from Sarkar Remark 3. The class of estimates of π 0 provided by those in 3 is a special case of that in xample 2. Definition Level adaptive Bonferroni method.. Define an estimate π 0 P satisfying Property. 2. Reject H i if P i /n π 0 P. Theorem. Under the conditional independence model, the FWR of the an adaptive Bonferroni method is controlled at. Proof. FWR = n pr P i pr P i n π 0 P, H i = 0 n π 0 P i, 0, H i = 0 IHi = 0 π 0 P i, 0.. The equality follows from the conditional independence of the p-values given H,..., H n. The second and third inequalities follow, respectively, from the first and second conditions of Property satisfied by the estimate of π Adaptive Holm methods An appropriate adaptive Holm method, as discussed before, would be the one that rejects H i if P i P î, where î = maxi : P j / minn π 0, n j +, for all j i, for some suitable estimate π 0 of π 0. However, a careful study of the results in Guo 2009 proving asymptotic control of his version of adaptive Holm method would reveal that this adaptive Holm method may not ultimately control the FWR at with a finite n unless the method is suitably modified. So, we modify it and refer to such a modification as -level adaptive Holm method with finite n in this article. Definition 2 -level adaptive Holm method.

7 . Define π 0 λ = n Rλ + n λ for any fixed λ 0,. 2. Reject H i if P i P ˆk, where k = max i : P j for all j i. + λ minn π 0 λ, n j + 7 Remark 4. It is important to note the difference between the above and Guo s 2009 adaptive Holm methods, even though Guo s 2009 method controls the FWR approximately as n. Guo s 2009 adaptive Holm method rejects H i if P i P ˆr, where r = max i Rλ : P j / nˆk, where k = max 0 j < n : nj+ n j n 0, if the maximum exists, otherwise it is 0, and n j = #P i > / n j, for j =,..., n, and n 0 = n π 0 λ. The following result holds for the above adaptive Holm method. Theorem 2. Under the conditional independence model, the FWR of an adaptive Holm method is controlled at. Proof. From Lemma and Remark, FWR Pr P i H,..., H n, 5 + λ minn π 0 λ, n 0 i I 0 where I 0 = i : H i = 0 and n 0 = I 0, the cardinality of I 0. Now, notice that n π 0 λ n 0 V λ + λ n 0 V i λ, λ where V λ = i I 0 IP i λ and V i λ = j i I 0 IP j λ. So, using the Bonferroni inequality, we see that the conditional probability in the right-hand side of 5 is less than or equal to i I 0 Pr n 0 P i + λ min n 0 V i λ λ, n 0 + λ min n 0 X λ, n 0 H..., H n, 6 where X Binomialn 0, λ. The inequality holds since V i λ st X. The expectation in 6 is less than or equal to /n 0, as seen from the following lemma, and hence the FWR is less than or equal to. The following lemma is a refined version of a similar result used in Guo We will skip its proof.

8 8 Lemma 2. Let X Binn 0, λ. Then min n 0 X λ, n = + λpr X = [n 0 λ] λ n, 0 n 0 where [n 0 λ] is the largest integer contained in n 0 λ Adaptive Hochberg methods As in adaptive Holm method, rejecting H i if P i P, where î = maxi : P î i / minn π 0, n i +, would be an appropriate adaptive version of the Hochberg method, but again it would not control the FWR for finite n unless suitably modified. We propose the stepup analog of the above -level adaptive Holm method and define that as -level adaptive Hochberg method with finite n in this article. Definition 3 -level adaptive Hochberg method.. Define π 0 λ as in Definition Reject H i if P i P ˆk, where k = max i : P i. + λ minn π 0 λ, n i + Theorem 3. Under the conditional independence model, the FWR of an adaptive Hochberg method is controlled at. Proof. First, consider the FWR conditionally given H,..., H n. Let us denote by P i the p-value corresponding to H i = 0, i =,..., n 0, and by P P n0 their ordered versions. Then, we have From Lemma and Remark, where FWR H,..., H n Pr Pr Pr n0 n0 n0 P i P i P i n 0λ = n 0 V λ +. λ + λ min n 0 λ, n 0 i + + λ minn 0 λ, n 0 i + + λ minn 0 λ, n, 7 0 i The probability in the right-hand side of 7 is the type I error probability of an adaptive version of Simes 986 test for testing the intersection of n 0 null hypotheses with critical values i = i/ + λ minin 0 λ, n 0, i =,..., n 0. The fact that this probability is less than or equal to under independence, and hence the FWR is controlled at unconditionally, thus proving the theorem, follows from the following lemma. This lemma will be proved in the appendix.

9 Result. Consider testing the intersection of n null hypotheses H i, i =,..., n, based on their respective p-values P i, i =,..., n. Define n λ = [n Rλ + ]/ λ. Use the adaptive version of Simes test rejecting the intersection null hypothesis if P i i for at least one i =,..., n, where i = i/ + λ minin λ, n. The type I error rate is controlled at when these p-values are independent with each being stochastically larger than U0, Adaptive Hommel methods As said in the previous section, the Hommel method can be adapted to the data through an estimate of π 0 as in developing adaptive versions of Holm and Hochberg methods. Before we do that, it is useful to see that the Hommel method can be described in the framework of a stepwise method. Let Q j = min P j, P j+,..., 2 P n, j =,..., n. n j + In terms of this Q j, which is increasing in j, the Hommel method finds ĵ = min j : Q j, n j + and accepts the null hypotheses with the p-values greater than /n j +, that is, those corresponding to P ĵ,..., P n, provided the above minimum exists, otherwise, rejects all hypotheses. This is like a stepdown test with the Q j s treated as the p-values and using the critical constants c j = /n j +, j =,..., n. Remark 5. Note that n j + Q j is increasing in j, since [ ] n j + Pi n j + Q j = min n j + P j, min j+ i n i j + n jpi min = n jq j+ i n i j j+. Therefore, the Hommel method can also be described as a stepup test in terms of the Q j s and the same critical constants. In fact, this stepdown or stepup test is same as the single-step test using n j + Q j, j =,..., n, and the common critical constant. We will, however, treat the Hommel method as the above stepdown test for convenience. Definition 4 -level adaptive Hommel method.. Define π 0 λ as in Definition Accept the null hypotheses corresponding to P ˆk,..., P n, where k = min j : Q j. + λ minn π 0 λ, n j + 9 Theorem 4. Under the conditional independence model, the FWR of an adaptive Hommel method is controlled at.

10 0 Proof. The probability of V = 0, that is, - FWR, of the method in the theorem, conditionally given H,..., H n, is equal to n pr Qi i V = 0 = pr n n0 + Qi i V = 0 since, for i = n n 0 + 2,..., n, the event Q i i allows less than n0 acceptances, hence its intersection with V = 0 is a null set. It is not difficult to see from Remark 5 that minn π 0 λ, n i + Q i is increasing in i. Therefore, = pr n n0 + Qi i V = 0 = pr Q n n0 + n n0 + V = 0 Now, the intersection event Q n n0 + n n0 + V = 0 is equivalent to saying that the n 0 null hypotheses that correspond to the ordered p-values P n n0 + P n defining Q n n0 + and are accepted due to Q n n0 + n n0 + are same as the n 0 true null hypotheses corresponding to the ordered p-values P P n0. Thus, we have, FWR H,..., H n P 2 Pn0 = pr min P, 2,..., n n0 + = n 0, + λ minn π 0 λ, n 0 which is greater than or equal to due to Result. The theorem then follows once the expectation over H,..., H n is taken. 4. Concluding remarks This paper has been focused on developing adaptive FWR procedures with proven control of the FWR with finite number of tests under independence or some form of dependence of the p-values. For adaptive Bonferroni and Holm methods, we have revisited Guo 2009 to improve or adjust his results, while for adaptive Hochberg and Hommel methods, we present some new results. It is important to point out that, if n n IH i = 0 π 0 in probability, as n, for some π 0 0,, the approximate versions of the present adaptive Holm, Hochberg and Hommel methods would be those without the factor + λ in the denominator of k in each of these methods. Acknowledgement This research is supported by the NSF Grant DMS

11 Appendix Proof of Lemma For the stepdwon method, n n0+ n n0+ FWR = pr Pi c i V = 0 = pr Pi c i P c i Now, pr n n0+ Pi c i P c n n0 +. Pi c i P c n n0 + P c n n0 + for i = n n 0 + A i P c n n0 + for i =,..., n n 0, for some events A i, i =,..., n n 0, defined in terms of the p-values corresponding to the false null hypotheses. Therefore, with A n n0 + as the sure event, we have n n0+ FWR pr A i P c n n0 + = pr P c n n0 +. Similarly, for the stepup method, n n0 + FWR pr Pi c i,..., P n c n P c n n0 +,..., P n0 c n, and, since Pi c i,..., P n c n P c n n0 +,..., P n c n0, B i P c n n0 +,..., P n0 > c n, where B n n0 + is the sure event and B i is an event defined in terms of the p-values corresponding to the false null hypotheses for i =,..., n n 0, we have FWR pr P c n n0 +,..., P n0 c n. Proof of Result This results can be proved using Lemma 2 and the following additional lemma: Lemma 3 Blanchard and Roquain Given a random variable U satisfying pru u u, u 0,, any positive valued non-decreasing function g, and a fixed constant c, we have I U c gu c. gu Define P = P,..., P n, and R i n R i n P the number of rejections in the stepup test based on P i = P,..., P n \ P i and the critical values ĉ i P = i/ + λ minin λ, n,

12 2 i =,..., n. Then, the type I error rate is equal to I P i i R n + n R i n + = I Since P i n λ ñλ = n R i λ where R i n = n j i= IP j λ, and R i n than or equal to I P i [R i n +] +λ min[r i n +]n λ,n n λ R i n +,. 0, we see that the expectation in A. is less [R i n P+] +λ minñλ,n R i n P +. A We now apply Lemma 3 to this expectation conditional on P i to see that it is less than or equal to + λ minñ, n + λ min n X λ, n, where X Binomialn, λ. The inequality follows since R i st n λ X. This expectation is less than or equal to /n, as seen from Lemma 2, and thus the result is proved. References Benjamini, Y, Krieger, K. & Yekutieli, D Adaptive linear step-up procedures that control the false discovery rate. Biometrika 93, Blanchard, G. & Roquain, Adaptive FDR control under independence and dependence. J. Mach. Learn. To appear. Guo, W A note on adaptive Bonferroni and Holm procedures under dependence. Biometrika. To appear. Hochberg, Y A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, Hochberg, Y. & Benjamini, Y More powerful procedures for multiple significance testing. Statist. Med. 9, Hochberg, Y., & Tamhane, A. C Multiple Comparison Procedures. New York: Wiley. Holm, S A simple sequentially rejective multiple test procedure. Scand. J. Statist. 6, Hommel, G A stagewise rejective multiple test procedure based on a modifed Bonferroni test. Biometrika 75, Hsu, J Multiple Comparisons: Theory and Methods. New York: Chapman & Hall. Lehmann,. L. & Romano, J. P Generalizations of the familywise error rate. Ann. Statist. 33, Marcus, R., Preitz,. & Gabriel, K. R On closed testing procedures with special reference to ordered analysis of variance. Biometrika 63, Romano, J. P. & Shaikh, A. M Stepup procedures for control of generalizations of the familywise error rate. Ann. Statist. 34, Sarkar, S. K. & Chang, C-K The Simes method for multiple hypothesis testing with positively dependent test statistics. J. Amer. Statist. Assoc. 92, Sarkar, S. K Some probability inequalities for ordered MTP 2 random variables: a proof of the Simes conjecture. Ann. Statist. 26, Sarkar, S. K On methods controlling the false discovery rate. Sankhyā, Ser. A 70, Part 3. In press.

13 Schweder, T. & Spjφtvoll, Plots of p-values to evaluate many tests simulataneously. Biometrika 69, Storey, J. D., Taylor, J.. & Siegmund, D Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach. J. Roy. Statist. Soc., Ser. B 66,