Modified Simes Critical Values Under Positive Dependence

Modified Simes Critical Values Under Positive Dependence Gengqian Cai, Sanat K. Sarkar Clinical Pharmacology Statistics & Programming, BDS, GlaxoSmithKline Statistics Department, Temple University, Philadelphia Abstract A modification of the critical values of Simes test is suggested in this article when the underlying test statistics are multivariate normal with a common nonnegative correlation, yielding a more powerful test than the original Simes test. A step-up multiple testing procedure with these modified critical values, which is shown to control false discovery rate (FDR), is presented as a modification of the traditional Benjamini-Hochberg (BH) procedure. Simulations were carried out to compare this modified BH procedure with the BH and other modified BH procedures in terms of false non-discovery rate (FNR), 1 FDR FNR and average power. The present modified BH procedure is observed to perform well compared to others when the test statistics are highly correlated and most of the hypotheses are true. Key words: Multivariate normal with non-negative common correlation; Adjusted Simes test; False Discovery Rate; Modified Benjamini-Hochberg procedures 1 Introduction Simes (1986) test has been receiving considerable attention recently by researchers as well as practitioners in multiple comparisons. This is because it performs much better than Bonferroni and Sidak procedures and, more importantly, its critical values are the same ones in the Benjamini-Hochberg (1995) step-up procedure that controls the false discovery rate (FDR). Suppose that we have a set of test statistics X 1,..., X n for testing some null hypotheses H 1,..., H n, respectively, and that P 1,..., P n are the corresponding p-values. Simes (1986) proposed his test for testing the overall null hypothesis The research is supported by NSF Grant DMS-0306366. Email address: sanat@temple.edu (Sanat K. Sarkar). Preprint submitted to Elsevier Science 12 April 2005

H = n i=1h i, as a modification of the Bonferroni test, based on the ordered p-values P (1) P (n) as follows: Reject H at level α if P (i) iα/n for at least one i. (1) Benjamini-Hochberg s (1995) FDR-controlling procedure (BH procedure) is a step-up test with these same critical values, i.e., it rejects H i for all i k and accepts the rest, where k = max{j : P (j) jα/n}. Simes (1986) proved that the type I error rate of his method, i.e, the probability of (1) under the null hypotheses, is exactly α when the X i s are iid and conjectured that it is conservative for a variety of commonly used multivariate distributions exhibiting positive dependence. Sarkar and Chang (1997) and Sarkar (1998) noticed that many of the commonly used positively dependent multivariate distributions arising in multiple comparisons are characterized by the MTP 2 (multivariate totally positive of order two) property of Karlin and Rinott (1980) and proved Simes conjecture for these distributions. Regarding the BH procedure, Benjamini and Yekutieli (2001) and Sarkar (2002) proved that the FDR of this procedure is less than or equal to n 0 α/n, where n 0 is the (unknown) number of true null hypotheses, for multivariate distributions that are positively dependent in the sense of being PRDS (positive regression dependent on subset), a slightly weaker version of the MTP 2 condition, and is exactly equal to n 0 α/n under independence. Readers are referred to Benjamini and Yekutieli (2001) and Sarkar (2002) for details of PRDS and to the Appendix for a brief review of the MTP 2 property. Notice that the FDR-controlling property of the BH procedure, considering n 0 = n, implies that Simes test controls the type I error rate. Therefore, Simes test is actually conservative even for the larger class of PRDS distributions. Multivariate normal with positive correlations that commonly arise in many multiple testing situations is an example of such a distribution. Test statistics having multivariate normal distribution with a known nonnegative common correlation are often encountered in multiple comparisons; for instance, in many-to-one comparisons using a balanced one-way layout (Hochberg and Tamhane, 1987; Hsu, 1996). This paper is motivated by the fact that, for such a distribution, since Simes test is conservative, it can potentially be improved in terms of power if its critical values are modified incorporating the correlation. These modified Simes critical values can also potentially improve the BH procedure. Of course, the idea of adjusting the critical values of Simes test by incorporating the dependence structure is not new; it was implemented before by Kwong et al. (2002) in an attempt to modify the BH procedure. There is, however, a methodological difference in our approach to adjusting the critical values. Rather than modifying only one critical value, as done in Kwong et al. (2002), we modify all but one critical value. This results in a modified Simes 2

procedure that gets more powerful than the unmodified one as the correlation increases. This is in sharp contrast with Kwong et al. s version of the modified Simes procedure which has an inconsistency in its power performance, particularly for large n, in that it peaks up quite dramatically for small correlation but becomes weak as correlation increases. We propose a modification of the BH procedure based on the adjusted Simes critical values derived in this article. We prove that this modified BH procedure controls the FDR. Kwong et al. s version of the modified BH procedure also controls the FDR, but, as discussed above, it provides minimal improvement over the unmodified BH procedure for large correlation. Yekutieli and Benjamini (1999) and Troendle (2000) also offered ideas of improving the BH procedure incorporating the dependence structure. Yekutieli-Benjamini s method, however, relies on resampling technique, and the FDR-controlling property can be verified only empirically. While Troendle proposed powerful step-up and step-down procedures based on normal theory, they are based on an assumption, valid only asymptotically, that the ordering of test statistics correspond to that of the means. How good is the performance of our version of the modified BH procedure, referred to as the extended BH (EBH) procedure, compared to the original BH procedure and Kwong et al. s (2002) version of the modified BH procedure? We investigate that in terms of three different concepts of power. One is the average power (Benjamini and Liu, 1999; Kwong et al., 2002; Storey, 2002). The second one is based on the concept of false non-discovery rate (FNR) (Genovese and Wasserman, 2002; Sarkar, 2005, 2004; Storey, 2003). The third one is 1 FDR FNR proposed in Sarkar (2004), which reflects the strength of unbiasedness of an FDR procedure. The setup of this paper is as follows. In Section 2, we explain our method of modifying Simes critical values using Sarkar s (1998) formula. The fact that the BH procedure based on these modified Simes critical values controls the FDR is established in Section 3. Section 4 compares EBH to the BH and Kwong et al. s procedures. Section 5 shows the simulation results when the multivariate normal with common correlation assumption is not met. The paper concludes with some remarks. Proofs of some supporting lemmas are deferred to the Appendix. 2 Modified Simes Critical Values As mentioned in the introduction, we will assume that the set of test statistics X = (X 1,..., X n ) is distributed as multivariate normal with mean vector Θ = (θ 1,..., θ n ) and covariance matrix Σ = (1 ρ)i + ρj, for some known 3

ρ 0. We are interested in testing H i : θ i = 0 versus K i : θ i > 0, for i = 1,, n. Simes test at level α for testing the overall null hypothesis H : n i=1h i against the alternative K : n i=1k i, in terms of the X i s, rejects H if X (i) Φ 1 (1 n i+1 α), where X n (1) X (n) are the ordered components of X, and Φ(z) is the cdf of N(0, 1) at z. This test, under the present set-up, is conservative, i.e., its type I error rate is less than or equal to α, and hence can be improved if we modify the critical values by c 1,α (ρ) c n,α (ρ); satisfying 1 P H {X (i) < c i,α (ρ), i = 1,..., n} = α. (2) For the time being, we will simply write c i instead of c i,α (ρ). From Lemma 2.1 in Sarkar (1998), we note that the left-hand side of (2) can be expressed as 1 P H {X (i) < c i, i = 1,..., n} = 1 n n P H {X i c 1 } + n i=1 { I(Xi c j+1 ) n j n 1 i=1 j=1 E H [ P H { X ( i) (1) < c 1,..., X ( i) (j) < c j X i } I(X i c j ) n j + 1 } ], (3) where X ( i) (1) X ( i) (n 1) denote the ordered components of the (n 1)- dimensional random vector X ( i) obtained by eliminating X i from X. Since, under H, X is exchangeable, we have n 1 1 P H {X (i) < c i, i = 1,..., n} = [1 Φ(c 1 )] + n K j, (4) j=1 where { I(X1 c j+1 ) K j =E H [Ψ j (X 1 ) I(X }] 1 c j ) n j n j + 1 { I(x cj+1 ) = Ψ j (x)φ(x) I(x c } j) dx, n j n j + 1 (5) with Ψ j (x) = P H { X ( 1) (1) < c 1,..., X ( 1) (j) < c j X 1 = x } (6) and φ(.) being the density of standard normal distribution. Note that K j depends only on c 1,..., c j+1. With given c 1,..., c j, unique c j+1 (> c j ) can be found from equation K j = 0. Therefore, from (4), we can derive the following procedure to calculate the unique desired critical values c 1,α (ρ) c n,α (ρ): (1) Let [1 Φ(c 1,α (ρ))] = α, that is, c 1,α (ρ) = Φ 1 (1 α) for any ρ 0. 4

(2) Make K j = 0 for j = 1, 2,, n 1, and from the n 1 equations sequentially solve for c i,α (ρ) for i = 2,, n. Obviously, for different ρ, we will get different critical values except c 1. Notice that when ρ = 0, we get Simes critical values with this method. Theorem 1 Simes test with modified critical values c i,α (ρ), i = 1,..., n, controls the type I error rate exactly at α when the test statistics follow a multivariate normal distribution with a common nonnegative correlation ρ. Proof. The result is obvious because of (4) and the way we obtain c i,α (ρ) s. In the process of computing the modified critical values, we need to evaluate (6). For this, we will employ the same algorithm as in Dunnett and Tamhane (1992) using an R program. Table 1 gives the modified critical values in terms of p-value for n = 2, 3, 4, 5 corresponding to different values of ρ at level α = 0.05. From Table 1, we note that c i,α (ρ) is a decreasing function of ρ, providing an empirical evidence that Simes test with these critical values become more powerful as ρ increases. A theoretical proof of this monotonicity property appears to be difficult. Nevertheless, we prove in the following that c i,α (ρ) < c i (0), for any i > 1 and ρ > 0, which proves that the modified Simes test is more powerful than the original Simes test. Theorem 2 For any ρ > 0 and i = 2,, n, c i,α (ρ) < c i,α (0). Proof. We prove this theorem by induction. For this, first note that c i,α (0), i = 2,..., n, are obtained from the following equations: φ(x) { I(x ci+1,α (0)) n i I(x c } i,α(0)) dx = 0, i = 1,... n 1. n i + 1 Let us write Ψ j (x) as Ψ ρ j(x). Since the distribution of X has the PRDS property, Ψ ρ j(x) is a decreasing function of x. Also, for any c j < c j+1, I(x c j+1 ) I(x c j) or 0, according as x or c n j n j+1 j+1. Therefore, we have { I(x Ψ ρ cj+1,α (0)) j(x)φ(x) I(x c } j,α(0)) dx n j n j + 1 { I(x Ψ ρ cj+1,α (0)) j(c j+1,α (0)) φ(x) I(x c } j,α(0)) dx n j n j + 1 = 0 { I(x = Ψ ρ cj+1,α (ρ)) j(x)φ(x) I(x c } j,α(ρ)) dx. n j n j + 1 Now, assume that the result is true for i = j; i.e, c j,α (ρ) < c j,α (0). Using this (7) 5

Table 1 Modified Simes Critical Values p i (ρ) = 1 Φ[c i (ρ)] with α = 0.05 p i n i 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 2 2.02500.02511.02529.02558.02600.02662.02750.02880.03082.03440 3 2.03333.03335.03339.03350.03369.03401.03454.03537.03672.03921 3.01667.01677.01697.01730.01781.01856.01966.02128.02385.02853 4 2.03750.03750.03751.03755.03765.03783.03816.03873.03971.04160 3.02500.02502.02509.02525.02555.02605.02684.02808.03009.03377 4.01250.01260.01279.01312.01364.01442.01556.01729.02006.02520 5 2.04000.04000.04000.04002.04007.04018.04040.04082.04157.04307 3.03000.03000.03003.03010.03028.03061.03118.03213.03373.03674 4.02000.02002.02011.02030.02066.02125.02218.02364.02600.03034 5.01000.01009.01027.01059.01110.01187.01302.01477.01761.02299 Note: p 1 (ρ) = 1 Φ[c 1 (ρ)] = α = 0.05. in (7), we see that c j+1,α (0) Ψ ρ j(x)φ(x)dx ρ c j+1,α (ρ) Ψ ρ j(x)φ(x)dx, (8) which implies that c j+1,α (ρ) < c j+1,α (0), i.e., the result must be true for i = j + 1 also. This proves the theorem. 3 Extended BH Procedure The BH procedure, in terms of X, is a step-up procedure that rejects H i, for all i k, where k = min{i : X (i) Φ 1 (1 n i+1 α)}. It is designed to control n the FDR, which is the expected ratio of false rejections to total number of rejections, exactly at n 0 α/n when ρ = 0, where n 0 is the number of true null hypotheses. We will modify this procedure by replacing its critical values by the modified Simes critical values obtained in the above section, and refer to that as the extended BH (EBH) procedure. Thus, the EBH procedure rejects H i, for all i k, where k = min{i : X (i) c i,α (ρ)}. We are going to show in this section that the EBH procedure also controls the FDR at α. More specifically, we prove the following theorem. Theorem 3 The FDR of the EBH procedure is less than or equal to n 0 α/n, where n 0 is the number of true null hypotheses, if X follows multivariate normal distribution with a common nonnegative correlation ρ. Proof. Our proof requires two supporting lemmas that will be stated and proved in Appendix. If n 0 = 0, the result is clearly true as FDR = 0. So let s assume that n 0 > 0 and, without any loss of generality, assume that the 6

first n 0 of the null hypotheses are true and the rest are false. For notational convenience, we will simply write c i instead of c i,α (ρ). According to Sarkar (2002), the FDR of a step-up procedure with any set of critical values c 1 c n is FDR = 1 [ n 0 n 0 n 1 P {X i c 1 } + E n i=1 i=1 j=1 { I(Xi c j+1 ) n j I(X i c j ) n j + 1 P { X ( i) (1) < c 1,..., X ( i) (j) < c j X i } Since the distribution of X is invariant under the permutations within the set of first n 0 X i s, the FDR in (9) simplifies to } ]. FDR = n n 1 0 n [1 Φ(c 1)] + n 0 K j (Θ 1 ), (10) j=1 (9) where K j (Θ 1 ) =E [ P { X ( 1) (1) < c 1,..., X ( 1) } (j) < c j X 1 { I(X1 c j+1 ) I(X } ] 1 c j ), n j n j + 1 (11) and Θ 1 = (θ 1,..., θ n ), with θ i = 0 for i = 1,..., n 0. Writing X i = 1 ρz i ρz 0 + θ i, i = 1,..., n, where Z i N(0, 1) for i = 0, 1,..., n, we can simplify K j (Θ 1 ) as follows: K j (Θ 1 ) =E [ =E P P { { X ( 1) (1) c 1,..., X ( 1) } P (X1 c j+1 Z 0 ) (j) c j Z 0 { n j Y (1) c 1 + ρz 0,..., Y (j) c j + } ρz 0 1 ρ 1 ρ { cj + 1 Φ } ρz 0 n j 1 ρ (n j + 1) Φ { c j+1 + } ρz 0, 1 ρ P (X }] 1 c j Z 0 ) n j + 1 { cj+1 + } ρz 0 Φ 1 ρ (12) for j = 1,..., n 1, where Y (1) Y (n 1) are the ordered components of (n 1)-dimensional normal with mean vector ( θ 2 θ 1 ρ,..., 1 ρ n ) and covariance matrix I n 1, and Φ = 1 Φ. Let { G Θ1 (Z 0 ; j) = P Y (1) c 1 + ρz 0,..., Y (j) c j + } ρz 0 1 ρ 1 ρ 7

and so that we have [ K j (Θ 1 ) =E =E =E [ [ G Θ1 (Z 0 ; j) Φ r(z 0 ; j) = R(Z 0 ; j) G 0 (Z 0 ; j) Φ G 0 (Z 0 ; j) Φ Φ { c j + } ρz 0 1 ρ Φ { c j+1 + ρz 0 }, 1 ρ { cj+1 + } { ρz 0 1 1 ρ { cj+1 + } ρz 0 1 ρ { cj+1 + }] [ ρz 0 E R(Z 1 ρ 0; j) n j r(z }] 0; j) n j + 1 { 1 n j r(z }] 0; j) n j + 1 { }] 1 n j r(z 0; j), n j + 1 (13) where R(z 0 ; j) = G Θ1 (z 0 ; j)/g 0 (z 0 ; j), and Z0 has the following probability density at z: G 0 (z; j) Φ { c j+1 + } ρz 1 ρ E [ G 0 (Z 0 ; j) Φ { c j+1 + ρz 0 }]. 1 ρ Since 1 r(z 0 ;j) is a decreasing function of n j n j+1 Z 0 (from Lemma 1) and R(Z0; j) is an increasing function of Z0 (from Lemma 2), the expectation of the product of these two functions with respect to the distribution of Z0 is less than or equal to the product of their expectations. But, the expectation of the first function with respect to Z0 is K j (0) E [ G 0 (Z 0 ; j) Φ { c j+1 + ρz 0 }], 1 ρ which is zero when these c i s are the modified Simes critical values. This proves the theorem. 4 Power Comparisons In this section, we will examine how much improvement over the BH procedure we can achieve by using our proposed EBH procedure. Also, we will compare the EBH procedure with Kwong et al. s version of the modified BH procedure, referred to as the MBH procedure. Towards comparing different FDR-controlling procedures, it is important to keep in mind that one can conceptualize power in many different ways. Three particular concepts have received attention in the literature. One is the average power (see, e.g. Benjamini and Liu, 1999; Kwong et al., 2002; Storey, 8

2002), which is the expected proportion of false null hypotheses that are correctly rejected. Another one is based on the concept of false non-discovery rate (FNR), which is an analog of FDR and defined in terms of Type II errors as the expected proportion of falsely accepted null hypotheses among those that are accepted (Genovese and Wasserman, 2002; Sarkar, 2005, 2004; Storey, 2003). Basically, between two FDR-controlling procedures, the one with smaller FNR is considered more powerful. The third one is 1 FDR FNR proposed in Sarkar (2004), which reflected the strength of unbiasedness of an FDR procedure. Out of two FDR-controlling procedures, the one with larger 1 FDR FNR is considered better. We will use these three concepts of power in this section. 4.1 Comparisons with the BH procedure Tables 2 and 3 show part of the extensive simulation carried out for n = 5. These tables reveal that the proposed EBH procedure can have much better performance than the unmodified BH procedure when the correlation among the test statistics is large and many of the null hypotheses are true. It should be noted that Table 3 also provides an idea how good our modification of the Simes test is in terms of controlling the Type I error rate. Each value in Table 2 and Table 3 is based on 50,000 simulations and the test statistics corresponding to false null hypotheses are assumed to have a common mean θ = 2. Table 2 Comparison of BH and EBH for n = 5, n 0 = 3 and α = 0.05 Method 0.0 0.1 0.3 0.5 0.7 0.9 FDR BH 0.0306 0.0297 0.0291 0.0287 0.0281 0.0261 EBH 0.0297 0.0294 0.0293 0.0295 0.0297 FNR BH 0.2501 0.2478 0.2433 0.2395 0.2356 0.2282 EBH 0.2476 0.2416 0.2335 0.2206 0.1930 1-FDR-FNR BH 0.7193 0.7225 0.7277 0.7318 0.7363 0.7458 EBH 0.7226 0.7290 0.7371 0.7500 0.7773 Average BH 0.4309 0.4315 0.4324 0.4310 0.4304 0.4369 Power EBH 0.4321 0.4374 0.4478 0.4716 0.5300 ρ 9

Table 3 Comparison of BH and EBH for n = 5, n 0 = 5 and α = 0.05 Method 0.0 0.1 0.3 0.5 0.7 0.9 FDR BH 0.0502 0.0498 0.0481 0.0447 0.0389 0.0330 EBH 0.0503 0.0508 0.0507 0.0500 0.0501 4.2 Comparison with the MBH procedure ρ Since Kwong et al. (2002) does not provide critical values for ρ > 0.5, we did simulation for ρ 0.5 and n = 5 to compare the two procedures in terms of FDR, FNR, 1 FDR FNR and average power. Table 4 presents these simulated values. As expected, the MBH procedure appears to dominate when ρ and n 0 are both small and the EBH procedure dominates when both are large. Each number in the table is based on 50,000 simulations and the mean corresponding to each alternative hypothesis is θ = 2. 5 Robustness Simes critical values are modified under the assumption of normality and equal correlation. For other distributions and/or unequal correlation, the conditional distributions in (3) become complicated and hard to simplify further. Therefore, at least three types of questions can be raised about the robustness of the proposed the EBH procedure: What if the common correlation assumption is not satisfied? What if the common correlation is not known? What if the distribution involved is not multivariate normal? In this section, we will carry out simulations to investigate these robustness issues of the EBH procedure. 5.1 Unequal Correlations Let us first look at the case where the correlation matrix has AR(1) structure; that is, when our test statistics follow an n-dimensional multivariate normal with unit variances and the following covariance (or correlation) matrix Σ = { σ ij = ρ i j, i = 1,..., n, j = 1,..., n }. Table 5 shows the simulation results based on this AR(1) correlation structure with n = 5, where the modified Simes critical values c i (ρ n 1 ) s corresponding 10

Table 4 Comparison of the EBH and MBH procedures with n = 5, α = 0.05 FDR n 0 ρ = 0.1 ρ = 0.2 ρ = 0.3 ρ = 0.4 ρ = 0.5 1 0.010 (0.024) 0.010 (0.027) 0.010 (0.029) 0.010 (0.031) 0.010 (0.028) 2 0.020 (0.027) 0.019 (0.029) 0.019 (0.032) 0.019 (0.034) 0.020 (0.032) 3 0.030 (0.033) 0.029 (0.034) 0.029 (0.036) 0.029 (0.039) 0.029 (0.037) 4 0.041 (0.042) 0.040 (0.042) 0.040 (0.044) 0.040 (0.045) 0.040 (0.044) 5 0.050 (0.050) 0.050 (0.050) 0.051 (0.051) 0.051 (0.051) 0.051 (0.049) FNR n 0 ρ = 0.1 ρ = 0.2 ρ = 0.3 ρ = 0.4 ρ = 0.5 0 0.856 (0.525) 0.818 (0.481) 0.783 (0.447) 0.746 (0.418) 0.707 (0.434) 1 0.561 (0.517) 0.544 (0.499) 0.528 (0.486) 0.510 (0.475) 0.492 (0.471) 2 0.385 (0.378) 0.377 (0.370) 0.369 (0.363) 0.361 (0.357) 0.352 (0.353) 3 0.248 (0.247) 0.245 (0.244) 0.242 (0.242) 0.238 (0.240) 0.234 (0.239) 4 0.125 (0.125) 0.124 (0.124) 0.123 (0.124) 0.122 (0.124) 0.120 (0.124) 1 FDR FNR n 0 ρ = 0.1 ρ = 0.2 ρ = 0.3 ρ = 0.4 ρ = 0.5 0 0.144 (0.475 ) 0.182 (0.519 ) 0.217 (0.553 ) 0.254 (0.582 ) 0.293 (0.566) 1 0.429 (0.459 ) 0.446 (0.474 ) 0.462 (0.484 ) 0.480 (0.495 ) 0.499 (0.501) 2 0.595 (0.595 ) 0.603 (0.601 ) 0.612 (0.605 ) 0.620 (0.609 ) 0.629 (0.615) 3 0.723 (0.721 ) 0.726 (0.722 ) 0.729 (0.722 ) 0.733 (0.722 ) 0.737 (0.724) 4 0.835 (0.834 ) 0.836 (0.834 ) 0.837 (0.832 ) 0.838 (0.831 ) 0.841 (0.833) 5 0.950 (0.950 ) 0.950 (0.950 ) 0.949 (0.949 ) 0.949 (0.949 ) 0.949 (0.951) Average Power n 0 ρ = 0.1 ρ = 0.2 ρ = 0.3 ρ = 0.4 ρ = 0.5 0 0.553 (0.688) 0.551 (0.690) 0.550 (0.690) 0.550 (0.689) 0.552 (0.663) 1 0.517 (0.546) 0.516 (0.545) 0.517 (0.541) 0.519 (0.537) 0.521 (0.528) 2 0.476 (0.483) 0.477 (0.484) 0.480 (0.484) 0.482 (0.482) 0.486 (0.479) 3 0.432 (0.434) 0.434 (0.434) 0.437 (0.435) 0.442 (0.435) 0.448 (0.432) 4 0.382 (0.382) 0.384 (0.382) 0.389 (0.383) 0.394 (0.382) 0.403 (0.381) Note: The values in parentheses are for the MBH procedure. to the smallest value of the correlation are used in the EBH procedure. From the simulation result, the FDR is still controlled at n 0 α/n. Each number in the table is based on 50000 simulations with the mean corresponding to each alternative hypothesis being θ = 2. Table 6 shows the simulation results based on two randomly generated positive correlation structures with n = 5. Each number in the table is based on 50000 simulations with multivariate normal test statistics with unit variance and the mean corresponding to each alternative hypothesis is θ. The correlation 11

Table 5 Simulated FDR With AR(1) Correlation Structure Minimun Correlation ρ 4 n 0 0.1 0.3 0.5 0.6 0.9 3 0.0281 0.0272 0.0271 0.0269 0.0275 4 0.0380 0.0356 0.0343 0.0342 0.0355 5 0.0467 0.0431 0.0413 0.0411 0.0438 matrices used in the simulation are Σ 1 and Σ 2, where 1.000 0.842 0.328 0.407 0.910 1.000 0.510 0.710 0.795 0.636 0.842 1.000 0.786 0.836 0.990 0.510 1.000 0.968 0.927 0.988 Σ 1 = 0.328 0.786 1.000 0.996 0.690 Σ 2 = 0.710 0.968 1.000 0.992 0.995 0.407 0.836 0.996 1.000 0.749 0.795 0.927 0.992 1.000 0.974 0.910 0.990 0.690 0.749 1.000 0.636 0.988 0.995 0.974 1.000 The simulation results in Table 5 and Table 6 show that for any positive correlation structure, if the critical values corresponding to the smallest correlation are used in the EBH procedure, the FDR can still be controlled at n 0 α/n. 5.2 Unknown Common Correlation Sometimes in practice, even though it might make sense to assume that the correlations are the same, this common value may not be exactly known. In a situation like this, one can use a conservative (small) estimate of this common correlation. This will still control FDR based on the simulation in Section 5.1. Table 6 Simulated FDR With Arbitrary Correlation Structure Mean for the Alternative Hypotheses (θ) Σ c i n 0 1.0 1.5 2.0 2.5 3.0 Σ 1 c i (0.3) 3 0.0267 0.0271 0.0273 0.0274 0.0274 4 0.0335 0.0335 0.0335 0.0335 0.0335 5 0.0383 0.0383 0.0383 0.0383 0.0383 Σ 2 c i (0.5) 3 0.0266 0.0271 0.0273 0.0273 0.0273 4 0.0322 0.0324 0.0325 0.0326 0.0327 5 0.0389 0.0389 0.0389 0.0389 0.0389 12

Table 7 Simulated FDR With Multivariate t Distribution Common Correlation ρ Among Test Statistics n 0 df 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 3 5 0.0303 0.0305 0.0304 0.0307 0.0307 0.0311 0.0315 0.0325 0.0341 10 0.0297 0.0294 0.0292 0.0292 0.0296 0.0296 0.0296 0.0300 0.0309 30 0.0297 0.0288 0.0286 0.0287 0.0281 0.0284 0.0287 0.0289 0.0291 50 0.0297 0.0286 0.0284 0.0286 0.0285 0.0287 0.0289 0.0292 0.0294 100 0.0293 0.0291 0.0293 0.0289 0.0290 0.0289 0.0288 0.0290 0.0295 500 0.0299 0.0297 0.0298 0.0296 0.0296 0.0292 0.0291 0.0291 0.0293 4 5 0.0401 0.0408 0.0411 0.0418 0.0427 0.0437 0.0452 0.0475 0.0518 10 0.0391 0.0390 0.0393 0.0391 0.0396 0.0404 0.0409 0.0424 0.0447 30 0.0399 0.0387 0.0385 0.0387 0.0388 0.0387 0.0393 0.0397 0.0407 50 0.0400 0.0394 0.0386 0.0381 0.0382 0.0387 0.0388 0.0389 0.0397 100 0.0398 0.0394 0.0388 0.0388 0.0386 0.0390 0.0389 0.0392 0.0396 500 0.0398 0.0399 0.0396 0.0392 0.0392 0.0389 0.0385 0.0389 0.0390 5 5 0.0503 0.0513 0.0525 0.0541 0.0566 0.0598 0.0637 0.0699 0.0796 10 0.0485 0.0490 0.0495 0.0504 0.0520 0.0544 0.0565 0.0602 0.0659 30 0.0491 0.0488 0.0494 0.0495 0.0502 0.0507 0.0519 0.0533 0.0559 50 0.0484 0.0484 0.0489 0.0487 0.0492 0.0498 0.0507 0.0520 0.0532 100 0.0501 0.0503 0.0508 0.0508 0.0508 0.0514 0.0514 0.0514 0.0525 500 0.0494 0.0498 0.0495 0.0501 0.0502 0.0501 0.0499 0.0503 0.0503 5.3 Multivariate t Distribution If the population follows a multivariate normal distribution with unknown variance σ 2, then the test statistics used for inferences follow multivariate t distributions. How does the EBH procedure perform in such cases? Table 7 shows the results of simulations based on the modified critical values with n = 5 in terms of p-value as listed in Table 1. Based on the simulation results in this table, the EBH procedure appears to perform well when n 0 < n and the degrees of freedom is not too small ( 10). When n 0 = n, the EBH procedure cannot control FDR at α for small degrees of freedom. Larger the correlation, the larger is the necessary degrees of freedom for the EBH procedure to control the FDR at α. Each number in the table is based on 50000 simulations with multivariate t test statistics with common correlation structure and the mean corresponding to each alternative hypothesis is θ = 2. 6 Concluding Remarks We have suggested in this article an alternative method of adjusting the critical values of Simes test, making it more powerful, when the underlying test 13

statistics are known to be equicorrelated normal. The step-up procedure based on these adjusted critical values, which is shown to control the FDR, provides a modification of the BH procedure and is different from Kwong et al. (2002). This newer modified BH procedure performs much better, compared to the BH procedure, when the correlation is high and a few of the null hypotheses are actually true, as opposed to Kwong et al. s version of the modified BH procedure that is designed to work well for small correlation and small number of true null hypotheses. The normality and equal-correlation assumptions are very crucial in our proposed modifications of Simes test and the BH procedure, as it is true in Kwong et al. (2002) also. Without these assumptions, that is, for other distributions and/or unequal-correlation cases, it seems difficult to propose such modifications. However, simulation results show that for unequal correlated cases, if the conservative (small) correlation estimates are used for deriving the modified critical values, the proposed EBH procedure can still control FDR at n 0 /nα for multivariate normal distributions. The EBH procedure performs well also for multivariate t distributions if the modified critical values in terms of p-value obtained based on normal distribution are used. It is important to point out that the computing time involved in obtaining the modified critical values increases dramatically as n increases, as they require evaluations of joint probabilities of the ordered components of a dependent random vector. It took us about 10 hours to get these critical values for n = 6 using R on a Pentium III 800MHz PC. While it limits our ability to further study the proposed procedure for larger number of tests given the present state of computing facilities available to us, it does not, however, limit the scope of this procedure, as quite often in practice, particularly in safety assessment and pharmacology studies in drug development, small number of statistics are involved; see, for example Zhang et al. (1997) and Dmitrienko et al. (2003). We have tried with the algorithm in Kwong and Liu (2000), but no improvement in terms of computing time is noted over the one in Dunnett and Tamhane (1992) we have used here. A faster algorithm for evaluating joint probabilities of the ordered components of a dependent random vector needs to be formulated. Or, an approximation formula needs to be developed to accurately calculate the adjusted Simes critical values for large n. 7 Acknowledgements We thank an Associate Editor and two referees for their valuable comments. 14

8 Appendix Lemma 1 For any fixed θ < θ, Φ(x θ ) Φ(x θ) is an increasing functions of x. Lemma 1 follows from the TP 2 (totally positive of order two) property of Φ(x θ) in (x, θ); see Karlin and Rinott (1980) and Das Gupta and Sarkar (1984). The next lemma will be proved using certain results on multivariate totally positive of order two (MTP 2 ) distributions. For the sake of a better understanding of the proof, we will now recall the definition of the MTP 2 property, due to Karlin and Rinott (1980), and then briefly review some basic related results that will be used in the proof. Definition. A non-negative real-valued function f(x 1,..., x n ) defined on R n is MTP 2 in (x 1,..., x n ) if, for any two points (x 1,..., x n ) and (y 1,..., y n ) in R n, f(max(x 1, y 1 ),..., max(x n, y n ))f(min(x 1, y 1 ),..., min(x n, y n )) f(x 1,..., x n )f(y 1,..., y n ). When n = 2, this is referred to as the totally positive of order two (TP 2 ) property of Karlin (1968). Result A.1. If f is MTP 2 in (x 1,..., x n ), then the ratio f(x 1,..., x k, x k+1,..., x n) f(x 1,..., x k, x k+1,..., x n ) is increasing in (x 1,..., x k ), for any fixed (x k+1,..., x n ) and (x k+1,..., x n) with x i > x i for i = k + 1,..., n. Result A.2. If ψ 1 (x 1,..., x n ) and ψ 2 (y 1,..., y n ) are both MTP 2, then the product ψ 1 ψ 2 is MTP 2 in (x 1,..., x n, y 1,..., y n ). The above two results follow easily from the definition of MTP 2. The MTP 2 properties of two particular functions arising in the proof of our next lemma follow from Result A.2. The first function is i=1 φ(x i θ i ), where φ(x θ) is the density of N(θ, 1) and is TP 2. The other function is I(x 1 x n ) = n 1 i=1 I(x i x i+1 ), where I(x y), which is 1 if x y, and 0 otherwise, is also TP 2. 15

Result A.3. If f(x 1,..., x n ) is MTP 2, then n f(x 1,..., x n ) dx i i=k+1 is MTP 2 in (x 1,..., x k ). A proof of this is given in Karlin and Rinott (1980). Lemma 2 Let X N n (Θ, I n ). Then, for any fixed a 1 a n, P Θ {X (1) a 1 + x,..., X (n) a n + x} P 0 {X (1) a 1 + x,..., X (n) a n + x} (14) is an increasing function of x. Proof. The joint cdf of (X (1),..., X (n) ) at (x 1,..., x n ), under any Θ, is given by P Θ {X (1) x 1,..., X (n) x n } = k i1,...,i n (Θ)P Θ {X i1 x 1,..., X in x n X i1 X in }, i 1,...,i n P (15) where P is the set of all permutations (i 1,..., i n ) of (1,..., n) and k i1,...,i n (Θ) = P Θ {X i1 X in }. When Θ = 0, the conditional probabilities in (15) are all equal. Therefore, the ratio of the probabilities in (14) can be written as i 1,...,i n P k i1,...,i n (Θ) P Θ{X i1 x 1,..., X in x n X i1 X in } P 0 {X i1 x 1,..., X in x n X i1 X in }. (16) We will now prove that the ratio of the conditional probabilities in (16), for each permutation (i 1,..., i n ), is increasing in (x 1,..., x n ). The lemma follows once this is proved. We will, however, give a proof of this result in the following only when (i 1,..., i n ) = (1,..., n); it can be proved similarly for other permutations. The joint density of (X 1,..., X n ) conditional on X 1 X n, under any Θ, is given by n g Θ (x 1,..., x n ) = k1,...,n(θ) 1 φ(x i θ i )I(x 1 x n ). (17) i=1 16

Hence, we have P Θ {X 1 x 1,..., X n x n X 1 X n } n n = I(y i x i )g Θ (y 1,..., y n ) dy i. i=1 As discussed above following Result A.2, g Θ (y 1,..., y n ) is MTP 2 in (y 1,..., y n, θ 1,..., θ n ) and n i=1 I(y i x i ) is MTP 2 in (y 1,..., y n, x 1,..., x n ). Hence, from Result A.3, we see that the conditional probability i=1 P Θ {X 1 x 1,..., X n x n X 1 X n } is MTP 2 in (x 1,..., x n.θ 1,..., θ n ). This implies, as stated in Result A.1, that the ratio P Θ {X 1 x 1,..., X n x n X 1 X n } P 0 {X 1 x 1,..., X n x n X 1 X n } is increasing in (x 1,..., x n ). This completes the proof. References Benjamini, Y. and Y. Hochberg (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 57, 289 300. Benjamini, Y. and W. Liu (1999). A step-down multiple hypotheses testing procedure that controls the false discovery rate under independence. Journal of Statistical Planning and Inference 82, 163 170. Benjamini, Y. and D. Yekutieli (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 29, 1165 1188. Das Gupta, S. and S. K. Sarkar (1984). Inequalities in Statistics and Probability, Volume 5 of IMS Lecture Notes Monograph Series, Chapter On TP2 and log-concavity, pp. 54 58. Institute of Mathematical Statistics. Dunnett, C. W. and A. C. Tamhane (1992). A step-up multiple test procedure. Journal of the American Statistical Association 87, 162 170. Genovese, C. and L. Wasserman (2002). Operating characteristics and extensions of the false discovery rate procedure. Journal of the Royal Statistical Society, Series B 64, 499 517. Hochberg, Y. and A. C. Tamhane (1987). Multiple Comparison Procedure. John Wiley & Sons. Hsu, J. C. (1996). Multiple Comparisons: Theory and Methods. Chapman & Hall. Karlin, S. (1968). Total Positivity. Stanford, CA: Stanford University Press. Karlin, S. and Y. Rinott (1980). Classes of orderings of measures and related correlation inequalities. I. multivariate totally positive distributions. Journal of Multivariate Analysis 10, 467 498. 17

Kwong, K. S., B. Holland, and S. H. Cheung (2002). A modified Benjamini- Hochberg multiple comparisons procedure for controlling the false discovery rate. Journal of Statistical Planning and Inference 104, 351 362. Kwong, K. S. and W. Liu (2000). Calculation of critical values for Dunnett and Tamhane s step-up multiple test procedure. Statistics & Probability Letters 49, 411 416. Sarkar, S. K. (1998). Some probability inequalities for ordered MTP 2 random variables: a proof of the Simes conjecture. Annals of Statistics 26 (2), 494 504. Sarkar, S. K. (2002). Some results on false discovery rate in stepwise multiple testing procedures. Annals of Statistics 30, 239 257. Sarkar, S. K. (2004). FDR-controlling stepwise procedures and their false negatives rates. Journal of Statistical Planning and Inference 125, 125 137. Sarkar, S. K. (2005). False discovery and false non-discovery rates in singlestep multiple testing procedures. To appear in Annals of Statistics. Sarkar, S. K. and C.-K. Chang (1997). The Simes method for multiple hypothesis testing with positively dependent test statistics. Journal of the American Statistical Association 92, 1601 1608. Simes, R. J. (1986). An improved Bonferroni procedure for multiple tests of significance. Biometrika 73, 751 754. Storey, J. D. (2002). A direct approach to false discovery rates. Journal of the Royal Statistical Society, Series B 64, 479 498. Storey, J. D. (2003). The positive false discovery rate: a bayesian interpretation and the q-value. Annals of Statistics 31 (6), 2013 2035. Troendle, J. F. (2000). Stepwise normal theory multiple test procedures controlling the false discovery rate. Journal of Statistical Planning and Inference 84, 139 158. Yekutieli, D. and Y. Benjamini (1999). Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. Journal of Statistical Planning and Inference 82, 171 196. 18