Controlling Bayes Directional False Discovery Rate in Random Effects Model 1

Transcription

1 Controlling Bayes Directional False Discovery Rate in Random Effects Model 1 Sanat K. Sarkar a, Tianhui Zhou b a Temple University, Philadelphia, PA 19122, USA b Wyeth Pharmaceuticals, Collegeville, PA 19426, USA Abstract Starting with a decision theoretic formulation of simultaneous testing of null hypotheses against two-sided alternatives, a procedure controlling the Bayesian directional false discovery rate (BDFDR) is developed through controlling the posterior directional false discovery rate (PDFDR). This is an alternative to Lewis and Thayer (2004) with a better control of the BDFDR. Moreover, it is optimum in the sense of being the non-randomized part of the procedure maximizing the posterior expectation of the directional per-comparison power rate given the data, while controlling the PDFDR. A corresponding empirical Bayes method is proposed in the context of one way random effects model. Simulation study shows that the proposed Bayes and empirical Bayes methods perform much better from a Bayesian perspective than the procedures available in the literature. 1 Introduction In simultaneous testing of null hypotheses against two-sided alternatives, it is often required to make directional decisions corresponding to those that are rejected. This may result in Type III errors. For instance, in testing the null hypothesis H i : θ i = θ i0 against the corresponding alternative K i : θ i < θ i0 or K + i : θ i > θ i0, simultaneously for i = 1,..., n, a Type III error occurrs once H i is rejected if θ i < θ i0 (or θ i > θ i0 ) is the true situation but one falsely claims that K + i (or K i ) is true. Controlling an error rate measuring directional errors would be a desirable objective in such a multiple testing situation. 1 Research of the first author was supported by NSF Grants DMS and DMS The work of the second author, who worked under the first author s supervision for her PhD, was supported by NSF Grant DMS MSC : Primary: 62J15, 62F03; Secondary: 62F15, 62C10, 62C12 Keywords: Multiple hypotheses testing; Directional decisions; Bayesian decision theory; False discovery rate 1

2 Different error rates, directional as well as non-directional and from frequentist as well as Bayes points of view, can be defined in terms of the different outcomes given in the following table. Table 1: Outcomes in testing n null hypotheses against two-sided alternative hypotheses True Situation Decision accept H accept K accept K + θ = θ 0 U V 1 V 2 n 0 θ < θ 0 T 1 S 1 S 2 n θ > θ 0 T 2 S 3 S 4 n + Total A R 1 R 2 n The quantity S 2 + S 3 is the number of directional (Type III) errors that occurred among the total number of rejections R = R 1 + R 2, with the ratio DFDP = S 2 + S 3 R 1, (1) where R 1 = max(r, 1), representing the pure Directional False Discovery Proportion. It is an analog of FDP = V 1 + V 2 R 1, (2) the False Discovery Proportion defined only in terms of the Type I errors. The Mixed Directional False Discovery Proportion is defined by combining these two proportions as follows MDFDP = FDP + DFDP. (3) Two types of directional error rates in the FDR framework, the Directional False Discovery Rate (DFDR) = E(DFDP) and the Mixed Directional False Discovery Rate (MDFDR) = E(MDFP), have been defined by Benjamini et al. (1993) from a frequentist point of view, considering these expectations with respect to data given parameters. Benjamini and Yekutieli (2005) have shown that that these error rates can be controlled by suitably 2

3 augmenting the original FDR procedure of Benjamini and Hochberg (1995), providing a proof of what is conjectured in Benjamini and Hochberg (2000), Shaffer (2002) and Williams et al. (1999). The idea of controlling directional false discoveries from a Bayesian point of view was considered by Lewis and Thayer (2004) and Shaffer (1999). Considering a one-way random effects model, which provides a Bayesian framework, Lewis and Thayer have shown that the DFDR, with its expectation taken with respect to both data and parameters, can be controlled by a multiple decision rule that minimizes a per-comparison Bayes risk defined in terms of an additive 0-1 loss function, providing a theoretical support of Shaffer (1999) s simulation-based findings. It is often argued that a point null hypothesis is never true and that it is only conventionally emphasized; see, for example, Jones and Tukey (2000), Shaffer (2002), and Williams et al. (1999). Or, when we use a Bayesian model with continuous prior, as in Shaffer (1999) and Lewis and Thayer (2004), the probability of a point null is zero. In such instances, the DFDR and MDFDR are same. In this article, we take another look at the problem considered by Lewis and Thayer (2004) of developing a procedure that controls the DFDR from a Bayesian perspective. Starting with a decision theoretic formulation of the underlying multiple testing problem in a Bayesian framework and calling E(DFDP) the Bayes DFDR (BDFDR) when the expectation is taken with respect to both data and parameter, we construct an alternative procedure that controls the BDFDR. Our procedure is developed through controlling the posterior DFDR (PDFDR), which is the conditional expectation of the DFDP given the parameter. While this is similar to what Lewis and Thayer (2004) also did, this new procedure, however, offers a better control of the BDFDR. Moreover, it is optimum in that it maximizes the posterior expectation E {(S 1 + S 4 )/n} given the data, while controlling the PDFDR. We will refer to this conditional expectation as the Posterior Directional Per- Comparison Power Rate (PDPCPR). We organize the article as follows. In Section 2, we present a Bayesian decision theoretic formulation of multiple hypothesis testing problem with directional decisions, and formulate the BDFDR and PDFDR. The new procedure controlling the BDFDR is then developed and properties of it are discussed in Section 3. In Section 4, we go back to the example of one-way random effects model considered by Lewis and Thayer (2004) and Shaffer (1999) and illustrate our Bayes procedure in that context, first assuming that both within and between variances are known and then considering 3

4 the case when the between variance is unknown. With unknown between variance, we incorporate the estimate of it considered by Lewis and Thayer (2004) and Shaffer (1999) into our Bayes procedure and discuss the behavior of the resulting new empirical Bayes procedure with respect to the actual value of the between variance. In Section 5, we discuss the findings of a simulation study comparing our Bayes and empirical Bayes procedures with other procedures that control the BDFDR for the multiple testing problem with directional decisions in a one-way random effects model. We demonstrate through this study that our proposed Bayes and empirical Bayes procedures provide much better control of directional false discoveries from a Bayesian perspective than those that are currently available in the literature. The article concludes with some final remarks. 2 A decision theoretic formulation In this section, we first present a general decision theoretic formulation of a multiple testing problem with directional decisions allowing the decisions to be randomized before we restrict ourselves in the rest of the paper to only non-randomized decisions. Suppose we have a multiple testing problem involving a set of statistics X = (X 1,..., X n ) with a probability distribution P θ with θ = (θ 1,..., θ n ) Θ R n, which is being used to test H i : θ i Θ 0 against K i : θ i Θ or K + i : θ i Θ +, simultaneously for i = 1,..., n. Let d i = 0, 1 or 1 according as H i, K i or K + i is accepted. Then, d = (d 1,..., d n ) represents a decision vector with D = {(d 1,..., d n ) : d i = 0, 1 or 1 i} being the decision space. Given X = x, we consider choosing the decision vector d according to a probability distribution over D: δ(d x) = n [ {δ 0 i (x)} I(di=0) {δ i (x)}i(d i= 1) {δ ] + i (x)}i(d i=1), d D, (4) i=1 for some 0 δ i (x), δ+ i (x) 1, i = 1,..., n, allowing the decisions to be made independently of each other given X. The vector δ(x) = (δ1 (X), δ 1 + (X), 4

5 ..., δn (X), δ n + (X)) is referred to as a multiple decision rule or multiple testing procedure. If 0 < δ i (X) < 1 or 0 < δ+ i (X) < 1, for at least one i, then δ(x) is randomized; otherwise, it is non-randomized. The main objective in a multiple testing problem is to determine δ(x), the choice of which is typically assessed based on a risk as measured by averaging a loss L(θ, δ), which one incurs in selecting d, over uncertainties. In a frequentist approach, only the uncertainty in X given θ is considered; whereas, in a Bayesian approach one would like to further utilize prior information on θ. Let h = (h 1,..., h n ), with h i = 0, 1 or 1 according as θ i Θ 0, θ i Θ or θ i Θ +, represent the true state of nature. Given Q(h, d), a measure of error providing an overall discrepancy between h and d, the loss function is given by L(θ, δ(x)) = d D Q(h, d)δ(d X). (5) The frequentist risk is given by R δ (θ) = E X θ L(θ, δ(x)), (6) and, given a prior distribution of θ, the posterior risk is and the Bayes risk is Π δ (X) = E θ X L(θ, δ(x)), (7) r δ = E θ R δ (θ) = E X Π δ (X). (8) Among different possible choices of Q(h, d) providing different concepts of error rates in multiple testing, the one we are interested in here is the following: n i=1 DFDP = I(h id i = 1) n i=1 I( d i = 1) 1, (9) 5

6 with the corresponding loss (as defined in 5) given by n i=1 I(h id i = 1) n i=1 I( d i = 1) 1 δ(d X) d D = [ { 1 } I(h i = +1) + I(h i = 1) J:>0 i J i J { + δ i (X) δ + i (X) }] [1 δ i (X) δ+ i (X)] i J i J + i J c = { 1 } I(h i = +1) + I(h i = 1) φ J,J +(X), (10) J:>0 i J i J + where J = {i : d i = 1}, J + = {i : d i = 1}, J = J J +, is the cardinality of J, and φ J,J +(X) = δ i (X) δ + i (X) [1 δ i (X) δ+ i (X)] (11) i J i J + i J c is the probability, given X, of rejecting the set of null hypotheses {H i, i J}. Under a prior distribution of θ, the posterior DFDR (PDFDR) is given by PDFDR = E θ X (DFDP) = { 1 s + i (X) + } s i (X) J:>0 i J i J + φ J,J +(X), (12) where s i (X) = P {h i = 1 X} and s + i (X) = P {h i = +1 X}, the posterior probabilities of negative and positive alternatives, respectively. The Bayes DFDR (BDFDR) is the expectation of (12) with respect to X. A non-randomized multiple testing procedure controlling the BDFDR is going to be constructed in the next section through controlling the PDFDR under a continuous prior. Before we do that, it is important to note that for a non-randomized rule δ, d can be replaced by δ in the above formulation. Furthermore, since the prior is continuous, the posterior probability of h i = 0 is zero. 3 Controlling Bayes directional FDR Let s i (X) = min{s i (X), s+ i (X)}, i = 1,..., n. Then, for any procedure with J = {i : s + i (X) s i (X)} and J + = {i : s i (X) s+ i (X)}, the PDFDR in 6

7 (12) can be equivalently expressed as P DF DR = { 1 } s i (X) φ J (X), (13) J:>0 with φ J (X) φ J,J +(X). Let s 1:n(X)... s n:n (X) be the ordered values of s i (X), i = 1,..., n and H i:n be the the null hypothesis corresponding to s i:n. Define A j (X) as the average of the first j smallest s-values, that is, A j (X) = 1 j i J j s i:n (X). (14) i=1 Our proposed new procedure controlling the BDFDR is then given in the following: Theorem 1 Let K(X) = { max {j : Aj (X) α}, if the maximum exists 0, otherwise. (15) Given K(X) = k, reject {H 1:n,..., H k:n } and accept the rest. Among these rejected hypotheses, positive sign decision is made if s i (X) < s+ i (X) and negative sign decision is made otherwise. The BDFDR of this procedure is less than or equal to α. Proof. The theorem follows by noting that the PDFDR of the procedure in this theorem is A K (X), which is less than or equal to α. It is important to note that, in the procedure of Lewis and Thayer (2004), a control of the BDFDR is achieved by selecting (with probability one) J and J +, given X, as follows: J = {i : s + i (X) α}, J + = {s i (X) α}. (16) Whereas, these subsets are chosen in our procedure as follows: where J = {i J : s + i (X) s i (X)}, 1 J + = {i J : s i (X) s+ i (X)}, min{s i (X), s+ i (X)} α. (17) i J 7

8 i J s + i i J + s i Clearly, the union of J and J +, that is, the set of rejected hypotheses, is smaller in (16) compared to that in (17), implying that our procedure is more powerful in the sense of allowing more rejections, yet keeping the BDFDR controlled at the same level. The formula for the PDFDR in (12) allows one to construct other BDFDR procedures. For instance, one might consider choosing J and J + by sepa- 1 1 rately controlling each of (X) and (X). This will J J + again be more powerful than Lewis and Thayer (2004), even though it is not going to be better than our procedure. In fact, one can see from (12) that our procedure provides in a certain sense an optimum choice of these subsets subject to a control of the PDFDR. More specifically, we have the following result. Proposition 1 The procedure in Theorem 1 is the non-randomized part of the procedure that maximizes the posterior directional per-comparison power rate (PDPCPR), defined as PDPCPR = E θ X ( S1 + S 4 n 1 ), (18) among all procedures with J = {i : s + i (X) s i (X)} and J + = {i : s i (X) s + i (X)} and subject to a control of the PDFDR at α. Proof. For any procedure φ, PDPCPR = { 1 [ 1 s + i n (X)] + [ 1 s i (X)]} φ J,J +(X) J:>0 i J i J + = { } 1 [1 s i (X)] φ J (X), (19) n J:>0 i J which, given s i:n (X), i = 1,..., n, can be expressed as PDPCPR = { } 1 [1 s i:n (X)] φ J (X). (20) n J:>0 Also, given s i:n (X), i = 1,..., n, the PDFDR is PDFDR = { } 1 s i:n (X) φ J (X). (21) J:>0 8 i J i J

9 Hence, it follows from the Neyman-Pearson lemma that the following procedure { 1 if φ 0 i J J(X) = [1 s 1 i:n(x)] > C α i J s i:n(x) 0 if i J [1 s 1 i:n(x)] < C α i J s (22) i:n(x), with some C α > 0 satisfying { } 1 s i:n (X) φ 0 J(X) = α, (23) J:>0 i J maximizes the PDPCPR subject to a control of the PDFDR at level α, given s i:n (X), i = 1,..., n, and hence given X. Since, for every > 0, [1 s i:n (X)] > C α s i:n (X) i J i J 1 s i:n (X) <, (24) + C α i J i J s i:n(x) and we can always find a C α the φ 0 1 J (X) in (22) is based on satisfying (23). The non-randomized part of this is our procedure in Theorem 1. Remark 1. Obviously, the randomized procedure in Proposition 1 has a better control of the PDFDR and has higher PDPCPR than its nonrandomized part. Nevertheless, we propose using the slightly more conservative non-randomized procedure in order to avoid the difficulty in offering a practical justification for using a randomized test instead of its nonrandomized part. A concept of power, called the average power, is defined from a frequentist point of view in terms of the proportion, (S 1 + S 4 )/(n + n + ), of alternatives that are correctly rejected [Shaffer (1999) and Dudoit et al. (2003)]. This proportion is same as (S 1 + S 4 )/n, when n 0 = 0. This is what Lewis and Thayer (2004) consider before taking its expectation with respect to both data and parameter to define what they call the directional per-comparison power rate as a measure of power from a Bayesian perspective. We will use the same measure in this paper, but refer to it as the Bayes directional percomparison power rate (BDPCPR), to compare the power performance of different procedures controlling the BDFDR in the context of the one-way random effects model considered by Lewis and Thayer (2004). 9

10 4 One-way random effects model Consider data from m independent studies with w j being the number of observations from study j. The vector of sample means for these studies is X = (X 1,..., X m ). Assume that the X j s are independently distributed with ) X j µ j, σ 2 N (µ j, σ2, j = 1,..., m. (25) w j The population means µ 1,..., µ m are considered random and assumed to be independently and identically distributed as µ j θ, τ 2 N(θ, τ 2 ), j = 1,..., m. (26) We consider θ and σ 2 to be known. Regarding τ 2, we will first assume that it is known and illustrate our BDFDR procedure. Then, assuming it unknown, we develop our empirical BDFDR procedure in terms of an estimate of it. Based on Bayes theorem, it is known that, conditionally given X = x, θ, σ 2, and τ 2, µ j s are independently distributed as follows: where and µ j x j, θ, σ 2, τ 2 N(ˆµ j, v j ), j = 1,..., m, (27) ˆµ j = τ 2 x j + (σ 2 /w j )θ τ 2 + σ 2 /w j v j = τ 2 σ 2 /w j τ 2 + σ 2 /w j. We consider the following multiple testing problem involving all pairwise comparisons among the µ j s: H ij : µ i µ j = 0 v.s. K ij : µ i µ j < 0 or K + ij : µ i µ j > 0, (28) for all 1 i < j m. The conditional marginal distributions of µ i µ j given x are µ i µ j x N (ˆµ i ˆµ j, v i + v j ), 1 i < j m, (29) 10

11 from which we can determine s ij and s+ ij as follows: ( ) s ij (x) = 1 s+ ij (x) = 1 Φ ˆµi ˆµ j, (30) vi + v j with Φ being the cdf of N(0, 1), before developing our Bayes procedure as in Theorem 1, assuming, of course, that θ, σ 2 and τ 2 are known. Note that the procedure of Lewis and Thayer (2004) rejects H ij in favor of K + ij (or K ij ) if ˆµ i ˆµ j vi + v j Z α (or Z α ), (31) that is, if s ij (x) (or s+ ij (x)) α, and accepts H ij otherwise. With unknown τ 2, we follow the idea in Shaffer (1999) and Lewis and Thayer (2004), and consider estimating τ 2 using the estimator ˆτ 2 = (F 1)σ 2 [(m 1) w j ]/[( w j ) 2 w 2 j ], (32) where F = max { w j (x j x) 2 /[(m 1)σ 2 ], 1}, and ignore the estimation of θ and σ. The resulting procedure obtained by replacing τ 2 by this ˆτ 2 is our proposed empirical Bayes procedure; of course, when ˆτ = 0, we accept all null hypotheses. To see how this empirical Bayes procedure performs in terms of controlling the BDFDR when τ 2 is replaced by its estimate ˆτ 2, we carried out a simulation study with α = We noticed that the BDFDR for this procedure is a decreasing function of τ 2, and when τ 0, the BDFDR is not controlled for m between 4 and 40. The limiting behavior of the BDFDR for τ 0 is presented in Table 2. Table 2: The limiting BDFDR as τ 2 0 for our empirical Bayes procedure m BDFDR Standard errors are.0001, based on 100,000 replications. To achieve BDFDR α for small τ 2, a second critical value F is determined through simulation, as in Table 3. 11

12 Table 3: Value of F in New EB to control DFDR at.025 m F When ˆτ 2 is small or F F, indicating that the µ j s are very close to each other, we recommend, as in Lewis and Thayer (2004), that all the null hypotheses be accepted, otherwise, our new empirical Bayes procedure be conducted by replacing τ by ˆτ. Compared with the corresponding values in Lewis and Thayer (2004), the values in Table 2 are the same, while those in Table 3 are slightly smaller. Remark 2. Before we proceed to numerically investigate in the next section the performance of our proposed procedures relative to others in the context of the one-way random effects model, including the frequentist procedures (assuming fixed effects) considered in Lewis and Thayer (2004), we want to make a few observations on these frequentist procedures. A frequentist procedure is one that is meant to control the DFDR and is developed typically by suitably augmenting a FDR procedure for testing the null hypotheses against the two-sided alternatives; see, for example, Benjamini and Hochberg (2000), Benjamini and Yekutieli (2005), Shaffer (2002) and Williams et al. (1999). For instance, with P ij = 2 min(p ij, 1 P ij ), the two-sided p-values corresponding to the test-statistics Z ij = X i X j, 1 i < j m, (33) σ 1 w i + 1 w j and the one-sided p-values P ij = 1 Φ(Z ij ), the following procedure is a directional version of the α-level FDR Bonferroni procedure: Directional Bonferroni Procedure: (i). Apply the Bonferroni procedure at level α to test the n = m(m 1)/2 null hypotheses H ij against the corresponding two-sided alternatives. (ii). Reject H ij in favor of K ij if P ij < α/n and P ij > 1/2. (iii). Reject H ij in favor of K + ij if P ij < α/n and P ij < 1/2. Assuming that n 0 = 0, the above procedure controls the DFDR at α/2. 12

13 The other frequentist DFDR procedure considered by Lewis and Thayer (2004) is the following directional version of the original BH FDR procedure [Benjamini and Yekutieli (2005)]: Directional BH Procedure: (i). Apply the BH procedure at level α to test the n = m(m 1)/2 null hypotheses H ij against the corresponding two-sided alternatives. Let R be the number of null hypotheses rejected. (ii). Reject H ij in favor of K ij if P ij < Rα/n and P ij > 1/2. (iii). Reject H ij in favor of K + ij if P ij < Rα/n and P ij < 1/2. It is important to note that, even though Lewis and Thayer (2004) considered it, the above directional BH procedure is not known to control the DFDR in the present context. This is due to the fact that the underlying p-values are dependent, unlike in Benjamini and Yekutieli (2005) where they are assumed to be independent. Had these p-values been independent, the DFDR would have been controlled at α/2, when n 0 = 0, as in the case of the directional Bonferroni procedure. This is why Lewis and Thayer (2004) have considered controlling the DFDR at level α through a 2α-level FDR procedure, and we will do that too in this paper. Benjamini and Yekutieli (2005) have proposed an alternative approach to controlling the DFDR in the dependence case based only on the one-sided p-values. Again, there is no guarantee that this alternative procedure is going to work, as the required positive regression dependence condition of the one-sided p-values is not met in the present context involving pairwise differences. Thus, the directional Bonferroni procedure seems to be the only frequentist procedure that is known to control the DFDR, and hence the BDFDR (with random effects), in the present context. Nevertheless, we will follow Lewis and Thayer (2004) and consider both this and the directional BH procedure, along with the unadjusted frequentist procedure and the Bayes procedure of Lewis and Thayer (2004), and compare them numerically with our proposed Bayes and empirical Bayes procedures. The Empirical Bayes procedure of Lewis and Thayer (2004) is not considered, as its performance is very close to that of their Bayes procedure. Also, it should be noted that the unadjusted procedure is not going to control the DFDR. It is kept in our comparative study, as in Lewis and Thayer (2004), only to provide a complete picture of how different procedures would perform. 13

14 5 Numerical Comparisons Simulation studies in a one-way random effects setup were conducted based on 25,000 replications of x and µ, with m = (2, 3, 4, 5, 10, 50), σ 2 /w j = 1 and τ = (.01,.50, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0, 8.0). The values of both the BDFDR and BDPCPR were computed for each of the proposed Bayes and empirical Bayes procedures (referred to as New Bayes and New EB, respectively), the unadjusted procedure (labelled Unadjusted), the directional Bonferroni procedure (labelled Bonferroni), the directional BH procedure (labelled BH), and Lewis-Thayer s Bayes procedure (labelled LT), all designed to control the BDFDR at α = Figures 1 and 2 present the comparisons in terms of the BDFDR and BDPCR, respectively, among these different procedures with m = 10. The unadjusted procedure does not control the BDFDR, as expected; although, it seems to do so for large τ. All other procedures control the BDFDR, interestingly, including the BH procedure for which, as discussed before, we have no theory to support its control of the BDFDR. While the New Bayes procedure is quite conservative for small values of τ (less than.5), it provides much better control of the BDFDR with a value close to 0.025, compared to any other procedure, when τ > 1. The performance of the New EB procedure in terms of controlling the BDFDR is very similar to that of the New Bayes procedure, except for small τ when it appears to work the best. The LT procedure is always very conservative whatever the value of τ is, although it is slightly better than the BH procedure for large τ. In terms of power (BDPCPR), when τ is small, there is not much difference among all of the procedures. However, when τ is not small, the New Bayes and Empirical Bayes procedures are both noticeably more powerful than every other procedure. Figure 3 compares the powers of the New Bayes and the LT procedures for different number of means. The New Bayes procedure is the same as the LT procedure when there is only one comparison (m = 2). With more than one comparison, the New Bayes procedure is uniformly more powerful than the LT procedure for different τ and the power gain is increasing in m. Figure 4 presents a comparison of powers between the New EB and the BH procedures for different number of means. When there is only one comparison (m = 2), the BH procedure is more powerful than New EB procedure. With more than one comparison (m > 2), however, the New EB procedure is more powerful and the power gain is increasing in m. 14

15 In conclusion, simulation results confirm that new Bayes and empirical Bayes methods proposed in this paper are more powerful directional error controlling procedures for pairwise comparisons in a one-way random effects model than those available in the literature. 6 Concluding remarks We like to clarify here a few points about this paper as prompted by the referees. First, we should emphasize that our proposed procedure is not claimed to be a Bayes decision procedure in the sense of minimizing Bayes risk under a certain loss function. In fact, in the Bayesian paradigm, there is no concept of controlling a posterior expected loss, instead of trying to minimize it. The idea of controlling an error rate is basically a frequentist notion, even when it is defined by averaging over both parameters and data. On the other hand, the procedure in Lewis and Thayer (2004) is a Bayes decision procedure that is developed using a loss function of the form L(λ) = i=1 {I(h id i = 1) + λi(h i 0, d i = 0)} and minimizing per-comparison Bayes risk r = E X,θ L(λ) with λ = α. Second, it is worth mentioning that it is a problem with almost all current multiple comparison procedures that, for large enough values of τ, even small differences with conventional p-values approaching 1.0 may be declared significant because of great majority of p- values for the pairwise comparisons will be very close to zero. Our procedure, by not controlling a per comparison error rate, does not have this problem. 7 Acknowledgements We thank the referees for their valuable comments. References Benjamini, Y. and Y. Hochberg (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 57, Benjamini, Y. and Y. Hochberg (2000). On the adaptive control of the false 15

16 discovery rate in multiple testing with independent statistics. Journal of educational and behavioral statistics 25, Benjamini, Y., Y. Hochberg, and Y. Kling (1993). False discovery rate control in pairwise comparisons. Research Paper, Dept. of Statistics and O.R., Tel Aviv University. Benjamini, Y. and D. Yekutieli (2005). False discovery rate-adjusted multiple confidence intervals for selected paramaters. Journal of the American Statistical Association 100, Dudoit, S., J. P. Shaffer, and J. C. Boldrick (2003). Multiple hypothesis testing in microarray experiment. Statistical Science 18 (1), Jones, L. V. and J. W. Tukey (2000). A sensible formulation of the significance test. Psychological Method 5, Lewis, C. and D. T. Thayer (2004). A loss function related to the fdr for random effects multiple comparison. Journal of Statistical Planning and Inference 125, Shaffer, J. P. (1999). A semi-bayesian study of Duncan s Bayesian multiple comparison procedure. Journal of Statistical Planning and Inference 82, Shaffer, J. P. (2002). Multiplicity, directional(type III) errors, and the null hypothesis. Psychological Methods 7, Williams, V. S., L. V. Jones, and J. W. Tukey (1999). Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. Journal of educational and behavioral statistics 24,

17 0.04 Unadjusted New Bayes New EB LT BH Bonferroni 0.03 BDFDR τ Figure 1: The BDFDR for all pairwise comparisons in one-way random effects setup with m=10 and σ2 w j = 1 17

18 BDPCPR Unadjusted New Bayes New EB LT BH Bonferroni τ Figure 2: The BDPCPR for all pairwise comparisons in one-way random effects setup with m=10 and σ2 w j = 1 18

19 0.20 m=2 m=3 m=4 m=5 m=10 m= Power Bayes Power LT τ Figure 3: The difference in the BDPCPR between the New Bayes procedure and Lewis and Thayer s Bayes procedure (Power Bayes - Power LT) with = 1 as a function of τ for pairwise comparisons of m means. σ 2 w j 19

20 Power Empirical Bayes Power BH m=2 m=3 m=4 m=5 m=10 m= τ Figure 4: The difference in the BDPCPR between the New Empirical Bayes procedure and Benjamini and Hochberg s procedure (Power Empirical Bayes - Power BH ) with σ2 w j = 1 as a function of τ for pairwise comparisons of m means. 20