A BAYESIAN STEPWISE MULTIPLE TESTING PROCEDURE. By Sanat K. Sarkar 1 and Jie Chen. Temple University and Merck Research Laboratories

Transcription

1 A BAYESIAN STEPWISE MULTIPLE TESTING PROCEDURE By Sanat K. Sarar 1 and Jie Chen Temple University and Merc Research Laboratories Abstract Bayesian testing of multiple hypotheses often requires consideration of all configurations of the true and false null hypotheses, which is a difficult tas when the number of hypotheses is even moderately large. In this article, a Bayesian stepwise approach is proposed. After ordering the null hypotheses in terms of their marginal Bayes factors, a determination is made of all possible configurations of true and false ordered null hypotheses. The most plausible configuration is then tested in a stepwise manner starting with the intersection of all the null hypotheses. The present Bayesian approach provides considerable reduction in the size of the set of families within which we restrict our posterior search for the right family of true and false null hypotheses, from to only + 1, given null hypotheses. The hierarchical prior structure is considered in general. The stepwise Bayes factors are derived in situations involving point null hypotheses as well as those arising in one-sided testing problems under normal distributional settings. Our procedure when applied to three different examples results in conclusions that are similar to those obtained using an analogous frequentist approach. 1. Introduction. Simultaneous testing of multiple null hypotheses, simply referred to as multiple testing, is a common inference problem in many statistical experiments. For instance, in pharmaceutical investigations of the efficacy of an experimental drug, the statistical assessment of the drug s superiority over existing drugs or over placebo on multiple endpoints is generally carried out through 1 Research was supported in part by NSA Grant MDA AMS 000 subject classification. Primary 6J15, 6F15 Key words and phrases: Hierarchical prior, simultaneous testing, multiple hypotheses, stepwise Bayes factor, marginal Bayes factor. 1

2 a multiple testing formulation. Such testing is often performed using frequentist stepwise multiple test procedures in which the ordered test statistics or the associated p-values are compared with a set of critical values in a stepwise fashion toward identifying the set of true and false null hypotheses. The critical value used in each step incorporates the decision made in the preceding step. Therefore, they provide better control of the familywise error rate, and hence are more powerful, compared to a single-step procedure where the determination of true and false hypotheses are made by simply comparing each test statistic or the corresponding p-value with a single critical value. A considerable amount of research has taen place on stepwise multiple testing from frequentist s viewpoint Hochberg and Tamhane 1987; Dunnett and Tamhane 1991; Dunnett and Tamhane 199; Finner 1993; Westfall and Young 1993; Hsu 1996; Liu 1996; Liu 1997; Holm 1999), with many of the proposed methodologies having been incorporated into statistical pacages Westfall and Tobias 1999; Westfall, Tobias, Rom, Wolfinger, and Hochberg 1999). Basically, there are two types of frequentist stepwise multiple testing procedures that are most commonly used step-up and step-down procedures. After ordering the hypotheses according to increasing values of their test statistics or p-values, a step-down procedure starts with testing the most significant hypothesis and continues until an acceptance occurs or all the hypotheses are rejected. On the other hand, a step-up test starts with testing the least significant hypothesis and continues until a rejection occurs or all the hypotheses are accepted. A generalized version of step-up and step-down procedures has received some attention recently Tamhane, Liu, and Dunnett 1998; Sarar 00). What would be a Bayesian analog of a frequentist stepwise testing? This seems to be an interesting question to answer as no such analog is yet available in the literature. We develop in this article a Bayesian stepwise procedure for multiple testing of null hypotheses that are non-hierarchical, i.e., no hypothesis is implying any other. The procedure involves two main steps, specification of a set of target families of true and false null hypotheses and a stepwise search for the most plausible one of these families. Given data and prior specifications of the parameters, an ordering is established among the null hypotheses according to increasing values

3 of their marginal Bayes factors, which will guide the determination of the target families. Once these target families are formed, the stepwise search is carried out using stepwise Bayes factors toward identifying the most plausible family. Given null hypotheses, the total number of target families is + 1. The idea used in configuring these different families is that, among all families with r false and r true null hypotheses, the one in which the first r ordered null hypotheses are false and the last r ordered null hypotheses are true is the most plausible, where the ordering of hypotheses is made according to an increasing magnitude of their marginal Bayes factors. The stepwise search begins with the target family with no false hypotheses. The strength of evidence of this family against all other families is checed by computing the Bayes factor of the intersection of the hypotheses in the family. If the evidence is strong, the search stops by declaring all of the null hypotheses to be true; otherwise, it declares the null hypothesis with the smallest marginal Bayes factor to be false and goes to the next step. The next step involves the rest target families where the null hypothesis rejected in the beginning step is considered false. Within this group of families, the evidentiary strength of the family in which the previously rejected null hypothesis is false and the rest are true is assessed by the Bayes factor, conditional on this group, of the corresponding intersection hypothesis. If this evidence is strong, the search stops by declaring the null hypothesis with the smallest marginal Bayes factor to be false and the rest to be true; otherwise, it continues to the next step. These conditional Bayes factors are referred to as stepwise Bayes factors in this article. The search continues until a stepwise Bayes factor provides a strong evidence for the corresponding intersection hypothesis. There is an operational similarity between our proposed Bayesian stepwise procedure and classical step-down multiple test procedure. Target families are also formed, of course, in terms of ordered test statistics or p-values, in classical step-down procedure using similar idea. Also the search for the statistically most significant target family in classical step-down procedure proceeds from the family with no false hypotheses toward those with higher number of false hypotheses. In this sense, our proposed method is a Bayesian step-down multiple test procedure. 3

4 It is important to emphasize certain positive features of the proposed method relative to those that exist in the Bayesian literature. Multiple testing is typically viewed as a model selection problem when it is approached from a Bayesian perspective, with different configurations of true and false null hypotheses being the competing models. The existing Bayesian model selection methods when applied to such a formulation of multiple testing, however, have to search for the right model or configuration within the set of all configurations of true and false null hypotheses Bertolino, Piccinato, and Racugno 1995; Berger and Pericchi 1996; Berger 1999; Berger and Pericchi 001, and references therein). This is an increasingly difficult tas when becomes large. In our method, on the other hand, the initial process of specifying the target families, allows us to disregard configurations, which we call families, that are undesirable in the sense of being relatively less evident a posteriori. This provides considerable reduction in the size of the set of families within which we restrict our posterior search for the right family of true and false null hypotheses, from to only + 1. Because of this, the computational tas greatly reduced. Another existing Bayesian approach to multiple testing is to use simultaneous credible regions Gelman and Tuerlincx 000). As opposed to such a single-step Bayesian approach, where the test for a given family does not depend on those for the others, our proposed step-down method offers a systematic improvement of the search for the right family by incorporating information gathered at every step. This seems to be a more effective way of adjusting for the multiplicity in the present multiple testing problem; see Berry 1988); Breslow 1990); and Berry and Hochberg 1999) for discussions on the general issue of handling multiplicity from Bayesian viewpoint. The proposed Bayesian step-down procedure is developed for simultaneous testing of multiple point as well as one-sided null hypotheses against the corresponding complementary alternatives. For these types of null and alternative hypotheses involving parameters of a number of different populations, formulas are derived for both marginal and stepwise Bayes factors based on a dataset that consists of independent samples from the populations and a prior under which the parameters are independent conditional on some hyper-parameter. The hierarchical structure 4

5 of the prior also provides a way of adjusting for multiplicity. These formulas are further developed in detail for situations where means of several normal populations are simultaneously compared with a nown value or with the mean of another population, assuming a common unnown variance of the populations. The scope of our method is illustrated by applying it to three datasets, one in Romano 1977, p. 48), one in White and Froeb 1980) and the other in Steel and Torrie 1980). The formulas for simultaneous testing of multiple point null hypotheses against one-sided alternatives have been developed in Chen and Sarar 00). A number of other papers, many of them recently, have appeared in the literature where Bayesian approach has been taen to address some problems in multiple comparisons. Duncan 1965) proposed a Bayesian decision-theoretic approach for multiple comparisons, in which a decision is made using relative magnitudes of losses due to Type I and Type II errors. An extension of this idea can be found in Waller and Duncan 1969) who used a hyper-prior distribution for the unnown ratio of between-to-within variances. Shaffer 1999) modified Duncan s procedure semi-bayesian) and found considerable similarity in both ris and average power between this modified procedure and the FDR-controlling procedure due to Benjamini and Hochberg 1995). Tamhane and Gopal 1993) derived Bayes decision rules for comparing treatments with a control under an additive overall loss function with either constant or linear loss functions for component losses. Westfall, Johnson, and Utts 1997) studied prior probability adjustment to account for multiple hypotheses and showed that the adjusted posterior probabilities correspond to Bonferroni adjusted p-values. Gopalan and Berry 1998) proposed a Bayesian multiple comparisons procedure for means in terms of posterior probabilities of all possible hypotheses of equality and inequality among means under Dirichlet process priors. The paper is organized as follows. Section provides some preliminaries. Section 3 describes the proposed Bayesian stepwise procedure with necessary formulas for simultaneous testing of multiple point as well as one-sided null hypotheses against the corresponding complementary alternatives. Further developments of the formulas for normal populations using priors recommended in Berger, Bouai, and Wang 5

6 1997a); Berger, Bouai, and Wang 1997b); and Berger, Bouai, and Wang 1999) are given in Section 4 before applying them to three data sets. The article is concluded in Section 5 with some remars and discussion. Proofs of some formulas are presented in the Appendix.. Preliminaries. Suppose that we have a data set X = {X 1,, X } consisting of independent samples X i : n i 1, i = 1,,, from populations and that X i has the following density n i fx i θ i ) = fx ij θ i ), i = 1,...,, where θ = θ 1,..., θ ) Ω R. We are interested in testing H i : θ Θ i against H i : θ Θ i, simultaneously for i = 1,...,, where Θ i Θ i = and Θ i Θ i = Ω. A classical frequentist procedure to individually test H i against H i is typically based on some test statistic T i T i X), with large values of T i indicating rejection of H i. With F i t) denoting the cdf of T i at t, the p-value corresponding to T i is given by P i = 1 F i T i ). To test H i against H i simultaneously for i = 1,...,, a frequentist stepwise procedure is generally applied, where ordered values of T i s or P i s are compared with a set of critical values in a stepwise fashion. These critical values are designed to control some types of error associated with false rejections, such as familywise error rate or false discovery rate Hochberg and Tamhane 1987; Dunnett and Tamhane 1991; Dunnett and Tamhane 199; Benjamini and Hochberg 1995; Hsu 1996; Tamhane and Dunnett 1999; Sarar 00). On the other hand, a Bayesian approach to testing H i against H i utilizes posterior probability of H i given the data and some suitable prior distribution of the parameters. A hierarchical model for the prior Berger 1985, pp ), which is appropriate in order to account for the multiplicity in the present multiple testing problem, is going to be considered in this article. Specifically, let the θ i, i = 1,...,, be independent with the first stage prior distribution π 1 θ i λ). The second stage prior for λ = λ 1, λ ) is π λ) = π 1 λ 1 λ )π λ ). The posterior density of θ given 6

7 X is given by.1) where πθ X) = mx)] 1 fx θ)πθ), fx θ) = fx i θ i ), i=1 and πθ) = i=1 mx) = Ω π 1 θ i λ)π λ)dλ, fx θ)πθ)dθ. To individually test H i against H i from a Bayesian viewpoint, one simply calculates the marginal posterior probability P H i X) = πθ X)dθ Θ i and P H i X) = 1 P H i X), and decides between H i and H i accordingly. Another commonly used approach is to use the Bayes factor, which is the ratio of marginal posterior odds to marginal prior odds of H i to H i, i.e.,.) with B i = P H i X) 1 P H i X).1 π i0 π i0, π i0 = πθ)dθ Θ i being the marginal prior probability of H i. We will call B i the marginal Bayes factor of H i. For testing the intersection of the null hypotheses, that is H = i=1 H i against H = H i=1 i, the Bayes factor in favor of H over H is H B = πθ X)dθ 1 H πθ X)dθ 1.3) H πθ)dθ H πθ)dθ. When λ = λ 1, λ ) is nown and H corresponds to a product space in Ω, as in the case when H i : θ i θ i0 for some nown θ i0, i = 1,...,, then.3) reduces to i=1 B = B i i=1 B iπ i0 i=1 1 π i0) + B i π i0 ] i=1 B. iπ i0 7

8 When an equal probability is assigned to H i and H i, as it is usually done in default Bayesian analysis, then.3) further simplifies to B = i=1 1) B i i=1 1 + B i) i=1 B. i It is worth noting that the use of Bayes factor in the literature is almost exclusively for comparing two competing hypotheses or models), i.e., either hypothesis i or model i) or hypothesis j or model j) is true. Berger and Pericchi 001) provided an introductory, yet comprehensive, review on objective Bayes model selection. Many authors Berger and Pericchi 1996; Berger, Bouai, and Wang 1997b; Dass and Berger 1998) considered Bayesian testing of multiple hypotheses; however, they have formulated the problem in terms of only two extreme cases; i.e., all hypotheses are true H = i=1 H i) versus all hypotheses are not true H = i=1 H i ). Simultaneous testing of H i against H i, i = 1,...,, i.e., the problem of deciding which of the null hypotheses are true and which are false, can be formulated as a model selection problem in terms of suitable disjoint partitions of the parameter space. Bayesian model selection was discussed by Berger and Pericchi 1996) and Berger and Pericchi 001) using pairwise Bayes factors and by Bertolino, Piccinato, and Racugno 1995) using multiple and partial Bayes factors. However, since there are in all configurations of true and false null hypotheses, any statistical analysis directly involving all of these configurations becomes an impossible tas when is large. A stepwise simultaneous testing of the hypotheses in some order of configurations, determined in an objective manner, seems to be not only practically feasible in terms of eeping trac of different configurations of hypotheses but also computationally economic. 3. A Bayesian Stepwise Multiple Testing Procedure. We will first develop in this section the proposed Bayesian procedure. Then, the formulas that are necessary for simultaneous testing of point as well as one-sided null hypotheses against the corresponding complementary alternatives will be presented. 8

9 3.1. The Proposed Procedure. Let B 1) B ) be the ordered values of the marginal Bayes factors B 1,..., B, and B j) correspond to H j). If the strength of evidence for H j) as indicated by B j) is wea, then the strength of evidence for H i) should be weaer for all i < j. Similarly, strong evidence for H j) would indicate stronger evidence for H l) for all l > j. In other words, among all r) possible configurations of the null hypotheses with r of them having wea and r having strong evidences, the most plausible configuration is H 1) H r) H r+1) H ). This leads us to the consideration of the following set of + 1 families { H1),..., H ) }, { H1), H ),..., H ) },..., { H1),..., H 1), H ) }, { H1),..., H ) }, within which we will restrict our search for the most plausible family of true and false null hypotheses. For this, let us define { r } { H r) = H i) i=1 i=r+1 H i) } for r = 0, 1,..., with H 0 = Ω. Also, the Bayes factor, called the stepwise Bayes factor, of H r) to any of H r+1),..., H ) is defined as B r) P H r) X) = i=r+1 P Hi) X) i=r+1 πhi) ), πh r) ) where πh r) ) is the prior probability of H r), r = 0, 1,..., 1. The Bayesian step-down procedure for selecting the most plausible target family then proceeds as follows: Step 0. Start with r = 0, i.e., the intersection of all the null hypotheses, calculate B 0). If B 0) > c, then accept H 0) = i=1 H i) and stop; if B 0) c, then reject H 1) go to the next step. Step r. Calculate B r). If B r) > c, then accept H r) and stop; if B r) c, then reject all H i) for i r + 1 and go to the next step. Step -1. Calculate B 1). If B 1) > c, then accept H 1) and stop; if B 1) c, then reject all H i) for i. 9,

10 In each step of the above procedure, the stepwise Bayes factor of the hypothetical target family to any of the remaining target families is calculated and compared with a predetermined constant c. In testing a single hypothesis against an alternative using Bayes factor, different authors Berger, Brown, and Wolpert 1994; Berger, Bouai, and Wang 1997b) have suggested using different constants for rejecting the null hypothesis. We will, however, adopt the principle of Berger, Bouai, and Wang 1997b) and use 1 for the constant c. This maes sense because a Bayes factor less than or equal to 1 simply would imply that the null model is less liely to be true as compared to the alternative model. Notice that a rejection of the intersection of null hypotheses suggests that at least one of the individual null hypotheses is not true; which naturally leads us to reject the null hypothesis with the smallest posterior probability or marginal Bayes factor. 3.. Testing Multiple Point Null Hypotheses. We will derive in this subsection the formulas necessary to perform the proposed procedure for simultaneous testing of point null hypothesis H i : θ i = θ i0, i = 1,...,, against the corresponding complementary alternatives H i : θ i θ i0, i = 1,...,. Since we have point null hypotheses, we cannot use a continuous prior density. The most common approach in this case is to assign H i a positive probability π i0, while giving H i the density 1 π i0 )g 1 θ i λ), conditional on λ, with g 1 θ i λ) being a proper prior Berger 1985). Note that θ i0 can be a nown, fixed number or an unnown parameter with a prior distribution. We will consider in the following both cases with nown and unnown θ i0. For each θ i, the conditional prior given λ is π 1 θ i λ) = π i0 Iθ i = θ i0 ) + 1 π i0 )g 1 θ i λ)iθ i θ i0 ). If θ i0 is nown, the posterior probability of H i given X is 3.4) P H i X) = mx)] 1 π i0 fx i θ i0 ) i) { π j0 fx j θ j0 ) +1 π j0 )f X j λ)} ] π λ)dλ, 10

11 where { mx) = πj0 fx j θ j0 ) + 1 π j0 )f X j λ) } π λ)dλ, f X j λ) = fx j θ j )g 1 θ j λ)dθ j, j = 1,...,, and i) means that the product is taen over j from 1 to, excluding i. Note that, for each j, the density fx j θ j ) may also involve the parameter λ, as in the case of a normal with unnown variance Section 4). To avoid notational complications in obtaining the formula for P H r) X), the posterior probability of H r) given X, we will assume here and also in the rest of the paper that H i is the ith ordered hypothesis H i), for i = 1,...,. Then we have, 3.5) r P H r) X) = mx)] 1 { 1 πi0 )f X i λ) } i=1 i=r+1 { πi0 fx i θ i0 } ] π λ)dλ, using which the stepwise Bayes factor for testing H r) can be obtained as follows: B r) P H r) X) j ) = P Hj) X) 1 π i0 3.6). π i0 i=r+1 When λ = λ 1, λ ) is nown, the 3.6) can be expressed in terms of marginal Bayes factors: 3.7) B r) = j i=r+1 ) 1 π i0 1 1 π i0 B i j i=r+1 1 π i0 π i0 ). If θ 10 = = θ 0 θ 0, where θ 0 is an unnown parameter corresponding to some other commonly referenced group such as a control group, and let fx 0 θ 0 ) denote the density function of the observed data X 0 and π 1 θ 0 λ) the prior pdf of θ 0 for the control group, then the posterior probability of H i given X becomes 3.8) P H i X) = m 0 X)] 1 fx 0 θ 0 )π i0 fx i θ 0 ) i) { πj0 fx j θ 0 ) +1 π j0 )f X j λ) } π 1 θ 0 λ)dθ 0 ]π λ)dλ, 11

12 where m 0 X) = fx 0 θ 0 ) { π j0 fx j θ 0 ) ] +1 π j0 )f X j λ) }π 1 θ 0 λ)dθ 0 π λ)dλ. The posterior probability of H r) given X is { r 3.9) P H r) X) = m 0 X)] fx 1 0 θ 0 ) 1 π j0 )f X j λ) ] π j0 fx j θ 0 ) }π 1 θ 0 λ)dθ 0 π λ)dλ Testing Multiple One-Sided Null Hypotheses. Consider now the problem of testing H i : θ i θ i0 versus H i : θ i > θ i0, simultaneously for i = 1,...,. Let the prior pdf of θ i, conditional on λ, be π 1 θ i λ), i = 1,...,. If θ i0 is nown, the posterior probability of H i and H r) given X are given by: 3.10) and 3.11) P H i X) = m X)] 1 f 0 X i λ) i) r P H r) X) = m X)] 1 { f 1 X j λ) } { f X j λ) }] π λ)dλ { f 0 X j λ) }] π λ)dλ, respectively, where and m { X) = f X j λ) }] π λ)dλ, f X j λ) = fx j θ j )π 1 θ j λ)dθ j, f0 X j λ) = fx j θ j )π 1 θ j λ)dθ j, θ j θ j0 f1 X j λ) = fx j θ j )π 1 θ j λ)dθ j. θ j >θ j0 1

13 The stepwise Bayes factor for H r) is then obtained from the following formula: B r) P H r) X) = i=r+1 P Hi) X) i=r+1 π 0H i) ) 3.1), π 0 H r) ) where r { } π 0 H r) ) = π 1 θ i λ)dθ i θ i >θ i0 i=1 is the prior probability of H r). i=r+1 { θ i θ i0 π 1 θ i λ)dθ i } ] π λ)dλ If θ 10 = = θ 0 θ 0 is unnown and assumes a prior distribution π 1 θ 0 λ), then the posterior probability of H i and H r) are, respectively, 3.13) P H i X) = m 0X)] 1 fx 0 θ 0 )f0 X i λ)π 1 θ 0 λ)dθ 0 and 3.14) where i) P H r) X) = m 0X)] 1 ] {f X j λ)} π λ)dλ fx 0 θ 0 ) r {f1 X j λ)} { f 0 X j λ) } π 1 θ 0 λ)dθ 0 ] π λ)dλ, ] m 0X) = {f X j λ)} π λ)dλ. j=0 4. Applications to normal data. The proposed Bayesian procedure is now applied to samples from normal populations with a common variance, that is, for each i, the pdf of X i is assumed to be n i fx i θ i, σ ) = { 1 σ π exp 1 x ij θ i ) ]}. Three different multiple testing problems comparing the populations in terms of their means are considered, assuming unnown σ. In each case, the necessary formulas are further developed from those presented in Section 3 for normal samples before applying to a real data set. All computations are carried out using compiled SAS R macros. 13

14 4.1. Multiple testing with a standard using point null hypotheses. This type of multiple testing is frequently encountered, for instance, in comparisons of multiple groups with a gold standard. The problem of interest is that of testing of the point null hypotheses H i : θ i = θ 0 versus the corresponding alternative H i : θ i θ 0, i = 1,...,, for some nown θ 0. We consider the prior density g 1 θ i ξ, σ ) to be that of Nµ, ξσ ), for some nown µ and ξ, with σ having a noninformative prior π σ ) σ ) 1. Let X i = 1 n i n i X ij, i = 1,...,, and S = Si = i=1 n i X ij X i ), i=1 be the sample means and the within sum of squares, respectively. Let n = i=1 n i be the total sample size of the samples. Then, the marginal Bayes factor for H i over H i is given by 4.15) B i = n i i) ω j 1 + F ) n + 1 l=1 1 + F ) n + 1 l=1 J i : J i =l ω 1 J i ) 1 + F Ji ) n J i: J i =l ω J i ) 1 + F Ji ) n, where J i is an ordered subset of {1,..., } {i}, n j = n j ξ + 1, ω j = π j0 n j 1 π j0 ), j = 1,..., F = T j, F = T j, ω Ji = j J i ω j, F Ji = j J i T j + j J c i T j, FJi = j Ji T j + T j, j J c i nj T X j θ 0 ) nj j =, T S j = X j µ), j = 1,...,, n j S 14

15 Table 1 Summary Statistics and Marginal and Stepwise Bayes Factors for Ball Bearing Data Process Mean Sample Variance n B i r B r) Stop and J c i = {1,..., } J i see the Appendix for a proof). The stepwise Bayes factor for testing H r) is 3.6) with the posterior probability of H r) being 4.16) r P H r) X) = 1 π j0) mx) πs ) n r n j r 1 + T j + π j0 T j n. The choice of ξ = is recommended Berger, Bouai, and Wang 1997b). For simplicity, we will assign 0.5 to each null hypothesis and its alternative. Example 1. Romano 1977, p. 48) Four production lines are set to produce a specific type of ball bearing with a diameter of 1 mm. At the end of a day s production, ten ball bearings are randomly selected from each of the four production lines. An F test indicates that at least one process is out of control. By applying the proposed Bayesian stepwise simultaneous testing procedure to the data with µ = θ 0 = 1 mm, we conclude that process is out of control producing ball bearings with an average diameter other than 1 mm Table 1). If one uses Holm s step-down method, the same conclusion could be reached. 4.. Multiple testing with an unnown control using point null hypotheses. More often the θ 0 itself is an unnown parameter, e.g., multiple testing with a control group or many-to-one multiple testing where the mean for the control group is 15

16 unnown. Let the prior distribution of θ i be π 1 θ i ξ, σ ) = Nµ, ξσ ), i = 0,,. With a noninformative prior for σ being π σ ) σ ) 1, it can be shown that the marginal Bayes factor for H i over H i is given by 4.17) B i = n i i) + 1 l=1 + 1 l=1 ω j n F ) n 1 + F ) n n J i : J i =l n J i ω 1 J i ) 1 + F Ji ) n J i: J i =l n 1 J i ω Ji ) 1 + F J i ) n where n = j=0 n j is the total sample size of the + 1 samples, n = ξ n j + 1, n Ji = ξn 0 + ξ n j + 1, n Ji = ξn 0 + ξ n j + 1, j J j j=0 j=0 j=0 j J c i F = n j X j + 1 ξ µ ξ n j Xj + 1 ξ µ / n /S, F = T J i = n 0 X 0 + F J i = T j + T J i, F Ji = T J i + T j, j J i j J c i n j X j + 1 ξ µ ξ n 0 X0 + n j Xj + 1 ξ µ, T j, j=0 /n Ji /S, j J c i j J c i T J i = n 0 X 0 + n j X j + 1 ξ µ ξ n 0 X0 + n j Xj + 1 ξ µ / n Ji /S j J i j Ji See the Appendix for a proof). The stepwise Bayes factor for testing H r) is 3.6) with the posterior probability of H r) being r P H r) ) = 1 π j0) π j0 4.18) m 0 X) πs ) n 1 + r n r n j r T j + T r n, where n r = n 0 ξ + ξ n j + 1, 16

17 Table Summary Statistics and Marginal and Stepwise Bayes Factors for Smoing and Pulmonary Health Data Group #) Mean Std. Dev. n B i r B r) NS 0) HS 5) MS 4) LS 3) PS ) NI 1) Stop and T r = n 0 X 0 + n j X j + 1 ξ µ ξ n 0 X0 + n j Xj + 1 ξ µ / n r /S. Example. White and Froeb 1980) The effect of smoing on pulmonary health is investigated in a retrospective study in which subjects who had been evaluated during a physical fitness profile were assigned, based on their smoing habits, to one of six groups non-smoers NS), passive smoers PS), non-inhaling smoers NI), light smoers LS), moderate smoers MS), and heavy smoers HS). A sample of 1050 female subjects, 50 from non-inhaling group and 00 from each of the remaining groups, were selected and data on their pulmonary function forced vital capacity, FVC) were recorded. One of the objectives of the study is to determine smoing effects on individual s pulmonary health relative to non-smoers. With prior mean µ = 3 and ξ =, our Bayesian stepwise procedure stops at r = 3, rejects H 5, H 4 and H 3 and concludes that heavy, moderate, and light smoers are significantly different from non-smoers in terms of mean FVC Table ). An application of Dunnett s two-sided method to the data suggests the same conclusion Multiple testing with an unnown control using one-sided hypotheses. We continue our discussion using the same distributional setups in the previous section, 17

18 but now we are interested in testing H i : θ i θ 0 versus H i : θ i > θ 0, i = 1,,, with θ 0 being the unnown mean of the control group. Note that there is no discrete part of the prior distribution π 1 θ i ξ, σ ). To avoid technical difficulty when calculating prior probability of H i, we will use IGσ ; a/, b/) instead of noninformative prior) as the prior density function of σ and gξ) as the prior density of ξ. It can be shown that the marginal Bayes factor for H i over H i is given by 4.19) B i = 1 π i0) π i0 1 j=0 n j β α IGσ ; α, β) φz 0 )Φz i0 )dz 0 dσ gξ)dξ 1 j=0 n j β α IGσ ; α, β), φz 0 ) 1 Φz i0 )] dz 0 dσ gξ)dξ where π i0 = φz 0 )Φz 0 )dz 0 is the prior probability of H i, Φ and φ are, respectively, the cumulative distribution and density functions of standard normal, α = n + a, β = S 1 + T j + b S, and z 0 = n0 ξ z i0 = n i + 1 z 0σ ξ n ξ j=0 θ 0 n 0ξ X ) 0 + µ /σ, n 0 + n 0ξ X 0 + µ n iξ X i + µ n 0 See the Appendix for a proof.) The posterior probability of H r) given X is P H r) X) = π) n Γα)b/) a/ m 0 X)Γa/) n 1 4.0) j β α IGσ ; α, β) j=0 r φz 0 ) 1 Φzj0 ) ) n i ] Φz j0 ) dz 0 dσ gξ)dξ See the Appendix for a proof.) Per the recommendation of Berger, Bouai, and Wang 1997b), the prior gξ) is chosen to be an inverse gamma gξ) = 1 ξ 3/ exp 1 ) 4.1). π ξ The computation of the marginal Bayes factor for H i and the posterior probability for H r) is carried out using the method of Monte Carlo integration. 18

19 Table 3 Summary Statistics and Marginal and Stepwise Bayes Factors for Mouse Growth Data Group Mean Std. Dev. n B i r B r) Stop Example 3. Steel and Torrie 1980) The toxicological effects of six different chemical solutions on young mice are studied and compared with a control group in terms of weight change. The interest focuses on the comparisons of the six solutions with the control and not on the comparisons among the six solutions. Since it is generally believed that drugs are potentially toxic, it is appropriate to test the null hypotheses of no mean difference in weight gain against the one-sided alternative hypotheses of a lower weight gain. An application of Dunnett s one-sided step-down method to the data reveals that solutions 3, 6, and are significantly different from the control Westfall, Tobias, Rom, Wolfinger, and Hochberg 1999, pp ). Using our proposed procedure with prior mean µ = 85 and a = b = 1, we conclude that solutions 3, 6,, and 5 are significantly more toxic than the control Table 3). 5. Concluding remars. While there exist Bayesian methodologies to address multiple testing problems, they either become too complicated to implement for large number of hypotheses or do not fully utilize the information that become available in the process of simultaneous testing of the hypotheses. Our idea of giving Bayesian formulations of the classical stepwise procedures is an attempt to overcome these shortcomings. Although we have decided in this article to propose a Bayesian version of the classical step-down method, we could have chosen the 19

20 classical step-up method and given its Bayesian version. In this step-up Bayesian method, once the target families were formed, the stepwise search would start from the family with no true null hypotheses and proceed towards those with higher number of true null hypotheses. Some recent papers Efron 003; Efron, Tibshirani, Storey, and Tusher 001; Storey 00; Storey 003) have looed at multiple testing problems using a Bayesian formulation. More specifically, let X i be the test statistics and Z i be Bernoulli random variables with the value 0 indicating that H i is true and 1 indicating that it is false. They assume that X i Z i 1 Z i )f i0 x)+z i f i1 x) and P {Z i = 0} = π i0, for i = 1,..., n. However, they provide Bayesian justifications of the frequentist measures of false discovery and positive false discovery rates. The idea of this article is completely different; we develop a stepwise decision procedure using Bayesian measures of evidence of different competing hypotheses. 6. Acnowledgments. The authors than Larry Ma, Hong Qi, and Alice Cheng for helpful hints on SAS R macros for computation of the examples. 0

21 APPENDIX A. Proofs. APPENDIX A.1. Proof of 4.16). By 3.5), the posterior probability of H i : θ i = θ 0 given X can be written as P H i X) π i0 πσ ) { n i exp 1 ni σ X i θ 0 ) + Si ] } i) { 1 π j0 ) πσ ) n j +1 ξ 1 exp 1 σ n j X j θ j ) + 1 )] ξ θ j µ) + Sj dθ j +π j0 πσ ) n j exp 1 nj σ X j θ 0 ) + Sj ) ]} σ ) 1 dσ = π j0 π) n σ ) n 1 exp 1 σ n j X j θ 0 ) + S i) { π j0 π j0 nj ξ + 1 exp n j X j θ 0 ) )]}dσ. 1 n j X j µ) σ n j ξ + 1 To simplify the proof, let d j = n j X j θ 0 ), and d j = n j X j µ), j = 1,...,. n j ξ + 1 Notice that the expansion of the polynomial i) 1 + α j ) = 1 + i) α j + 1 j 1 = l=1 i) 1 j 1 <j i) α j J i: J i =l α Ji, 1 α j1 α j + i) 1 j 1 <j <j 3 α j1 α j α j3

22 where α Ji = j J i α j. Then one has A.) P H i X) π j = l=1 π j0 π) n σ ) n 1 exp 1 d σ j + S { J i: J i =l j J i πs ] n 1 + F ) 1 n + l=1 n ) Γ J i : J i =l ω 1 j exp ω 1 J i 1 ]} dj σ d ) j dσ ) ) n 1 + F Ji, where F, ω Ji and F Ji are as defined in 4.16). Similarly, one can show that the posterior probability of θ i θ 0 given X is A.3) P H i X) i) { 1 π j0 ) fx j θ j, σ )g 1 θ j ξ, σ )dθ j } +π j0 fx j θ j = θ 0, σ ) 1 π i0 ) fx i θ i, σ )g 1 θ i ξ, σ )dθ i π σ )dσ = π j0 ω 1 j πs ] n F ) n + l=1 J i : J i =l n ) Γ ω Ji ) 1 + F ) n Ji, where F and F Ji are as defined in 4.16). Hence, the marginal Bayes factor for testing H i : θ i = θ 0 is in the form of 4.16).

23 APPENDIX A.. Proof of 4.17). According to 3.6), the posterior probability of H r) given X is A.4) P H r) X) = r 1 π j0 ) mx) r exp exp r 1 π j0 ) π j0 1 σ π) n+r ξ r σ ) n+r 1 n j X j θ j ) + 1 )] ξ θ j µ) + Sj dθ j 1 )] σ n j θ j θ 0 ) + Sj dσ π j0 = mx) exp 1 r σ π) n n j X j µ) n j ξ + 1 r + n 1 j σ ) n 1 n j θ j θ 0 ) dσ, which is an inverse gamma function of σ and the integration over σ results in 4.17). APPENDIX A.3. Proof of 4.18). by 3.9), the posterior probability of H i : θ i = θ 0 given X is P H i X) π i0 πσ ) n 0 +n i +1 ξ 1 exp 1 σ n j X j θ 0 ) j=0,i + 1 ξ θ 0 µ) + S ]} i) exp 1 σ { 1 π j0 ) πσ ) n j +1 ξ 1 n j X j θ j ) + 1ξ )] θ j µ) dθ j + π j0 πσ ) n j exp 1 σ n j X j θ 0 ) ]} dθ 0 σ ) 1 dσ 3

24 = π j0 π) n+1 σ ) n+1 1 ξ 1 exp 1 σ = l=1 π j0 { J i : J i =l j J i πs ) n A.5) n F ) n n ) Γ 1 + l=1 ω 1 j exp J i: J i =l n j X j θ 0 ) + 1 ξ θ 0 µ) + S j=0 1 dj σ n j X j θ 0 ) ) ]} dθ 0dσ n 1 J i ) ω 1 J i 1 + F Ji ) n. where n, F, n Ji, and F J i are as defined in 4.18). Similarly, one can show that the posterior probability of H i : θ i θ 0 given X is A.6) P H i X) i) { 1 π j0 ) fx j θ j, σ )g 1 θ j ξ, σ )dθ j } +π j0 fx j θ j = θ 0, σ ) fx 0 θ 0, σ )π 1 θ 0 ξ, σ )dθ 0 1 π i0 ) fx i θ i, σ )g 1 θ i ξ, σ )dθ i π σ )dσ = where F, n j1 j l, and F j 1 j l π j0 ω 1 j πs ) n n F ) n + factor for H i is in the form of 4.18). l=1 n ) Γ J i : J i =l n 1 J i ω Ji ) 1 + F J i ) n. are as defined in 4.18). Therefore, the marginal Bayes 4

25 APPENDIX A.4. Proof of 4.18). According to 3.10), the posterior probability of H r) given X is r 1 π j0 ) P H r) X) = A.7) exp r m 0 X) 1 σ π j0 ) n 0 X 0 θ 0 ) + π) n+r+1 ξ r+1 σ ) n+r+1 exp 1 ] σ S n j X j θ j ) + 1 ξ θ 0 µ) )]dθ 0 exp 1 n σ j X j θ j ) + 1ξ )] θ j µ) dθ j dσ r 1 π j0 ) = m 0 X) π) n n 0 ξ + exp 1 σ n 0 X r n j X j µ) ξn j S )] π j0 ) n j ξ + 1 ) r n j ξ + 1) n j X j + 1 ξ µ n0 X0 + n 0 + σ ) n n j Xj + 1 ξ µ) n j + 1 ξ which is the inverse gamma function of σ and the integration of A.7) results in 4.18). 5

26 APPENDIX A.5. Proof of 4.0). According to 3.14), the posterior probability of H i : θ i θ 0 given X is P H i X) π) n++1 A.8) θ0 exp 1 σ exp i) = π) n+ 1 σ { ξ +1 σ ) n++a+1 1 exp 1 S σ + b ) ] n 0 X 0 θ 0 ) + 1ξ )] θ 0 µ) n i X i θ i ) + 1ξ θ i µ) )] dθ i dθ 0 exp 1 n σ j X j θ j ) + 1ξ )] } θ j µ) dθ j dσ gξ)dξ i) ξ 1 n j ξ + 1) 1 σ ) n+a+ 1 exp 1 σ exp 1 σ n ξ exp 1 σ θ0 dθ i dθ 0 dσ gξ)dξ = π) n+ n j ξ + 1) 1 j=0 i) n j X j µ) n j ξ + 1 ) θ 0 n 0ξ X 0 + µ n 0 ξ + 1 n i + 1 ) θ i n iξ X i + µ ξ n i ξ + 1 σ ) n+a 1 exp 1 σ j=0 n j X j µ) n j ξ S + b ) + n 0 X )] 0 µ) n 0 ξ + 1 ) + n i X )] i µ) n i ξ S + b exp 1 ) zi0 σ z 0 exp 1 ) σ z i dz i dz 0 dσ gξ)dξ = π) n Γα) n 1 j β α j=0 IGσ ; α, β) φz 0 )Φz i0 )dz 0 dσ gξ)dξ. 6

27 Similarly, one can show that the posterior probability of H i : θ i > θ 0 given X is { P H i X) A.9) i) ] fx 0 θ 0, σ )π 1 θ 0 ξ, σ ) fx i θ i, σ )π 1 θ i ξ, σ )dθ i dθ 0 θ 0 = π) n Γα) n 1 j β α fx j θ j, σ )π 1 θ j ξ, σ )dθ j ] } π ξ, σ )dσ dξ IGσ ; α, β) φz 0 ) 1 Φz i0 )] dz 0 dσ gξ)dξ. The prior probability of H i is given by A.30) π i0 = θ i θ 0 π 1 θ j ξ, σ )dθ j π ξ, σ )dσ dξ j=0 = πξσ ) 1 θ0 exp 1 exp 1 ξσ θ i µ) = π ξ, σ )dσ dξ ) ξσ θ 0 µ) ) ] dθ i dθ 0 π ξ, σ )dσ dξ π) 1 exp 1 ) z0 z 0 exp 1 ) ] z i dz i dz 0 = φz 0 )Φz 0 )dz 0. Similarly, one can show that the prior probability of H i is A.31) 1 π i0 = θ i>θ 0 j=0 = φz 0 ) 1 Φz 0 )] dz 0. π 1 θ j ξ, σ )dθ j π ξ, σ )dσ dξ Combining A.8) - A.31) results in 4.0). 7

28 APPENDIX A.6. Proof of 4.1). By 3.1), the posterior probability of H r) is P H r) b/) a/ X) = m 0 X)Γa/)π) ξ +1 exp 1 σ r θ j θ 0 exp n++1 σ ) n++a+1 1 exp θ j >θ 0 exp 1 ] σ S + b) n 0 X 0 θ 0 ) + 1ξ θ 0 µ) )] 1 n σ j X j θ j ) + 1ξ )] θ j µ) dθ j 1 n σ j X j θ j ) + 1ξ )] θ j µ) dθ j A.3) = dθ 0 dσ gξ)dξ b/) a/ n++1 m 0 X)Γa/)π) exp 1 σ r j=0 z j z j0 exp z j>z j0 exp n j X j µ) n j ξ + 1 j=0 1 ) σ z j dz j n j ξ + 1) 1 )] + S + b 1 ) σ z j dz j dθ 0dσ gξ)dξ σ ) n+a 1 exp 1 ) σ z 0 which simplifies to 4.1). References Benjamini, Y. and Hochberg, Y. 1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B 57, Berger, J. O. 1985). Statistical Decision Theory and Bayesian Analysis nd ed.). New Yor: Springer-Verlag. 8

29 Berger, J. O. 1999). Bayes factors. In S. Kotz, C. B. Read, and D. L. Bans Eds.), Encyclopedia of Statistical Sciences, Volume 3, pp New Yor: Wiley. Berger, J. O., Bouai, B., and Wang, Y. 1997a). Properties of unified bayesianfrequentist tests. In S. Panchapaesan and N. Balarishnan Eds.), Advances in Statistical Decision Theory and Applications, pp Boston: Birhauser. Berger, J. O., Bouai, B., and Wang, Y. 1997b). Unified frequentist and bayesian testing of a precise hypothesis. Statistical Science 1, Berger, J. O., Bouai, B., and Wang, Y. 1999). Simultaneous bayesianfrequentist sequential testing of nested hypotheses. Biometria 86, Berger, J. O., Brown, L. D., and Wolpert, R. L. 1994). A unified conditional frequentist and Bayesian test for fixed and sequential simple hypothesis testing. The Annals of Statistics, Berger, J. O. and Pericchi, L. R. 1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association 91, Berger, J. O. and Pericchi, L. R. 001). Objective bayesian methods for model selection: Introduction and comparison. In P. Lahiri Ed.), Model Selection. Beachwood, Ohio: Institute of Methematical Statistics Lecture Notes - Monograph Series, Volume 38. Berry, D. A. 1988). Multiple comparisons, multiple tests, and data dredging: A Bayesian perspective. In J. M. Bernardo, M. H. Degroot, D. V. Lindley, and A. F. M. Smith Eds.), Bayesian Statistics 3, pp Berry, D. A. and Hochberg, Y. 1999). Bayesian perspectives on multiple comparisons. Journal of Statistical Planning and Inference 8, Bertolino, F., Piccinato, L., and Racugno, W. 1995). Multiple Bayes factors for testing hypotheses. Journal of the American Statistical Association 90,

30 Breslow, N. 1990). Biostatistics and bayes. Statistical Science 5, Chen, J. and Sarar, S. K. 00). Multiple testing of response rates with a control: A Bayesian stepwise approach. submitted. Dass, S. and Berger, J. O. 1998). Unified Bayesian and conditional frequentist testing of composite hypotheses. Due University: ISDS Discussion Paper Duncan, D. B. 1965). A Bayesian approach to multiple comparisons. Technometrics 7, 171. Dunnett, C. W. and Tamhane, A. C. 1991). Step-down multiple tests for comparing treatments with a control in unbalanced one-way layouts. Statistics in Medicine 11, Dunnett, C. W. and Tamhane, A. C. 199). A step-up multiple test procedure. Journal of the American Statistical Association 87, Efron, B. 003). Robbins, empirical Bayes and microarrays. Annals of Statistics 31, Efron, B., Tibshirani, R., Storey, J. D., and Tusher, V. 001). Empirical bayes analysis of a microarray experiment. Journal of the American Statistical Association 96, Finner, H. 1993). On a monotonicity problem in step-down multiple test procedures. Journal of the American Statistical Association 88, Gelman, A. and Tuerlincx, F. 000). Type S eerror rates for classical and Bayesian single and multiple comparison procedures. Computational Statistics Formerly: Computational Statistics Quarterly] 15 3), Gopalan, R. and Berry, D. A. 1998). Bayesian multiple comparisons using dirichlet process priors. Journal of the American Statistical Association 93, Hochberg, Y. and Tamhane, A. C. 1987). Multiple Comparison Procedures. New Yor: John Wiley & Sons, Inc. 30

31 Holm, S. 1999). Multiple confidence sets based on stagewise tests. Journal of the American Statistical Association 94, Hsu, J. C. 1996). Multiple Comparisons: Theory and Methods. Washinton, D.C.: Chapman & Hall/CRC. Liu, W. 1996). Multiple tests of a non-hierarchical finite family of hypotheses. Journal of the Royal Statistical Society, Series B 58, Liu, W. 1997). Stepwise tests when the test statistics are independent. The Australian Journal of Statistics 39, Romano, A. 1977). Applied Statistics for Science and Industry. Boston, MA: Allyn and Bacon. Sarar, S. K. 00). Some results on false discovery rate in stepwise multiple testing procedures. Annals of Statistics 30, Shaffer, J. P. 1999). A semi-bayesian study of Duncan s Bayesian multiple comparison procedures. Journal of Statistical Planning and Inference 8, Steel, R. G. D. and Torrie, J. H. 1980). Principles and Procedures of Statistics: A Biometrical Approach. New Yor: McGraw Hill Boo Co. Storey, J. D. 00). A direct approach to false discovery rates. Journal of the Royal Statistical Society, Series B 64, Storey, J. D. 003). The positive false discovery rate: A bayesian interpretation and the q-value. to appear in Annals of Statistics. Tamhane, A. C. and Dunnett, C. W. 1999). Stepwise multiple test procedures with biometric applications. Journal of Statistical Planning and Inference 8, Tamhane, A. C. and Gopal, G. V. S. 1993). A Bayesian approach to comparing treatments with a control. In F. M. Hoppe Ed.), Multiple Comparisons, Selection and Applications in Biometry. New Yor: Marcel Deer, Inc. Tamhane, A. C., Liu, W., and Dunnett, C. W. 1998). A generalized step-updown multiple test procedure. The Canadian Journal of Statistics 6,

32 Waller, R. A. and Duncan, D. B. 1969). A Bayes rule for the symmetric multiple compsrison problem. Journal of the American Statistical Association 64, Westfall, P. H., Johnson, W. O., and Utts, J. M. 1997). A Bayesian perspective on the bonferroni adjustment. Biometria 84, Westfall, P. H. and Tobias, R. D. 1999). Advances in multiple comparisons and multiple tests using the SAS system. In Proceedings of the Twenty-Fourth Annual SAS Users Group International Conference, pp Cary, NC: SAS Institute, Inc. Westfall, P. H., Tobias, R. D., Rom, D., Wolfinger, R. D., and Hochberg, Y. 1999). Multiple Comparisons and Multiple Tests Using the SAS R System. Cary, NC: SAS Institute, Inc. Westfall, P. H. and Young, S. S. 1993). Resampling-Based Multiple Testing: Examples and Methods for p-value Adjustment. New Yor: John Wiley & Sons. White, J. R. and Froeb, H. F. 1980). Small-airways dysfunction in nonsmoers chronically exposed to tobacco smoe. New England Journal of Medicine 30, Sanat K. Sarar Department of Statistics Temple University Philadelphia, PA 191 U. S. A. sanat@sbm.temple.edu Jie Chen Merc Research Laboratories P. O. Box 4, WP37C-305 West Point, PA U. S. A. jie chen@merc.com 3