Statistica Sinica 24 (2014), 1-23 doi:http://dx.doi.org/10.5705/ss.2011.205 ON LEAST FAVORABLE CONFIGURATIONS FOR STEP-UP-DOWN TESTS Gilles Blanchard 1, Thorsten Dickhaus 2, Étienne Roquain3 and Fanny Villers 3 1 Potsda University, 2 Huboldt University and 3 UPMC University Abstract: This paper investigates an open issue related to false discovery rate (FDR) control of step-up-down (SUD) ultiple testing procedures. It has been established that for this type of procedure, under soe broad conditions and in an asyptotic sense, the FDR is axiu when the signal strength under the alternative is axiu. In other words, so-called Dirac unifor configurations are asyptotically least favorable in this setting. It is known that this property also holds in a nonasyptotic sense (for any finite nuber of hypotheses) for the two extree versions of SUD procedures, naely step-up and step-down (under additional conditions for the step-down case). It is therefore natural to conjecture that this nonasyptotic least favorable configuration property could ore generally be true for interediate fors of SUD procedures. We prove that this is not the case. The arguent is based on the exact calculations proposed earlier by Roquain and Villers (2011a); we extend the by generalizing Steck s recursion to the case of two populations. Furtherore, we quantify the agnitude of this phenoenon by providing a nonasyptotic upper bound and explicit vanishing rates as a function of the total nuber of hypotheses. Key words and phrases: False discovery rate, least favorable configuration, ultiple testing, Steck s recursions, step-up-down. 1. Introduction 1.1. Least favorable configurations for ultiple testing In atheatical statistics, so-called least favorable paraeter configurations (LFCs) play a pivotal role. For a statistical decision proble over a paraeter space Θ with risk R(, ) and a decision rule δ, an LFC is any eleent θ (δ) of Θ that axiizes the risk of δ over Θ. When available, the knowledge of an LFC allows one to obtain a bound on the risk over a possibly large paraeter space, including non- or sei-paraetric cases where Θ has infinite diensionality. In practice, LFCs are of interest during the planning phase of an experient when the ai is to design a procedure δ with an a priori guaranteed axiu risk ax R(θ, δ) = θ Θ R(θ (δ), δ) = α. (1.1)
2 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS In particular, if there is a one-paraeter faily (δ β ) β of candidate decision rules, (1.1) can be used for adequate calibration of β I R. If the right-hand side (RHS) of (1.1) is invertible w.r.t. β, we can derive a closed for for β(α); if not, we can still approxiate β by using Monte-Carlo ethods siulating the distribution corresponding to the LFC. While LFC considerations naturally occur in hypothesis testing probles, they are particularly delicate for ultiple hypothesis testing. The latter issue has been investigated by any authors, see Finner and Roters (2001); Benjaini and Yekutieli (2001); Lehann and Roano (2005); Finner, Dickhaus, and Roters (2007); Roano and Wolf (2007); Guo and Rao (2008); Soerville and Heelann (2008); Finner, Dickhaus, and Roters (2009); Finner and Gontscharuk (2009); Gontscharuk (2010). In that setting, a faily of 2 null hypotheses H 1,..., H is to be tested siultaneously under a coon statistical odel with paraeter space Θ, and soe type I error criterion is used to account for ultiplicity. In applications, it is relevant to deterine LFCs over the restricted paraeter spaces Θ,0 where exactly 0 out of of the null hypotheses are true, for a given nuber 0 that is fixed when axiizing over θ Θ,0. In the present work, we restrict our attention to ultiple testing procedures that depend on the observed data only through a collection of arginal p-values, each associated with an individual null hypothesis. This is a coon setting for ultiple testing probles in high diension. Moreover, we consider procedures that reject exactly those null hypotheses having their p-value less than a certain coon threshold t, which can possibly be data-dependent. We call such procedures threshold-based for short. LFCs for ultiple testing depend crucially on the type I error criterion considered. We first discuss criteria defined through loss functions that only depend on the nuber of type I errors, R(θ, δ) := E θ [ϕ(v )], (1.2) where V is the nuber of type I errors of ultiple testing procedure δ: V = V (θ, δ) := {1 i : H i is true for θ and gets rejected by δ}. (1.3) Assue that ϕ is a nondecreasing function and that, for threshold-based procedures, t is a nonincreasing function in each p-value. When the p-values are jointly independent, it is known that the LFC over Θ,0 is a Dirac-unifor (DU) distribution, with p-values corresponding to the 0 true nulls independent unifor variables and the 0 p-values under alternatives with point ass 1 at zero. This result is forally recalled in Section S1.2 of the suppleent Blanchard et al. (2013). For exaple, this holds under the above assuptions for
ON LEAST FAVORABLE CONFIGURATIONS 3 the faily-wise error rate (FWER), which is the usual type I error concept in traditional ultiple hypothesis testing theory. Over the last two decades, alternative type I error criteria, introduced in applications to genoics, proteoics, neuroiaging, and astronoy, have led to assive ultiple testing probles with large systes of hypotheses, see Dudoit and van der Laan (2008); Pantazis et al. (2005); Miller et al. (2001). Here, a less stringent notion of type I error control is needed in order to ensure reasonable power of the corresponding ultiple tests. In particular, the false discovery rate (FDR) introduced by Benjaini and Hochberg (1995) has becoe a standard criterion for type I error control in large-scale ultiple testing probles. It does not fall into the class of type I error easures defined by (1.2), and the LFC proble for the FDR criterion turns out to be a challenging issue even for siple classes of ultiple tests and under independence assuptions. A possible approach to LFCs under the FDR criterion is in an asyptotic sense, when the nuber of hypotheses tends to infinity, see Finner, Dickhaus, and Roters (2009) and Gontscharuk (2010). In practice, however, it is desirable to have precise inforation about LFCs for a fixed nuber of hypotheses, in particular for design, planning, and calibration purposes. LFCs for the FDR criterion under arbitrary dependencies have also been studied by Lehann and Roano (2005); Guo and Rao (2008). 1.2. Contributions We focus on the nonasyptotic theory of LFCs under the FDR criterion for so-called step-up-down ultiple tests (SUD procedures, for short). These procedures constitute a subclass of threshold-based ultiple testing procedures wherein the threshold t is obtained by coparing the ordered p-values to a fixed set of critical values, see Tahane, Liu and Dunnett (1998); Sarkar (2002). We provide a survey of known LFC results for SUD procedures in specific odel classes, in Section 3. New results for LFCs of SUD procedures are derived in Section 4. We establish that the DU configuration is not, generally, a nonasyptotic LFC. Then, since it is known that the DU configuration is the least favorable in an asyptotic sense, we derive precise, nonasyptotic, upper bounds on the difference between the FDR under an arbitrary alternative and under the DU configuration. In particular we analyze, for soe specific situations, the rate at which these bounds decay to zero as the nuber of hypotheses grows. In Section 5, we give exact forulas for coputing the FDR under the so-called two-group fixed and rando ixture odels for the p-values. This is a toolbox section for the rest of the paper; the forulas are used to disprove nuerically the conjecture studied in Section 4. This section builds on the previous work of Roquain and Villers (2011a,b), which is suarized. A novel addition
4 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS is an extension of Steck s recursion to copute the joint cuulative distribution function (c.d.f.) of order statistics of two ixed populations, that is used to handle the case of the fixed ixture odel. This point had been left open by Roquain and Villers (2011a,b) and is of intrinsic interest, because the coputational coplexity of existing ethods for this kind of function is exponential with k (Glueck et al. (2008)) while Steck s recursion is polynoial. Finally, the exact forulas derived in Section 5 are used to discuss the appropriateness of the FDR criterion in Section 6.2. 2. Matheatical Setting 2.1. Models Let F be the set of continuous c.d.f.s fro [0, 1] into [0, 1]. We consider a set of 2 null hypotheses H 1,..., H, and assue the existence of a corresponding collection of tests with associated p-value faily p := (p i, i {1,..., }), that constitutes the only observed inforation. Two closely related odels for the joint distribution of p-values are coon in the literature. The first is the (two group) fixed ixture odel, denoted FM(, 0, F ), with paraeters 2, 1 0, F F. It odels p = (p i, i {1,..., }) as a faily of utually independent variables, with, for all i, { U(0, 1) if 1 i 0, p i F if 0 + 1 i, where U(0, 1) denotes the unifor distribution on (0, 1). The second odel is the (two group) rando ixture odel, denoted by RM(, π 0, F ), with paraeters 2, π 0 [0, 1], F F. Under this odel, 0 is an (unobserved) binoial rando variable B(, π 0 ), and p follows the FM(, 0, F ) odel conditionally on 0. Here, the true nulls are assigned to the 0 (rando or not) first coordinates. This is assued without loss of generality by independence, and because we consider procedures that only depend on the order statistics of the p-values. Coon additional assuptions are that F (x) x, for all x, and that F is concave. These are satisfied in the Gaussian location odel: F (t) = Φ(Φ 1 (t) µ), for a given alternative ean µ > 0, where Φ(z) = P(Z z) for Z N (0, 1). This is the alternative distribution of p-values when testing for µ 0 under a Gaussian location shift odel with unit variance. The assuptions are also satisfied for the Dirac δ 0 distribution, as introduced by Finner and Roters (2001). The corresponding distribution in the FM odel is called Dirac-unifor (DU) configuration and denoted by FM(, 0, F 1), or siply DU(, 0 ). We define siilarly RM(, π 0, F 1). The DU configuration can be seen as a
ON LEAST FAVORABLE CONFIGURATIONS 5 liit of the Gaussian location odel for an alternative ean µ =. It is often considered a natural candidate for an LFC of several global type I error rates, see, e.g., Finner, Dickhaus, and Roters (2007); Roano and Wolf (2007); Soerville and Heelann (2008). 2.2. Procedures We consider step-up-down (SUD) procedures introduced by Tahane, Liu and Dunnett (1998), see also Sarkar (2002). Let a threshold or critical value collection be any nondecreasing sequence t = (t k ) 1 k [0, 1], with t 0 = 0. Definition 1. The step-up-down procedure SUD λ (t) of order λ {1,..., } with threshold collection t, given a sequence of reordered p-values p (1) p (2) p (), with p (0) = 0, rejects the ith hypothesis if p i t = tˆk, with k = { ax{k {λ,..., } : k {λ,..., k}, p (k ) t k } if p (λ) t λ ; ax{k {0,..., λ} : p (k) t k } if p (λ) > t λ. (2.1) For convenience, we identify procedures with their rejection sets, e.g., SUD λ (t) = {1 i : p i tˆk}. Here λ = 1 and λ = correspond to the traditional step-down (SD) and step-up (SU) procedures, respectively. An illustration is provided in Figure 1. A standard choice for t is Sies (1986) linear critical values t k = αk/ for a pre-specified level α (0, 1). The corresponding step-up-down procedure is called the linear step-up-down procedure and denoted by LSUD λ. For λ = 1 and λ =, LSUD λ is siply written as LSD and LSU, respectively. LSU is the procedure of Benjaini and Hochberg (1995). More generally, threshold collections are coonly of the for t k = ρ(k/) for a function ρ : [0, 1] [0, 1] assued to satisfy ρ : [0, 1] [0, 1] is continuous and nondecreasing; (2.2) x (0, 1] ρ(x)/x is nondecreasing. (2.3) The function ρ is called the critical value function (and its inverse, the rejection curve, see, e.g., Finner, Dickhaus, and Roters (2009)). Assuptions (2.2) and (2.3) are eaningful in particular for analyzing asyptotics, wherein ρ is independent of. For a fixed, (2.3) is equivalent to k t k /k is nondecreasing ; Finner, Gontscharuk, and Dickhaus (2012) call such threshold collections feasible critical values. 2.3. False discovery rate and LFCs Introduced by Benjaini and Hochberg (1995), the FDR of a ultiple testing procedure is the averaged ratio of the nuber of erroneous rejections to the total
6 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS Figure 1. Value of k (vertical solid line), defined by (2.1), for several procedures of the SUD type. The rejection set of the botto-right SUD procedure (using λ = 2/3) coincides with that of the SU procedure for this realization of the p-value faily. nuber of rejections. In our setting, for P either FM(, 0, F ) or RM(, π 0, F ), the FDR of a step-up-down procedure can be written as FDR(SUD λ (t), P ) = E p P [FDP(SUD λ (t), 0, p)], (2.4) where the false discovery proportion (FDP) is FDP(SUD λ (t), 0, p) = {1 i 0 : p i tˆk} {1 i : p i tˆk} 1 ; (2.5) here denotes cardinality and 0 is fixed or rando according as P is FM(, 0, F ) or RM(, π 0, F ). We use the short notation FDR(SUD λ (t), 0, F ) for the quantity FDR(SUD λ (t), FM(, 0, F )), resp. FDR(SUD λ (t), π 0, F ) for the quantity FDR(SUD λ (t), RM(, π 0, F )), or siply FDR(SUD λ (t), F ) when the odel is unabiguous. Definition 2. A c.d.f. F F is a least favorable configuration (LFC) for the FDR of SUD λ (t) in the fixed ixture odel with 0 true hypotheses out of if F F, FDR(SUD λ (t), 0, F ) FDR(SUD λ (t), 0, F ). A siilar definition holds for the rando ixture odel with hypotheses and proportion π 0 of true hypotheses.
ON LEAST FAVORABLE CONFIGURATIONS 7 This definition can be restricted to a subclass G F (typically, the class of concave c.d.f.s). Finally, if F is an LFC that is coon to all values of 0 for the FM(, 0, F ) odel, then F is also an LFC in the RM(, π 0, F ) odel for any value of π 0 (by integrating over 0 B(, π 0 )). 3. Survey of FDR Coparison Results for SUD Procedures We briefly survey (and suarize in Figure 2) existing coparison results concerning the FDR of step-up-down procedures under the FM and RM odels. They concern inequalities between the FDR of different procedures under the sae odel, or of the sae procedure under different alternatives. Consider first the proble of the onotonicity of FDR(SUD λ (t)) in λ (vertical arrows). Provided F is concave, it was established by Theore 4.1 of Zeisel, Zuk and Doany (2011) that the FDR grows as the rejection set grows. In particular, since SUD λ (t) SUD λ+1 (t) for any λ {1,..., 1}, we have FDR(SUD λ (t)) FDR(SUD λ+1 (t)), (3.1) both for the FM(, 0, F ) and RM(, π 0, F ) odels. This iplies in particular that FDR(SD(t)) FDR(SU(t)) for a concave F. Siilar inequalities have been obtained under a condition on the threshold collection t, rather than on F : Theore 4.3 of Finner, Dickhaus, and Roters (2009) and Theore 3.10 of Gontscharuk (2010) establish that, when k t k /k is nondecreasing, for any λ {1,..., 1}, FDR(SUD λ (t)) FDR(SU(t)), (3.2) both for the FM(, 0, F ) and RM(, π 0, F ) odels. Though (3.1) and (3.2) suggest that FDR(SU(t)) is generally larger than FDR(SD(t)), we show by a counterexaple in Section S1.1 of Blanchard et al. (2013) that this is cannot hold in the absence of any assuptions on F or t. We turn to the onotonicity of FDR(SUD λ (t), F ) in F. For the step-up, Theore 5.3 of Benjaini and Yekutieli (2001) states that F F iplies FDR(SU(t), F ) FDR(SU(t), F ) if k t k /k is nondecreasing, with the inequality reversed if k t k /k is nonincreasing. For the step-down and under the RM(, π 0, F ) odel, Theore 4.1 of Roquain and Villers (2011a) states that the Dirac-unifor configuration is an LFC under a (coplicated) condition on t, that is fulfilled in particular by the linear threshold collection t k = αk/, α (0, 1) over the class of concave c.d.f.s. These results thus establish the LFC property of the DU(, 0 ) distribution for SU and SD procedures under appropriate assuptions (F concave and linear threshold faily being sufficient).
8 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS For SUD procedures that are neither step-up nor step-down (i.e. λ / {1, }), the only coparison result we know of is asyptotic in. Precisely, cobining Theore 4.3 of Finner, Dickhaus, and Roters (2009) and Lea 3.7 of Gontscharuk (2010), we obtain the following: Theore 1 (Gontscharuk (2010)). Let t be a threshold collection of the for t k = ρ(k/), where ρ satisfies (2.2) and (2.3). Let λ be a sequence such that λ / κ [0, 1], and 0 () a sequence such that 0 ()/ ζ [0, 1].If SUD λ (t) / converges in probability as under the DU(, 0 ()) distribution, then we have under the odel FM(, 0 (), F ), for any F F: li sup { FDR(SUDλ (t), F ) FDR(SUD λ (t), F 1) } 0, (3.3) for all ζ [0, 1] if κ > 0 or for all ζ [0, 1) if κ = 0. These results leave open the question of (nonasyptotic) LFCs for SUD procedures. Still, as a whole they see to point towards DU(, 0 ) as an LFC at least for a linear threshold faily and concave F, thus propting the following conjecture that otivates the contributions of this paper: Conjecture 1. For any 2, the Dirac-unifor configuration is an LFC for the FDR of any linear step-up-down procedure in the RM(, π 0, F ) and FM(, 0, F ) odels, at least over the class of concave F. 4. Analysis of Conjecture 1 4.1. Disproving the conjecture with a nuerical counterexaple The exact calculations described in Section 5 allow us the nuerical coputation of FDR(LSUD λ (t)), which leads to the following: Nuerical result. Put = 10, α = 0.5, and take F 0 (x) = x. Then, for any λ {4, 5, 6, 7} we have FDR(LSUD λ, F ) > FDR(LSUD λ, F 1), (4.1) under the FM(, 0, F ) odel with 0 = 7 and F = F 0, and under the FM(, 0, F ) odel with π 0 = 7/10 and F = F 0. We display the corresponding values graphically in Figure 3 (top), along with other FDR values coputed for the Gaussian alternative c.d.f. F = Φ(Φ 1 ( ) µ), providing additional configurations for which (4.1) holds (e.g., µ = 1 and λ = 5). The case F (x) = x corresponds to µ = 0, while F 1 corresponds to µ = +. The results for the FM(, 0, F ) (top left panel) and RM(, π 0, F ) (top right panel) odels are qualitatively the sae.
ON LEAST FAVORABLE CONFIGURATIONS 9 Figure 2. An arrow A B eans FDR(A) FDR(B). These results hold for the fixed ixture (FM) odel except when RM is written. The brackets are a shortened reference to the corresponding literature, see ain text for ore details. While this disproves Conjecture 1, as expected fro the asyptotic analysis of Theore 1, as becoes larger, the aplitude of the phenoenon vanishes, see Figure 3 (botto).the next question is therefore a ore precise quantification of the difference of the two sides of (4.1). Reark 1. The results were double-checked via extensive and independent Monte-Carlo siulations in order to exclude the possibility that the reported phenoenon could be an artifact produced by accuulated rounding errors in the nuerical coputation. Reark 2. For counterexaple 4.1, we used the identity c.d.f. F 0. By continuity of the exact forulas obtained in Section 5 as a function of the values F (t i ), we conclude that the LFC conjecture is also false over any restricted subclass F F as soon as F 0 is an adherent point of F (in the sense of weak convergence of probability easures). Many standard classes have this property, such as the
10 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS Fixed ixture 0 / = 0.7 Rando ixture π 0 = 0.7 Figure 3. The LFC of LSUD is not always DU. FDR(LSUD λ ) as a function of the order λ {1,..., }. Left: fixed ixture; right: rando ixture. The graphs were obtained under a one-sided Gaussian location odel with paraeter µ. The target FDR level is α = 0.5. one-sided location class F = {F µ : F µ (x) = D(D 1 (x) µ), µ > 0} and the scale class F = {F σ : F σ (x) = 2D(D 1 (x/2)/σ), σ > 1}, for which D is soe upper tail distribution function (see, e.g., Section 2.1 of Neuvial and Roquain (2012) for details). 4.2. Nonasyptotic bound We now derive a nonasyptotic version of Theore 1. The ain idea is to consider the SUD procedure as a function operating on c.d.f.s, and develop a perturbation analysis when the epirical c.d.f. of the p-values is δ-close to the
ON LEAST FAVORABLE CONFIGURATIONS 11 population c.d.f. (which happens with large probability). In this perspective, we introduce the following notation: Definition 3. Let ρ : [0, 1] [0, 1] be a continuous and nondecreasing function. For any nondecreasing function G : [0, 1] [0, 1], and l [0, 1], define { in {u [l, 1] : G(ρ(u)) u} if G(ρ(l)) l; U(l, G) := (4.2) ax {u [0, l] : G(ρ(u)) u} if G(ρ(l)) < l. Here, the infiu and supreu are well-defined, since the considered sets are nonepty. Since G is nondecreasing, U(l, G) is a fixed point of the function G ρ (so that the infiu is indeed a iniu and the supreu, a axiu). Denote by Ĝ(x) := 1 i=1 1{p i x} the epirical c.d.f. of the p-values. The following lea establishes the close connection between the functional U and the SUD λ procedure: Lea 1. For t k := ρ(k/), it holds ( ) λ U, Ĝ k ( ) λ U, (Ĝ + 1 ) 1, (4.3) where k is the nuber of hypotheses rejected by the SUD λ procedure (2.1). The ain result of this section is proved in Section 7.2: Theore 2. Let ρ : [0, 1] [0, 1] satisfy (2.2) (2.3) and let t k := ρ(k/). For ζ, δ (0, 1) arbitrary constants, let ( ) ( ) λ λ := U, (GDU ζ + δ) 1 and u δ := U, (GDU ζ δ) 0, u + δ where G DU ζ (x) := (1 ζ) + ζx; and, for any y (0, 1), let ( ρ(u + ε(δ,, ζ, y) := δ ) ρ(u δ ) u + δ ) ζ + 4ζ 1 ζ e 2(δ y 1/)2 + (1 y/ζ) +. (4.4) Then, for any F F and λ {1,..., }, in the FM(, 0, F ) odel with 0 < 0 <, we have ( FDR(SUD λ (t), 0, F ) FDR(SUD λ (t), 0, F 1) + ε δ,, 0, 1 ). (4.5) In the RM(, π 0, F ) odel, for 3 we have ( FDR(SUD λ (t), π 0, F ) FDR(SUD λ (t), π 0, F 1)+ε δ,, π 0, log ) + 2. (4.6)
12 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS Figure 4. Illustration of u + δ and u δ for the LSUD and the SUD based on the AORC. Here, the horizontal axis is on the threshold scale t = ρ(u). ζ = 0.5; δ = 0.03. The area between (G DU ζ δ) 0 and (G DU ζ + δ) 1 is displayed in gray. The bounds are valid for any δ (0, 1), so one can iniize (4.4) over δ to get the sharpest bound. The first ter in (4.4) is a bound on the perturbation of the FDR copared to the population case when G G DU ζ δ. The second ter bounds the probability of the copleent of the latter event. As grows, δ = δ should be chosen to decrease to zero, while ensuring that the second ter reains negligible. We now analyze the behavior of the first ter as δ becoes sall; and in Section 4.3, we give the resulting rate as and δ is chosen appropriately. As a first exaple, consider ρ(x) = αx (LSUD, Figure 4, left panel); then u δ = (1 ζ δ)/(1 αζ) 0 and u+ δ = (1 ζ + δ)(1 αζ) 1. Hence, (ρ(u+ δ ) ρ(u δ ))ζ/u+ δ (2αζδ)/(1 ζ + δ). As a result, (4.5) and (4.6) hold by replacing the first ter of ε(δ,, ζ, y) by (2αζδ)/(1 ζ + δ). Consider now ρ(u) = αu 1 u(1 α), that is, ρ 1 (t) = t α + t(1 α), (4.7) with ζ > α. The rejection curve ρ 1 (Figure 4 right panel), is the asyptotically optial rejection curve (AORC), introduced by Finner, Dickhaus, and Roters (2009). Here, equation (G DU ζ + δ)(u) = ρ(u) defining u + δ is quadratic with two roots in [0, 1]; let v δ be the largest one. If λ/ v δ, we have u + δ = v δ and obtain a singular situation where the bound ρ(u + δ ) ρ(u δ ) reains large for sall δ (see Figure 4 for an illustration). This is related to the known fact that
ON LEAST FAVORABLE CONFIGURATIONS 13 AORC-based SUD procedures becoe unstable if λ/ is too close to 1. In the regular case where λ/ < v δ, u + δ is the saller root; the two points ρ(u δ ) and ρ(u + δ ) are converging to each other as δ becoes sall and the bound given in Theore 2 vanishes. The exact expressions of u δ, u+ δ, and v δ in this case can be derived by solving the corresponding quadratic equations. We then have v δ = 1 δα/(ζ α) + O(δ 2 ), u + δ = (1 ζ)/(1 α) + δζ/(ζ α) + O(δ2 ), and u δ = (1 ζ)/(1 α) δζ/(ζ α) + O(δ2 ). Since ρ ((1 ζ)/(1 α)) = α/ζ 2, we have ρ(u + δ ) ρ(u δ ) = 2αδ/(ζ2 αζ) + O(δ 2 ). Assuing λ/ < v δ, we find that the first ter of ε(δ,, ζ, y) is equivalent to [2α(1 α)/(ζ α)(1 ζ)]δ as δ tends to zero. 4.3. Convergence rate when tends to infinity We use Theore 2 to obtain an explicit bound on the convergence rate of the LHS expression in (3.3) for specific critical value functions. Corollary 1. Let α (0, 1). Let t k := ρ(k/), for either ρ(x) := αx or ρ given by (4.7) (AORC). Consider the SUD procedure with threshold collection t = (t k ) 1 k and of order λ = λ possibly depending on. With (ζ ) (α, 1), consider either the FM(, 0, F ) odel with 0 = ζ or the RM(, π 0, F ) odel with π 0 = ζ. Assue [1/(1 ζ )] (log )/ = o(1); for the AORC case, assue additionally li inf ζ > α and [1/(1 λ /)] (log )/ = o(1). Then for any F F, ( 1 log ) (FDR(SUD λ (t), F ) FDR(SUD λ (t), F 1)) + =O. (4.8) 1 ζ The assuption ζ > α is not restrictive, since when ζ α, controlling the FDR is a trivial proble: the procedure rejecting all the hypotheses has an FDR (and even an FDP) saller than ζ α. While liited for siplicity to the linear and AORC rejection curves, the conclusion of Corollary 1 is ore inforative than that of Theore 1. For ζ = ζ (0, 1) fixed independently of, the convergence in (4.8) occurs at a paraetric rate, up to a log factor. Furtherore, the ultiplicative constant in the O( )-notation can be derived explicitly using the arguents fro the previous section. Finally, for ζ tending to 1 slower than (log )/ ( oderately sparse case), the bound still converges to zero, though at a slower rate. The assuptions of Corollary 1 are ore restrictive than those of Theore 1 however, excluding in particular the case where ζ tends to 1 faster than (log )/ ( highly sparse case). 5. Exact Forulas We suarize here soe forulas derived by Roquain and Villers (2011a,b) to calculate the joint distribution of the nuber of false discoveries and the nuber of discoveries, and additionally contribute a new recursion that akes
14 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS these forulas fully usable in all considered odels. These calculations were used to state the nuerical result of Section 4.1, and also provide an exact expression for the FDP distribution. 5.1. A generalization of Steck s recursion We need as an auxiliary result the ultidiensional cuulative distribution function of order statistics in a two population odel. Let k 0. For any t = (t 1,..., t k ), let Ψ k (t) = Ψ k (t 1,..., t k ) := P [ U (1) t 1,..., U (k) t k ], (5.1) where (U i ) 1 i k is a sequence of i.i.d. rando variables uniforly distributed on (0, 1), and with Ψ 0 ( ) 1. These functions can be evaluated using Steck s recursion Ψ k (t) = (t k ) k k 2 ( k ) j=0 j (tk t j+1 ) k j Ψ j (t 1,..., t j ) (Shorack and Wellner (1986)). We generalize this to the case of two populations. For 0 k 0 k and any threshold collection t = (t 1,..., t k ), let Ψ k,k0,f (t 1,..., t k ) := P [ U (1) t 1,..., U (k) t k ], (5.2) where (U i ) 1 i k is a sequence of independent rando variables, with (U i ) 1 i k0 uniforly distributed on (0, 1), and (U i ) k0 +1 i k having c.d.f. F. By convention, Ψ 0,0,F ( ) = 1. To our knowledge, existing forulas for coputing Ψ k,k0,f have a coplexity exponential with k (Glueck et al. (2008)). We propose a substantially less coplex coputation: Proposition 1. For 0 k 0 k, Ψ k,k0,f (t 1,..., t k ) = (t k ) k 0 F (t k ) k k 0 0 j 0 j k 2 j 0 k 0 j j 0 k k 0 ( k0 j 0 )( ) k k0 j j 0 (t k t j+1 ) k 0 j 0 (F (t k ) F (t j+1 )) k k 0 j+j 0 Ψ j,j0,f (t 1,..., t j ). (5.3) The above forula uses 0 0 := 1. This proposition is proved in Section 7.4. The case k = k 0 reduces to the standard Steck s recursion. 5.2. Reforulating the result of Roquain and Villers (2011) For any threshold collection t = (t k ) 1 k, any F F, and for any π 0 [0, 1], 0 k, 0 j k, set ( )( ) j P,π0,F (t, k, j) = π j 0 j k j πk j 1 (t k ) j (F (t k )) k j
ON LEAST FAVORABLE CONFIGURATIONS 15 Ψ k (1 G(t ),..., 1 G(t k+1 )); (5.4) ( )( ) j P,π0,F (t, k, j) = π j 0 j k j πk j 1 (1 G(t k+1 )) k Ψ k,j,f (t 1,..., t k ), (5.5) where G(t) = π 0 t + (1 π 0 )F (t). For any 0 {0,..., }, k 0, k, j 0, k j 0, set ( )( ) 0 0 Q,0,F (t, k, j) = (t k ) j (F (t k )) k j j k j Ψ k,0 j,f (1 t,..., 1 t k+1 ); (5.6) ( )( ) 0 0 Q,0,F (t, k, j) = (1 t k+1 ) 0 j (1 F (t k+1 )) 0 k+j j k j Ψ k,j,f (t 1,..., t k ), (5.7) where F (t) = 1 F (1 t). The following theore suarizes soe of the results of Roquain and Villers (2011a,b). Theore 3 (Roquain and Villers (2011)). Let R = SUD λ (t) be a step-up-down procedure of order λ {1,..., } with a threshold collection t. Let V := R {1,..., 0 } be the set of false rejections. Then, (i) In the RM(, π 0, F ) odel, for any π 0 [0, 1], F F, 0 k, 0 j k, { P,π0,F (t t λ, k, j) for k < λ, P( V = j, R = k) = (5.8) P,π0,F (t t λ, k, j) for k λ. (ii) In the FM(, 0, F ) odel, for any 0 {0,..., }, F F, 0 k, 0 (k + 0 ) j 0 k, { Q,0,F (t t λ, k, j) for k < λ, P( V = j, R = k) = (5.9) Q,0,F (t t λ, k, j) for k λ. The above forulas in cobination with Proposition 1 allow the exact coputation of the full joint distribution of ( V, R ) in the considered odels, therefore also of the distribution of the FDP (which equals V /( R 1)), and of the FDR, its expectation. The coputations were found to be nuerically tractable up to of the order of several hundreds. 6. Discussion 6.1. Rejection curve calibration We discuss soe practical consequences of our work. For concreteness, we consider the critical value function tα ρ β (t) := 1 + β t(1 α), (6.1)
16 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS and the associated threshold collection t β ; α corresponds to the target FDR and β is a tuning paraeter. This has been proposed as an ad hoc adjustent of the asyptotically optial rejection curve (4.7) by Finner, Dickhaus, and Roters (2009), further studied by Finner, Gontscharuk, and Dickhaus (2012). The AORC itself, obtained for β = 0, does not ensure strict FDR control at level α for any finite. The question is how to choose β as sall as possible while ensuring control of the FDR at level α. In the case of a step-up procedure based on this function, it is known that the LFC is a Dirac-Unifor distribution (see Section 3). Therefore, to calibrate β, we can apply the principle delineated at (1.1): find β(α) so that sup FDR(SU(t β ), 0, F 1) = α. 0 {0,...,} The above equation can be solved by nuerical search; for each value of β the lefthand side can be coputed either by exact coputation or by Monte-Carlo approxiation. This approach has been advocated by Finner, Dickhaus, and Roters (2009); Finner, Gontscharuk, and Dickhaus (2012) and Gontscharuk (2010), using the exact coputations for the FDR of SUD procedures under Dirac-Unifor distributions derived by Dickhaus (2008). In the case of a ore general step-up-down procedure, the exact LFC is not known; oreover, we have checked (using a nuerical counterexaple) that the negative result of Section 4.1 also holds for the critical value function ρ β. Therefore, the approach delineated above cannot rigorously be applied for exact calibration of β. There is then interest in alternative approaches based on an upper bound on the FDR. Here there is the in-depth analysis of Finner, Gontscharuk, and Dickhaus (2012) for SUD procedures, and the elegant result obtained by Gavrilov, Benjaini, and Sarkar (2009), naely, that β β = 1/ leads to an FDR upper bounded by α for the special case of the step-down procedure (λ = 1). 6.2. Liitations of FDR as a ultiple testing criterion While the FDR is widely used in practice, one can question how appropriate it is to base the ultiple type I error criterion solely on controlling the expectation of the FDP. The results of Section 5 ay be used to study nuerically this issue by exact coputation of the point ass function of the FDP under arbitrary configurations for the alternative. Based on this, we investigated the extent to which the distribution of the FDP concentrates around its expectation for a siple Gaussian location odel with paraeter µ. A graphical representation of this distribution in soe specific cases is reported in Blanchard et al. (2013).
ON LEAST FAVORABLE CONFIGURATIONS 17 We have found that the distribution of the FDP is not concentrated around the corresponding FDR if the effect size µ is close to zero (weak signal) or if the proportion π 0 of true null hypotheses is close to 1 (sparse signal). Thus, even though joint independence of the p-values holds, controlling the FDR alone does not guarantee a sall FDP in these cases. Furtherore, when µ and π 0 are fixed with, such a spread of FDP distribution can arise when adding dependencies between test statistics, see Finner, Dickhaus, and Roters (2007); Delattre and Roquain (2011). To alleviate such shortcoings, control of the false discovery exceedance (i.e., of the probability that the FDP exceeds a given threshold) has recently been proposed, see Farcoeni (2008) for a review. This brings the question of the corresponding LFC: are Dirac-unifor configurations least favorable for, e.g., P(FDP(LSU) > x)? We have found nuerically that this is not the case for any x. Hence, finding (possibly approxiate) LFCs for the false discovery exceedance reains an open proble. 7. Proofs 7.1. Proof of Lea 1 Let Ĝ = (Ĝ + 1 ) 1. Note that for any k {1,..., }, p (k) t k is equivalent to Ĝ(ρ(k/)) k/. We first analyze the case where p (λ) > t λ, that is, Ĝ(ρ(λ/)) < λ/, and SUD λ behaves as a step-up, so that: k { k = ax {0,..., λ ( ( k } : Ĝ ρ )) k } = ax { u [ 0, λ ] } λ ) : Ĝ (ρ(u)) u = U(, Ĝ, where the second equality holds because U(l, Ĝ), being a fixed point of the function Ĝ ρ, belongs to {0, 1/,..., /}. This iplies (4.3). Assue now p (λ) t λ, that is, Ĝ(ρ(λ/)) λ/, and SUD λ behaves as a step-down. First, suppose k < holds. Then, on the one hand, k + 1 { k { λ + 1 = in,..., } ( k : Ĝ (ρ < )) k } { k { λ = in,..., ( k : Ĝ (ρ < } )) k } { k { λ = in,..., ( ( k : Ĝ } ρ )) k }, (7.1) because Ĝ(ρ(k/)) is an integer. On the other hand, since Ĝ (ρ(λ/)) λ/ and U(λ/, Ĝ ) is an integer, we have λ ) { U(, Ĝ = in u [ λ ( ) }, 1] : Ĝ ρ(u) u
18 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS { { λ = in u,..., } ( ) : Ĝ ρ(u) } u. (7.2) Cobining (7.1) and (7.2), we conclude that ( k +1)/ = U(λ/, Ĝ ), and (4.3) holds. Finally, if k =, then for any k/ {λ/,..., /} we have Ĝ (ρ(k/)) k/. Hence, for all k/ {λ/,..., ( 1)/}, Ĝ (ρ(k/)) > k/ which entails U(λ/, Ĝ ) = 1. Hence, k/ = U(λ/, Ĝ ), also iplying (4.3). 7.2. Proof of Theore 2 Consider first the FM(, 0, F ) odel. We establish the following slightly ore general inequality: for arbitrary δ, ζ (0, 1), with ν 0 = ax k {0 1, 0 } { k/ ζ } [0, 1], we have FDR(SUD λ (t), 0, F ) FDR(SUD λ (t), 0, F 1) + 0 ζ ε (δ,, ζ, ν 0). (7.3) Inequality (4.5) is then obtained for ζ = 0 / and ν 0 = 1/. Preliinaries. With Ĝ(x) the epirical p-value c.d.f., and G DU ζ (x) := (1 ζ) + ζx, put û := ˆk/, where ˆk is defined by (2.1). Definition 3 iplies that if G, G are two nondecreasing functions such that x [0, 1], G(x) G (x), then U(λ/, G) U(λ/, G ). Based on the bound (4.3), we deduce that δ (0, 1), { sup x [0,1] Ĝ(x) G DU ζ (x) δ 1 } { u δ û u+ δ }. (7.4) We now assue the DU(, k) odel. In this case, we have (a.s.) p k = 0 for k k + 1, so that Ĝ = ( k)/ + k/ĝk, iplying in turn G DU ζ (x) Ĝ(x) = (k/ ζ)(1 Ĝk(x)) ζ(ĝk(x) x). For any ν > 0 satisfying k/ ζ ν, we deduce fro (7.4) the following relation, which is valid up to a negligible set: { Ω δ (k) := sup x [0,1] Ĝk(x) x ζ 1 (δ ν 1 ) } { u δ û u+ δ }. (7.5) Under the DU(, k) odel, Ĝk is the epirical c.d.f. of k i.i.d. unifor variables. Using the DKW inequality with Massart s (1990) optial constant, we get { P DU(,k) [(Ω δ (k)) c ] 2 exp 2k (δ ν 1/)2 } + ζ 2 { ( 2 exp 2 δ ν 1 ) 2 (1 ν } + ζ ) +, (7.6) because k/ ζ ν and ζ 1.
ON LEAST FAVORABLE CONFIGURATIONS 19 Upper bound. Let q(x) = ρ(x)/x when x (0, 1] and q(0) = li x 0 + ρ(x)/x (this liit exists since q is nondecreasing). Applying Theore 4.3 of Finner, Dickhaus, and Roters (2009), we obtain FDR(SUD λ (t), 0, F ) 0 E DU(, 0 1)[q(û)] 0 ρ(u + δ ) 0 u + δ ρ(u + δ ) u + δ + 0 P DU(, 0 1)[(Ω δ ( 0 1)) c ], + 0 2 exp { 2 ( δ ν 1 ) 2 (1 ν } + ζ ) +, (7.7) using (7.5) (7.6) with k := 0 1 and for any ν >0 satisfying ν [( 0 1)/] ζ. Lower bound. In the odel DU(, 0 ) with 0 <, we have û > 0 a.s. and thus, using (7.5) (with k := 0 ), FDR(SUD λ (t), 0, F 1) = ( 0 E Ĝ0 (ρ(û)) ) DU(, 0 ) û ( 0 E Ĝ0 (ρ(û)) ) DU(, 0 ) 1{Ω δ ( 0 )}. û Now, the latter is larger than or equal to 0 E DU(, 0 ) 0 ρ(u δ ) u + δ ( Ĝ0 (ρ(u δ )) u + δ 0 ) 1{Ω δ ( 0 )} 1 1 ζ P DU(, 0 ) [(Ω δ ( 0 )) c ], because u + δ 1 ζ. Fro (7.6), we obtain the lower bound FDR(SUD λ (t), 0, F 1) 0 ρ(u δ ) u + δ 0 1 { 1 ζ 2 exp 2 ( δ ν 1 (1 ν } + ζ ) +, (7.8) for any ν > 0 satisfying ν 0 / ζ. Finally, (7.7) and (7.8) entail (7.3). Proof for rando ixture odel. In the RM(, π 0, F ) odel, conditional on 0, we recover the FM(, 0, F ) odel and (7.3) holds. Taking ζ = π 0, the distribution of 0 is binoial with paraeters (, ζ). In particular, ν 0 is rando. However, we have for any γ (0, 1): E[FDP(SUD λ (t), 0 )] E [FDP(SUD λ (t), 0 )1{ν 0 γ}] ) 2
20 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS ( 0 +P π 0 > γ 1 ). (7.9) Using Hoeffding s (1963) inequality, we can write ( 0 P π 0 > γ 1 ) 2e 2(γ 1/)2 +. (7.10) We now cobine (7.9) and (7.10) with (4.5), and take γ := log / log /2 + 1/ as soon as 3. This finishes the proof. 7.3. Proof of Corollary 1 Consider the FM(, 0, F ) odel with 0 < (the case 0 = is trivial). Consider δ (2/, 1) satisfying for large, ( 2 1 1 )(δ 2 ) 2 log = ζ, (7.11) so that e 2(δ 2/)2 + (1 1/(ζ)) + = 1/ for large. Since ζ > α > 0 by assuption, we have δ (log )/. Fro Theore 2, it is sufficient to prove that ρ(u + δ ) ρ(u δ ) ( δ ) = O. 1 ζ u + δ Fro the exaples of Section 4.2, this holds for the linear critical value function. This also holds for the AORC as soon as λ / < v δ for large and 1/(ζ α) = O(1), which is the case by assuption (the forulas provided at the end of Section 4.2 are also valid when ζ depends on, because the O( ) are unifor in ζ, for ζ bounded away fro α). The proof in the RM(, π 0, F ) odel is siilar. 7.4. Proof of Proposition 1 We follow the proof of the regular Steck s recursion, see Shorack and Wellner (1986) p. 366 369. By using the convention U (0) = t 0 = 0 and by considering the sallest j for which U (j+1) > t j+1, we can write P(U (k) t k ) P [ U (1) t 1,..., U (k) t k ] k 2 = P( i j, U (i) t i, U (j+1) > t j+1, U k t k ) = j=0 k 2 j=0 X {1,...,k}, X =j P( i j, U (i) t i, i / X, t j+1 U i t k ).
ON LEAST FAVORABLE CONFIGURATIONS 21 Hence, if U (i:x) denotes the ith sallest eber of the set {U i, i X}, we obtain (t k ) k 0 F (t k ) k k 0 Ψ k (t 1,..., t k ) = P(U (k) t k ) P [ U (1) t 1,..., U (k) t k ] = = k 2 j=0 k 2 X {1,...,k}, X =j j j=0 j 0 =0 X {1,...,k}, X =j P( i / X, t j+1 U i t k ) ( )( k0 k k0 = j 0 j j 0 0 j 0 j k 2 j 0 k 0 j j 0 k k 0 P( i j, U (i:x) t i )P( i / X, t j+1 U i t k ) 1{ X {1,..., k 0 } = j 0 }Ψ j,j0,f (t 1,..., t j ) ) Ψ j,j0,f (t 1,..., t j ) (t k t j+1 ) k 0 j 0 (F (t k ) F (t j+1 )) k k 0 j+j 0. Acknowledgeents This work was supported by the French Agence Nationale de la Recherche (ANR grant references: ANR-09-JCJC-0027-01, ANR-PARCIMONIE, ANR-09- JCJC-0101-01) and the French inistry of foreign and european affairs (EGIDE - PROCOPE project nuber 21887 NJ). References Benjaini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to ultiple testing, J. Roy. Statist. Soc. Ser. B 57, 289-300. Benjaini, Y. and Yekutieli, D. (2001). The control of the false discovery rate in ultiple testing under dependency, Ann. Statist. 29, 1165-1188. Blanchard, G., Dickhaus, T., Roquain, E. and Villers, F. (2013). On least favorable configurations for step-up-down tests. Suppleent. Delattre, S. and Roquain, E. (2011). On the false discovery proportion convergence under Gaussian equi-correlation, Statist. Probab. Lett. 81, 111-115. Dickhaus, T. (2008). False Discovery Rate and Asyptotics, PhD thesis, Heinrich-Heine- Universität Düsseldorf. Dudoit, S. and van der Laan, M. J. (2008). Multiple Testing Procedures with Applications to Genoics, Springer Series in Statistics, Springer, New York. Farcoeni, A. (2008). A review of odern ultiple hypothesis testing, with particular attention to the false discovery proportion, Statist. Methods Med. Res. 17, 347-388. Finner, H., Dickhaus, T. and Roters, M. (2007). Dependency and false discovery rate: asyptotics, Ann. Statist. 35, 1432-1455. Finner, H., Dickhaus, T. and Roters, M. (2009). On the false discovery rate and an asyptotically optial rejection curve, Ann. Statist. 37, 596-618.
22 G. BLANCHARD, T. DICKHAUS, É. ROQUAIN AND F. VILLERS Finner, H. and Gontscharuk, V. (2009). Controlling the failywise error rate with plug-in estiator for the proportion of true null hypotheses, J. Roy. Statist. Soc. Ser. B 71, 1031-1048. Finner, H., Gontscharuk, V. and Dickhaus, T. (2012). False discovery rate control of step-updown tests with special ephasis on the asyptotically optial rejection curve, Scandinavian Journal of Statistics 39, 382-397. Finner, H. and Roters, M. (2001). On the false discovery rate and expected type I errors, Bio. J. 43, 985-1005. Gavrilov, Y., Benjaini, Y. and Sarkar, S. K. (2009). An adaptive step-down procedure with proven FDR control under independence, Ann. Statist. 37, 619-629. Glueck, D. H., Karipour-Fard, A., Mandel, J., Hunter, L. and Muller, K. E. (2008). Fast coputation by block peranents of cuulative distribution functions of order statistics fro several populations, Coun. Statist. Theory Methods 37, 2815-2824. Gontscharuk, V. (2010). Asyptotic and Exact Results on FWER and FDR in Multiple Hypothesis Testing, PhD thesis, Heinrich-Heine-Universität Düsseldorf. Guo, W. and Rao, M. B. (2008). On control of the false discovery rate under no assuption of dependency, J. Statist. Plann. Inference 138, 3176-3188. Hoeffding, W. (1963). Probability inequalities for sus of bounded rando variables, J. Aer. Statist. Assoc. 58, 13-30. Lehann, E. L. and Roano, J. P. (2005). Generalizations of the failywise error rate, Ann. Statist. 33, 1138-1154. Massart, P. (1990). The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality, Ann. Probab. 18, 1269-1283. Miller, C. J., Genovese, C., Nichol, R. C., Wasseran, L., Connolly, A., Reichart, D., Hopkins, A., Schneider, J. and Moore, A. (2001). Controlling the false-discovery rate in astrophysical data analysis, The Astronoical Journal 122, 3492-3505. Neuvial, P. and Roquain, E. (2012). On false discovery rate thresholding for classification under sparsity, Ann. Statist. 40, 2572-2600. Pantazis, D., Nichols, T. E., Baillet, S. and Leahy, R. M. (2005). A coparison of rando field theory and perutation ethods for statistical analysis of MEG data, NeuroIage 25, 383-394. Roano, J. P. and Wolf, M. (2007). Control of generalized error rates in ultiple testing, Ann. Statist. 35, 1378-1408. Roquain, E. and Villers, F. (2011a). Exact calculations for false discovery proportion with application to least favorable configurations, Ann. Statist. 39, 584-612. Roquain, E. and Villers, F. (2011b). Suppleent to Exact calculations for false discovery proportion with application to least favorable configurations, Ann. Statist. Available online. Sarkar, S. K. (2002). Soe results on false discovery rate in stepwise ultiple testing procedures, Ann. Statist. 30, 239-257. Shorack, G. R. and Wellner, J. A. (1986). Epirical processes with applications to statistics, Wiley Series in Probability and Matheatical Statistics: Probability and Matheatical Statistics, John Wiley & Sons Inc., New York. Sies, R. J. (1986). An iproved Bonferroni procedure for ultiple tests of significance, Bioetrika 73, 751-754. Soerville, P. N. and Heelann, C. (2008). Step-up and step-down procedures controlling the nuber and proportion of false positives, Coput. Statist. Data Anal. 52, 1323-1334.
ON LEAST FAVORABLE CONFIGURATIONS 23 Tahane, A. C., Liu, W. and Dunnett, C. W. (1998). A generalized step-up-down ultiple test procedure, Canad. J. Statist. 26, 353-363. Zeisel, A., Zuk, O. and Doany, E. (2011). FDR control with adaptive procedures and FDR onotonicity, Ann. Appl. Statist. 5, 943-968. Universität Potsda, Institut für Matheatik, A neuen Palais 10, 14469 Potsda, Gerany. E-ail: gilles.blanchard@ath.uni-potsda.de Huboldt-University Berlin, Departent of Matheatics, Unter den Linden 6, D-10099 Berlin, Gerany. E-ail: dickhaus@ath.hu-berlin.de Université de Paris 6 (UPMC), Case courrier 188, 4, Place Jussieu, 75252 Paris Cedex 05, France. E-ail: etienne.roquain@upc.fr Université de Paris 6 (UPMC), Case courrier 188, 4, Place Jussieu, 75252 Paris Cedex 05, France. E-ail: fanny.villers@upc.fr (Received August 2011; accepted October 2012)