Statistical Properties of Bernstein Copulae with Applications in Multiple Testing

Transcription

1 Mathematical Statistics Stockholm University Statistical Properties of Bernstein Copulae with Applications in Multiple Testing André Neumann Taras Bodnar Dietmar Pfeifer Thorsten Dickhaus Research Report 2016:13 ISSN

2 Postal address: Mathematical Statistics Dept. of Mathematics Stockholm University SE Stockholm Sweden Internet:

3 Mathematical Statistics Stockholm University Research Report 2016:13, Statistical Properties of Bernstein Copulae with Applications in Multiple Testing June 2016 André Neumann a1, Taras Bodnar b, Dietmar Pfeifer c, Thorsten Dickhaus a a Institute for Statistics, University of Bremen, Bibliothekstraße 1, D Bremen, Germany b Department of Mathematics, Stockholm University, Roslagsvägen 101, SE Stockholm, Sweden c Institute of Mathematics, Carl von Ossietzky University of Oldenburg, D Oldenburg, Germany Abstract A general way to estimate continuous functions consists of approximations by means of Bernstein polynomials. Sancetta and Satchell 2004) proposed to apply this technique to the problem of approximating copula functions. The resulting so-called Bernstein copulae are nonparametric copula estimates with some desirable mathematical features like smoothness. In the present paper, we extend previous statistical results regarding bivariate Bernstein copulae to the multivariate case and study their impact on multiple tests. In particular, we utilize them to derive asymptotic confidence regions for the family-wise error rate FWER) of simultaneous test procedures which are empirically calibrated by making use of Bernstein copulae approximations of the dependency structure among the test statistics. This extends a similar approach by Stange at al. 2015) in the parametric case. A simulation study quantifies the gain in FWER level exhaustion and, consequently, power which can be achieved by exploiting the dependencies, in comparison with common threshold calibrations like the Bonferroni or the Šidák correction. Finally, we demonstrate an application of the proposed methodology to real-life data from insurance. Keywords: Asymptotic oscillation behavior, Bonferroni correction, family-wise error rate, p-value, risk management, Šidák correction, simultaneous test procedure 1 Corresponding author. address: neumann@uni-bremen.de.

4 1 Introduction Copula-based modeling of dependency structures has become a standard tool in applied multivariate statistics and quantitative risk management; see, e. g., [23], [17], [14], [12], and Chapter 5 of [22]. The estimation of an unknown copula is key to a variety of modern multivariate statistical methods. In particular, applications of copulae to the calibration and the analysis of multiple tests have been considered in [10], [3], [31], [6], [28], and [27]; see also Sections and 4.4 in [9]. Specifically, the copula-based construction of simultaneous test procedures STPs) in the sense of [13] developed in [10] and [31] under parametric assumptions regarding the type of dependencies among test statistics considerably extends previous approaches as in [15] which are confined to asymptotic Gaussianity and, consequently, linear dependencies. In the case of a parametric copula, generic estimation techniques like the generalized) method of moments or maximum likelihood estimation are established notions; cf. Section 3.2 of [31] and references therein. The empirical copula as well as its asymptotic properties as a nonparametric estimator have been studied, among others, in [25], [8], [33], and, more recently, in [5], [29], and [4], to mention only a few references. However, similarly as multivariate histogram estimators, the empirical copula in dimension m has some undesirable properties. For example, it is discontinuous, and it typically assigns zero mass to large subsets of [0, 1] m, even if the sample size n is large, due to the concentration of measures phenomenon. One way to tackle these issues consists of smoothing of the empirical copula. In particular, in [26] smoothing by Bernstein polynomials has been proposed, leading to so-called Bernstein copulae. Approximation theory for Bernstein copulae has been derived in [7], and asymptotic statistical properties of Bernstein copula estimators in the bivariate case m = 2) have been proven in [16] and [2]. Applications of Bernstein copulae to modeling dependencies in non-life insurance have been considered in [11]. In the present work, we contribute to theory and applications of Bernstein copulae in the case of a general dimension m 2. In Section 2, we extend the asymptotic theory regarding the Bernstein copula estimator by proving its rate of convergence in infinity norm as well as its asymptotic normality, together with explicit expressions for the limit variance for arbitrary m. Section 3 is then devoted to applications of Bernstein copulae in multiple testing, in particular to the calibration of multivariate STPs for control of the family-wise error rate FWER), avoiding restrictive parametric dependency assumptions. The application of the central limit theorem derived in Section 2 allows for a precise quantification of the uncertainty about the realized FWER in the case that the copula of test statistics is pre-estimated prior to calibrating the significance threshold of the STP. This extends the results in [31] to the case of nonparametric copula pre-estimation. Sec- 2

5 tion 4 demonstrates by means of a simulation study that the latter pre-estimation approach leads to a better exhaustion of the FWER level and thus enhances the power of the STP compared with traditional approaches which only take univariate marginal distributions of test statistics into account. Finally, we apply the proposed multiple testing methodology to real-life data from insurance Section 5), and we conclude with a discussion in Section 6. Lengthy proofs and some auxiliary results are deferred to Section 7. 2 Oscillation behavior of empirical Bernstein copulae Let X = X 1,..., X m ) be a random vector taking values in the probability space X, F, P ), where X R m, F is a σ-field over X, and P denotes the oint) distribution of X. The univariate marginal cumulative distribution functions cdfs) of X we denote by F, = 1,..., m, whereas C stands for the copula related to the distribution P. Assume that X 1,..., X n are stochastically independent and identically distributed i.i.d.) random vectors with X 1 P. Then, the empirical copula Ĉn pertaining to X 1,..., X n is given by Ĉ n u) = ˆF Ĥn n u) ) with ˆF n u) = ˆF 1,nu 1 ),..., ˆF m,nu m )). In this, ˆF,n denotes the generalized inverse of the marginal empirical cumulative distribution function ecdf) in coordinate 1 m, given by ˆF,n x ) := 1 n ni=1 1,x ] X i, ), and Ĥ n x) := 1 n 1,x] X i ). n i=1 The symbol 1 A denotes the indicator function of set A and, x] =, x 1 ]..., x m ]. An analogous bold-face notation for vectors will be used throughout the remainder. The Bernstein copula estimation is based on the Bernstein polynomial approximation, which is for a fixed copula C given by K B K u) := C k/k) P k,k u ), k=0 where K k=0 := K 1 k 1 =0 K m k, k/k := ) k 1 m=0 K 1,..., km K m, ) K P k,k u) := u k 1 u) K k, k 3

6 and K 1,..., K m are given positive integers. It has been proved in [7, Corollary 3.1] that lim K B K u) = C u). The Bernstein copula estimator for C is then given by K ˆB n,k u) := Ĉ n k/k) P k,k u ). k=0 Theorem 2.1 establishes the consistency of ˆB n,k. Theorem 2.1 Chung-Smirnov consistency rate). Let K = Kn) such that m K 1/2 = O n 1/2 log log n) 1/2). Then ˆB n,k C = O n 1/2 log log n) 1/2) a.s. for n, where g := sup u [0,1] g u) for g : [0, 1] R. Proof. By the triangle inequality, it holds that ˆB n,k C ˆB n,k B K + B K C. 1) From Lemma 7.2 we get that B K C = O n 1/2 log log n) 1/2). For the first summand in 1), we get ˆB n,k B K sup K u [0,1] k=0 max k {0,...,K} Ĉn k/k) C k/k) m Ĉn k/k) C k/k), P k,k u ) where {0,..., K} := {0,..., K 1 }... {0,..., K m }. Let F,n denote the marginal ecdf of U i, := F X i, ) for = 1,..., m and i = 1,..., n and let H n stand for the ecdf of U 1,..., U n. Application of the identity see, e.g. [34, Section 3]) F,n u ) = F F,n u ) ) leads to Ĉn k/k) = H n F n k/k) ) and ˆB n,k B K max k {0,...,K} max k {0,...,K} + H n F n k/k) ) C k/k) H n F n k/k) ) C F n k/k) ) 2) ) max F k,n k. 3) k {0,...,K } From [18, Theorem 2] we get that the summand in 2) is of order O n 1/2 log log n) 1/2) as well as that each summand in 3) is of order O n 1/2 log log n) 1/2). This completes the proof. 4 K K

7 In Theorem 2.2 we prove a pointwise central limit theorem for the Bernstein copula estimator. Theorem 2.2 Asymptotic normality). Assume bounded second order partial derivatives for C on 0, 1). If n 1/2 m K 1/2 0, n, then it holds for all u 0, 1) and sequences u n ) n N in 0, 1) with lim n u n = u that n 1/2 ˆBn,K u n ) C u n ) ) d N 0, σ 2 u) ), where σ 2 u) :=C u) 1 C u)) + C u)) 2 u 1 u ) 2 C u) C u) 1 u ) + 2 C u) C u) P [U i, u, U i, u ] u u ). =+1 Proof. It holds that n 1/2 ˆBn,K u n ) C u n ) ) = n 1/2 ˆBn,K u n ) B K u n ) ) +n 1/2 B K u n ) C u n )). From Lemma 7.2 we have n 1/2 B K u n ) C u n )) = O n m 1/2 K 1/2 = o 1) by our assumption on K. Let U i, = F X i, ). Since the remainder term in Lemma 7.3 does not depend on u, we conclude that n 1/2 ˆBn,K u n ) B K u n ) ) = n 1/2 n Y i,k u n )+O n 1/4 log n) 1/2 log log n) 1/4), i=1 where K Y i,k u) := 1,k/K] U i ) C k/k) k=0 C k/k) 1,k /K ] U i, ) k ) P k,k K u ). From Lemma 7.5 we get completing the proof. n 1/2 n i=1 Y i,k u n ) d N 0, σ 2 u) ), 5

8 Remark. If the marginal distributions of X are known, then the Bernstein copula estimator can be constructed by replacing Ĉn u) with Ĥn F u)). In this case, the Y i,k can be simplified to K Y i,k u) := 1,k/K] U i ) C k/k) ) m P k,k u ) k=0 with variance and it holds V [Y i,k u)] = C u) 1 C u)) + O K 1/2, n 1/2 ˆBn,K u) B K u) ) n = n 1/2 Y i,k u). i=1 3 Calibration of multivariate multiple test procedures In this section, we assume that we have uncertainty about the distribution of X. We thus consider a statistical model of the form X, F, P ϑ,c : ϑ Θ, C C)). The probability measure P ϑ,c is indexed by two parameters. The parameter C denotes the copula of X, and ϑ is a vector of marginal parameters which refer to F 1,..., F m. The model for the i.i.d. sample X 1,..., X n consequently reads as X n, F n, P ϑ,c : ϑ Θ, C C)), where P ϑ,c = Pϑ,C n. Based on this model, we consider multiple test problems of the form X n, F n, P ϑ,c : ϑ Θ, C C), H), where H = {H 1,..., H m } with H Θ for all 1 m denotes a family of m null hypotheses regarding the parameter ϑ. The copula C is not the primary target of statistical inference, but a nuisance parameter in the sense that it does not depend on ϑ. This is a common setup in multiple test theory. We will mainly consider a semi-parametric situation, where Θ is of finite dimension, while C is a function space. Remark 3.1. The assumption that the number of tests equals the dimension of X is only made for notational convenience. The case that these two quantities differ can be treated with obvious modifications. A multiple test for H is a measurable mapping ϕ = ϕ 1,..., ϕ m ) : X n, F n ) {0, 1} m, where ϕ x 1,..., x n ) = 1 means reection of the -th null hypothesis H in favor of the alternative K = Θ \ H, 1 m. We restrict our attention to multiple tests which are such that ϕ = 1 c, )T ), where T = T 1,..., T m ) : X n R m denotes a vector of real-valued test statistics which tend to larger values 6

9 under alternatives, and c = c 1,..., c m ) are the corresponding critical values. In many problems of practical interest, T will only use the data x i, ) 1 i n, for every 1 m. In particular, this typically holds true if ϑ = ϑ 1,..., ϑ m), ϑ represents a set of functionals of F, and H only concerns ϑ, for every 1 m. For the calibration of c, we aim at controlling the FWER in the strong sense. For given ϑ Θ and C C, the FWER is defined as the probability for at least one false reection type I error) of ϕ under P ϑ,c, i. e., FWER ϑ,c ϕ) = P ϑ,c I 0 ϑ) {ϕ = 1}, where I 0 ϑ) = {1 m : ϑ H } denotes the index set of true null hypotheses under ϑ. The multiple test ϕ is said to control the FWER at level α [0, 1], if sup FWER ϑ,c ϕ) α. ϑ Θ,C C Notice that, although the trueness of the null hypotheses is determined by ϑ alone, the FWER depends on ϑ and C, because the dependency structure in the data typically influences the distribution of ϕ when regarded as a statistic with values in {0, 1} m. Throughout the remainder, we assume that the following set of conditions is fulfilled. Assumption 3.2. a) Letting H 0 = m H denote the global hypothesis of H, there exists a ϑ H 0 such that C C : ϑ Θ : FWER ϑ,c ϕ) FWER ϑ,cϕ). Notice that this assumption can be weakened by considering closed test procedures, where our proposed methodology is applied to every non-empty intersection hypothesis in H; cf. Remark 1 in [32] for details. b) The vector of marginal cdfs of T = T 1,..., T m ) depends on ϑ only, and is at least asymptotically as n ) known under ϑ. We denote the vector of marginal cdfs of T = T 1,..., T m ) under ϑ by G = G 1,..., G m ). c) Letting C ϑ denote the copula of T under ϑ, there exists a continuously differentiable function h : [0, 1] [0, 1] such that C ϑ u) = h Cu)) for all u [0, 1] m. The function h may be unknown. Notice that, if T only uses the data x i, ) 1 i n, for every 1 m, then C ϑ C T is independent of ϑ. 7

10 Before we start to explain the proposed method for the calibration of c, let us illustrate prototypical example applications of our general setup. Example 3.3. a) Let Θ = R m and assume that ϑ R is the expected value of X for every 1 m. The -th null hypothesis may be the one-sided null hypothesis H = {ϑ 0} with corresponding alternative K = {ϑ > 0}. Assume that the scale parameter of the marginal distribution of each X is known and without loss of generality equal to one. A suitable test statistic T is then given by T X 1,..., X n ) = n i=1 X i, / n. Under ϑ = 0 R m, we have that G = Φ the cdf of the standard normal law on R) is the cdf of the asymptotic) null distribution of T for every 1 m. If the considered copula family C consists of multivariate stable copulae meaning that the observables follow a multivariate stable distribution), then the copula C ϑ is of the same type as C, hence all parts of Assumption 3.2 are fulfilled. b) Let X = [0, ) and assume that the stochastic representations X d = ϑ Z with ϑ > 0 hold true for all 1 m, where Z is a random variable taking values in [0, 1]. The parameter of interest in this problem is ϑ = ϑ 1,..., ϑ m ) Θ = 0, ) m. For each coordinate, we consider the pair of hypotheses H : {ϑ ϑ } versus K : {ϑ > ϑ }, for a given vector ϑ = ϑ 1,..., ϑ m) 0, ) m of hypothesized upper bounds for the supports or right end-points of the distributions) of the X s. This has applications in the context of stress testing in actuarial science and financial mathematics cf., e. g., [21]). Suitable test statistics are given by the component-wise maxima of the observables, i. e., T X 1,..., X n ) = max 1 i n X i, /ϑ, 1 m. Assuming that the tail behavior of each X is known such that the marginal limiting) extreme value distribution of T under ϑ can be derived and letting C consist of max-stable copulae, all parts of Assumption 3.2 are fulfilled here, too. Let us remark here that these two examples have been treated under the restrictive assumption of one-parametric copula families C in [31]. The following lemma is taken from [10] and connects FWER ϑ,c ϕ) with C ϑ. Lemma 3.4. Let Assumption 3.2 be fulfilled and assume that ϕ is a simultaneous test procedure STP) in the sense of [13], meaning that all m critical values are identical, say c = cα) for all 1 m. Then we have that FWER ϑ,c ϕ) 1 C ϑ 1 α loc ), 8

11 where α ) loc = 1 G cα)) denotes a local significance level for the -th marginal test problem. In practice, it is convenient to carry out the STP in terms of p-values p = 1 G T ) such that ϕ = 1 [0,α ) loc )p ). Lemma 3.4 shows that the problem of calibrating the local significance levels corresponding to cα) is equivalent to the problem of estimating the contour line of C ϑ at contour level 1 α. Any point on that contour line defines a valid set of local significance levels. Thus, one may weight the m hypotheses for importance by choosing particular points on the contour line. If all m hypotheses are equally important it is natural to choose equal local levels α ) loc α loc for all 1 m. This amounts to finding the point of intersection of the contour line of C ϑ at contour level 1 α and the main diagonal in the m-dimensional unit hypercube. Recall that we assume that C and, consequently, C ϑ are unknown. Based on our investigations in Section 2 and making use of Assumption 3.2.c), we thus propose to calibrate ϕ empirically. If h is known, this can be done by solving the equation h ˆBn,K 1 α loc ) ) = 1 α 4) for α loc, for fixed weights. If for a given α the solution of 4) with the chosen weights is not unique, one should choose the smallest set of local significance levels such that 4) holds. To facilitate the argumentation, let us from now on assume that we choose equal local levels. We denote the solution of 4) with α ) loc α loc by ˆα loc,n. This leads to the definition ˆα loc,n := 1 ˆB n,k h 1 α)), where ˆB n,k p) := inf { u [0, 1] ˆBn,K u,..., u) p } for p 0, 1). Since ˆB n,k depends on the data, ˆα loc,n is a random variable and FWER ϑ,cϕ) = 1 C ϑ 1 ˆα loc,n,..., 1 ˆα loc,n ) is a random variable, too, which is distributed around the target FWER level α. The following theorem is the main result of this section and quantifies the uncertainty about the realized FWER if the empirical calibration of ϕ is performed via 4). Theorem 3.5. Let Assumption 3.2 be fulfilled. Then FWER ϑ,c has the following properties. a) Consistency: C C : FWER ϑ,cϕ) α almost surely. 9

12 b) Asymptotic Normality: where C C : n FWER ϑ,cϕ) α ) d N 0, σ 2 α), σα 2 = σ C ϑ 1 α),..., C ϑ 1 α)) C Cϑ 1 α)) C ϑ Cϑ 1 α))) 2 and C u) := C u,..., u), C ϑ u) := C ϑ u,..., u). c) Asymptotic Confidence Region: δ 0, 1) : C C : lim n P ϑ,c FWER n ϑ,cϕ) α ˆσ n z 1 δ = 1 δ, Proof. where ˆσ 2 n : X n 0, ) is a consistent estimator of the asymptotic variance σ 2 α. In this, z β = Φ 1 β) denotes the β-quantile of the standard normal distribution on R. a) Let C C be arbitrary, but fixed. Since h is continuously differentiable, h is also Lipschitz-continuous with Lipschitz constant L > 0. Therefore, with Theorem 2.1 we get FWER ϑ,cϕ) α = 1 α Cϑ 1 ˆα loc,n ) = h ˆBn,K 1 ˆα loc,n ) ) h C 1 ˆα loc,n )) h ˆBn,K ) h C) L ˆBn,K C =O n 1/2 log log n) 1/2) a.s. b) Letting p := h 1 α), Lemma 7.6 yields that n 1 αloc,n C p)) = n ˆB n,k p) C p) ) d N 0, σ C p),..., C ) p)). C C p)) 10

13 Therefore, applying the Delta Method to C ϑ, we have that n FWER ϑ,cϕ) α ) = n Cϑ 1 ˆα loc,n ) 1 α) ) = n Cϑ 1 ˆα loc,n ) C ϑ C p)) ) d N The result follows from the definition of p. 0, σ C p),..., C p)) C C p)) c) Follows directly from part b) using Slutsky s Theorem. C ϑ C p)) ) ) 2. If the function h is unknown, one may approximate the value of ˆα loc,n with high precision by a Monte Carlo simulation for a given number M of Monte Carlo repetitions. To this end, generate M n pseudo-random vectors which follow the estimated oint) distribution of X under ϑ, by combining ˆB n,k and the marginal cdfs F 1,..., F m of X 1,..., X m under the global hypothesis. From these, calculate a pseudo-sample T 1,..., T M from the distribution of T under ϑ. Then, GT 1 ),..., GT M ) constitutes a pseudo-random sample from the estimator of C ϑ, and the empirical equi-coordinate 1 α)-quantile of this pseudo-sample approximates ˆα loc,n. Since the number M of pseudo-random vectors to be generated is in principle unlimited, Theorem 3.5 continues to hold true if this strategy is pursued. 4 Simulation study In this section we report the results of a simulation study regarding the FWER and the power of multiple tests which are empirically calibrated as proposed in Section 3. Assume w.l.o.g. that I 0 ϑ) := {1,..., m 0 } and let m 1 := m m 0. The empirical FWER and the empirical power, respectively, are given by and FWER ϑ,c ϕ) := L 1 power ϕ) := L 1 L l=1 L m 1 1 l=1 =m { } m0 x l) ) ϕ l) 1,..., x l) =1 n 1 { } x l) ) ϕ l) 1,..., x l) =1 n, where x l) ) 1,..., x l) n X n denotes the pseudo-sample in the l-th simulation run. In this simulation we use the setting given in [31, Section 4.3]. This means that we 11

14 are in the situation of Example 3.3.b) except that the copula C of X is not maxstable. The distribution of Z is given by an m-variate t 4 -copula with Beta3, 4) marginal distributions, where Σ i, := 1 {i=} + ρ1 {i }, 1 i, m. The extreme value copula of the multivariate t-distribution has been derived in [24]. It has an analytically intractable form. In [31], this extreme value copula has been approximated by a Gumbel-Hougaard copula with an estimated copula parameter. In contrast, we approximated this extreme value copula nonparametrically with the Bernstein copula estimator. The calculation of Bernstein copulae has been performed as in [7, Example 4.2], using K := n for all {1,..., m}. Since the function h is intractable, we calibrate the test with the following algorithm which was outlined at the end of Section 3. Algorithm 4.1. a) Choose a number M of Monte Carlo repetitions. b) For each b = 1,..., M draw a sample U #b 1,..., U #b n of ˆB n,k and calculate X #b i, = F ) U #b i,, 1 i n, 1 m. c) For all 1 m, compute T #b ) = T X #b 1,..., X #b n and obtain the pseudosample V #b ) = FZ n T #b from the copula of T. Notice that, due to the stochastic independence of the observables, the marginal cdfs of T are given by F n Z. d) Finally, calibrate ˆα loc,n by solving # { V #b 1 ˆα loc,n } = 1 α) M. 5) Notice that in Algorithm 4.1, we implicitly weight the hypotheses. This means that the weights corresponding to the obtained ˆα loc,n depend on the simulation data, for convenience of implementation. In comparison, the classical Bonferroni and Šidák corrected local significance levels are given by respectively. α ) loc = α m and α) loc = 1 1 α)1/m, 1 m, 12

15 m = 6 ρ = 0.2 ρ = 0.5 ρ = 0.8 m 0 = 2 m 0 = 4 m 0 = 2 m 0 = 4 m 0 = 2 m 0 = 4 Empirical FWER Bonferroni Šidák Bernstein Empirical power Bonferroni Šidák Bernstein m = 12 ρ = 0.2 ρ = 0.5 ρ = 0.8 m 0 = 5 m 0 = 10 m 0 = 5 m 0 = 10 m 0 = 5 m 0 = 10 Empirical FWER Bonferroni Šidák Bernstein Empirical power Bonferroni Šidák Bernstein Table 1: Comparison of empirical FWER and power regarding Bonferroni, Šidák and Bernstein corrections with m {6, 12}, varying m 0, ρ {0.2, 0.5, 0.8}, α = 0.05, L = 2500, M = 400, and n = 100. With the notation introduced in Example 3.3.b), we put ϑ = 2,..., 2) and ϑ = ϑ 1,..., ϑ m ) with 2, for m 0, ϑ = 2 + U, U UNI0, 0.5], otherwise. The results of the simulation study for different values of m, m 0, and ρ are displayed in Table 1. It reveals that the Bernstein correction performs better in all cases, 13

16 i. e., its empirical FWER is closer to α and its empirical power is higher than those of the generic calibrations. Of course, one could increase the difference in empirical powers between the Bernstein calibration and the Bonferroni or Šidák calibrations by choosing other values for the ϑ under the alternatives. Figure 1: Distribution of the empirical FWER regarding Bonferroni, Šidák and Bernstein correction with m = 6, m 0 = 4, ρ = 0.5, α = 0.05, L = 2500, M = 400, and n = 100. Figure 1 and Figure 2 provide a more detailed view on the distribution of the empirical FWER and the empirical power, respectively, of the three concurring calibrations from Table 1. Namely, we performed the entire simulation algorithm comprising L simulation runs with M Monte Carlo repetitions for the approximation of h in each of these runs 200 times. In the figures, we display results for one 14

17 particular parameter configuration m, m 0, ρ). For other parameter configurations, the results are very similar. Figure 2: Distribution of the empirical power regarding Bonferroni, Šidák and Bernstein correction with m = 6, m 0 = 4, ρ = 0.5, α = 0.05, L = 2500, M = 400, and n = 100. As expected from Theorem 3.5, the histograms look normal, each with a small variance. 5 Application In this section, we analyze insurance claim data from m = 19 adacent geographical regions see Table 4). For every region {1,..., 19} these claims have, 15

18 for confidentiality reasons, been adusted to a neutral monetary scale. The claim amounts and types have been aggregated to full years, such that temporal dependencies are considered negligible. However, strong non-linear spatial dependencies are likely to be present in the data. Hence, we treat each of the n = 20 rows in Table 4 as an independent repetition X i = x i of an m-dimensional random vector X = X 1,..., X m ), where 1 i 20 is the time index in years and m = 19 refers to the regions. An important quantity for regulators and risk managers is the region-specific value-at-risk VaR). The VaR at level p for region is defined as the p-quantile of the marginal) distribution of X, i. e., VaR p) := F X p). In insurance mathematics, typically considered values of p are close to one. Here, we chose p = Our goal is to derive multiplicity-corrected confidence intervals for ϑ = VaR 0.995), 1 m = 19 which are compatible with i. e., dual to) the Bonferroni, Šidák and Bernstein copula-based correction methods discussed } before. To this end, let auxiliary point hypotheses be defined as H ϑ : { ϑ = ϑ for fixed ϑ > 0. According to the Extended Correspondence Theorem see [9, Section 1.3]), the set of all values ϑ for which H ϑ is retained by a multiple test at FWER level α leading to a local significance level α ) loc in coordinate ) constitutes a confidence region at simultaneous confidence level 1 α for ϑ, 1 m. We set α = 5%. In quantitative risk management, it is common practice to model the excess distribution of X over some given threshold u by a generalized Pareto distribution GPD) cf., e. g., [22, Section 7.2.2]). Definition 5.1 [22, Definition 7.16]). For shape parameter ξ R and scale parameter β > 0, the cdf of the GPD is given by ξx/β) 1/ξ, ξ 0, G ξ,β x) = 1 exp x/β), ξ = 0, where x 0 if ξ 0 and 0 x β/ξ if ξ < 0. In the remainder, we make the following assumption. Assumption 5.2. For every 1 m = 19 there exists a threshold u parameter values ξ and β such that and P [X u x X > u ] G ξ,β x) for all x R. 16

19 Under Assumption 5.2, an approximation of the VaR at level p for region is given by VaR ξ,β p) u + β ) ξ 1 p 1 =: q ξ, β ), ξ 1 F X u ) provided that p F X u ). For ease of notation, we let ϑ = q ξ, β ) in the sequel. For computational convenience, we carried out the test for H ϑ as a confidenceregion test in the sense of [1] based on the family : { } ξ = ξ, β = β β > 0, ξ R ) 6) Hξ,β of point hypotheses. Namely, the test procedure works as follows. Algorithm Test each H ξ,β by an arbitrary level α) loc test, where α) loc denotes a multiplicitycorrected significance level based on the Bonferroni, Šidák or Bernstein copula calibration, respectively. 2. Let a confidence region C ξ,β x 1,..., x n ) at confidence level 1 α ) loc for ξ, β ) be defined as the set of all parameter values ξ, β ) for which H ξ,β is retained. 3. Reect H ϑ at level α ) loc, if the set {ξ, β ) : q ξ, β ) = ϑ } has an empty intersection with C ξ,β x 1,..., x n ). Due to Algorithm 5.3, it suffices to construct point hypothesis tests for 6). A standard technique for testing parametric hypotheses is to perform a likelihood ratio test. In the risk management context, this method is described in [22, A.3.5]. Define the random variable N u := # {1 i n X i, > u } and let X 1,,..., X Nu, denote the corresponding sub-sample for region. Then the excesses Y 1,,..., Y Nu, over u are defined by Y i, := X i, u. The test statistic for testing H ξ,β is then given by T Y1,,..., Y Nu,; ξ, β ) := 2 log Λ Y1,,..., Y Nu, ; ξ, β ), where the likelihood ratio Λ is defined by Λ ) L Y Y 1,,..., Y Nu,; ξ, β 1,,..., Y Nu,; ξ, β := sup ξ,β) L Y 1,,..., Y Nu,; ξ, β ) 17 )

20 with log-likelihood function log L Y 1,,..., Y Nu,; ξ, β ) = N u log β ξ ) Nu i=1 log 1 + ξ Y i, β ). Figure 3: Raw data and mean excess plots for regions 2 and 4. The graphs in the upper panel display the data from Table 4 for {2, 4}, respectively. The graphs in the lower panel show the corresponding mean excess plots. Under H ξ,β, T is asymptotically χ 2 -distributed with two degrees-of-freedom. This means that the asymptotic) confidence interval C ξ,β x 1,..., x n ) in the 18

21 second step of Algorithm 5.3 is given by C ξ,β X 1,..., X n ) = { ξ, β ) : T Y1,,..., Y Nu, ; ξ, β ) F 1 ) χ 1 α loc)} ) ] for ϑ based on the third step of = min q ξ, β ) Utilizing 7), the confidence region [ϑ lower, ϑ upper Algorithm 5.3 is constructed by finding the minimum value ϑ lower and the maximum value ϑ upper = max q ξ, β ), where ξ, β ) are located on the boundary of C ξ,β x 1,..., x n ). A graphical method for the determination of a suitable threshold u is based on the mean excess plot in coordinate ; see [22, Section 7.2.2] for details. Namely, all possible values u of u are plotted against the mean of the values of Y 1,,..., Y Nu,. If the GPD model is appropriate, the plot should yield an approximately linear graph for arguments exceeding u. Usually the few largest values of u are ignored, because they lead to very small values of N u. For example, Figure 3 shows the mean excess plots for the two regions 2 and 4. The mean excess plot for region 2 is approximately linear when ignoring the three smallest and the four largest values of u. This means that a suitable threshold u 2 would be between and Similarly, the mean excess plot for region 4 is approximately linear when ignoring the two largest values of u, hence u 4 < Based on such considerations, we chose the thresholds u = u 1,..., u 19 ) given by u := 1.0, 28.0, 9.0, 0.3, 0.2, 0.4, 2.6, 1.2, 0.4, 1.1, 0.1, 0.2, 22.5, 1.6, 3.2, 0.2, 12.5, 1.2, 0.5). Finally, it remains to determine the local significance levels α ) ) loc. In the 1 19 case of the Bonferroni or the Šidák method, this is trivial. To calibrate the local significance levels with the Bernstein method, we employed a modified version of Algorithm 4.1 based on the empirical excess distribution. Algorithm 5.4 yields a resampling-based approximation of the copula of the vector T = T 1,..., T m ) of the region-specific likelihood ratio test statistics. Algorithm 5.4 Calibration of the local significance levels for FWER level α). a) For every 1 m, estimate the parameters ξ and β of the excess distribution of X via maximum likelihood and calculate N u. b) Choose a number M of Monte Carlo repetitions. c) For each 1 b M draw a pseudo sample U #b 1,..., U #b n from the empirical) Bernstein copula ˆB n,k and calculate the corresponding GPD excesses Y #b i, = G ˆξ, ˆβ U #b i),), 1 i Nu, 1 m, where U #b i), denotes the i-th reverse order statistic of U #b ) i,. 1 i n 19

22 d) For each 1 m, compute T #b = T Y #b 1,,..., Y #b N u,; ˆξ, ˆβ ), and obtain the pseudo-sample from the copula of T. V #b = Ĝ,M ) T #b, 1 m e) Finally, calibrate ˆα loc,n by solving # { V #b 1 ˆα loc,n } = 1 α) M. 1 m Table 2 displays the parameter estimates for the region-specific GPD models, and Table 3 displays the lower bounds ) ϑ lower of the region-specific confidence intervals for the 99.5% VaR obtained by the Bonferroni, Šidák and Bernstein copula calibration, respectively. ˆξ ˆβ Table 2: Estimated parameters ˆξ, ˆβ, 1 19, for the region-specific GPD models. Estimation has been performed via maximum likelihood. Bonferroni Šidák Bernstein Table 3: Lower confidence bounds ϑ lower for the 99.5% VaR, 1 19, obtained by the Bonferroni, the Šidák and the Bernstein copula method, respectively. Similarly as in Algorithm 4.1, an implicit weighting has been employed for the determination of the local significance levels α ) ) loc in Algorithm 5.4. Therefore, the confidence bounds obtained with the Bernstein copula method are not 1 m guaranteed to be more informative i. e., larger) than the ones obtained by the Bonferroni or the Šidák methods for all regions. However, we observe improvements in almost all regions. It is remarkable that this expected behavior of 20

23 the Bernstein copula calibration can already be verified for the rather moderate sample size of n = 20, because the likelihood ratio tests and the Bernstein copula calibration are both based on asymptotic considerations. Raw data x i, region year i Table 4: Insurance claim data from 19 adacent geographical regions over 20 years. 21

24 We omitted the values of ) ϑ upper, because they are uninformative extremely large). This is in line with the fact that all scale parameter estimates ˆξ in 1 m Table 2 are positive. For ξ 0, the GPD has infinite support, thus the modeled 99.5% VaR tends to be very large. 6 Discussion We have derived a nonparametric approach to the calibration of multivariate STPs which take the oint distribution of test statistics into account. In contrast to previous approaches which were restricted to cases with low-dimensional copula parameters, the Bernstein copula-based approximation of the local significance levels proposed in the present work can be applied under almost no assumptions regarding the dependency structures among test statistics or p-values, respectively. This makes the proposed methodology an attractive choice for data which have not been explicitly modeled prior to the statistical analysis. Furthermore, our empirical results on simulated as well as on real-life data indicate the gain in power which is possible by an explicit consideration of the dependency structure among test statistics in the calibration of the multiple test. This is particularly important for modern applications with high dimensionality of, but also pronounced dependencies in the data. On the other hand, Theorem 3.5 provides a precise asymptotic performance guarantee for the empirically calibrated multiple test, meaning that a sharp upper bound for its realized FWER can be obtained, at least asymptotically for large sample sizes. This is in contrast to most of the existing resampling-based multiple test procedures like the max T and min P tests proposed in [35], which are obvious competitors of our approach. Future work shall explore the case that some qualitative assumptions regarding the dependency structure are at hand. For example, it will be interesting to quantify the uncertainty of the FWER of an STP which is calibrated by assuming an Archimedean p-value copula as in [3]. In this case, nonparametric estimation of the copula generator function as for instance proposed in [19] will lead to an empirical calibration of the multiple test. 7 Appendix In this section several lemmas are formulated and proved which are used in the proofs of Theorem 2.1 and Theorem 2.2. Lemma 7.1. It holds that 22

25 a) sup Kk=0 ) m k 2 u [0,1] K u m P k,k u ) 1 m K 1 2 b) sup Kk=0 m k u [0,1] K u m P k,k u ) 1 m K 1/2 2 c) sup Kk=0 Kk m k k u [0,1] m =0 K u P k,k u ) P k,k u ) m K 1/2 Proof. a) Using the fact that P k,k u ) is the probability function of the binomial distribution for each u [0, 1] and = 1,..., m, we get ) K 2 k K sup u P k,k K u ) = sup u [0,1] k=0 b) Similarly, with the Jensen inequality we get K k sup u K P k,k u ) = u [0,1] k=0 c) Finally, we obtain K K sup u [0,1] k=0 k =0 K K = k =0 k =0 K K k = k =0 k =k K K k + k =0 k =k +1 K k u k =0 K k k K k k K = = u [0,1] k =0 sup u [0,1] K k sup u [0,1] k =0 K sup u [0,1] sup u [0,1] ) 2 k u P K u k,k ) u 1 u ) K = 1 4 k =0 u K P k,k u ) k K u u 1 u ) K u P k,k u ) P k,k u ) u P k,k u ) P k,k u ) u K P k,k u ) P k,k u ) u K P k,k u ) P k,k u ) P k,k u ) + K k k =0 u K P k,k u ) K 1. ) 2 P k,k u ) ) 1/2 = 1 2 K 1/2, 1/2 K 1/2. 23

26 where the last inequality follows from part b) of the lemma. Lemma 7.2. It holds that B K C 1 2 where g := sup u [0,1] g u) for g : [0, 1] R. Proof. We get K 1/2, K B K C sup C k/k) C u) P k,k u ) u [0,1] k=0 K k sup u u [0,1] k=0 K P k,k u ), where the last inequality follows from the Lipschitz property of multivariate copula cf., [26, Section 2]). The rest of the proof follows from part b) of Lemma 7.1. Lemma 7.3. Assume bounded second order partial derivatives for C on 0, 1) and let u 0, 1). Then, n 1/2 ˆBn,K u) B K u) ) = n 1/2 n Y i,k u)+o n 1/4 log n) 1/2 log log n) 1/4) where Y i,k u) := K k=0 and U i, = F X i, ). Proof. It holds that i=1 1,k/K] U i ) C k/k) C k/k) 1,k /K ] U i, ) k K ) ) P k,k u ) n 1/2 ˆBn,K u) B K u) ) K = n 1/2 Ĉ n k/k) C k/k) ) m P k,k u ). k=0 From [33, Section 4] we get n 1/2 Ĉ n k/k) C k/k) ) =ᾱ n k/k) C k/k) ᾱ n 1,..., k /K,..., 1) + O n 1/4 log n) 1/2 log log n) 1/4) where ᾱ n k/k) := n 1/2 n i=1 1,k/K] U i ) C k/k) ) and U i = F X i ) C. The result of the lemma now follows directly from the definition of Y i,k. 24

27 Lemma 7.4. Assume bounded second order partial derivatives for C on 0, 1) and let u 0, 1). Then, for all i {1,..., n} and K N m we get that E [Y i,k u)] = 0 and V [Y i,k u)] = σ 2 u) + O K 1/2 where σ 2 u) :=C u) 1 C u)) + C u)) 2 u 1 u ) 2 C u) C u) 1 u ) + 2 C u) C u) P [U i, u, U i, u ] u u ). =+1 Remark. The variance of Y i,k u) only depends on n. Proof. Since E [ 1,k/K] U i ) ] = C k/k) and E [ 1,k /K ]U i, ) ] = k K, we get E [Y i,k u)] = 0. Let f C k) := m C k/k) 1,k /K ]U i, ) k K ). It holds that V [Y i,k u)] =E [ Y i,k u) 2] K K = C k k ) /K) P k,k u ) P k,k u ) k=0 k =0 K K 2 C k/k) C k /K) P k,k u ) P k,k u ) k=0 k =0 K K 2 C k k1 /K) C,..., k k,..., k ) m k=0 k =0 K 1 K K m ) C k/k) k P k,k K u ) P k,k u ) K K + C k/k) C k /K) P k,k u ) P k,k u ) k=0 k =0 K K + 2 C k/k) E [f C k )] P k,k u ) P k,k u ) k=0 k =0 K K + E [f C k) f C k )] P k,k u ) P k,k u ) k=0 k =0 25

28 Since E [f C k)] = 0, we have K K V [Y i,k u, )] = C k k ) /K) P k,k u ) P k,k u ) k=0 k =0 K K C k/k) C k /K) P k,k u ) P k,k u ) k=0 k =0 K K 2 C k k1 /K) C,..., k k,..., k ) m k=0 k =0 K 1 K K m ) C k/k) k P k,k K u ) P k,k u ) K K + C k/k) C k /K) k=0 k =0 =1 [ P U i, k ] k, U i, k ) k P k,k K K K K u ) P k,k u ) I) II) III) IV) By the Lipschitz property of multivariate copula cf., [26]) and Lemma 7.1 we conclude for term I) I) =C u) + =C u) + O K K O k=0 k =0 K 1/2. By Lemma 7.2 we get for term II) k k K II) = B K u) 2 = C u) + O u P k,k u ) P k,k u ) = C u) 2 + O K 1/2 2 K 1/2. 26

29 For term III), we get by Taylor series expansion that K K III) = 2 C u) + O K 1/2 k=0 k =0 =1 C u) + O k k u k ) + O u K =1, K C u) + O K 1/2 k P k,k K u ) P k,k u ) =1 = 2 C u) [C u) C u) u ]) + O K 1/2 And, finally, for term IV) we have K K IV) = C u) + O K 1/2 C u) + O K 1/2 k=0 k =0 =1 =1 =1 ) k k ) P [U i, u, U i, u ] + O u + O u K k ) k K K K P k,k u ) P k,k u ) = C u) C u) P [U i, u, U i, u ] + 2O K 1/2 u u =1 =1 + O K 1/2 = C u) C u) P [U i, u, U i, u ] u u ) + O K 1/2. =1 Hence, V [Y i,k u)] =C u) 1 C u)) + C u)) 2 u 1 u ) 2 C u) C u) 1 u ) + 2 C u) C u) P [U i, u, U i, u ] u u ) =+1 + O K 1/2. 27

30 The first term comes from I) and II), the second from IV), the third from III), the forth from IV). Lemma 7.5. Let m K 1/2 0, n. Then for all u 0, 1) and all sequences u n ) n N in 0, 1) with lim n u n = u n 1/2 n i=1 Y i,k u n ) d N 0, σ 2 u) ). Proof. We want to use the Lindeberg Central Limit Theorem see [30, Section 1.9.3] or [20, Theorem ]). Let X i,n := n 1/2 Y i,k u n ), 1 i n. Because of Lemma 7.1, the remainder term in Lemma 7.4 does not depend on u. Therefore, we have E [X i,n ] = 0 and [ n ] σn 2 := V X i,n i=1 = V [Y 1,K u n )] = σ 2 u n ) + O K 1/2 n σ 2 u), because σ 2 is continuous. From the convergence X 1,n 0 a.s. and the dominated convergence theorem, we get the Lindeberg Condition, i.e. n E [ ] Xi,n1 2 { Xi,n >ɛσ n} σ 2 n = E [ ] Y1,n1 2 { X1,n >ɛσ n} σ 2 n i=1 n 0 for all ɛ > 0. The result follows from [30, Section 1.9.3]. Lemma 7.6. Let p 0, 1). Suppose that C C p)) > 0 exists, then n ˆB n,k p) C p) ) d N 0, σ C p),..., C ) p)), C C p)) where C p) := inf {u [0, 1] C u,..., u) p} and C u) := C u,..., u). Proof. We argue similarly to the proof of Theorem A in [30, Section 2.3.3]. Fix p 0, 1) and let G n t) := P n1/2 ˆB n,k p) C p) ) t, σ 28

31 where σ 2 := σc p),...,c p)). Let u C C p)) n := t σn 1/2 + C p). We have G n t) = P [ ˆB n,k p) u n ] = P [ p ˆB n,k u n,..., u n ) ] Put c nt := n1/2 Cu n,...,u n) p) σu n). Then it holds that G n t) = P [ c nt Z n ], where Z n := n1/2 B n,k u n,...,u n) Cu n,...,u n)) σu n). Furthermore, we get Φ t) G n t) = P [Z n < c nt ] 1 Φ t)) = P [Z n < c nt ] Φ c nt ) + Φ t) Φ c nt ) 1) Since C and C, 1 m are continuous, we have lim c nt = lim n n = t = t. t σ σ u n,..., u n ) C u n ) C C ) p)) t σn 1/2 σ σ C p),..., C p)) C C p)) Further, by Theorem 2.2 and Polya s Theorem see [30, Section 1.5.3]), since Φ is continuous, we have lim sup P [Z n x] Φ x) = 0. n x R Using these two properties, 1) results in lim Φ t) G n t) lim n sup n x R = 0. P [Z n < x] Φ x) + lim n Φ t) Φ c nt ) References [1] Aitchison, J. 1964). Confidence-region tests. J. R. Stat. Soc., Ser. B, 26:

32 [2] Belalia, M. 2016). On the asymptotic properties of the Bernstein estimator of the multivariate distribution function. Stat. Probab. Lett., in press. [3] Bodnar, T. and Dickhaus, T. 2014). False discovery rate control under Archimedean copula. Electron. J. Stat., 82): [4] Bouzebda, S. and Zari, T. 2013). Strong approximation of empirical copula processes by Gaussian processes. Statistics, 475): [5] Bücher, A. and Dette, H. 2010). A note on bootstrap approximations for the empirical copula process. Stat. Probab. Lett., ): [6] Cerqueti, R., Costantini, M., and Lupi, C. 2012). A copula-based analysis of false discovery rate control under dependence assumptions. Economics & Statistics Discussion Paper 065/12, Università degli Studi del Molise, Dipartimento di Scienze Economiche, Gestionali e Sociali SEGeS). [7] Cottin, C. and Pfeifer, D. 2014). From Bernstein polynomials to Bernstein copulas. J. Appl. Funct. Anal., 93-4): [8] Deheuvels, P. 1979). La fonction de dependance empirique et ses propriétés. Un test non paramètrique d independance. Bull. Cl. Sci., V. Sér., Acad. R. Belg., 65: [9] Dickhaus, T. 2014). Simultaneous Statistical Inference with Applications in the Life Sciences. Springer-Verlag Berlin Heidelberg. [10] Dickhaus, T. and Gierl, J. 2013). Simultaneous test procedures in terms of p-value copulae. Proceedings on the 2nd Annual International Conference on Computational Mathematics, Computational Geometry & Statistics CMCGS 2013), pages Global Science and Technology Forum GSTF). [11] Diers, D., Eling, M., and Marek, S. D. 2012). Dependence modeling in nonlife insurance using the Bernstein copula. Insur. Math. Econ., 503): [12] Embrechts, P., Lindskog, F., and McNeil, A. 2003). Modelling dependence with copulas and applications to risk management. In Rachev, S., editor, Handbook of Heavy Tailed Distributions in Finance, pages Elsevier Science B.V. [13] Gabriel, K. R. 1969). Simultaneous test procedures - some theory of multiple comparisons. Ann. Math. Stat., 40: [14] Härdle, W. K. and Okhrin, O. 2010). De copulis non est disputandum - Copulae: an overview. AStA Adv. Stat. Anal., 941):

33 [15] Hothorn, T., Bretz, F., and Westfall, P. 2008). Simultaneous inference in general parametric models. Biom. J., 503): [16] Janssen, P., Swanepoel, J., and Veraverbeke, N. 2012). Large sample behavior of the Bernstein copula estimator. J. Statist. Plann. Inference, 1425): [17] Joe, H. 2014). Dependence modeling with copulas. Boca Raton, FL: CRC Press. [18] Kiefer, J. 1961). On large deviations of the empiric D. F. of vector chance variables and a law of the iterated logarithm. Pacific J. Math., 11: [19] Lambert, P. 2007). Archimedean copula estimation using Bayesian splines smoothing techniques. Comput. Stat. Data Anal., 5112): [20] Lehmann, E. L. and Romano, J. P. 2005). Testing statistical hypotheses springer texts in statistics). [21] Longin, F. M. 2000). From value at risk to stress testing: The extreme value approach. Journal of Banking & Finance, 24: [22] McNeil, A. J., Frey, R., and Embrechts, P. 2005). Quantitative risk management. Concepts, techniques, and tools. Princeton, NJ: Princeton University Press. [23] Nelsen, R. B. 2006). An introduction to copulas. 2nd ed. Springer Series in Statistics. New York, NY: Springer. [24] Nikoloulopoulos, A. K., Joe, H., and Li, H. 2009). Extreme value properties of multivariate t copulas. Extremes, 122): [25] Rüschendorf, L. 1976). Asymptotic distributions of multivariate rank order statistics. Ann. Stat., 4: [26] Sancetta, A. and Satchell, S. 2004). The bernstein copula and its applications to modeling and approximations of multivariate distributions. Econometric Theory, 2003): [27] Schmidt, R., Faldum, A., and Gerß, J. 2015). Adaptive designs with arbitrary dependence structure based on Fisher s combination test. Stat. Methods Appl., 243): [28] Schmidt, R., Faldum, A., Witt, O., and Gerß, J. 2014). Adaptive designs with arbitrary dependence structure. Biom. J., 561):

34 [29] Segers, J. 2012). Asymptotics of empirical copula processes under nonrestrictive smoothness assumptions. Bernoulli, 183): [30] Serfling, R. J. 1980). Approximation theorems of mathematical statistics. New York: Wiley. [31] Stange, J., Bodnar, T., and Dickhaus, T. 2015). Uncertainty quantification for the family-wise error rate in multivariate copula models. AStA Adv. Stat. Anal., 993): [32] Stange, J., Dickhaus, T., Navarro, A., and Schunk, D. 2016). Multiplicityand dependency-adusted p-values for control of the family-wise error rate. Stat. Probab. Lett., 111: [33] Stute, W. 1984). The oscillation behavior of empirical processes: the multivariate case. Ann. Probab., 122): [34] Swanepoel, J. W. H. 1986). A note on proving that the modified) bootstrap works. Comm. Statist. A Theory Methods, 1511): [35] Westfall, P. H. and Young, S. S. 1993). Resampling-based multiple testing: examples and methods for p-value adustment. Wiley Series in Probability and Mathematical Statistics. Applied Probability and Statistics. Wiley, New York. 32