Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk

Size: px

Start display at page:

Download "Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk"

Benedict Green
6 years ago
Views:

1 Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk Axel Gandy Department of Mathematics Imperial College London user! 2009, Rennes July 8-10, 2009

2 Introduction Test statistic T, reject for large values. Observation: t. p-value: p = P(T t) Often not available in closed form. Monte Carlo Test: ˆp naive = 1 n where T, T 1,... T n i.i.d. Examples: Bootstrap, Permutation tests. Goal: Estimate p using few X i n I(T i t), i=1 Mainly interested in deciding if p α for some α. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

3 Introduction Test statistic T, reject for large values. Observation: t. p-value: p = P(T t) Often not available in closed form. Monte Carlo Test: ˆp naive = 1 n where T, T 1,... T n i.i.d. Examples: Bootstrap, Permutation tests. Goal: Estimate p using few X i n I(T i t), }{{} =:X i B(1,p) i=1 Mainly interested in deciding if p α for some α. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

4 Introduction Test statistic T, reject for large values. Observation: t. p-value: p = P(T t) Often not available in closed form. Monte Carlo Test: ˆp naive = 1 n where T, T 1,... T n i.i.d. Examples: Bootstrap, Permutation tests. Goal: Estimate p using few X i n I(T i t), }{{} =:X i B(1,p) i=1 Mainly interested in deciding if p α for some α. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

5 Introduction Test statistic T, reject for large values. Observation: t. p-value: p = P(T t) Often not available in closed form. Monte Carlo Test: ˆp naive = 1 n where T, T 1,... T n i.i.d. Examples: Bootstrap, Permutation tests. Goal: Estimate p using few X i n I(T i t), }{{} =:X i B(1,p) i=1 Mainly interested in deciding if p α for some α. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 2

6 Sequential approaches based on S n = n i=1 X i S n Stop once S n U n or S n L n τ: hitting time Compute ˆp based on S τ and τ. Hit B U : decide p > α, Hit B L : decide p α, n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 3

7 Sequential approaches based on S n = n i=1 X i S n Stop once S n U n or S n L n τ: hitting time Compute ˆp based on S τ and τ. Hit B U : decide p > α, Hit B L : decide p α, n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 3

8 Sequential approaches based on S n = n i=1 X i S n Stop once S n U n or S n L n τ: hitting time Compute ˆp based on S τ and τ. Hit B U : decide p > α, Hit B L : decide p α, n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 3

9 Previous Approaches Besag & Clifford (1991): h S n 0 n 0 m (Truncated) Sequential Probability Ratio Test, Fay et al. (2007) h S n 0 0 m R-package MChtest. n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 4

10 What do we really want? Is p α? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk Want: RR p (ˆp) { P p (ˆp > α) if p α, P p (ˆp α) if p > α. sup RR p (ˆp) ɛ p [0,1] for some (small) ɛ > 0. For Besag & Clifford (1991), SPRT: sup p RR P 0.5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

11 What do we really want? Is p α? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk Want: RR p (ˆp) { P p (ˆp > α) if p α, P p (ˆp α) if p > α. sup RR p (ˆp) ɛ p [0,1] for some (small) ɛ > 0. For Besag & Clifford (1991), SPRT: sup p RR P 0.5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

12 What do we really want? Is p α? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk Want: RR p (ˆp) { P p (ˆp > α) if p α, P p (ˆp α) if p > α. sup RR p (ˆp) ɛ p [0,1] for some (small) ɛ > 0. For Besag & Clifford (1991), SPRT: sup p RR P 0.5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

13 What do we really want? Is p α? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk Want: RR p (ˆp) { P p (ˆp > α) if p α, P p (ˆp α) if p > α. sup RR p (ˆp) ɛ p [0,1] for some (small) ɛ > 0. For Besag & Clifford (1991), SPRT: sup p RR P 0.5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

14 What do we really want? Is p α? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk Want: RR p (ˆp) { P p (ˆp > α) if p α, P p (ˆp α) if p > α. sup RR p (ˆp) ɛ p [0,1] for some (small) ɛ > 0. For Besag & Clifford (1991), SPRT: sup p RR P 0.5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

15 What do we really want? Is p α? Two individuals using the same statistical method on the same data should arrive at the same conclusion. First law of applied statistics, Gleser (1996) Consider the resampling risk Want: RR p (ˆp) { P p (ˆp > α) if p α, P p (ˆp α) if p > α. sup RR p (ˆp) ɛ p [0,1] for some (small) ɛ > 0. For Besag & Clifford (1991), SPRT: sup p RR P 0.5 Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 5

16 Recursive Definition of the Boundaries Want: sup RR p (ˆp) ɛ p Suffices to ensure P α (hit B U ) ɛ P α (hit B L ) ɛ Recursive definition: Given U 1,..., U n 1 and L 1,..., L n 1, define U n as the minimal value such that P α (hit B U until n) ɛ n and L n as the maximal value such that P α (hit B L until n) ɛ n where ɛ n 0 with ɛ n ɛ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

17 Recursive Definition of the Boundaries Want: sup RR p (ˆp) ɛ p Suffices to ensure P α (hit B U ) ɛ P α (hit B L ) ɛ Recursive definition: Given U 1,..., U n 1 and L 1,..., L n 1, define U n as the minimal value such that P α (hit B U until n) ɛ n and L n as the maximal value such that P α (hit B L until n) ɛ n where ɛ n 0 with ɛ n ɛ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

18 Recursive Definition of the Boundaries Want: sup RR p (ˆp) ɛ p Suffices to ensure P α (hit B U ) ɛ P α (hit B L ) ɛ Recursive definition: Given U 1,..., U n 1 and L 1,..., L n 1, define U n as the minimal value such that P α (hit B U until n) ɛ n and L n as the maximal value such that P α (hit B L until n) ɛ n where ɛ n 0 with ɛ n ɛ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

19 Recursive Definition of the Boundaries Want: sup RR p (ˆp) ɛ p Suffices to ensure P α (hit B U ) ɛ P α (hit B L ) ɛ Recursive definition: Given U 1,..., U n 1 and L 1,..., L n 1, define U n as the minimal value such that P α (hit B U until n) ɛ n and L n as the maximal value such that P α (hit B L until n) ɛ n where ɛ n 0 with ɛ n ɛ (spending sequence). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 6

20 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n P α (S n =k, τ n) 0 k= 3 k= 2 k= 1 k= 0 1 ɛ n 0 U n 1 L n -1 n = Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

21 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n P α (S n =k, τ n) 0 1 k= 3 k= 2 k= 1.2 k= ɛ n 0.07 U n 1 2 L n -1-1 n = Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

22 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n P α (S n =k, τ n) k= 3 k= 2.04 k= k= ɛ n U n L n n = Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

23 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n P α (S n =k, τ n) k= 3 k= k= k= ɛ n U n L n n = Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

24 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n n = P α (S n =k, τ n) k= 3 k= k= k= ɛ n U n L n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

25 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n n = P α (S n =k, τ n) k= 3.02 k= k= k= ɛ n U n L n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

26 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n n = P α (S n =k, τ n) k= k= k= k= ɛ n U n L n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

27 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n n = P α (S n =k, τ n) k= k= k= k= ɛ n U n L n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

28 Recursive Definition - Example α = 0.2, ɛ n = 0.4 n 5+n. U n =the minimal value such that L n = maximal value such that P α (hit B U until n) ɛ n P α (hit B L until n) ɛ n n = P α (S n =k, τ n) k= k= k= k= ɛ n U n L n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 7

29 Sequential Decision Procedure - Example α = 0.2, ɛ n = 0.4 n 5+n n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 8

30 Influence of ɛ on the stopping rule ɛ = 0.1, 0.001, 10 5, 10 7 ; ɛ n = ɛ n 1000+n n Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 9

31 Sequential Estimation based on the MLE One can show: S τ ˆp = τ, τ < α, τ =, hitting the upper boundary implies ˆp > α, hitting the lower boundary implies ˆp < α. Hence, sup RR p (ˆp) ɛ p Furthermore, random interval I n s.t. In only depends on X 1,..., X n, ˆp In. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 10

32 Example - Two-way sparse contingency table H 0 : variables are independent. Reject for large values of the likelihood ratio test statistic T T d χ 2 (7 1)(5 1) under H 0. Based on this: p = Matrix sparse - approximation poor? Use parametric bootstrap based on row and column sums. Naive test statistic ˆp naive with n = 1,000 replicates: p = < Probability of reporting p > 0.05: roughly Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 11

33 Example - Two-way sparse contingency table H 0 : variables are independent. Reject for large values of the likelihood ratio test statistic T T d χ 2 (7 1)(5 1) under H 0. Based on this: p = Matrix sparse - approximation poor? Use parametric bootstrap based on row and column sums. Naive test statistic ˆp naive with n = 1,000 replicates: p = < Probability of reporting p > 0.05: roughly Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 11

34 Example - Two-way sparse contingency table H 0 : variables are independent. Reject for large values of the likelihood ratio test statistic T T d χ 2 (7 1)(5 1) under H 0. Based on this: p = Matrix sparse - approximation poor? Use parametric bootstrap based on row and column sums. Naive test statistic ˆp naive with n = 1,000 replicates: p = < Probability of reporting p > 0.05: roughly Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 11

35 Example - Bootstrap and Sequential Algorithm > dat <- matrix(c(1,2,2,1,1,0,1, 2,0,0,2,3,0,0, 0,1,1,1,2,7,3, 1,1,2,0,0,0,1, + 0,1,1,1,1,0,0), nrow=5,ncol=7,byrow=true) > loglikrat <- function(data){ + cs <- colsums(data);rs <- rowsums(data); mu <- outer(rs,cs)/sum(rs) + 2*sum(ifelse(data<=0.5, 0,data*log(data/mu))) + } > resample <- function(data){ + cs <- colsums(data);rs <- rowsums(data); n <- sum(rs) + mu <- outer(rs,cs)/n/n + matrix(rmultinom(1,n,c(mu)),nrow=dim(data)[1],ncol=dim(data)[2]) + } > t <- loglikrat(dat); > library(simctest) > res <- simctest(function(){loglikrat(resample(dat))>=t},maxsteps=1000) > res No decision reached. Final estimate will be in [ , ] Current estimate of the p.value: Number of samples: 1000 > cont(res, steps=10000) p.value: Number of samples: 8574

36 Further Uses of the Algorithm Simulation study to evaluate whether a test is liberal/conservative. Determining the sample size to achieve a certain power. Iterated Use: Determining the power of a bootstrap test. Simulation study to evaluate whether a bootstrap test is liberal/conservative. Double bootstrap test. Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 13

37 Expected Hitting Time Result: E p (τ) < p α Example with α = 0.05, ɛ n = ɛ n 1000+n : E p (τ) ε = ε = 1e 05 ε = 1e 07 E p (τ) µ p p µ p = theoretical lower bound on E p (τ). Note: 1 0 µ pdp = ; for iterated use: Need to limit the number of steps.

38 Expected Hitting Time Result: E p (τ) < p α Example with α = 0.05, ɛ n = ɛ n 1000+n : E p (τ) ε = ε = 1e 05 ε = 1e 07 E p (τ) µ p p µ p = theoretical lower bound on E p (τ). Note: 1 0 µ pdp = ; for iterated use: Need to limit the number of steps.

39 Summary Sequential implementation of Monte Carlo Tests and computation of p-values. Useful when implementing tests in packages. After a finite number of steps: ˆp or interval [ˆp L n, ˆp U n ] in which ˆp will lie. Guarantee (up to a very small error probability): ˆp is on the correct side of α. R-package simctest available on CRAN. (efficient implementation with C-code) For details see Gandy (2009). Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 15

40 References Besag, J. & Clifford, P. (1991). Sequential Monte Carlo p-values. Biometrika 78, Davison, A. & Hinkley, D. (1997). Bootstrap methods and their application. Cambridge University Press. Fay, M. P., Kim, H.-J. & Hachey, M. (2007). On using truncated sequential probability ratio test boundaries for Monte Carlo implementation of hypothesis tests. Journal of Computational & Graphical Statistics 16, Gandy, A. (2009). Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk. Accepted for publication in JASA. Gleser, L. J. (1996). Comment on Bootstrap Confidence Intervals by T. J. DiCiccio and B. Efron. Statistical Science 11, Axel Gandy Sequential Implementation of Monte Carlo Tests with Uniformly Bounded Resampling Risk 16

arxiv: v4 [stat.me] 20 Oct 2017

arxiv: v4 [stat.me] 20 Oct 2017 A simple method for implementing Monte Carlo tests Dong Ding, Axel Gandy, and Georg Hahn Department of Mathematics, Imperial College London arxiv:1611.01675v4 [stat.me] 20 Oct 2017 Abstract We consider