Bayesian Average Error Based Approach to Sample Size Calculations for Hypothesis Testing

Transcription

1 Bayesian Average Error Based Approach to Sample Size Calculations for Hypothesis Testing Eric M Reyes and Sujit K Ghosh Department of Statistics North Carolina State University NCSU Bayesian Seminar Series 01 September 2011

2 A Motivating Example

3 Example: Binary Response, Two Independent Samples Sampling Distribution Two independent groups that have event rates θ 1 and θ 2. Hypothesis of Interest H 0 : θ 1 = θ 2 vs. H 1 : θ 1 θ 2 Decision Rule For test statistic T (X ), we reject H 0 if T (X ) > t for some t.

4 Classical Approach The sample size is given by the following rule: n Z ( ) 2 Z α 2 θ(1 θ) + Zβ θ1 (1 θ 1 ) + θ 2 (1 θ 2 ) (θ 2 θ 1 ) 2 where α and β represent the Type-I and Type-II error, respectively, and θ = (θ 1 + θ 2 ) /2 [Friedman 1998, Fund. of Clin. Trials].

5 Limitations of Classical Approach n Z ( ) 2 Z α 2 θ(1 θ) + Z β θ1 (1 θ 1 ) + θ 2 (1 θ 2 ) (θ 2 θ 1 ) 2 Calculations rely on posited values for the parameters of interest. Pivot quantitites not guaranteed to exist [Adcock 1997, Stat.]. Asymptotic normal approximations may be questionable [M Lan 2008, Bayes. Anal.].

6 Methodology

7 Problem Statement Sampling Distribution Hypothesis of Interest Decision Rule X f (x θ) where θ π(θ) and θ Θ H 0 : θ Θ 0 vs. H 1 : θ Θ 1, where Θ 0 Θ 1 = and Θ 0 Θ 1 Θ For test statistic T (X ), we reject H 0 if T (X ) > t for some t. Key Assumption Pr (θ Θ j ) = Θ j π(θ)dθ > 0 for j = 0, 1

8 Bayesian Average Errors Density of Test Statistic T(X) t 0 Density Under H 0 Density Under H 1 AE 1 (t 0 ) AE 2 (t 0 ) Average Bayesian Type-I Error: AE 1(t) = Pr(T (X ) > t θ Θ 0) Average Bayesian Type-II Error: AE 2(t) = Pr(T (X ) t θ Θ 1) Cutoff t

9 Optimal Choice of Cut-off t For a given w (0, 1), define the Total Weighted Error (TWE) as TWE(t, w) = wae 1 (t) + (1 w)ae 2 (t).

10 Optimal Choice of Cut-off t For a given w (0, 1), define the Total Weighted Error (TWE) as TWE(t, w) = wae 1 (t) + (1 w)ae 2 (t). Definition of Optimal Cut-off t 0 (w) = arg min t {TWE(t, w)}

11 Bayes Factor The Bayes Factor BF(X ) is often used to quantify the evidence in favor of H 1 [Weiss 1997, Stat.]: ( ) ( ) Pr(θ Θ1 X ) Pr(θ Θ0 ) BF(X ) = Pr(θ Θ 0 X ) Pr(θ Θ 1 )

12 Primary Result Theorem Consider testing the hypothesis as described previously. Let BF(X ) denote the Bayes Factor and let ϕ(x ) : x [0, 1] represent a randomized test for the hypothesis. Then, for a given value of w (0, 1), ˆϕ(X ) minimizes TWE where Implications ˆϕ = I (log(bf(x )) > log(w/(1 w))). T (X ) = log(bf(x )) and t 0 = log ( w ) 1 w

13 Rule for Sample Size Determination Define Total Error (TE) as TE(t) = AE 1(t) + AE 2(t).

14 Rule for Sample Size Determination Define Total Error (TE) as TE(t) = AE 1(t) + AE 2(t). Specify a bound α (0, 1) on TE. Specify a weight w (0, 1).

15 Rule for Sample Size Determination Define Total Error (TE) as TE(t) = AE 1(t) + AE 2(t). Specify a bound α (0, 1) on TE. Specify a weight w (0, 1). Choose minimum n > 1 such that TE(t 0(w)) α. Total Error t 0 = log w 1 w n = n 1 n = n 2 n = n 3 Total Error Bound α Cutoff t

16 Example

17 Example: Binary Response, Two Independent Samples Sampling Distribution X = (x 1, x 2 ) where x k θ k ind. Bin (n, θ k ), for k = 1, 2 θ = (θ 1, θ 2 ) [0, 1] 2. Hypothesis of Interest Prior Distribution H 0 : θ 1 = θ 2 ( Θ0 = {θ [0, 1] 2 : θ 1 = θ 2 } ) H 1 : θ 1 θ 2 ( Θ1 = {θ [0, 1] 2 : θ 1 θ 2 } ) π(θ) = ui (θ 1 = θ 2 = η) p (a0,b 0 )(η)+ (1 u)i (θ 1 θ 2 ) p (a1,b 1 )(θ 1 )p (a2,b 2 )(θ 2 ) where u = Pr(θ 1 = θ 2 ) and p (a,b) (θ) denotes a Beta(a, b) density.

18 Choices for Classical Approach n Z ( ) 2 Z α 2 θ(1 θ) + Z β θ1 (1 θ 1 ) + θ 2 (1 θ 2 ) Set θ 1 and θ 2 such that θ = 0.5. Choose α = 0.05 and β = Choose α = (θ 2 θ 1 ) 2

19 Prior Parameters Density d = θ 2 θ 1 θ 1 = θ 2 θ 1 θ θ

20 Results: Binary Response, Two Independent Samples d = θ 2 θ n Z n w= n w= n w=

21 Discussion

22 Critiques Critique 1: Computation Increasing complexity and/or sample sizes increases computation. An observed relationship between log n and log α may decrease the computational burden.

23 log n - log α Relationship w = 0.9 w = 0.5 w = 0.1 logn^ (α) Log Total Error Bound α

24 Critiques Critique 2: Sensitivity of Priors Bayes factor is sensitive to choice of priors. Our key assumption does not allow for use of improper reference priors. Using alternative Bayes factor for T (X ) may yield a promising option.

25 Future Work Use of alternative Bayes factor for T (X ). Better understanding the relationship between n and α. Using different conditional errors: Pr (θ Θ j X )

26 Conclusions This methodology is a general approach to hypothesis testing and sample size determination that is broadly applicable to simple and complex hypothesis tests. R package available at emreyes/

27 Conclusions This methodology is a general approach to hypothesis testing and sample size determination that is broadly applicable to simple and complex hypothesis tests. R package available at emreyes/ This research was supported by NIH grant T32HL

28 References Adcock CJ. Sample size determination: a review. The Statistician, 46: , Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical trials. 3rd edition. Springer-Verlag, M Lan CE, Joseph L, Wolfson DB. Bayesian sample size determination for binomial proportions. Bayesian Analysis, 3: , Weiss R. Bayesian sample size calculations for hypothesis testing. The Statistician, 46: , 1997.

29 Total Weighted Error TWE(t, w) = wae 1 (t) + (1 w)ae 2 (t) = (1 w) I (BF(x) > t) ( BF(x) w ) (1 w)m 0 (x)dx 1 w m 0 (X ) = Θ 0 f (x θ)π(θ)dθ Θ 0 π(θ)dθ BF(X ) = m 1(X ) m 0 (X )