Uniformly and Restricted Most Powerful Bayesian Tests

Transcription

1 Uniformly and Restricted Most Powerful Bayesian Tests Valen E. Johnson and Scott Goddard Texas A&M University June 6, 2014 Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

2 Overview of Null Hypothesis Significance Testing (NHST) Central to scientific publication Tens of thousands of hypothesis tests are reported every year Prior beliefs of investigators, editors, other scientists, patients, general public, etc. vary regarding alternative hypotheses implicit to each hypothesis test Ditto for loss functions Scientific hypothesis tests are normally conducted under frequentist paradigm using standard threshold for significance (i.e, 0.05). Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

3 Towards routine Bayesian testing For Bayesian testing procedures to find routine application in NHST, practitioners need Standard evidence thresholds for rejection of null hypothesis, based on Bayes factors Default alternatives hypotheses Simple expressions/programs to compute Bayes factors Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

4 Further requirements for routine Bayesian testing For specified evidence threshold, Scientists want default alternatives that provide highest probability for rejection of null hypothesis Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

5 Further requirements for routine Bayesian testing For specified evidence threshold, Scientists want default alternatives that provide highest probability for rejection of null hypothesis Uniformly most powerful Bayesian tests provide highest probability of rejection in one-parameter exponential family models Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

6 Further requirements for routine Bayesian testing For specified evidence threshold, Scientists want default alternatives that provide highest probability for rejection of null hypothesis Uniformly most powerful Bayesian tests provide highest probability of rejection in one-parameter exponential family models Restricted most powerful Bayesian tests provide high probability of rejection when UMPBTs do not exist, and they often have simple expressions Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

7 Notation and assumptions H 0, H 1 denote models/hypotheses f (y θ) denotes the sampling density under all models m i (y) denotes the marginal density of data under model i Θ denotes parameter space π i (θ) denotes the prior density for θ Θ under model i BF 10 (y) denotes the Bayes factor between H 1 and H 0 γ denotes evidence threshold, i.e., reject H 0 if BF 10 > γ Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

8 Definition A uniformly most powerful Bayesian test for a given evidence threshold γ, in favor of an alternative hypothesis H 1 against a fixed null hypothesis H 0 is a Bayesian hypothesis test in which the Bayes factor for the test satisfies the following inequality P θt [BF 10 (y) > γ] P θt [BF 20 (y) > γ] (1) for any θ t Θ and for all alternative hypotheses H 2 : θ π 2 (θ): Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

9 Binomial data Suppose y Binom(n, π) H 0 : π = π 0 Suppose under alternative H 1 : π = π 1 > π 0 Let π t denote true (i.e., data-generating) value of π Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

10 Binomial data (cont) For a point alternative hypothesis, log of Bayes factor is [ ] [ ] π1 /(1 π 1 ) 1 π1 log[bf 10 (y)] = y log + n log π 0 /(1 π 0 ) 1 π 0 Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

11 Binomial data (cont) For a point alternative hypothesis, log of Bayes factor is [ ] [ ] π1 /(1 π 1 ) 1 π1 log[bf 10 (y)] = y log + n log π 0 /(1 π 0 ) 1 π 0 P πt [BF 10 (y) > γ] can be expressed [ P πt y > log(γ) n log[(1 π ] 1)/(1 π 0 )] log [π 1 (1 π 0 )]/[(1 π 1 )π 0 ] Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

12 Binomial data (cont) For a point alternative hypothesis, log of Bayes factor is [ ] [ ] π1 /(1 π 1 ) 1 π1 log[bf 10 (y)] = y log + n log π 0 /(1 π 0 ) 1 π 0 P πt [BF 10 (y) > γ] can be expressed [ P πt y > log(γ) n log[(1 π ] 1)/(1 π 0 )] log [π 1 (1 π 0 )]/[(1 π 1 )π 0 ] Minimizing the RHS of inequality maximizes P[BF 10 (y) > γ], regardless of the value of π t. Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

13 One parameter exponential family models One parameter exponential family models Suppose x = {x 1,..., x n } are iid with joint density function [ ] n n f (x) = exp η(θ) T (x i ) na(θ) h(x i ), i=1 where η(θ) is strictly monotonic Consider a one-sided test of a point null hypothesis that H 0 : θ = θ 0 against an arbitrary alternative hypothesis. i=1 Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

14 One parameter exponential family models UMPBT(γ) for one parameter exponential family models Theorem Define g γ (θ, θ 0 ) = log(γ) + n[a(θ) A(θ 0)], η(θ) η(θ 0 ) and define u = ±1 according to whether η(θ) is monotonically increasing or decreasing, and define v = ±1 according to whether the alternative hypothesis requires θ to be greater than or less than θ 0, respectively. Then a UMPBT(γ) can be obtained by restricting the support of π 1 (θ) to values of θ that belong to the set arg min θ uv g γ (θ, θ 0 ). Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

15 One parameter exponential family models Implications Like classical uniformly most powerful tests, UMPBTs exist for all common 1PEFs Unique UMPBTs are often defined by simple alternative hypotheses; exceptions occur when several values of parameter define the same rejection region Rejection regions for UMPBTs in exponential family models can generally be matched to rejection regions of UMPTs by appropriate choice of γ and Type I error Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

16 One parameter exponential family models Implications Like classical uniformly most powerful tests, UMPBTs exist for all common 1PEFs Unique UMPBTs are often defined by simple alternative hypotheses; exceptions occur when several values of parameter define the same rejection region Rejection regions for UMPBTs in exponential family models can generally be matched to rejection regions of UMPTs by appropriate choice of γ and Type I error This property establishes a connection between BFs and p-values Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

17 One parameter exponential family models Asymptotic Properties Theorem Let θ denote a value of θ that defines a UMPBT(γ) test. Then under certain regularity conditions, the following are true. 1. For some t (θ 0, θ ), 2. Under the null hypothesis, θ θ 0 = 2 log(γ) na (t). (2) log(bf 10 ) N ( log(γ), 2 log(γ)) as n. (3) Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

18 One parameter exponential family models Asymptotics For fixed γ, as n the UMPBT(γ) alternative converges to null hypothesis. In practice, very large samples are collected for hypothesis tests when either 1. A very small effect size is being tested, or 2. Very strong evidence against H 0 is required Asymptotic properties of UMPBTs seem consistent with actual statistical practice Evidence in favor of true null is probabilistically bounded by log(γ) Rates at which to increase γ with n are topic for additional research Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

19 Examples Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

20 Binomial data Suppose y Binom(n, π) H 0 : π = 0.3, n = 10, γ = 3; H 1 : π > 0.3 Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

21 Binomial data Suppose y Binom(n, π) H 0 : π = 0.3, n = 10, γ = 3; H 1 : π > 0.3 UMPBT(γ) value of π 1 satisfies π 1 = log(γ) n[log(1 π) log(1 π 0 )] arg min π log[π/(1 π)] log[π 0 /(1 π 0 )] = Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

22 P[BF 10 > 3] vs data-generating parameter Probability Bayes factor exceeds p Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

23 Normal data Suppose x 1,..., x n iid N(µ, σ 2 ), σ 2 known UMPBT(γ) test of H 0 : µ = µ 0 is given by 2 log γ µ 1 = µ 0 ± σ, n depending on whether µ 1 > µ 0 or µ 1 < µ 0. Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

24 P[BF 10 > 10] vs data-generating parameter for σ 2, n = 1 Probability Bayes factor exceeds µ Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

25 Comparison to classical UMPT of normal mean Classical one-sided test s rejection region is x µ 0 ± z α σ n Equating the rejection regions for the UMPBT(γ) test and the UMPT of size α leads to γ = exp ( z 2 α/2 ) UMPBT places µ 1 on boundary of classical UMPT rejection region alen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

26 Comparison to classical UMPT of normal mean Classical one-sided test s rejection region is x µ 0 ± z α σ n Equating the rejection regions for the UMPBT(γ) test and the UMPT of size α leads to γ = exp ( z 2 α/2 ) UMPBT places µ 1 on boundary of classical UMPT rejection region UMPBT the most subjective of objective Bayesian hypothesis tests? alen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

27 Other Exact UMPBTs UMPBTs exist for Simple tests regarding coefficients in linear models with known observational variances Chi-squared tests on one degree of freedom when H 0 : λ = 0 and H 1 : λ > 0, λ the non-centrality parameter Two-sided tests in 1PEF, under constraint of symmetric alternative. Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

28 Approximate UMPBTs Approximate UMPBTs can be obtained in normal model hypothesis tests with unknown variances (require data dependent alternative hypotheses). 1. T-tests (one-sample, paired, two-sample) 2. Simple tests of linear regression coefficients with unknown observational variance Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

29 T-tests For one-sample t-test, P(BF 10 > γ) can be expressed as P µt [a < x < b] For two-sample t-test, P(BF 10 > γ) can be expressed as P µ1 µ 0 [c < x 1 x 2 < d] The parameters (a, b, c, d) depend on n, γ, and s 2. Upper bounds b, d with n Ignoring upper bound, data-dependent (s 2 ) approximate UMPBT can be obtained by minimizing a or c. Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

30 Bayes evidence thresholds versus test size Evidence threshold versus size of test γ α Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

31 Bayes evidence thresholds versus test size Under assumption of equipoise (i.e., P(H 0 ) = P(H 1 )), p = 0.05 γ (3, 5) P(H 0 x) (.17,.25) p = 0.01 γ (12, 20) P(H 0 x) (.05,.08) p = γ (25, 50) P(H 0 x) (.02,.04) p = γ (100, 200) P(H 0 x) (.005,.001) Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

32 Bayes evidence thresholds versus test size Standard definitions of significant and highly significant results correspond to only weak evidence against null hypotheses. Definition of significant or highly significant should require evidence of > 25 : 1 or > 100 : 1 against the null p-values of or Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

33 π-restricted Most Powerful Bayesian Tests A π-restricted most powerful Bayesian test for evidence threshold γ > 0 in favor of the alternative hypothesis H 1 : θ π(θ ψ 1 ) against a fixed null hypothesis H 0 is a Bayesian hypothesis test in which the Bayes factor for the test satisfies P θt [BF 10 (x) > γ] P θt [BF 20 (x) > γ], for any θ t Θ and for all alternative hypotheses H 2 : θ π(θ ψ 2 ), where π is a density function parameterized by ψ, and ψ 1, ψ 2 Ψ. Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

34 g-prior-restricted Most Powerful Tests Suppose y N(Xβ, σ 2 I ), H 0 : β = 0, H 1 : β N [ 0, gσ 2 (X X) 1] WIth non-informative prior on σ 2, value of g that maximizes P[BF 10 > γ] is argmin g (g + 1) g [1 (g + 1) p/n γ 2/n] Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

35 g-prior-restricted Most Powerful Tests Restricted most powerful Bayesian tests have applications in ANOVA, where they provide correspondence to F tests One and two sample t-tests Bayesian variable selection, where γ can be set according to p and n To get corresponding size α test, set g = F 1, where F is 1 α quantile from corresponding F distribution. Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

36 Bayes factors of RMPBT s in linear models ANOVA: Two-sample t: One-sample t: [ F + J 1 ˆF ] (n 1)/2 BF 10 (ˆF ) = F (n J)/2 n J 1 + J 1 ˆF n J BF 10 (ˆt) = t n 2 [ t ] n 2ˆt 2 (n 1)/ n 2ˆt 2 BF 10 (ˆt) = t n 1 [ t ] n 1ˆt 2 n/ n 1ˆt 2 Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

37 Summary UMPBTs provide default objective Bayes factors for the most common of statistical hypothesis tests Large sample behavior is reasonable Approximately mimic the subjective alternative hypothesis implicit to classical tests (for matched γ and Type I error) Correspondence between UMPBTs and UMPTs provide guidance on appropriate definition of significant and highly significant findings, and insight into the non-reproducibility of scientific studies Restricted most powerful Bayesian tests can provide default settings for hyperparameters for parametric alternative hypotheses Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31

38 The End Emanuel Parzen 1819 Laura Lane College Station, TX Valen E. Johnson and Scott Goddard Texas A&MUniformly University Most Powerful Bayes Tests June 6, / 31