Uniformly Most Powerful Bayesian Tests and Standards for Statistical Evidence

Transcription

1 Uniformly Most Powerful Bayesian Tests and Standards for Statistical Evidence Valen E. Johnson Texas A&M University February 27, 2014 Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

2 Motivation Multi-agent, Multi-cohort End-Stage Melanoma trial, Standard-of-care survival times: Biomarker Group Tmt 1 Tmt 2 Tmt 3 a b c d e Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

3 Assumptions: Suppose survival time for patient i in biomarker group b under treatment t is Y i Exp(µ bt ) You wish to test a null hypothesis H 0 : µ bt = µ 0 versus H 1 : µ bt = µ 1 > µ 0 Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

4 Alternative hypotheses: Suppose you know the true mean survival time is µ t. If you want to maximize expected weight of evidence, you take H 1 : µ = µ t, because 1 0 m t (y) log [ ] mt (y) dy m 0 (y) = [ m1 (y) m t (y) log m 0 (y) [ m1 (y) m t (y) log m 1 (y) ] dy ] dy > 0 This choice of µ = µ t makes all posterior inferences exactly correct, even in a repeated sampling sense Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

5 Problems for subjective Bayesian analysis µ t is generally not known. There is not a unique prior density for survival times of patients (i.e., drug sponsors, physicians, medical centers, patients, regulatory agencies) Similarly for decision theoretic analysis; there is no unique loss function Decision to proceed to next trial phase not based on Bayes factor, but whether Bayes factor (or significance level) for particular treatment combination exceeds a threshold. Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

6 Probability of exceeding threshold In practice, we usually reject H 0 if the Bayes factor exceeds a threshold, say γ. In adaptive trial, we may also reject H 1 if BF 10 < 1/γ If we believe null is false, then we really want to maximize P µt [BF 10 (y) > γ]. For exponential data and a point alternative hypothesis, the log of the Bayes factor is ( 1 log[bf 10 (y)] = n [log(µ 1 ) log(µ 0 )] 1 ) n y i µ 1 µ 0 i=1 Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

7 Probability of exceeding threshold Probability that log(bf 10 ) exceeds log(γ) can be written [ n ] P µt y i > log(γ) + n [log(µ 1) log(µ 0 )] 1 i=1 µ 0 1 µ 1 Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

8 Probability of exceeding threshold Probability that log(bf 10 ) exceeds log(γ) can be written [ n ] P µt y i > log(γ) + n [log(µ 1) log(µ 0 )] 1 i=1 µ 0 1 µ 1 Minimizing the RHS maximizes P µt [BF 10 (y) > γ], regardless of the value of µ t Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

9 Notation and assumptions H 0, H 1 denote models/hypotheses f (y θ) denotes the sampling density under all models m i (y) denotes the marginal density of data under model i Θ denotes parameter space π i (θ) denotes the prior density for θ Θ under model i BF 10 (y) denotes the Bayes factor between H 1 and H 0 Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

10 Definition A uniformly most powerful Bayesian test for a given evidence threshold γ, in favor of an alternative hypothesis H 1 against a fixed null hypothesis H 0 is a Bayesian hypothesis test in which the Bayes factor for the test satisfies the following inequality P θt [BF 10 (y) > γ] P θt [BF 20 (y) > γ] (1) for any θ t Θ and for all alternative hypotheses H 2 : θ π 2 (θ): Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

11 One parameter exponential family models One parameter exponential family models Suppose x = {x 1,..., x n } are iid with joint density function [ ] n n f (x) = exp η(θ) T (x i ) na(θ) h(x i ), i=1 where η(θ) is strictly monotonic Consider a one-sided test of a point null hypothesis that H 0 : θ = θ 0 against an arbitrary alternative hypothesis. i=1 Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

12 One parameter exponential family models UMPBT(γ) for one parameter exponential family models Theorem Define g γ (θ, θ 0 ) = log(γ) + n[a(θ) A(θ 0)], η(θ) η(θ 0 ) and define u = ±1 according to whether η(θ) is monotonically increasing or decreasing, and define v = ±1 according to whether the alternative hypothesis requires θ to be greater than or less than θ 0, respectively. Then a UMPBT(γ) can be obtained by restricting the support of π 1 (θ) to values of θ that belong to the set arg min θ uv g γ (θ, θ 0 ). Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

13 One parameter exponential family models Implications Like classical uniformly most powerful tests, UMPBTs exist for all common 1PEFs Unique UMPBTs are often defined by simple alternative hypotheses; exceptions occur when several values of parameter define the same rejection region Rejection regions for UMPBTs in exponential family models can generally be matched to rejection regions of UMPTs by appropriate choice of γ and Type I error Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

14 One parameter exponential family models Implications Like classical uniformly most powerful tests, UMPBTs exist for all common 1PEFs Unique UMPBTs are often defined by simple alternative hypotheses; exceptions occur when several values of parameter define the same rejection region Rejection regions for UMPBTs in exponential family models can generally be matched to rejection regions of UMPTs by appropriate choice of γ and Type I error This property establishes a connection between BFs and p-values Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

15 One parameter exponential family models Asymptotic Properties Theorem For a one parameter natural exponential family density, suppose that A(θ) has three bounded derivatives in a neighborhood of θ 0, and let θ denote a value of θ that defines a UMPBT(γ) test and satisfies dg γ (θ, θ 0 ) dθ = 0. (2) Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

16 One parameter exponential family models Asymptotic Properties Theorem Then the following statements are true. 1. For some t (θ 0, θ ), 2. Under the null hypothesis, θ θ 0 = 2 log(γ) na (t). (3) log(bf 10 ) N ( log(γ), 2 log(γ)) as n. (4) Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

17 One parameter exponential family models Asymptotics As n, UMPBT(γ) alternative converges to null hypothesis, for fixed γ. In practice, very large samples are collected for hypothesis tests when either 1. A very small effect size is being tested, or 2. Very strong evidence against H 0 is required Asymptotic properties of UMPBTs seem consistent with actual statistical practice Evidence in favor of true null is probabilistically bounded by log(γ) Rates at which to increase γ with n are topic for additional research Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

18 Examples Examples Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

19 Examples Binomial data Suppose y Binom(n, π) H 0 : π = 0.3, n = 10, γ = 3; H 1 : π > 0.3 UMPBT(γ) value of π 1 satisfies π 1 = log(γ) n[log(1 π) log(1 π 0 )] arg min π log[π/(1 π)] log[π 0 /(1 π 0 )] = Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

20 Examples P[BF 10 > 3] vs data-generating parameter Probability Bayes factor exceeds p Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

21 Normal data Examples Suppose x 1,..., x n iid N(µ, σ 2 ), σ 2 known UMPBT(γ) test of H 0 : µ = µ 0 is given by 2 log γ µ 1 = µ 0 ± σ, n depending on whether µ 1 > µ 0 or µ 1 < µ 0. Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

22 Examples P[BF 10 > 10] vs data-generating parameter for σ 2, n = 1 Probability Bayes factor exceeds µ Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

23 Examples Comparison to classical UMPT of normal mean Classical one-sided test s rejection region is x µ 0 ± z α σ n Equating the rejection regions for the UMPBT(γ) test and the UMPT of size α leads to γ = exp ( z 2 α/2 ) UMPBT places µ 1 on boundary of classical UMPT rejection region Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

24 Examples Comparison to classical UMPT of normal mean Classical one-sided test s rejection region is x µ 0 ± z α σ n Equating the rejection regions for the UMPBT(γ) test and the UMPT of size α leads to γ = exp ( z 2 α/2 ) UMPBT places µ 1 on boundary of classical UMPT rejection region UMPBT the most subjective of objective Bayesian hypothesis tests? Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

25 Examples Other Exact UMPBTs UMPBTs exist for Simple tests regarding coefficients in linear models with known observational variances Chi-squared tests on one degree of freedom when H 0 : λ = 0 and H 1 : λ > 0, λ the non-centrality parameter Two-sided tests in 1PEF, under constraint of symmetric alternative. Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

26 Approximate UMPBTs Examples Approximate UMPBTs can be obtained in normal model hypothesis tests with unknown variances (require data dependent alternative hypotheses). 1. T-tests (one-sample, paired, two-sample) 2. Simple tests of linear regression coefficients with unknown observational variance Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

27 Examples T-tests For one-sample t-test, P(BF 10 > γ) can be expressed as P µt [a < x < b] For two-sample t-test, P(BF 10 > γ) can be expressed as P µ1 µ 0 [c < x < d] The parameters (a, b, c, d) depend on n, γ, and s 2. Upper bounds b, d with n Ignoring upper bound, data-dependent (s 2 ) approximate UMPBT can be obtained by minimizing a or c. Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

28 Examples Bayes evidence thresholds versus test size Evidence threshold versus size of test γ α Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

29 Examples Bayes evidence thresholds versus test size Under assumption of equipoise (i.e., P(H 0 ) = P(H 1 )), p = 0.05 γ (3, 5) P(H 0 x) (.17,.25) p = 0.01 γ (12, 20) P(H 0 x) (.05,.08) p = γ (25, 50) P(H 0 x) (.02,.04) p = γ (100, 200) P(H 0 x) (.005,.001) Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

30 Examples Bayes evidence thresholds versus test size Standard definitions of significant and highly significant results correspond to only weak evidence against null hypotheses. Definition of significant or highly significant should require evidence of > 25 : 1 or > 100 : 1 against the null p-values of or Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

31 Examples Ongoing research: Scott Goddard, graduate student at Texas A&M, is currently developing restricted most power Bayesian tests Suppose y N(Xβ, σ 2 I ), H 0 : β = 0, H 1 : β N [ 0, gσ 2 (X X) 1] WIth non-informative prior on σ 2, value of g that maximizes probability that BF 10 > γ is (g + 1) argmin [1 (g + 1) p/n γ 2/n] g Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

32 Examples Restricted most powerful Bayesian tests have applications in ANOVA, where they provide correspondence to F tests Bayesian variable selection, where γ can be set according to p and n Goddard has developed analytic expressions for optimal g and found expressions to set g to control Type 1 error in ANOVA and Bayesian variable selection contexts. Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

33 Examples Summary UMPBTs provide default objective Bayes factors for the most common of statistical hypothesis tests Large sample behavior is reasonable Approximately mimic the subjective alternative hypothesis implicit to classical tests (for matched γ and Type I error) Correspondence between UMPBTs and UMPTs provide guidance on appropriate definition of significant and highly significant findings, and insight into the non-reproducibility of scientific studies Restricted most powerful Bayesian tests can provide default settings for hyperparameters for parametric alternative hypotheses Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31

34 Examples The End Valen E. Johnson Texas A&M University Uniformly most powerful Bayes Tests February 27, / 31