Permutation Test for Bayesian Variable Selection Method for Modelling Dose-Response Data Under Simple Order Restrictions

Transcription

1 Permutation Test for Bayesian Variable Selection Method for Modelling -Response Data Under Simple Order Restrictions Martin Otava International Hexa-Symposium on Biostatistics, Bioinformatics, and Epidemiology I-Biostat, UHasselt, Belgium

2 Research team Hasselt University (Belgium): Martin Otava Ziv Shkedy Interuniversity Institute for Biostatistics and statistical Bioinformatics Durham University (UK): Adetayo Kasim Tel Aviv University (Israel): Daniel Yekutieli Martin Otava 2

3 BVS model formulation Martin Otava 3

4 -response modeling Increasing dose of therapeutical compound. Variety of possible responses: Toxicity. Inhibition or stimulation. Gene expression level. Weight Goal: Determine if there is any relationship. If so, what is the shape of the profile. 0 Select 1 threshold 2 doses 3 (e.g. MED). Mutagenity Gene expression Gene expression Martin Otava 4

5 Order constraints Compound effect becomes stronger when dose is increased. Monotone restriction (non-decreasing or non-increasing). Zero effect is meaningful. Response No parametrical assumptions about dose-response curve shape Martin Otava 5

6 BVS model formulation Basic model: Y ij N(µ i, σ 2 ) Modeling of mean: i E(Y ij ) = µ i = µ 0 + z l δ l. Priors: l=1 Hyper Priors: σ 2 Γ(10 3, 10 3 ), η µ N(0, 10 6 ), σµ 2 Γ(10 3, 10 3 ), η δi N(0, 10 6 ), σ 2 δ i Γ(10 3, 10 3 ). π i U(0, 1). µ 0 N(η µ, σµ), 2 δ i N(η δi, σδ 2 i )I(0, A), z i Bernoulli(π i ), Martin Otava 6

7 Set of all models Response Response g_ g_ Response Response g_ g_ Model Up: Mean Structure z g 0 µ 0 = µ 1 = µ 2 = µ 3 (0,0,0) g 1 µ 0 < µ 1 = µ 2 = µ 3 (1,0,0) g 2 µ 0 = µ 1 < µ 2 = µ 3 (0,1,0) g 3 µ 0 < µ 1 < µ 2 = µ 3 (1,1,0) g 4 µ 0 = µ 1 = µ 2 < µ 3 (0,0,1) g 5 µ 0 < µ 1 = µ 2 < µ 3 (1,0,1) g 6 µ 0 = µ 1 < µ 2 < µ 3 (0,1,1) g 7 µ 0 < µ 1 < µ 2 < µ 3 (1,1,1) g_4 g_5 Response Response H up : µ 0 µ 1 µ 2... µ K 1 g_6 g_7 Response Response Martin Otava 7

8 Results of BVS Martin Otava 8

9 Example: BVS model Incorporating models with equal means results into less decreasing profile. Posterior means are averages of means of particular models at each MCMC iteration. ˆµ BV S = R P (g r data)ˆµ r r=0 Weight Order restricted BVS Connection to model averaging. Martin Otava 9

10 Posterior probability of model Vector z = (z 1,..., z K 1 ) = z r uniquely defines the model g r. In each MCMC iteration we sample one vector z = (z 1,..., z K 1 ). Posterior mean of indicator of z = z r translates into posterior probability of the model g r. = For posterior probabilities holds: P (z = z r data) = P (g r data). Martin Otava 10

11 Example: Posterior probabilities Posterior probabilities of particular models. Model g 0 represents H 0. Model g 1 is strongly supported by the data. Connection to model selection. g_0 g_1 g_2 g_3 g_4 g_5 g_6 g_7 Model Posterior probability Martin Otava 11

12 Results of BVS Model uncertainty taken into account! Model selection: P (gr data). Estimation of means: ˆµ = R r=0 P (g r data)ˆµ r. Inference: P (g0 data). BVS framework addresses all perspectives simultaneously. Martin Otava 12

13 Inference for BVS Martin Otava 13

14 Hypothesis testing Testing the hypothesis H 0 : µ 0 = µ 1 = µ 2 =... = µ K 1 against ordered alternative (one inequality strict) H up : µ 0 µ 1 µ 2... µ K 1 H dn : µ 0 µ 1 µ 2... µ K 1 Use P (g 0 data), estimation of P (H 0 data), to reject H 0. Depends on: data on hand, prior distributions, set of alternative hypotheses. Martin Otava 14

15 Hypothesis testing Testing the hypothesis H 0 : µ 0 = µ 1 = µ 2 =... = µ K 1 against ordered alternative (one inequality strict) H up : µ 0 µ 1 µ 2... µ K 1 H dn : µ 0 µ 1 µ 2... µ K 1 Use P (g 0 data), estimation of P (H 0 data), to reject H 0. Depends on: data on hand, prior distributions, set of alternative hypotheses. When to reject? Martin Otava 14

16 When to reject in Bayesian framework It boils down to same questions that arise for Bayes factor. Purely Bayesian: subjective choice. Martin Otava 15

17 When to reject in Bayesian framework It boils down to same questions that arise for Bayes factor. Purely Bayesian: subjective choice. No prior scientific knowledge: non-informative priors. But: how to define them. Can we do subjective decision then? Martin Otava 15

18 When to reject in Bayesian framework It boils down to same questions that arise for Bayes factor. Purely Bayesian: subjective choice. No prior scientific knowledge: non-informative priors. But: how to define them. Can we do subjective decision then? Is there quantity to use that does not depend on priors? Could we control errors? Martin Otava 15

19 Permutation test 1. Consider T = P (g 0 data) as test statistics. 2. Permute dose labels (under H 0 ) B times and perform BVS on new data. 3. Get collection of T b. 4. More against H 0 smaller T b, b = 1,..., B. 5. Compute quantity [ p Bayes = 1 B ] B I (T b < T ) b=1 6. Reject H 0 if p Bayes < α. Martin Otava 16

20 Example: Permutation test Weight Order restricted BVS g_0 g_1 g_2 g_3 g_4 g_5 g_6 g_7 Model Posterior probability Method p MCT Williams MCT Marcus LRT Permutation Martin Otava 17

21 Simulation study Martin Otava 18

22 Setting Simulated 10 5 experiments, 1400 under each of models g 1,..., g 7, rest under g 0. Permutation test was applied and H 0 rejected for p Bayes < Frequentist procedures LRT and MCTs were applied (α = 0.05). Is permutation test maintaining type I error in same way as LRT and MCTs? How powerful is the permutation test in comparison with LRT and MCTs? Martin Otava 19

23 Simulation study - Results Power by true profile Williams Marcus LRT Permutation Profile Martin Otava 20

24 Discussion Martin Otava 21

25 Comments Difficult to asses test statistics properties. Does not depend on priors put on hypotheses. Number of permutations depends on value of P (g 0 data). Computationally intensive. Simulation results suggest good behaviour. Martin Otava 22

26 Conclusions Alternative to frequentist tests with respect to objectivity. Unified framework. Impulse to further research. Martin Otava 23

27 Thank you for your attention! Martin Otava 24