Sequential Monte Carlo Algorithms for Bayesian Sequential Design

Transcription

1 Sequential Monte Carlo Algorithms for Bayesian Sequential Design Dr Queensland University of Technology Collaborators: James McGree, Tony Pettitt, Gentry White Acknowledgements: Australian Research Council Discovery Grant and organisers of MCMski January 6, 24

2 Sequential Experimental Design Adaptive decisions as new data are collected More robust to parameter and model uncertainty Natural to use Bayesian framework. Posterior becomes new prior Next decision obtained by looking forward to all future decisions (backward induction) Simplified by myopic design (one-at-a-time) Next design point d t+ = argmaxu(d Ý:Ø, :Ø). Ý:Ø collected data at design :Ø. U is utility function

3 Why SMC for Bayesian sequential design? Much more efficient than MCMC. Simple re-weighting step to incorporate new information. Parallel implementation possible (e.g. GPU) Increase in efficiency allows comparisons of utility functions Convenient estimation of important Bayesian utility functions (e.g. mutual information) Decisions in real time?

4 SMC For One static Model m Sample from sequence of targets Data annealing here π t (θ Ñ Ý:Ø, :Ø) = f(ý:ø θ Ñ, :Ø)π(θ Ñ )/Z m,t, for t =,...,T. Ý:Ø (independent) data up to t, :Ø design points up to t, θ Ñ parameter for model m. f is likelihood, π prior, π t posterior f(ý:ø m, :Ø) = Z m,t = f(ý:ø θ Ñ, :Ø)π(θ Ñ )dθ Ñ. θñ SMC: Generate a weighted sample (particles) for each target in the sequence via steps Reweight: particles as data comes in (efficient) Resample: when ESS small Mutation: diversify duplicated particles (can be efficient)

5 SMC For One STATIC Model m (Algorithm) Chopin (22) Have current particles {W i t,θ Ø }N i= based on data Ý :Ø Re-weight step to included y t+ W i t+ Wi t f(y t+ θ Ø,d t+). Check effective sample size: ESS = / N i= (Wi t+ )2 If ESS > E (e.g. E = N/2) go back to re-weight step for next observation If ESS < E do the following Resample proportional to weights. Duplicates good particles Mutation: Move all particles via MCMC kernel say R times (adaptive proposal)

6 SMC Estimate of Evidence Del Moral et al (26) It can be shown Z t+ /Z t = f(y t+ Ý:Ø,d t+ ) = θ f(y t+ θ,d t+ )π(θ Ý:Ø, :Ø)dθ. Using SMC particles to approximate posterior at t gives estimator N Z t+ /Z t Wtf(y i t+ θ,d Ø t+). i= Can then obtain approximation of Z t+ through Z t+ = Z t+ Z t Z. Z Z t Z t Z Also gives estimate of posterior predictive probability of y t+

7 Advantage : Efficiently comparing utilities (Drovandi et al 23) Discrete data (binary) example Need to compute utility for all possible d (then for all t =,...,T) U(d Ý:Ø, :Ø) = f(z Ý:Ø, :Ø,d)U(d,z Ý:Ø, :Ø). z {,} The whole process requires the computation (sampling) of many many posterior distributions SMC (IS) to rescue. Pretend z is the next observation collected at design d. Simple re-weight to incorporate this observation. Use weighted sample to estimate U(d, z). SMC also provides estimate of posterior predictive. There may be different choices for U(d,z), want to compare. Need to simulate the design many times.

8 The Design Problem Estimating Maximum tolerated dose, minimum effective dose (clinical trials) E[Y t ] = g (η t ) where dt λ η t = θ +θ, λ where d t is the dose assigned to the tth subject. Y t Binary(E[Y t ]). Uninformative prior π(θ,θ,λ) N(θ ;,)N(θ ;,)U(λ;,) Objective is precise estimation of ( d = λ T η θ T θ T Let θ = (θ,θ,λ). ) /λ T +. ( ) λ η θ + >, θ

9 The Utility Functions Possible choices for U(d,z Ý:Ø, :Ø) Posterior precision of d (Natural choice, but posterior of d can be unstable when little information is available) 2 Kullback-Leibler Divergence between prior and posterior for θ 3 Determinant of Posterior Covariance matrix of θ 4 Hybrid utility. Utility 3 for subjects. Utility thereafter.

10 Some Results.5 prec kld D post hybrid D * Subject Figure: Distributions of the estimated target stimulus over to subjects for the true parameter configuration of θ Ì = (,3,) producing d =.538. Solid horizontal line is the true d. Shown are the 2.5%, 5% and 97.5% quantiles over the 5 runs for each utility function.

11 Advantage 2: Estimating Difficult Utilities E.g. Mutual Information for Model Discrimination (Drovandi et al 24) Have set of K proposed models M =,...,K. Select design to maximise ability to discriminate between models Consider mutual information between model indicator M and predicted observation Z for y t+ Box and Hill (967). I(M;Z Ý:Ø,d) = H(M Ý:Ø) H(M Z; Ý:Ø,d). Therefore U(d Ý:Ø) = H(M Z; Ý:Ø,d) which is equal to U(d Ý:Ø) = K m= π(m Ý:Ø) f(z m, Ý:Ø,d)logπ(m Ý:Ø,z,d)dµ(z),

12 SMC for multiple models Effectively run an SMC algorithm for each model m =,...,K Have set of N particles for each model {W i m,t,θ Ñ,Ø }N i=. ESS for each model m resampling and within-model updates when required Design part: use data up to t, Ý:Ø, and particles of all models to compute the next design d t+

13 Estimating the Utility U(d Ý:Ø) = K m= π(m Ý:Ø) f(z m, Ý:Ø,d)logπ(m Ý:Ø,z,d)dµ(z), Borth (975) notes difficult computation SMC to the rescue. Potential observation z at potential design point d. Pretend this the observation for y t+.

14 Estimating the Utility (cont...) Estimate predictive probability using weights w i m,t (d,z) = Wi m,t f(z θ Ñ,Ø,d), ˆf(z m, Ý:Ø, :Ø,d) = N wm,t i (d,z). Z m,t denotes current evidence for model m, which integrates out posterior of θ at t. Estimate evidence including (d,z) Z m,t (d,z) using i= logẑm,t(d,z) = logẑm,t +log ˆf(z m, Ý:Ø, :Ø,d). Convert to ˆπ(m Ý:Ø,d,z)

15 Estimating the Utility (cont...) Therefore estimate of utility for discrete z is K Û(d Ý:Ø) = m=ˆπ(m Ý:Ø) ˆf(z m, Ý:Ø,d)logˆπ(m Ý:Ø,z,d). z S In continuous case approximate integral via MC integration. Draw zm,t i f(z m,θi m,t,d) for i =,...,N. Weighted sample {Wm,t i,θi m,t,zi m,t } from joint p(z,θ d,m,y :t,d :t ). Therefore estimate of utility for continuous z is Û(d y :t,d :t ) = K m= ˆπ(m y :t,d :t ) N Wm,t i logˆπ(m y :t,d :t,zm,t i,d). i= Could also be used for large counts.

16 Chemical Engineering Example Masoumi et al 23 Consider chemical reaction A B. Four competing models for the fraction of A remaining after time t minutes at temperature T ( ( Model : µ = exp θ texp θ )) 2 T [ ( Model 2: µ 2 = +θ texp θ )] 2 T [ ( Model 3: µ 3 = +2θ texp θ )] /2 2 T [ ( Model 4: µ 4 = +3θ texp θ )] /3 2 T y m N(µ m,σ 2 ) Two design variables = (t,t): t {,25,5,75,,25,5} and T {45,475,525,575,6} yielding 35 possible choices.

17 Chemical Example Continued 5 independent observations Four cases. Each model allowed to be true with parameter (θ,θ 2,σ) = (4,5,.) Prior distribution ( θ θ 2 ) N ([ 4 5 ] [ 7, 5 ]) (θ > )(θ 2 > ). σ IG(,) Comparing 3 different utility functions: Random design, mutual information and total separation (Masoumi et al 23)

18 Chemical Reaction Results (model true) Mutual information Total separation Random Figure: First order reaction as true.

19 Chemical Reaction Results (model 2 true) Mutual information Total separation Random Figure: Second order reaction as true.

20 Chemical Reaction Results (model 3 true) Mutual information Total separation Random Figure: Third order reaction as true.

21 Chemical Reaction Results (model 4 true) Mutual information Total separation Random Figure: Fourth order reaction as true.

22 Advantage 3: Embarrisingly Parallel Extend to design in presence of random effects model Pharmacokinetics Example. For subject t the model for samples collected at design t = (d t,d 2t ) is: y t MVN(g(β t, t),δ Á), β t MVN(µ,Ω), Here β Ø is random effect for tth subject Priors: g(β t, t) = exp(β 2t ) exp µ MVN(,Σ), for Σ known. ( exp(β t) exp(β 2t ) t Ω InvWish(Ψ,ν), for Ψ and ν known δ U(a,b), < a b <, ), Design objective is to learn about parameters: θ = (µ,ω,δ ).

23 Exact-Approximate SMC The (marginal) likelihood f(ý t θ, t) = f(ý t β t,δ, t)π(β t µ,ω)dβ t Can be estimated unbiasedly. For each particle θ (i) f(ý t θ (i), t) = M M j= f(ý t β (j) t,δ (i), t) () where β (j) t MVN(µ (i),ω (i) ), j =,...,M. SMC with unbiased estimate of likelihood an exact-approximate algorithm! (Duan and Fulop 23) Caveat: SMC can degenerate quicker compared to using exact likelihood (need large enough M)

24 Speeding things up on the GPU Serial implementation of SMC design too slow here Many ways to parallelise algorithm (over particles, M and/or design choices d) Here we chose calculating the likelihood over particles in parallel on GPU (required in all aspects of algorithm) Order of magnitude speed improvement over C implementation of likelihood Work still in progress...

25 Key References Box, G. E. P. and Hill, W. J. (967). Discrimination among mechanistic models. Technometrics, 9():57-7. Drovandi, C. C. et al. (24). A Sequential Monte Carlo Algorithm to Incorporate Model Uncertainty in Bayesian Sequential Design. JCGS To Appear. Drovandi, C. C. et al. (23). Sequential Monte Carlo for Bayesian sequentially designed experiments for discrete data. CSDA, 57(): Chopin, N. (22). A sequential particle filter method for static models. Biometrika, 89(3): Del Moral, P et al (26). Sequential Monte Carlo samplers. JRSS: Series B, 68(3):4-436.