Calibration and uncertainty quantification using multivariate simulator output

Transcription

1 Calibration and uncertainty quantification using multivariate simulator output LA-UR 4-74 Dave Higdon, Statistical Sciences Group, LANL Jim Gattiker, Statistical Sciences Group, LANL Brian Williams, Statistical Sciences Group, LANL

2 Inference combining a physics model with experimental data drop time drop height (floor) Data generated from model: d z =.3 dz dt dt + ǫ simulation model: d z = dt drop time statistical model: y(z) = η(z) + δ(z) + ǫ drop height (floor) drop time Improved physics model: d z = θ dz dt dt + ǫ statistical model: y(z) = η(z, θ) + δ(z) + ǫ drop height (floor)

3 Basic formulation borrows from Kennedy and O Hagan () 7 (a) (b) (c) (d) η(x), y(x). η(x), y(x). δ(x). y(x) x 3. x x x experimental conditions θ calibration parameters ζ(x) true physical system response given inputs x η(x, θ) simulator response at x and θ. y(x) experimental observation of the physical system δ(x) discrepancy between ζ(x) and η(x, θ) may be decomposed into numerical error and bias e(x) observation error of the experimental data y(x) = ζ(x) + e(x) y(x) = η(x, θ) + δ(x) + e(x) 3. x

4 A Bayesian approach for combining simulations and experimental data for forecasting, calibration and uncertainty quantification prior uncertainty posterior uncertainty 3 3 y(x), η(x,θ) y(x), η(x,θ). θ x. θ x A simple example... x model or system inputs θ model calibration parameters ζ(x) true physical system response given inputs x η(x, θ) simulator response at x and θ. y(x) experimental observation of the physical system e(x) observation error of the experimental data Assume: y(x) = ζ(x) + e(x) = η(x, θ) + e(x) θ unknown. Standard Bayesian estimation gives: π(θ y(x)) L(y(x) η(x, θ)) π(θ)

5 Accounting for limited simulator runs y(x), η(x,θ) 3 prior uncertainty. θ 3. x Borrows from Kennedy and O Hagan (). x model or system inputs θ calibration parameters ζ(x) true physical system response given inputs x η(x, θ) simulator response at x and θ. simulator run at limited input settings η = (η(x, θ),..., η(x m, θm)) T treat η(, ) as a random function use GP prior for η(, ) y(x) experimental observation of the physical system e(x) observation error of the experimental data y(x) = ζ(x) + e(x) y(x) = η(x, θ) + e(x)

6 Accounting for limited simulation runs prior uncertainty posterior realizations of η(x,t) posterior uncertainty 3 3 y(x), η(x,θ). θ 3. x Again, standard Bayesian estimation gives: η(x,t). x π(θ, η(, ) y(x)) L(y(x) η(x, θ)) π(θ) π(η(, )) t. y(x), η(x,θ). θ 3. x Posterior means and quantiles shown. Uncertainty in θ and η(x, θ) are incorporated into the forecast. Gaussian process models for η(, ).

7 Accounting for model discrepancy prior uncertainty Borrows from Kennedy and O Hagan (). y(x), η(x,θ) 3. θ 3. x x model or system inputs θ model or system inputs ζ(x) true physical system response given inputs x η(x, θ) simulator response at x and θ. y(x) experimental observation of the physical system δ(x) discrepancy between ζ(x) and η(x, θ) may be decomposed into numerical error and bias e(x) observation error of the experimental data y(x) = ζ(x) + e(x) y(x) = η(x, θ) + δ(x) + e(x) y(x) = η(x, θ) + δ n (x) + δ b (x) + e(x)

8 Accounting for model discrepancy prior uncertainty posterior model uncertainty y(x), η(x,θ) δ(x) 3. θ 3. x 3 posterior model discrepancy 3. x y(x), η(x,θ) y(x) 3. θ 3. x 3 calibrated forecast 3. x Again, standard Bayesian estimation gives: π(θ, δ n, δ b y(x)) L(y(x) η(x, θ), δ(x)) π(θ) π(η) π(δ) Posterior means and 9% CI s shown. Posterior prediction for ζ(x) is obtained by computing the posterior distribution for η(x, θ) + δ(x) Uncertainty in θ, η(x, t), and δ(x) are incorporated into the forecast. Gaussian process models for η(x, t) and δ(x)

9 Application: implosions of steel cylinders Neddermeyer 43 Initial work on implosion for fat man. Use high explosive (HE) to crush steel cylindrical shells Investigate the feasability of a controlled implosion

10 Some History Early work on cylinders called beer can experiments. Early work not encouraging:...i question Dr. Neddermeyer s seriousness... Deke Parsons. It stinks. R. Feynman Teller and VonNeumann were quite supportive of the implosion idea Data on collapsing cylinder from high speed photography. Symmetrical implosion eventually accomplished using HE lenses by Kistiakowsky. Implosion played a key role in early computer experiments. Feynman worked on implosion calculations with IBM accounting machines. Eventually first computer with addressable memory was developed (MANIAC ).

11 The Experiments

12 Neddermeyer s Model cm 3 4 X s 3 4 X s 3 4 X s 3 4 X s 3 4 X s cm 3 4 X s 3 4 X s 3 4 X s 3 4 X s 3 4 X s Energy from HE imparts an initial inward velocity to the cylinder v = m e u m + m e /m mass ratio m e /m of HE to steel; u energy per unit mass from HE. Energy converts to work done on the cylinder: s { work per unit mass = w = r ρ( λ) i log ri ro log ro + λ log λ } r i = scaled inner radius; r o = scaled outer radius; λ = initial r i /r o ; s = steel yielding stress; ρ = density of steel.

13 Neddermeyer s Model cm 3 4 X s 3 4 X s 3 4 X s 3 4 X s 3 4 X s where cm 3 4 X s 3 4 X s 3 4 X s 3 4 X s 3 4 X s ODE: dr dt = [ R f(r) { v s ρ g(r) }] r = inner radius of cylinder varies with time R = initial outer ( radius of cylinder f(r) = r r λ ln + λ ) r g(r) = ( λ ) [r lnr (r + λ ) ln(r + λ ) λ lnλ ] λ = initial ratio of cylinder r(t = )/R constant volume condition: r outer r = λ

14 Goal: use experimental data to calibrate s and u ; obtain prediction uncertainty for new experiment expt cm expt 3 t = µs cm expt t = µs cm t = 4 µs m e /m.3 m e /m.7 m e /m.36 Hypothetical data obtained from photos at different times during the 3 experimental implosions. All cylinders had a.in outer and a.in inner radius. (λ = 3 ).

15 Carry out simulated implosions using Neddermeyer s model Sequence of runs carried at m input settings (x, θ, θ) = (m e /m, s, u ) varying x over predefined ranges using an OA(3, 4 3 θ θ )-based LH design.... x m θm θm.. inner radius (cm). inner radius (cm) time (s) x 3 x 4 time (s) pi angle (radians) pi radius by time radius by time and angle φ. Each simulation produces a n η = 6 vector of radii for times 6 angles.

16 Generating OA-based LH designs Example: N = 6, 3 factors each at 4 levels OA(6, 4 3 ) design induced LH design x_ x x_ x_ x x Ensures some higher dimensional filling relative to standard LH designs.

17 Generating (nearly) OA-based LH designs Example: NOA(48, 4 8 ) N = 48, 8 factors each at 4 levels..4.8 x_ x_ x_ columns of NOA design matrix X are not exactly orthogonal allows more factors with good higher dimensional properties x_4 x_ x_6 x_ x_

18 PC representation of simulation output Ξ = [η ; ;η m ] a n η m matrix that holds output of m simulations SVD decomposition: Ξ = UDV T K η is st p η columns of [ m UD] columns of [ mv T ] have variance Cylinder example: PC (9.4% of variation) PC (.9% of variation) PC 3 (.7% of variation) r.. time angle r.. time angle r.. time angle p η = 3 PC s: K η = [k ;k ;k 3 ] each vector k i holds trace of PC i. k i s do not change with φ from symmetry of Neddermeyer s model. Simulated trace η(x i, θ i, θ i ) = K ηw(x i, θ i, θ i )+ǫ i, ǫ i s iid N(, λ η ), for any set of tried simulation inputs (x i, θ i, θ i ).

19 Gaussian process models for PC weights Want to evaluate η(x, θ, θ ) at arbitrary input setting (x, θ, θ ). Also want analysis to account for uncertainty here. Approach: model each PC weight as a Gaussian process: where w i (x, θ, θ ) GP(, λ wi R((x, θ), (x, θ ); ρ wi )) R((x, θ), (x, θ ); ρ wi ) = Restricting to the design settings p x k= ρ 4(x k x k ) wik p θ k= x θ θ... and specifying x m θm θm w i = (w i (x, θ, θ ),..., w i (x m, θ m, θ m)) T gives iid w i N (, λ wi R((x, θ ); ρ wi ) ), i =,..., p η where R((x, θ ); ρ wi ) m m is given by (). ρ 4(θ k θ k ) wi(k+p x ) () *note: additional nugget term w i iid N (, λ wi R((x, θ ); ρ wi ) + λ ǫi I m), i =,...,pη, may be useful.

20 Gaussian process models for PC weights At the m simulation input settings the mp η -vector w has prior disribution w λ w R((x, θ ); ρ w ) w =. N.,... w pη λ wp η R((x, θ ); ρ wpη ) w N(, Σ w ); note Σ w = I pη λ w R((x, θ ); ρ w ) can break down. Emulator likelihood: η = vec([η(x, θ, θ ); ;η(x m, θ m, θ m)]) mnη L(η w, λ η ) λη exp { λ η(η Kw) T (η Kw) }, λ η Γ(a η, b η ) where n η is the number of observations in a simulated trace and Equivalently K = [I m k ; ;I m k pη ]. L(η w, λ η ) λ mp η η exp { λ η(w ŵ) T (K T K)(w ŵ) } m(nη pη) λη exp { λ ηη T (I K(K T K) K T )η } mpη λη exp { λ η(w ŵ) T (K T K)(w ŵ) }, λ η Γ(a η, b η) a η = a η + m(n η p η ), b η = b η + ηt (I K(K T K) K T )η, ŵ = (K T K) K T η.

21 Gaussian process models for PC weights Resulting posterior can then be based on computed PC weights ŵ: ŵ w,λ η N(w, (λ η K T K) ) w λ w, ρ w N(, Σ w ) ŵ λ η, λ w, ρ w N(, (λ η K T K) + Σ w ) Resulting posterior is then: π(λ η, λ w, ρ w ŵ) (λη K T K) + Σ w exp{ ŵt ([λ η K T K] + Σ w ) ŵ} p η λ a η η e b ηλ η λ a w wi e b wλ wi p η i= p x j= i= ( ρ wij ) b ρ p θ ( ρ wi(j+px )) b ρ j= MCMC via Metropolis works fine here. Bounded range of ρ wij s facilitates MCMC.

22 Posterior distribution of ρ w.8 PC [x θ] PC. 3 [x θ] PC3. 3 [x θ] Separate models by PC More opportunity to take advantage of effect sparsity

23 Predicting simulator output at untried (x, θ, θ ) Want η(x, θ, θ ) = Kw(x, θ, θ ) For a given draw (λ η, λ w, ρ w ) a draw of w can be produced: ( ) (( ) [( ) ]) ŵ (λη K w N, T K) + Σ w,w (λ w, ρ w ) Define Then ( ) V V V = = V V [( ) (λη K T K) ] + Σ w,w (λ w, ρ w ) w ŵ N(V V ŵ, V V V V ) Realizations can be generated from sample of MCMC output. Lots of info (data?) makes conditioning on point estimate ( λ η, λ w, ρ w ) a good approximation to the posterior. Posterior mean or median work well for ( λ η, λ w, ρ w )

24 Exploring sensitivity of simulator output to model inputs Simulator predictions varing input, holding others at nominal

25 Basic formulation borrows from Kennedy and O Hagan () Experiment r, η δ x 4 time x 4 time r, η+δ x 4 time φ φ φ δ x = m e /m.3 θ = s? θ = u? (t, φ) simulation output space x experimental conditions θ calibration parameters ζ(x) true physical system response given conditions x η(x, θ) simulator response at x and θ. y(x) experimental observation of the physical system δ(x) discrepancy between ζ(x) and η(x, θ) may be decomposed into numerical error and bias e(x) observation error of the experimental data y(x) = ζ(x) + e(x) y(x) = η(x, θ) + δ(x) + e(x)

26 Kernel basis representation for spatial processes δ(s) Define p δ basis functions d (s),..., d pδ (s). basis Here d j (s) is normal density cetered at spatial location ω j : d j (s) = π exp{ (s ω j) } set δ(s) = p δ j= s d j (s)v j where v N(, λ v I pδ ). Can represent δ = (δ(s ),..., δ(s n )) T as δ = Dv where D ij = d j (s i )

27 v and d(s) determine spatial processes δ(s) d j (s)v j δ(s) basis... z(s) s s Continuous representation: δ(s) = p δ j= d j (s)v j where v N(, λ v I pδ ). Discrete representation: For δ = (δ(s ),..., δ(s n )) T, δ = Dv where D ij = d j (s i )

28 Basis representation of discrepancy time angle φ Represent discrepancy δ(x) using basis functions and weights p δ = 4 basis functions over (t,φ); D = [d ; ;d pδ ]; d k s hold basis. δ(x) = Dv(x) where v(x) GP (, λ v I pδ R(x, x ;ρ v ) ) with R(x, x ;ρ v ) = p x k= ρ 4(x k x k ) vk ()

29 Integrated model formulation Data y(x ),..., y(x n ) collected for n experiments at input conditions x,..., x n. Each y(x i ) is a collection of n yi measurements over points indexed by (t, φ). y(x i ) = η(x i, θ) + δ(x i ) + e i = K i ( w(x i, θ) + ( D i v(x i ) + ) e i ) v(xi ) y(x i ) w(x i, θ), v(x i ), λ y N [D i ;K i ], (λ w(x i, θ) y W i ) Since support of each y(x i ) varies and doesn t match that of sims, the basis vectors in K i must be interpolated from K η ; similary, D i must be computed from the support of y(x i ): x r. r.. r x time pi pi angle pi angle pi time 4 x pi pi angle time x *note: cubic spline interpolation over (time, φ) used here.

30 Define Integrated model formulation n y = n y + + n yn, the total number of experimental data points, y to be the n y -vector from concatination of the y(x i ) s, v = vec([v(x ); ;v(x n )] T ) and u(θ) = vec([w(x, θ, θ ); ;w(x n, θ, θ )] T ) ( ( ) ) v y v, u(θ), λ y N B, (λ u(θ) y W y ), λ y Γ(a y, b y ) (3) where W y = diag(w,..., W n ) and ( ) P T B = diag(d,..., D n, K,..., K n ) D PK T P D and P K are permutation matricies whose rows are given by: P D (j + n(i ); ) = e T (j )p δ +i, i =,..., p δ; j =,..., n P K (j + n(i ); ) = e T (j )p η +i, i =,..., p η; j =,..., n

31 Integrated model formulation (continued) Equivalently (3) can be represented ) ( ) (( ) ) (ˆv v v, λ û u(θ) y N, (λ u(θ) y B T W y B), λ y Γ(a y, b y) with n y = n y + + n yn, the total number of experimental data points ) (ˆv = (B T W û y B) B T W y y a y = a y + [n y n(p δ + p η )] [ ( b y = b y + )) T ( )) (ˆv (ˆv ] y B W û y y B û model dimension reduction simulator data and discrep standard n η m n y basis p η m n (p δ + p η ) Basis approach particularly efficient when n η and n y are large.

32 λ Marginal likelihood The (marginal) likelihood L(ˆv, û, ŵ λ η, λ w, ρ w, λ y, λ v, ρ v, θ) has the form ˆv Σ v û N, Λ y + ŵ Λ Σ η uw where Λ y = λ y B T W y B Λ η = λ η K T K Σ v = λ v I pη R(x, x;ρ v ) R(x, x;ρ v ) = n n correlation matrix from applying () to the conditions x,..., x n corresponding the the n experiments. Σ uw = w R((x, θ), (x, θ); ρ w) λ w R((x, θ), (x, θ ); ρ w ) λ wp η R((x, θ), (x, θ); ρ wpη ) λ wp η R((x, θ), (x, θ ); ρ wpη ) w R((x, θ ), (x, θ); ρ w ) λ w R((x, θ ), (x, θ ); ρ w ) λ wp η R((x, θ ), (x, θ); ρ wpη ) λ wp η R((x, θ ), (x, θ ); ρ wpη ) λ Permutation of Σ uw is block diagonal can speed up computations. Only off diagonal blocks of Σ uw depend on θ.

33 Posterior distribution Likelihood: L(ˆv, û, ŵ λ η, λ w, ρ w, λ y, λ v, ρ v, θ) Prior: π(λ η, λ w, ρ w, λ y, λ v, ρ v, θ) Posterior: π(λ η, λ w, ρ w, λ y, λ v, ρ v, θ ˆv, û, ŵ) L(ˆv, û, ŵ λ η, λ w, ρ w, λ y, λ v, ρ v, θ) π(λ η, λ w, ρ w, λ y, λ v, ρ v, θ) Posterior exploration via MCMC Can take advantage of structure and sparcity to speed up sampling. A useful approximation to speed up posterior evaluation: π(λ η, λ w, ρ w, λ y, λ v, ρ v, θ ˆv, û, ŵ) L(ŵ λ η, λ w, ρ w ) π(λ η, λ w, ρ w ) L(ˆv, û λ η, λ w, ρ w, λ y, λ v, ρ v, θ) π(λ y, λ v, ρ v, θ) In this approximation, experimental data is not used to inform about parameters λ η, λ w, ρ w which govern the simulator process η(x, θ).

34 Posterior distribution of model parameters (θ, θ )

35 Posterior mean decomposition for each experiment Experiment Experiment Experiment 3 r, η r, η r, η x 4 time φ x 4 time φ x 4 time φ δ δ δ x 4 time φ x 4 time φ x 4 time φ r, η+δ r, η+δ r, η+δ x 4 time φ x 4 time φ x 4 time φ

36 Posterior prediction for implosion in each experiment Experiment Experiment Experiment 3 r... x time φ r... x time φ r... x time φ Experiment r... x time φ r

37 9% prediction intervals for implosions at exposure times Experiment Experiment Experiment 3 Time µs Time 7 µs Time µs Time 4 µs Time 4 µs Time 4 µs Predictions from separate analyses which hold data from the experiment being predicted.

38 Quantifying uncertainty for simulation-based forecasts Simulation-based predictions accumulate uncertainty due to: parameter/calibration uncertainty; simulator discrepancy/inadequacy; observation error in data; sparseness in data Limits on dimensionality? Have dealt with up to -dimensional θ. Statistics typically uses the wrong model (eg. a regression line) to explain data. So the framework is nothing new, in principle. Calibrating a model with substantial inadequacy? only slight inadequacy? The slowness of the simulator and high dimensionality make things difficult. Extrapolation is often a goal in such investigations. Generally, the the closer to reality the simulator is, the more it can be trusted for extrapolation. Can this be more rigorously formalized?

39 Application: Ta Flyerplate Experiment PTW model governs features on the visar velocity profile. Use pricipal components (EOF s) to deal with high dimensionality.

40 Simulations and Data.3 velocity km/s..... standardized velocity.. 3 time µs.. 3 time µs PTW calibration parameters with input domains Parameter Description Domain Min Max θ Initial strain hardening rate Material constant in thermal activation κ energy term relates to the temperature dependence Material constant in thermal activation log(γ) energy term relates to the strain rate dependence y Maximum yield stress (at K) y Minimum yield stress ( melting)..8 s Maximum saturation stress (at K).4.36 s Minimum saturation stress ( melting).8.43

41 PC representation of simulation output.. standardized velocity time (µ s) fit using p η = 7% of variation velocity (km/s) time (µ s) standardized velocity.. 3 time (µ s)

42 Marginal Posteriors for spatial correlation parameters ρ wij PC.9.9 PC PC θ θ θ

43 PC - based sensitivities

44 .8 Discrepancy basis stadardized velocity time (µ s) Represent discrepancy δ(x) using basis functions and weights Here d j (s) is normal density cetered at spatial location ω j : d j (s) = exp{ π (s ω j) } set δ(s) = p δ j= d j (s)v j where v N(, λ v I pδ ). Can represent δ = (δ(s ),..., δ(s n )) T as δ = Dv where p δ = basis functions over t. D ij = d j (s i )

45 posterior distribution for PTW parameters

46 posterior predictive distribution for trace η Y time (µs) time (µs) standardized velocity standardized velocity 4 δ time (µs) time (µs) standardized velocity standardized velocity

47 posterior predictive distribution for trace (original scale) η Y velocity. velocity time (µs)... 3 time (µs).3 δ.3.. velocity. velocity time (µs)... 3 time (µs)