J. Beckman, TSA-1 Morrison, TSA-5. May 11-14, m4s

Transcription

1 LA-UR- a Title: Author@): Submitfed to: EVALUATNG UNCERTANTY N SMULATON MODELS Michael Richard John D. Stephen D. McKay TSA-1 J. Beckman TSA-1 Morrison TSA-5 C. Upton TSA-5 Predictability: Quantifying Uncertainty in Models of Complex Phenomena Center for Nonlinear Studies 1th Annual nternational Conference Los Alamos NM 7545 May m4s ~~~~~~~~~~ S Los Alamos National Laboratory an affirmative action/equal opportunity employer is operated by the University of California for the U.S. Department of Energy under contract W-745-ENG-36. By acceptance of this article the publisher recognizesthat the US. Government retains a nonexclusive royalty-free license to publish or reproduce the published form of this contnbution or to allow others to do so for U.S. Government purposes. The Los Alarnos National Laboratory requests that the publisher identtfy this article as work performed under the auspices of the US. Department of Energy. Form No. 36 A5 ST69 1OF91. -~._.

2 .' DECLAMER This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof nor any of their employees makes any warranty express or implied or assumes any legal liability or responsibility for the accuracy completeness or usefulness of any information apparatus product or process dmlosed or represents that its usc would not infringe privately owned rights. Reference hmin to any specific commercial product process or service by trade name trademark manufacturer or otherwise does not necessarily constitute or imply its endorsement mommendrtion or favoring by the United States Gomnment or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.

3 DSCLAMER Portions of this document may be illegible in electronic image products. mages are produced from the best available original document.

4 ... Predictability: Quantifying Uncertainty in Models of Complex Phenomena LA-TJR-9-.Center for Nonlinear StudiesLos Alamos National Laboratory 1th Annual nternatid Conference Los Alamos New Mexico May Evaluating Uncertainty in Simulation Models* Michael D. McKay Richard J. Beckman John D. Morrison Stephen C. Upton Los Alamos National Laboratory Los Alamos New Mexico USA 4 Questions related to input uncertainty 1 Topics Modeling * Uncertainty hportance Estimation Example time dependent output Example E subset selection Model structure Final Thoughts Models How do alternative input values affect model prediction? Are calculations driven by only a subset of inputs? How well does a subset of inputs predict relative to the full set? How much does a model component contribute to prediction error? 5 Example of prediction uncertainty C-SA Tons A mathematical model m ( - )is a formal statement of assumptions about a relationship between known inputs x and predicted outputs y. Structure of a model defines how charackristics y are determined from x. t is a mathematical algorithm or set of rules and depends on random number streams in stochastic simulation models. inputs x define initial conditions and parameter values in the rules of rn(.). Simulatwn variables z c<nrespond to random elements in the rules of m (. ). 3 Origins of uncertainty Simulation or sampling variability comes from simulation variables (random number streams). t is inherent in model prediction plausible runs form a band of prediction uncertainty for y (tons) due to dif mntinput values. nput uncertainty comes from plausible alternative input values and exists independently of the model. Structural uncertainty comes from plausible alternative model structures. *From an invited paper presented at the Second nternational Symposium on Sensitivity Analysis of Model Output (SAM '9) Wee taly April L 4 - e n q5o3

5 . 9 Decomposition of prediction variance (Panier's formula) 6 Abstraction of uncertainty y = m(z ; e) Simulation variability and the simulation distribution:.1f z + - Y fylx has simulation mean py(z) and variance a$(z) nput uncertainty and the prediction distribution: { D z ;.f Structural uncertainty: ( m E M m z - f+} 5) - sm} 4 Y Y fy Qy M={m(zz;6)~6EO}~the~~~al parametric sense might enumerate a known list of models + might index a hypothetical collection of models + 7 nput importance Let the input variables be partitioned into disjoint subsets x = x s u x?. nputs and subsets need not be stochastically independent. The full mode1 predictor is y = a ( x ). Let the predictor from the subset x S be -S = E ( Y XS) A linear form for the expectation need not be assumed. souroe / df Total -1 snr sum of squares SST = 5.? c i=l3=1 k=l (Yijk Approx &Sum of Squares) -T T k importance of x s rehive to x relates to the difference between 5 and y which we choose to measure by the quadratic loss function c = (y - %. The commonly called mean squared error (MSE) of prediction is given by E ( C ) = Var(y) - Var(9. mportance measure Relationship of prediction variances of y and y follows from and to give Y=5+(Y-5) Var(Y)= Var[E(Y E[WY zs)l Var(Y) = Var(3 +E(L). mportance of xs relative to z is measured by the correlation ratio. = Var(WVar(Y) Note: inputs z are not requjred to be statistically independent. 11 Estimation (continued) D e c ~ i t i o onf the prediction variance of y work for dependent inputs : and simulation variables z. (Biased) estimates available are: Correlation ratio R = $$ = SSB/SST Partial correlaton ratio A 7jz.x = ssw/sst Expectations under LHS would be approximate.

6 1 Example : time dependent output 14 Correlation ratios A discrete event simulation of time dependent movements of various cargos by various types of aircraft. nputs input variables: Use Rate Fuel FlowEuroute Time... outputs Tons of cargo delivered by aircraft type C-SA (and many more). %# USe Rate Flow T Eiuo ime MOG gz offl Time T i Hours On1 nit CV (.39) is a critical value from normal theory under a null hypothesis of independence of : and y. t is used here only to filter. 15 nput Wse Rate fixed at nominal value Sampling plan Replicated Latin hypercube sampling (LHS). Size s = 1 replicated n = 4 times with T = replicates on random number streams. Total N = 1 x 4 x = 96 runs. 13 All inputs vary C-SA Tons day The average value.7 of R for Use Rate for C-5A Tons relates to the reduction of variabiity achieved by holding Use Rate fixed. Campare Slide 15 to Slide day Objective of input uncatam * tyanalysisistoexplainthe predictionllnc&mq depicted in the bands of output curves. y is function of day =

7 1 Example 1 (continued) 16 Use Rate and?fuel Flow fixed at 4 extreme points Model m(.) predicts flow of material in an ecosystem. We investigate how uncertainty in input parameters affects predicted concentrations in different subsystems of the ecosystem. C-5A Tons y-a scalar-is concentration in compartment C3 at equilibrium. a E c is vector of 4 inputs which are parameters in differential equatims that govern the concentrations. fz is a joint independent uniform distribution for the inputs. The= are no simulation variables z day Bands of curves correspond to different combinations of values of Use Rate (high in top bands) and Fuel Flow (high within top and bottom ). Widths of bands c<nrespond to variability attributable to the other 6 inputs. 17 Example : subset selection Questions to answer: (1) what is the unceftainty in predicted concentration? and () is there a small subset of inputs that drives the calculation? 19 Sequential selection procedure Sequentially build nested subsets of top inputs as follows: o Select top individual inputs. o Consider all pairs cunstructed with each top singleton and one additional variable. Select the top of those. o Consider all triples constructed with each top pair and one additional variable. Select the top of those. o And so forth. Validate 6nal selections with respect to families of conditional distributions { fyp } and {fyz~}. All 4 inputs vary The variability we want to explain is described by the prediction density function of y. 1 CrnC nkation!gkg 3 4 prediction (marginal) distribution fy all inputs vary

8 1 Estimated correlation ratios R 3 Several inputs fixed at several different values Figure shows fraction of the variance of y explained by each input individually m CY R '. TW 3 3 e4 6S..-...".."..."."..."*-*- m A subset of 1 of the 4 of inputs has been ide&ed as drivers T..."."."*..."._."......e Ranked input 6 1 Estimated correlationratios for individual inputs cqncmmong/kg 3 1 conditional distributions fy1x xs = ( } nput 1 fixed at several different values nput 1 has some affect on y. but not enough to be considered the driver. The distribution of y is not wllapsed. 1 4 Variability from unimportant inputs Narrow baud of conditional distributions is due to 74 unimportant inputs. 1 Gxwnbation gkg 1 conditional distributions fylx6 3 4 xs = input subset (1) 1 concsnlr~ong K g conditional distributions fylzr. xt = complement of ( }

9 Toward an implicit construction of M 5 Can structural uncertainty be examined like input uncertainty? Let alternative activity models be r e k m e n t s of a compartmental model. Effect is to change residence time rj (and resource utilization which we have not discussed) ! However - C Except in limited cases the model domain M exists at most inferentially and usually only hypothetidy. The meaning of gm is an open question. Two possibilities for it are (1) gm is a sampling distribution on M and () gm represents the likelihood (objective or subjective) that any particular model in M is correct with respect to reality. 6 Special case: structural uncertainty for a class of discrete event simulation models Simulation moves actors through activities. The simulation is charactenzed by sequences of activities Ai from the set { Q Q Z.} event times ti and residence or duration times r; for each actor.. 1 t = a 1-1 A = a r A A 6 3 = a 5 - r -- L A = a t4 Actor s view: time line of 4 activities We assume that this characterization of the model (sequences of activities event times and residence times) is sufficient to allow analysis of structural uncertainty. 7 Some questions related to structural uncertainty Are activities in the correct sequence? Are there missing or extra activities? wbat is the unexkinty in activity went times? What is the un- in the activity residence times? 1 i - A highly aggregated model is indicated by box 1 above. Disaggregation or = h e m a t is indicated by boxes sand further by boxes ~ A 1 Structural model uflceftainty really addresses two issues: how well a model uses inputs z to determine the output y. and how well 1: alone in the limitcan predict y. Generally activity modeling and refinement relate to the first point and notions of aggregation and ensemble averaging in relate to the second. 9 Final thoughts We discussed some directions for research and development of methods for assessing simulation variability input lllk&m * Q and structural model uncertainty. Variancebased measures of importame for input and simulation variables arise naturally when using the quadratic loss function of the diffmnce between the full model prediction y and the restricted prediction 5. Generic methods for assessing structural model uncertainty do not now exist. Methods to analyze structural umxbmty * for particular classes of models like discrete event simulation models may be attainable.

10 .. 3 Bibliography C. L. Atwood. ndividual model evaluation and probabilistic weighting of models. n Proceedings of Workshop in Advanced Topics in Risk and Reliability Analysis Model Uncertainty: its Characterization and Quantijkation NUREG/CP-13 pages Annapolis MD U.S. Nuclear Regulatory Commission. David C. Cox. An analytical method for uncertainty analysis of nonlinear output functions with application to fault-tree analysis. EEE Transactions on Reliability R-31(5): R.. Cukier H. B. Levine and K. E. Shuler. Nonlinear sensitivity analysis of multiparameter model systems. Journal of Computational Physics 6: David Draper. Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society B 57(1): Jon C. Helton. Treatment of uncerkm 'tyin performance assessments of complex systems. Risk Analysis 14(4): Ronald L. man and Stephen C. Hora. A robust measure of uncertainty importance for use in fault tree system analysis. Risk Analysis 1(3): B. Krzykacz. Samos: A computer program for the derivation of empirical Sensitivity measures of results from large computer models. Technical Report GRS-A-17 Gesellschaft fur Reaktorsicherheit (GRS) mbh Garchin& Republic of Germany 199. Kathryn Blachond Laskey. Model unce-ty: theory and practical implications. EEE Transactions on Systems Man and Cybernetics 6(3): M. D. McKay W. J. Conover andr. J. Beclanaa A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 1(): Michael D. McKay. Sensitivity analysis. n Proceedings of the Workshop on Validation of Computer-based Mathematical Models in Energy Related Research and Development pages 3 6 Fort Worth TX197. Texas Christian University. Michael D. McKay. Aspects of modeling uncertainty and prediction. n Proceedings of Workshop i in Advanced Topics in Risk and Reliability Analysis Model Uncertainty: ts Characterization and Quantijication NUREG/CP-13 pages Armapolis MD U.S. Nuclear Regulatory Commission. Michael D. McKay. Evaluating prediction uncertainty. Technical Report NUREG/CR-6311 U.S. Nuclear Regulatory Commission and Los Alamos National Laboratory Michael D. McKay. Nonparametric variance-based methods of assessing uncertainty importance. Reliability Engineering and System SajGety Michael D. McKay and Richard J. Beclrmaa Using variance to identify important inputs. n Proceedings of the American Statistical Association Secn'on on Physical and Engineering Sciences Toronto Michael D. McKay and John D. Morrison. Strucml model uncertainty in stochastic simulation. n Proceedings of the 9th Symposium on the nerface: Computing Science and Statistics Houston TX1997. Max D. Morris. Factorial sampling plans for prelimimry computational experiments. Technomerrics 33(): H. H. Panjer. On the decompositionof moments by conditional moments. The American Statistician 7: Jerome Sacks William J. WelchToby J. Mitchell and Henry P. Wynn. Design and analysis of computer experiments. Statistical Science 4(4): M. Sobol'. Sesitivity estimates for nonlinear mathematical models. Mathematical Modelling and computational Experiment 1: Robert L. Winkler. Modeling Uncertainty: Probabilities for models? n Proceedings of Workshop in Advanced Topics in Risk and Reliability Analysis Model Uncertainty: ts Characterization and Quantijication NUREG/CP-13 pages Annapolis MD U.S. Nuclear Regulatory Commission.