Review of Methodology and Rationale of Monte Carlo Simulation

Transcription

1 Review of Methodology and Rationale of Monte Carlo Simulation Application to Metrology with Open Source Software Vishal Ramnath Mechanical Metrology Group National Metrology Institute of South Africa November 6, 2008

2 Overview of Presentation Introduction Review of GUM Methodology Review of Monte Carlo Methodology Analysing and Understanding the Data Reporting Results in GUM Terms

3 Review of GUM Methodology Review of Monte Carlo Methodology Introduction This is an introductory presentation to convey the basic ideas behind the mechanics of the Monte Carlo technique as applied to metrology measurement uncertainty problems. The rationale for the need to understand and implement Monte Carlo (MC) techniques in the context of metrology is that with the advance of science and technology more accurate measurements are for various reasons increasingly necessary in many economies and MC simulations present the most accurate and readily available numerical technology to solve such challenges taking into account certain limitations in existing approaches such as the well known ISO Guide to Uncertainty in Measurement (GUM).

4 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 1 - Essential Information Needed For an input quantity x i in the GUM framework three quantities are needed viz. the expectation of x i which is just the estimate of this input the standard deviation of x i which is the standard deviation of this input σ(x i ) the corresponding degrees of freedom ν i associated with x i If there are dependencies with other input quantities x j, j i then covariances are also required: in the case of two inputs x i and x j the covariance u(x i, x j ) and correlation coefficient r(x i, x j ) are related by (1) u(x i, x j ) = r(x i, x j )u(x i )u(x j ), 1 r(x i, x j ) 1

5 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 1 cont. - Essential Information Needed (2) u(x i, x j ) = 1 q(q 1) q (x i,k x i )(x j,k x j ) k=1 If r(x i, x j ) = 0 then there is no correlation and if r(x i, x j ) 1 then there is strong correlation Most uncertainty calculations assume r(x i, x j ) 0 for simplicity i.e. no correlation between input quantities but if necessary correlation can be explicitly incorporated into calculations In the case of correlation between more than two variables e.g. x i, x j, x k with i j k then a covariance matrix and not a scalar correlation coefficient is required

6 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 1 cont. - Essential Information Needed The GUM approach is the propagation of uncertainties associated with input quantities in a measurement model to provide estimates of the model output quantity (univariate) or quantities (multivariate) It should be noted that: Within the framework of the GUM a mathematical model of the measurand is a prerequisite in order to implement an uncertainty calculation

7 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 2 - Standard Calculation Technique (3a) (3b) (3c) (3d) (3e) y = f(x 1,..., x n ) math model c i = f sens coeff x i n [ ] 2 f u 2 (y) = u 2 (x i ) std unc x i u 4 (y) ν eff = k i=1 n i=1 t t c 4 i u 4 (x i ) ν i calc eff deg freedom Γ[ ν ν eff +1 ) eff +1 2 ] (1 πνeff Γ[ ν eff 2 ] + u2 2 du = p ν eff coverage factor

8 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 2 - Brief Comment on Sensitivity coeff s Various possibilities will arise in practise with real inputs x R n : univariate, explicit, real model or multivariate, explicit, real model univariate, implicit, real model or multivariate, implicit, real model In the case of an implicit model i.e. where an explicit functional relationship between the input and output(s) is not known then additional matrix algebraic manipulations are necessary and such manipulations require the solution of linear systems of equations In addition as per the above but with complex models i.e. with x C n require analogous sensitivity conterparts where now partial derivatives for both the real and imaginary components of an input are necessary

9 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 3 - Assumptions & Limitations of the GUM There are three chief requirements that limit the applicability of the GUM: the non-linearity for the measurand as modelled by a function f(x) must be insignificant [GUM Clause 5.1.2] the Central Limit Theorem must be assumed to apply for the model of the measurand i.e. the PDF for the output must be Gaussian (alternately in terms of a t-distribution) [GUM Clauses G.2.1 and G.6.6] and the necessary conditions for a Welch-Satterthwaite formula to calculate the effective degrees of freedom must apply [GUM Clause G.4.2]

10 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 4 - When & how will the GUM not work As per the three requirements for the GUM to adequately apply it will then by implication not function adequately when: non-linearities in the model are significant - when the model can not accurately be represented by a first order Taylor series expansion then the probability distribution of the measurand can similarly not be accurately represented in terms of the convolution integral of the distributions of the input quantities; the conditions for the validity of the Central Limit Theorem as applicable to the measurement model are not sufficiently strong - theoretically the CLT predicts a Gaussian distribution for the measurand only in the limit as the number of input quantities increases i.e. it is not necessarily a true or accurate representation of the measurand PDF for a small finite number of input parameters

11 Review of GUM Methodology Review of Monte Carlo Methodology GUM Review 4 cont. - When & how will the GUM not work As per the three requirements for the GUM to adequately apply it will then by implication not function adequately when: the conditions for the validity of the Welch-Satterthwaite formula are not present i.e. in the case for a univariate, real output y where the input quantities x are not mutually independent - the GUM does not state how ν eff is to be calculated when the input quantities are correlated i.e. even though correlation coefficients (alternately covariance matrix) may be modelled / calculated from experimental data there is no methodology to estimate ν eff and hence a corresponding coverage factor k unless one assumes u(x i, x j ) 0 i j Correlation coefficients r(x i, x j ) are used for calculating the combined standard uncertainty u c, cf. U = k(p, ν eff ) u c

12 Review of GUM Methodology Review of Monte Carlo Methodology Review of GUM Methodology - Background to why MC is being utilized With the three requirements for the GUM to adequately apply and with limitations and lack of applicability that arises when these conditions are not met for many practical measurement models of real measurement systems and standards we then see that: Due to the sometimes restrictive conditions on the limitations and applicability of the GUM that many NMI s and possibly even industrial metrology laboratories are starting to investigate and implement Monte Carlo simulations for their own laboratory standards and in inter-comparisons for e.g. CMC justifications

13 Review of GUM Methodology Review of Monte Carlo Methodology Review of MC Methodology In a measurement uncertainty analysis one is concerned with propagating uncertainties from inputs to outputs and the GUM propogates uncertainties from a first order approximation from a model of a measurement system with the assumption that the measurand has a Gaussian distribution whilst a MC method directly propogates PDF information without any prior assumptions. A MC method can be accurately described as a statistical sampling technique.

14 Review of GUM Methodology Review of Monte Carlo Methodology Outline of MC Process Used for a Univariate Model 1. select M Monte Carlo trials 2. generate M vectors by sampling from the PDF s for the set of N input quantities 3. for each vector evaluate the model to give the corresponding value of the output quantity 4. calculate the estimate of the output quantity i.e. the measurand and its associated standard uncertainty 5. use the simulation data to build a discrete representation of the distribution function 6. use the distribution function to calculate the coverage interval for the measurand

15 For continous random variables recall: f(x) is a PDF for a random variable x if (i) f(x) 0 x R, (ii) f(x)dx = 1, (iii) P(a < X < b) = b a f(x)dx The corresponding cumulative distribution function is F(x) = x f(t)dt For MC work we will use the following nomenclature: Let the PDF for input X i be g i (ξ i ), the PDF for the measurand Y be g(η), and G(η) = η g(z)dz denote the distribution function corresponding to g(η)

16 There are additional mathematical definitions and terminology that are necessary to more fully understand how a Monte Carlo simulation works in practice but for our purposes we will not delve too deeply into the finer details in this presentation and rather concentrate on some of the more practical considerations that are needed if one wishes to undertake and implement a MC measurement uncertainty analysis

17 A few preliminary points should be noted: a good random number generator is essential for reliable work - the MS Excel RNG is not satisfactory and will introduce problems the software code used should allow definition of a model and the parameters defining the PDF s for the input quantities symmetry in the output PDF is not assumed no derivatives are required there is an avoidance of the concept of effective degrees of freedom sensitivity coefficients are not calculated or needed: possible to modify post-processing to calc a sensitivity coeff

18 When developing a mathematical model for a MC simulation it should be noted that there is no distinction between Type A and Type B uncertainty contributors and that the measurand is simply defined in terms of a function e.g. (4) y = f(x 1, x 2,..., x n ) where the inputs x 1,..., x n directly model and describe the influence if an input is changed - it is this variation/change in the input parameters that is propogated through the model and hence influences the output expressible as an uncertainty.

19 A MC simulation is therefore different from the GUM in the sense that one has to have a full and complete understanding of the entire measurement system under investigation and one can not simply assign an input uncertainty and a unity sensitivity coefficient without adequate justification

20 In a MC simulation the PDF s of the input quantities g 1 (ξ 1 ),..., g N (ξ N ) are required and the following options are possible: if the input x i is a quantity that has been measured/calibrated then it will have a measurement/calibration certificate that was done with the GUM so the quoted value is the mean µ i and its standard uncertainty is obtained by dividing the expanded uncertainty by the applicable coverage factor - this is enough information to infer its PDF since the GUM result is always expressed in terms of a Gaussian PDF which is completely defined in terms of µ and σ similarly as above for rectangular (particularly if estimated from e.g. literature), triangular, U shaped distributions etc.

21 cont. a statistical analysis based on relevant theory may indicate an inputs PDF e.g. dimensional cosine terms and one then just has to estimate some parameters to fully define the PDF there may be discrete numerical data for an input parameter which means that input parameter s PDF can be built up with its frequency data (like a histogram)

22 Mass Meas Unc Example to Contrast the GUM vs. MC Example (Math Model Explanation & Details) Consider a mass calibration to illustrate the principles of a MC simulation and some of the differences with the standard GUM approach with a schematic illustration of the measurement system

23 Mass Meas Unc Example to Contrast the GUM vs. MC Example (Math Model Explanation & Details) Consider a mass calibration to illustrate the principles of a MC simulation and some of the differences with the standard GUM approach with a schematic illustration of the measurement system m W pivot δm R m R

24 Mass Meas Unc Example to Contrast the GUM vs. MC Example (Math Model Explanation & Details) Consider a mass calibration to illustrate the principles of a MC simulation and some of the differences with the standard GUM approach with a schematic illustration of the measurement system m W pivot δm R m R The principle of a mass measurement with an equal arm force balance is of a balance of moments generated which by applying a force balance with Archimedes principle for buoyancy effects we then get

25 Mass Meas Unc Example to Contrast the GUM vs. MC Example (Math Model Explanation & Details) Consider a mass calibration to illustrate the principles of a MC simulation and some of the differences with the standard GUM approach with a schematic illustration of the measurement system m W pivot δm R m R The principle of a mass measurement with an equal arm force balance is of a balance of moments generated which by applying a force balance with Archimedes principle for buoyancy effects we then get m W m R + δm R (5) m W g ρ air g = (m R + δm R )g ρ air g ρ W ρ R

26 Developing the Meas Model Example (Mass example cont.) Rearranging

27 Developing the Meas Model Example (Mass example cont.) Rearranging ( m W 1 ρ ) ( air = (m R + δm R ) 1 ρ ) air ρ W ρ R Symbol m W m R δm R ρ i Description mass of weight piece W mass of reference weight piece R small test mass to add onto m R to achieve force balance mass density with i respectively that of the weight W air medium air or that of the reference s density R

28 Developing the Meas Model cont. Example (Expressing model with lab specific stds) Mass is an invariant quantity as any physical parameter in metrology but the values quoted will differ depending on the lab system in use e.g. mass is heavier in air than in water which is why astronauts train under water to simulate weightlessness:

29 Developing the Meas Model cont. Example (Expressing model with lab specific stds) Mass is an invariant quantity as any physical parameter in metrology but the values quoted will differ depending on the lab system in use e.g. mass is heavier in air than in water which is why astronauts train under water to simulate weightlessness: In mass metrology laboratories use the concept of conventional mass Definition (Conventional Mass) The conventional mass m W,c of a weight W is the apparent mass of a hypothetical weight of density ρ W 0 = 8000 kg.m 3 that balances W in air when the air density is ρ air 0 = 1.2 kg.m 3 i.e. m W (1 ρ air 0 /ρ W ) = m W,c (1 ρ air 0 /ρ W 0 ) Fact (Usage of Conventional Mass) Conventional mass is simply a measurement tool used to incorporate the invariance of inertial mass i.e. compare apples with apples

30 Final Measurement Model We utilize the Open Source computer algebra system Maxima to simplify our calculations due to the substitions that the conventional mass introduces. The reason for why one may prefer to use a CAS is in the cases when fairly complicated expressions and hand calculations carry the risk of lengthy time and error generation.

31 Review of calculas theory - cf. GUM assumptions Recall that from multi-variable calculas that Taylor series expansions for the case of a single variable (6) f(x) = f(a)+ f (a) 1! (x a)+ f (a) (x a) f (n 1) (a) 2! (n 1)! (x a)n 1 +R n (x) can be generalized to the case for multiple variables. An example for two variables would be f(x, y) = f(a, b) + f x + 1 2! f (a, b)(x a) + (a, b)(y b) y [ 2 f x 2(a, b)(x a) f x y + 2 f y 2(a, b)(y b)2 ] + (a, b)(x a)(y b)

32 Approx. of functions - linearized in 1D 9 Let f(x) = exp(x 2 ) and expand about a = 1 f = exp(x 2 ) 8 y-axis function e x2 1st e + 2e(x 1) 2nd e + 2e(x 1) + 3e(x 1) x-axis

33 Approx. of functions - linearized in 2D Let f(x) = exp(x 2 + y 2 ) and expand about a = [1, 1] T Linear approx: L(f) = 2e 2 (y 1) + 2e 2 (x 1) + e 2 z y x d plot y x1-3 % error

34 Approx. of functions - linearized in 2D Let f(x) = exp(x 2 + y 2 ) and expand about a = [1, 1] T Linear approx: L(f) = 2e 2 (y 1) + 2e 2 (x 1) + e 2 Quadratic approx: f = L(f)+ 1 2 [6e2 (y 1) 2 +8e 2 (x 1)(y 1)+6e 2 (x 1) 2 ] 9 3d plot z y x d plot z y x y y x1-3 % error x % error

35 Approx. of functions - linearized in 2D Let f(x) = exp(x 2 + y 2 ) and expand about a = [1, 1] T Linear approx: L(f) = 2e 2 (y 1) + 2e 2 (x 1) + e 2 Quadratic approx: f = L(f)+ 1 2 [6e2 (y 1) 2 +8e 2 (x 1)(y 1)+6e 2 (x 1) 2 ] 9 3d plot z y x d plot z y x y x % error % error The error for a linear approximation of the very non-linear model function i.e. using the GUM method is 3% and for a quadratic approximation 0.3% y x

36 Taylor series for f(x 1,..., x d ) For n variables we then have the Taylor series T(x 1,..., x d ) for f(x 1,..., x d ) as (7) T(x 1,..., x d ) = n 1 =0 n d =0 n 1 x n 1 1 n d x n d d (x 1 a 1 ) n1 (x d a d ) n d f(a 1,...,a d ) n 1! n d! (8) T(x 1,...,x d ) = D α f(a) (x a) α α! i N 0

37 Special cases for Taylor series expansions For a model f(x) of the measurement system given with inputs x = [x 1,...,x n ] T (x is a column vector with dimensions n 1) and with nominal value a which is the state that the measurement ystem is in then making use of the general Taylor series expansion for multiple variables T(x) = D α f(a) α 0 α! (x a) α we note that: First order approximation: (9) f(x) f(a 1,..., a n ) + f x 1 (x 1 a 1 )+ + f a x n (x n a n ) a

38 Special cases for Taylor series expansions cont. In most cases it is seldom beneficial to construct a multiple variable Taylor series expansion for 3rd or higher order Second order approximation: [ ] f f(x) f(a 1,..., a n ) + f x a a x n a (10) + 1 2! 2 f x1 2 2 f x 2 x 1. 2 f x n x 1 2 f x 1 x 2 2 f 2 f x 2 2 x 1 x n 2 f x 2 x n f x n x 2 2 f xn 2 x 1 a 1. x n a n x 1 a 1 x n a n Comment: the n n square matrix above is the Hessian matrix for f and all the entries are evaluated at a

39 Final Measurement Model - Calc details Recall that we have a model for the mass to be measured ( (11) m W 1 ρ ) ( air = (m R + δm R ) 1 ρ ) air ρ W ρ R and we wish to write the model in terms of the conventional mass by subsituting the formulae ( m W = m W,c 1 ρ ) ( air0 1 ρ ) 1 air0 (12a) (12b) (12c) ρ W 0 m R = m R,c ( 1 ρ air0 ρ W 0 δm R = δm R,c ( 1 ρ air0 ρ W 0 ) ( 1 ρ air0 ρ W ) 1 ρ R ) ( 1 ρ air0 ρ R ) 1 Once the equations are substited we then want to solve for m W,c which is the mass of the weight that we wish to calibrate

40 Measurement Model - Formulating and solving in Maxima In Maxima we use the following computer code: LHS: mw*(1 - rhoair/rhow); RHS: (mr + deltamr)*(1 - rhoair/rhor); LHS1: ev(lhs, mw = mwc*(1 - rhoair0/rhow0)/(1 - rhoair0/rhow)); RHS1: ev(rhs, mr = mrc*(1 - rhoair0/rhow0)/(1 - rhoair0/rhor), deltamr = deltamrc*(1 - rhoair0/rhow0)/(1 - rhoair0/rhor)); soln: solve(lhs1 = RHS1, mwc); mwc: rhs(soln[1]); whence m W,c = 1 (ρ R ρ air 0 )ρ W ρ air ρ R + ρ air ρ air 0 [((m R,c + δm R,c )ρ R ρ air m R,c ρ air δm R,c )ρ W (13) +( ρ air 0 m R,c ρ air 0 δm R,c )ρ R +ρ air ρ air 0 m R,c + ρ air ρ air 0 δm R,c ]

41 Measurement Model - GUM Formulation The measurement model consists of five input parameters which we list below: symbol description PDF comments m R,c reference mass Gaussian from meas cert δm R,c balance mass Gaussian from meas cert ρ air density of air rectangular estimated from CIPM formula ρ W density of weight rectangular estimate that is equally likely ρ R density of reference rectangular from literature of physical properties Comment on parameters that are not included: the density ρ W 0 does not explicitly appear in the model equation as it cancels out when the model equation is algebraically solved for m W,c which would not be obvious in a spreadsheet the air density ρ air 0 is not included in the model as a variable but as a constant since this is known exactly

42 Practical Implementation of MC Model Inputs Definition (Model constants & parameters) In a measurement mathematical model working in SI units one should distinguish between how to incorporate constants and parameters. A parameter is a variable that one is uncertain of and which has a statistical uncertainty (however small) and PDF whilst a constant is exactly known. 1 Fact (Theories which use exact constants i.e. zero unc) An example would be the speed of light which was historically measured using various techniques with associated experimental uncertainties and with Einstein s Special Theory of Relativity fixed and then later defined as c 0 = m.s 1 where σ(c 0 ) def = 0 Fact (Theories which use approx constants i.e. finite unc) An example would be the Avogadro number N A = mol 1 which as a constant of nature is fixed but which is currently experimentally known to an accuracy of σ(n A ) = mol 1 1 For details see the CODATA website for physical and chemical reference values at

43 Math Model - GUM Calcs 1 We will consider both first and second order calc s using the GUM for comparison with a Monte Carlo simulation. 1st order: apply the GUM as usual with sums of products of gradients etc. 2nd order: must build up f with additional terms using Hessian matrix etc. Fact (Practical observation of GUM calc s) The GUM makes use of the assumption that there is a linearized model of the system to propogate the uncertainties and to be strictly consistent one should apply a linearized model when calculating the standard uncertainty in order not to mix of terms from different assumptions and approximations, however we note that in practice most metrologists would most likely take the original expression to evaluate the model and not its linearization - this is only valid if the model is approximately linear in a neighbourhood of a where a is the state space that the system is in.

44 Math Model - GUM Calcs 2 Since there are five input variables in our mathematical model the distinction and implications of 1st and 2nd order approximations are not immediately obvious to appreciate. Noting the variables m R,c, δm R,c, ρ air, ρ W, ρ R we can reasonably conclude that the two variables that are most likely to be uncertain and vary are δm R,c because this must be adequately controlled to achieve a balance and equlibrium on the force beam and the equilibrium can be a bit subjective if there isn t an exact balance and the beam is moving very slowly ρ air the actual air density which will depend and vary with the laboratory s ambient temperature and pressure

45 Math Model - GUM Calcs 2 cont Setting the nominal conditions for argument as a = (m R,c, δm R,c, ρ air, ρ W, ρ R ) = (0.099 kg, kg, 1.17 kg.m 3, 7800 kg.m 3, 8000 kg.m 3 ) for m W,c kg we can then see in a limited sense the implications of the GUM requirement for a linearized model.

46 Mass Unc Math Model - Approx Linear approx of f error [ppm] O( ) Quadratic approx of f error [ppm] O( ) ρair / kg.m ρair / kg.m e-008 4e-008 3e-008 2e-008 1e e e-008-3e e δm R,c / kg δm R,c / kg e-008

47 Uncertainty results using the GUM 1 Assuming for argument that all the inputs for m W,c = f(m R,c, δm R,c, ρ air, ρ W, ρ R ) have uncertainties of 0.1% and in addition are not correlated we then have that the uncertainty estimate for the mass being weighed m W,c reported in standard uncertainty is: 1st order: (14) u(m W,c ) = kg 2nd order: (15) u 2 (f) = N ( ) 2 f u 2 (x i ) i=1 + N i=1 = L + H x i N j=1 [ ( ) ] f + f 3 f u 2 (x 2 x i x j x i x i xj 2 i )u 2 (x j ) u(f) = kg difference in uncertainty is underestimated by approx. 113 ppm

48 Uncertainty results using the GUM 2 Full uncertainty can be misleading if linearization not accurate if f is very non-linear in the above it is not too significant since f is not too non-linear It should be noted that H is formed out of double sum over the number of variables N and that such a calculation can only realistically be performed in a computer algebra system due to the excessive number of partial derivatives that must be computed e.g. with N = 5 then 100 partial derivatives must be calculated. The computer code to perform this computation within a CAS e.g. Maxima is relatively straightforward to implement but it should be noted that the full expression can become algebraically large and unwieldy the non-linear terms correctly evaluate to zero when the model is indeed linear.

49 Including non-linear terms using the GUM Example (A nonlinear functional) Work out the uncertainty for f(x 1, x 2 ) = exp[x x 2 2 ] assuming u(x 1 ) = u(x 2 ) = 0.1% at the point a = [x 1 = 1, x 2 = 1] T and compare the linear and nonlinear answers using the GUM. The linearization of f is 2e 2 (x 2 1) + 2e 2 (x 1 1) + e 2 and now the nonlinear term H where u 2 (f) = L + H is H = u 2 1 ( u 2 2 ( (8 x 21 x22 e2 x 2 2 ) +x2 ( )) x 1 e x2 2 +x2 1 8 x 1 x 2 2 ex2 2 +x x 1 e x2 2 +x2 1 +u 2 1 ( ) 2 ( ) 4 x 2 2 x 1 e x2 2 +x2 1 8 x 1 3 ex2 2 +x x 1 e x2 2 +x2 1 ex2 2 +x e x2 2 +x u 2 2 ( u 2 1 ( (8 x 21 x22 e2 x 2 2 ) +x2 ( )) x 2 e x2 2 +x2 1 8 x 1 2 x 2 ex2 2 +x x 2 e x2 2 +x2 1 +u 2 2 ( ) 2 ( ) 4 x 2 2 x 2 e x2 2 +x2 1 8 x 2 3 ex2 2 +x x 2 e x2 2 +x2 2 ex2 2 +x e x2 2 +x

50 Non-linear calc - how significant are the GUM approx? For f = e x2 1 +x2 2 the non-linear contribution H is non-zero as indicated above, and the difference between the linear and non-linear uncertainty estimates is u(f linear ) = u(f non linear ) = the difference in uncertainty is therefore underestimated by approx. 11 ppm Fact (GUM linear model underestimates uncertainty) It is seen that a linearized model may underestimate the actual uncertainty by ppm

51 Comparison of PDF s for Math Model s Calc Unc When calculating uncertainties there are three PDF s that one must consider when interpreting the measurand s uncertainty: the actual PDF for the measurand as computed in terms of a convolution integral (Markov formula 2 ) (16) g(η) = g(ξ)δ(y f(ξ))dξ N dξ N 1 dξ 1 a Gaussian 3 like i.e. a t-distribution with ν eff degrees-of-freedom via. the Welch-Satterwaithe formula for the calculation of the measurand y s PDF as per the GUM approach a discrete PDF in a Monte Carlo simulation that is built up with sampled data from the input PDF s g 1 (ξ 1 ),..., g N (ξ N ) that will converge to the measurand s actual PDF (as calculated with a covolution integral) as the number of MC events M 2 adequate mathematical statistics working knowledge is necessary to fully understand the conditions/derivation of the Markov formula wrt. GUM 3 A Gaussian PDF i.e. N(µ = 0; σ 2 ) is entirely defined in terms of the variance σ 2 whilst a t-distribution needs ν to define its shape

52 Comment on Application of Markov formula The GUM is based on the application of the Markov formula to linearized models and all of the results and formulae in the GUM can be derived (with certain assumptions) via. application of the Markov formula Practical examples: 1. Higher order terms are necessary in the GUM for non-linear models where the GUM will not work yielding incorrect results e.g. Y = X 2 where u(y) = 2x u(x) x if just linear terms of the form u 2 (f) = N i=1 [ x f u(x i i)] 2 are used 2. The Markov formula will yield the correct result with u(y) = u(x) 4x u2 (x) which is true even for x = 0 In general a direct evaluation is only analytically possible for certain simple cases whilst symbolic evaluation is only feasible with a low order of variables requiring transformations and evaluation/calculation of Jacobians with a numerical approach preferred

53 Comment on Application of Markov formula cont. The joint PDF g X1,X 2,...(ξ 1, ξ 2,...) built up in terms of matrix multiplications requires the use of a Dirac delta function δ as defined in terms of a sum with derivative terms and in addition manipulation of the inputs ξ i wrt. the output η Such calculations in the GUM require the application of further matrix algebra and will not be considered in this presentation The direct application of the Markov formula is in practice awkward and difficult to implement particularly in the case of non-linear models and the use of a Monte Carlo approach is entirely consistent with the Markov formula and is in fact a more practical calculation method that does not rely on any of the assumptions inherent as in the GUM

54 Implementation of the MC Algorithm We proceed as per the six steps as previously discussed as follows: 1. set the number of simulations to a minimum number of events to yield adequate statistical data to analyse, say M = the model for our measurand is (17) (18) m W,c = f(m R,c, δm R,c, ρ air, ρ W, ρ R ) y = f(x 1, x 2, x 3, x 4, x 5 ) so there are 5 inputs for the model and all of these inputs are simple scalar quantities. The first two inputs have a Gaussian PDF whilst the remaining inputs all have rectangular PDF s. Our numerical calculations will be performed in GNU Octave for convenience

55 Implementation of the MC Algorithm cont. 2. implementation for sample draws in Octave: for Gaussian PDF with g N(µ, σ): set z = µ + σ randn() for a rectangular PDF with mean µ and half-width a: set z = µ + a 2 (rand() 0.5) 3. the model is just evaluated by specifying the functional f as a m-file of form

56 Implementation of the MC Algorithm cont. 3. sample m-file: function value = mwc(mrc,deltamrc,rhoair,rhow,rhor) rhoair0 = 1.2; value = ( ((mrc + deltamrc)*rhor - rhoair*mrc - rhoair*deltamrc)*rhow + (-rhoair0*mrc - rhoair0*deltamrc)*rhor + rhoair*rhoair0*mrc + rhoair*rhoair0*deltamrc ) / ( (rhor - rhoair0)*rhow - rhoair*rhor + rhoair*rhoair0 ); sample function call: test = mwc(0.9999, , 1.19, 7950, ) test =

57 Implementation of the MC Algorithm cont. 3. Short overview of a MC program (actual program 300 lines of code) clear all clc M = input( Enter the number of MC events: ); % nominal values in model mrc_ref = 0.099; deltamrc_ref = 0.001; rhoair_ref = 1.17; rhow_ref = 7800; rhor_ref = 8000; % uncertainties in model ux1 = 0.1E-2*mRc_ref; ux2 = 0.1E-2*deltamRc_ref; ux3 = 0.1E-2*rhoair_ref; ux4 = 0.1E-2*rhoW_ref; ux5 = 0.1E-2*rhoR_ref; data = zeros(m,1); tic for i=1:m x1 = mrc_ref + ux1*randn(); x2 = deltamrc_ref + ux2*randn(); x3 = rhoair_ref + ux3*2*(rand() - 0.5); x4 = rhow_ref + ux4*2*(rand() - 0.5); x5 = rhor_ref + ux5*2*(rand() - 0.5); f = mwc(x1,x2,x3,x4,x5); data(i) = f; end toc hist(data) print( C:/Draft/test.eps, -color ) Running the above program on a single 2.4 GHz CPU takes approx. 1.8 seconds for 10 4 MC events i.e. simple models can realistically be simulated with run times of a few minutes.

58 Implementation of the MC Algorithm cont. 4. the output quantity is estimated by applying a weighted sum of all the MC events this is generally indistinguishable from the arithmetic mean for large M e.g. M = O(10 6 ) 5. the simulation is used to build up a discrete representation of the underlying model s distribution function this is equivalent to calc s via. a PDF but is more accurate: a rough analogy would be the use of a histogram of measured data as a indicator of the underlying PDF as illustrated in the previous slide 6. once all the data is generated a coverage interval is estimated by inverse linear interpolation using Ĝ(η) a rough explanation would be a minima search of a parameter α for a function built up in terms of the inverse function of the distribution function where α would be used to calculate lower and upper bounds for y for a given confidence limit e.g %

59 Analysing and Understanding the Data Reporting Results in GUM Terms A measurement uncertainty calculation using the Monte Carlo method requires an underlying model of the measurand (univariate or multivariate) and could be in the form of an explicit or implicit formulation. The steps involved in order to perform post-processing and analysis are simply: formulate a model of the measurand apply multiple simulations of the model using sampled draws from the input PDF s to generate a data set of the measurand to analyse The data set to perform further operations on is simply a collection of the model evaluated M times e.g. if the model is univariate and explicit i.e. y = f(x 1,...,x N ) then the data set is just an array of numbers (19) data = [y 1,..., y M ]

60 Analysing and Understanding the Data Reporting Results in GUM Terms Analysing and Understanding the Data An statistical analysis of the data using formulae as defined in the techniques/algorithms inherent in a Monte Carlo measurement uncertainty calculation can be thought of as a numerical/statistical experiment that approximates a real physical experimental situation with simulated experimental conditions. The data that results from a MC simulation are: ŷ = the expected value of the measurand u(ŷ) = the measurand s standard deviation [y low, y high ] = the confidence interval of the measurand for a given probability e.g. p = 95.45% using the actual PDF g(η) for the measurand i.e. the expanded uncertainty The measurand s PDF g(η) is not necessarily symmetric or even Gaussian which means a simple ± for expanded uncertainty may be misleading with results more accurately reported of form ŷ +(y high ŷ) (ŷ y low )

61 Analysing and Understanding the Data Reporting Results in GUM Terms Reporting Results in GUM Terms In essence a MC simulation will yield the same outputs that are required in a measurement uncertainty calculation viz. standard and expanded uncertainties for the expected value of the measurand (ŷ µ): (20) (21) u(y) = u(ŷ) σ U(Y) 1 2 [(y high ŷ) + (ŷ y low )] the difference is that since the method does not involve the concept of effective degrees of freedom ν eff there is no corresponding coverage factor e.g. k = 2 for a p = 95.45% confidence level so an analogous quantity would be (22) k = U(y) u(y) Recall that in the GUM one must convert to standard uncertainties where the PDF is by definition Gaussian i.e. N(µ, σ 2 )

62 The GUM is based on certain assumptions/approximations and only be accurately applied in certain limiting cases Modifications to the standard GUM technique are available in certain limiting cases but are generally limiting and difficult to apply with only modest improved uncertainty accuracy improvement A Monte Carlo measurement uncertainty simulation will work for any problem of arbitary complexity and will always yield accurate and reliable results with sufficient MC simulation events (an adaptive MC algorithm to test for convergence is possible to avoid unnessary computational overhead) Questions