RECOMMENDED TOOLS FOR SENSITIVITY ANALYSIS ASSOCIATED TO THE EVALUATION OF MEASUREMENT UNCERTAINTY

Advanced Mathematical and Computational Tools in Metrology and Testing IX Edited by F. Pavese, M. Bär, J.-R. Filtz, A. B. Forbes, L. Pendrill and K. Shirono c 2012 World Scientific Publishing Company (pp. 1 12) RECOMMENDED TOOLS FOR SENSITIVITY ANALYSIS ASSOCIATED TO THE EVALUATION OF MEASUREMENT UNCERTAINTY A.ALLARD Laboratoire National de mtrologie et d Essais 1, rue Gaston Boissier, 75724 Paris Cedex 15, France E-mail: alexandre.allard@lne.fr www.lne.fr N.FISCHER Laboratoire National de mtrologie et d Essais 1, rue Gaston Boissier, 75724 Paris Cedex 15, France E-mail: nicolas.fischer@lne.fr www.lne.fr Sensitivity analysis is an important part of metrology, particularly for the evaluation of measurement uncertainty. Such an analysis allows to have a better knowledge on the contributions of the different input quantities to the variance of the output quantity. This article intends to give recommendations for the computation of the contributions to the variance, particularly when a Monte Carlo method is used. First in the case of an approximately linear (or monotonic measurement model), we propose to compute Spearman correlation coefficient. This sensitivity measure is easy to compute, doesn t need extra Monte Carlo trials and provides a good estimation of the sensitivity indices under the monotony of the measurement model assumption. Second, for a non linear measurement model, we propose to characterize the sensitivity through the variance of the conditional expectation. This quantity may be estimated by different methods, often known as importance measures. The principle is to decompose the variance of the output quantity into terms of increasing dimension. All these methods will be able to deal with sensitivity indices whatever the properties of the measurement model are, and also to provide an estimation for higher-order sensitivity indices if needed. The advantages and drawbacks of the different sensitivity indices are then discussed through a practical application case. Care should be taken to provide a sensitivity method that is consistent with the measurement model considered and also that minimizes the computational cost. Keywords: Sensitivity Analysis, Measurement Uncertainty, Monte Carlo Methods, Spearman coefficient, Sobol, Local polynomial Smoothers 1

1. Introduction Both the GUM and its Supplement 1 constitute reference documents to evaluate the measurement uncertainty. They provide guidance on such an evaluation whether a Taylor approximation or a Monte Carlo simulation is used. In both cases, the evaluation of measurement uncertainty could be represented by Figure 1. Fig. 1. The general method to evaluate measurement uncertainty. In such a context, the uncertainty budget associated to the evaluation of measurement uncertainty is very useful to the metrologist to have a better knowledge of the measurement process, through the contributions to the variance of the output quantity of each input quantity. Within the GUM Uncertainty Framework (GUF) 3 this is given by the partial derivatives of the measurement mode that are computed at the best estimate of each input quantity. It is a local sensitivity measure. It doesn t give the uncertainty budget if the input quantities depart from their best value. It would be preferable to evaluate the sensitivity all over the range of the different input quantities, which falls in the field of global sensitivity analysis. The recent use of Monte Carlo Methods, encouraged by the publication of the Supplement 1 to the GUM in 2008, doesn t need to compute those partial 2

derivatives. Another sensitivity measure should then be used and, if possible, a global sensitivity method that is more accurate than local ones. This paper aims at proposing some different methods to quantify the contributions to the variance of the output quantity. Also, the computation of sophisticated sensitivity indices goes often along with a high computational time. Consequently, the choice of a sensitivity method should take into consideration many aspects of the problem : linearity/monotony of the measurement model, computational time for one run of the measurement model and total number of runs needed to compute the indices. The first part of the paper deals with the basic sensitivity methods, that are to be used for simple measurement models, if the linearity (or monotony) of the model is assessed. In a second part, more complex methods are introduced, denoted as variance-based methods. Both the concept and estimation tools are given. All these methods are illustrated through the mass calibration example from the supplement 1 to the GUM (GUM-S1 4 ) and the calculations presented here have been performed using R software. 2. Basic Sensitivity Methods 2.1. The One At a Time Method (GUM-S1) The computation of a sensitivity measure associated with a Monte Carlo method is not straitforward. The GUM-S1 proposes to evaluate such contributions by holding all quantities but one fixed at their best value and performing a new Monte Carlo simulation. This method provides a quantification of the effect of the given input quantity X i on the standard deviation of the output quantity. Such a procedure is not an efficient method to evaluate the contributions to the variance of the output. First, it implies to make as many Monte Carlo simulations as the total number of input quantities n, which may be quite heavy if n is large. Second, it allows to evaluate the sensitivity while only one input quantity is varied in the same time. But in reality, all the input quantities are allowed to vary all together and if interactions are supposed to arise in a measurement process, they will only arise to the condition that all the quantities involved vary in the same time. Consequently this method is quite expensive from a computational point of view and fails to provide a complete uncertainty budget when interactions arise. However, if the measurement model is either linear or monotonic, there is no need to look for complicated methods to evaluate the contributions of interactions. In this case, a simple measure of the sensitivity may be 3

obtained through the computation of correlations coefficients. 2.2. Pearson and Spearman Correlation Coefficients If the measurement model is approximately linear, then the Pearson correlation coefficient as defined in equation 1 is suitable. M ( Xij X )( i Yj Ȳ ) j=1 ρ (X i ; Y )= M ( Xij X ) 2 M ( i Yj Ȳ ) 2 (1) j=1 j=1 where M is the total number of runs and X i and Y are the empirical means of respectively the input quantity X i and the output quantity Y. In order to be a bit less restrictive on the linear assumption, the Spearman correlation coefficient gives good results under the assumption of monotony of the measurement model. 6 Like the Pearson correlation coefficient, it can be easily compute from the original Monte Carlo data sample used to propagate the probability distributions, according to equation 2. r (X i ; Y )=1 6 M (RX ij RY j ) 2 j=1 M (M 2 1) (2) where RX ij denotes the rank of the j th observation of X i and RY j denotes the rank of the j th observation of Y. The corresponding sensitivity indices are obtained through a standardization of these correlation coefficient, and the contribution of the input quantity X i to the variance of the output quantity Y is given by the following quantity : S Spearman r 2 (X i ; Y ) i = n i=1 r2 (X i ; Y ) (3) These correlation coefficients are the suitable sensitivity methods to use in cases where there are no interactions arising in the measurement process. Their computation relies on the same samples as those used to compute the empirical distribution function of the output quantity, whereas more sophisticated sensitivity methods may require additional Monte Carlo runs and consequently a much higher computational time. 4

Whatever the software used to perform the Monte Carlo method according to the GUM-S1, they should be quite easy to compute, without any additional Monte Carlo trials. 2.3. Application Case : Mass Calibration Example The mass calibration example is provided by the GUM-S1. The measurand is the deviation δm of the conventional mass m W,c from the nominal value m nom = 100g : [ ( 1 δm =(m R,c + δm R,c ) 1+(ρ a ρ a0 ) 1 )] m nom (4) ρ W ρ R Five input quantities contribute to the variance of the measurand. They are assigned probability distribution functions as defined in table 1 : Table 1. Probability distributions of the input quantities for the mass calibration example Input quantity Probability Mean Value Stand. dev. Lower Bound Upper Bound X i (mg) distribution μ (mg) σ (mg) a (mg) b (mg) m R,c Gaussian 100 0.05 δm R,c Gaussian 1.234 0.02 ρ a Rectangular 1100 1300 ρ W Rectangular 7000000 9000000 ρ R Rectangular 7950000 8050000 A Monte Carlo method is performed in order to compute the empirical pdf for the output quantity. M =10 6 runs are used and the final results are given in Table 2. Table 2. Results of the propagation of distributions using a Monte Carlo method for the mass calibration example Mean value Standard deviation ylow yhigh (mg) (mg) (mg) (mg) 1.234 0.075 1.084 1.383 The goal is to compute the uncertainty budget, that is to say the contributions to the variance of δm of the different input quantities. Table 3 summarizes the first results obtained applying the basic sensitivity tools. 5

Table 3. Uncertainty budget for the mass calibration example according to the GUM at first order, the Pearson and Spearman correlation coefficients and the One-At-a-Time (OAT) method Input quantity GUM Pearson Spearman OAT X i 1st order Correlation Correlation method m R,c 0.86 0.86 0.86 0.86 δm R,c 0.14 0.14 0.14 0.14 ρ a 0 0 0 0 ρ W 0 0 0 0 ρ R 0 0 0 0 According to the results in table 3, m R,c is the most contributive input quantity. The interpretation of these results could lead to the conclusion that both ρ a, ρ W and ρ R have no contribution to the variance of the output. A simple calculation invalidates these conclusions. We performed five other Monte Carlo simulations. For each one, the standard uncertainty of one input quantity is decreased by 10% while the standard uncertainty of the other inputs remains at its original value and we observe the impact of such a change on the standard uncertainty ũ (δm) of the output. Table 4 summarizes the corresponding results. Table 4. Standard uncertainty of the output quantity while one input quantity has its standard uncertainty lowered by 10% Input quantity m R,c δm R,c ρ a ρ W ρ R Resulting ũ (δm) 0.722 0.750 0.720 0.719 0.754 Compared with the original value for the standard uncertainty of the output quantity, three input quantities have, in reality, an equivalent influence : m R,c ρ a and ρ W. Actually, the basic sensitivity tools fail to identify the contribution of both ρ a and ρ W, which is provided by the interaction effect. Then, another framework has to be defined to evaluate such contributions. This framework relies on the decomposition of the variance, using the variance of the conditional expectation. 6

3. Variance-Based Sensitivity Indices 3.1. Concepts Let s consider the total variance decomposition theorem, which states that thevarianceofaquantityy may be decomposed as the sum of the variance of the conditional expectation of Y given X i andtheexpectationofthe conditional variance of Y given X i. V (Y )=V [E (Y X i )] + E [V (Y X i )] (5) The first term V i = V [E (Y X i )] denotes the part of the variance of Y that is due to the variations of the input quantity X i while the second term denotes the part of the variance of Y that is due to all the input quantities but X i. The first order sensitivity index is then defined as S i = V i (6) V (Y ) A second order sensitivity index can also be deduced from the variance due to the couple of quantities X i and X j : S i,j = V [E (Y X i,x j )] V i V j (7) V (Y ) Higher order sensitivity indices can be obtained in the same manner, until n th order. Of course, it would be a hard work to compute them all, and moreover, the higher the order, the more likely the sensitivity index to be null. Hence, it is possible to define a total order sensitivity index T i which encloses all the effects of all orders in which the input X i is involved. T i = S i + j =is i,j +... + S 1,2,...,n (8) If an input quantity X i is involved is no interaction with another input in a measurement process, then T i = S i. Actually, when a measurement process is to be improved, the total sensitivity index should be considered in the general case. If a contributive standard uncertainty is reduced, the impact on the standard uncertainty of the output will result from the corresponding first order sensitivity index but also from the higher order sensitivity indices if any involving the same input quantity, as it is the case for the mass calibration example. 7

Different methods are available to estimate these quantities. We present here the Sobol method and the local polynomial smoothers method. As these methods may be computationally expensive, it s particularly interesting to identify the influent input quantities previously in order to reduce the set on influent input quantities of which we want to evaluate the contributions. The Morris s design can be used to this extent 6 3.2. Sobol Estimation The Sobol indices offer the ideal framework to evaluate the sensitivity associated to the evaluation of measurement uncertainty. 1 They allow to compute first and higher order sensitivity indices as well as total order ones. 7 They rely on two M-samples (x i,j ) i=1,...,n;j=1,...,m and ( ) x i,j i=1,...,n;j=1,...,m that are mixed together in order to obtain an estimate of the sensitivity index. The first order sensitivity index is then estimated by the quantity : where : S Sobol i = ˆD i ˆD (9) ˆD i = 1 M M f (x 1,k,..., x n,k ) f k=1 ( ) x 1,k,..., x (i 1),k,x i,k,x (i+1),k,x n,k ˆf 0 2 (10) is the estimator of the variance of the conditional expectation of Y given X i and ˆD = 1 M M k=1 f 2 (x 1,k,..., x n,k ) ˆf 2 0 is the estimator of the total variance of Y.Thequantity ˆf 0 = 1 M M k=1 f (x 1,k,..., x n,k ) denotes the empirical mean of y. ˆD ij may also be defined as above with both x i,k and x j,k taken from the first sample while the other inputs are taken from the two samples and enables to estimate the second order indices : S Sobol i,j = ˆD ij ˆD i ˆD j ˆD (11) Higher order indices can be obtained in the same manner. But as they are often null, it may be more efficient to compute the total order sensitivity indices, to visualize if some significant contributions are missing. The estimation of the total sensitivity indices relies on the estimation of the variance of the output due to the variations of all other input quantities than X i. Sobol proposes then the corresponding estimate : 8

T Sobol i =1 ˆD i ˆD (12) where : ˆD i = 1 M M f (x 1,k,..., x n,k ) f ( x 1,k,..., x i 1,k,x ) i,k,x i+1,k,x n,k 2 ˆf 0 k=1 (13) Such an estimation is computationally very expensive. Indeed, it needs a very large number of runs to reach the convergence of the estimation. For practical purposes, bootstrap samples may be used to compute mean values for the sensitivity indices as well as a standard deviation in order to control this convergence. Then, the use of the Sobol indices may not be possible if the measurement model f is a computational code for which one evaluation of the code may last one second or more. In this case, another faster method has to be used, such as the local polynomial smoothers. 2 3.3. Local Polynomial Smoothers The second method for the estimation of the variance-based sensitivity indices as proposed here rely on a non-parametric estimation using a kernel function of the relationship between the input quantity and the output quantity. We consider an heteroskedastic model that is estimated using local polynomial smoothers : Y k = m (X ik )+σ (X ik ) ɛ k,k =1,..., M (14) where m (X ik )=E (Y X i = x i )andσ 2 (X ik )=V (Y X i = x i )foreach single set of observation (X ik ; Y k ). The principle is to approximate the regression function m by a p th order polynomial, for z in a neighborhood of x : m (z) = p β j (z x) j (15) j=0 where β j denotes the coefficients of the polynomial. This one is then fitted to the observations (Y k,x ik ) by solving the least-squares problem 9

min β M Y k k=1 2 p ( ) β j (X ik x) j Xik x K h j=0 (16) The quality of this estimation is then dependent on the choice of the kernel function K as well as the smoothing parameter h. More details about this estimation are given by Da Veiga et al. 5 The ( estimate ) ˆm (x) of the function m (x) isthen: ˆm (x) = ˆβ 0 (x) Let Xit be another sample of the input quantity X i.this t=1,...,m sample is used to estimate the variance of ˆm, which is an estimate of the VCE : ˆT LP i = 1 M M t=1 ( ( ) 2 ˆm Xit ˆ m) (17) where ˆ m = 1 ( M M 1 t=1 ˆm Xj ). This estimation of the sensitivity can be standardized to obtain the contribution of X i to the variance of the output Y : Ŝ LP i = ˆT LP i ˆσ 2 z (18) where ˆσ 2 z is the estimator of the total variance of Y. In order to compute the second order sensitivity indices between X i and X j, the variance of conditional expectation of Y given both X i and X j is to be considered m (X i,x j )=E (Y X i,x j ). Then, the estimation of the corresponding regression model leads to the calculation of the global contribution of the couple (X i ; X j ): ˆT LP ij = 1 M M t=1 ( ( ˆm Xit, X ) 2 jt ˆ m) (19) and the contribution of the interaction is obtained as follows : Ŝ LP ij = ˆT LP ij LP ˆT i ˆσ z 2 ˆT LP j (20) The convergence for the estimation of the sensitivity indices using local polynomial smoothers is obtained with quite low sample sizes. However, it s 10

important to notice that two different samples are needed : the first one to to perform the estimation of the VCE ˆm, and the second one to compute the sensitivity estimates ˆT i. 3.4. Application Case : Mass Calibration Example For the mass calibration example, the application of these last two methods give the results summarized in table 5. Uncertainty budget a for the mass calibration ex- Table 5. ample Input quantity Sobol Local Polynomial Smoothers X i m R,c 0.47 0.46 δm R,c 0.02 0.07 ρ a 0 0 ρ W 0 0 ρ R 0 0 Interaction ρ a,ρ W 0.59 b 0.49 Interaction ρ a,ρ R 0.4 0 Note: a Performed with M=300 000 runs b Please note that for the Sobol estimation, the convergence of the indices is not achieved yet We note an important gap between the variance-based indices and the simple basic ones in Table 3, which is due to the presence of an important interaction in the measurement model. The basic sensitivity methods are not able to deal with such interactions and give a false estimation of the sensitivity. Both Sobol and LSP methods are indeed able to identify and quantify the interaction effect arising from ρ a and ρ W, which is approximately in the same proportion as the first order contribution of m R,c. Moreover, these sensitivity indices fit quite well with the observations made in table 4, about the reduction in the variance of the output quantity that can be obtained if we are able to reduce the variance in either m R,c either ρ a or ρ W. This is a strong evidence that a full decomposition of the variance is needed in such a case. 4. Conclusion As a conclusion, this paper identifies some good practice in performing a sensitivity analysis associated to the evaluation of measurement uncer- 11

tainty. We identify the theoretical quantity, the variance of the conditional expectation, that is to be estimated when the objective is to determine an uncertainty budget associated to an evaluation of measurement uncertainty. It is a measure of how the variations of a given quantity X i impact the variations of the output Y. Moreover, this is not only a measure in a neighborhood of the most probable value for the quantities involved (local sensitivity analysis), but all over their range (global sensitivity analysis). First, in the case of a simple measurement model (linear or linearized, monotonic), the Spearman correlation coefficient constitutes an easy and valid method to calculate the contributions to the variance of all the input quantities. Moreover, it is easy to implement in any software. Therefore, we strongly recommend to perform this index for such measurement models. Simple sensitivity measures should be used for simple models. Second, if the measurement model is more complex or implies any interaction effect, then the Spearman correlation can t deal with the second order effects. Then, both the Sobol indices and the estimation using local polynomial smoothers give a suitable estimation of the contributions to the variance. If the computational time to compute one run of the measurement model is low, then the Sobol estimation allows to estimate all orders sensitivity indices, included total order ones. Otherwise, the local polynomial estimation should be used to perform first and second order sensitivity indices. References 1. A.Allard and N.Fischer, Sensitivity analysis in metrology : study and comparison on different indices for measurement uncertainty, inadvanced Mathematical and Computational Tools in Metrology and Testing VIII, 78, 1-6 (2009). 2. A.Allard and N.Fischer and F.Didieux and E.Guillaume and B.Iooss, Evaluation of the most influent input variables on quantities of interest in a fire simulation, injournal of the French Society of Statistics, 152, 1,103-117 (2011). 3. BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, Evaluation of measurement data - Guide to the expression of Uncertainty in Measurement, (2008). 4. BIPM, IEC, IFCC, ISO, IUPAC, IUPAP, Evaluation of measurement data - supplement 1 to the Guide to the expression of Uncertainty in Measurement - Propagation of distributions using a Monte Carlo method, (2008). 5. S.Da Veiga and F.Wahl and F.Gamboa, Local Polynomial Estimation for Sensitivity Analysis on Models With Correlated Inputs, intechnometrics, 51, 4, 452-463 (2009). 6. A.Saltelli and K.Chan and E.M.Scott, Sensitivity Analysis, (Wiley, 2000). 7. I.M.Sobol,Sensitivity Estimates for Nonlinear Mathematical Models, inmathematical and Computational Experiments, 1, 407-414 (1993). 12