UNCERTAINTY AND SENSITIVITY IN STATISTICAL DATA

Size: px

Start display at page:

Download "UNCERTAINTY AND SENSITIVITY IN STATISTICAL DATA"

Sabrina Hopkins
5 years ago
Views:

1 UNCERTAINTY AND SENSITIVITY IN STATISTICAL DATA PhD Claudiu VAIDA-MUNTEAN Prof.univ.dr. Virgil VOINEAGU Conf. Univ. Dr Gabriela MUNTEANU Bucharest University of Economic Studies ABSTRACT Uncertainty and sensitivity analysis in statistical data is considered a necessary requirement in current statistical and econometric practice. Composite indicator development involves stages where subjective judgements have to be made: the selection of individual indicators, the treatment of missing values, the choice of aggregation model, the weights of the indicators, etc. All these subjective choices are the bones of the composite indicator and, together with the information provided by the data themselves, shape the message communicated by the composite indicator. Since the quality of a model depends on the soundness of its assumptions, good modelling practice requires an evaluation of the confidence in the model, assessing the uncertainties associated with the modelling process and the subjective choices taken. This is nothing but sensitivity analysis studying the relationship between information flowing in and out of the model. The main purpose of sensitivity analysis is to highlight how the variation in the output can be apportioned, quantitatively, to different sources of variation in the assumptions, and how the given composite indicator depends upon the information fed into it. Sensitivity analysis is closely related to implicit uncertainty analysis of the model. A combination of uncertainty and sensitivity analysis can help to gauge the robustness of the composite indicator ranking, to increase its transparency, to identify which countries are favoured or weakened under certain assumptions. In econometric practice, a priori assumptions have to be verified a posteriori, with adequate statistical data. In what follows, we shall describe how to apply uncertainty and sensitivity analysis to composite indicators, which has proven to be useful in dissipating some of the controversy. Key words: statistical data, uncertainty, sensitivity Revista Română de Statistică - Supliment nr. 2 /

2 Let CI be the index value for the country j ( j=, M ) CI j =f r j (I j,i 2j,...I Qj,W s,w S2,...,W SQ ) () according to the weighting model f rs, where the index r refers to the aggregation system (linear, geometric or multicriterial), and s to the weighting scheme. (e.g. budget allocation, analytical hierarchy, advantage... etc). The index is based on Q normalised individual indicators I j,i 2j,... I Qj, for that country and scheme-dependent weights W s,w S2,...,W SQ for the individual indicators. The most frequently used normalisation methods for the individual indicators are based on the Min-Max values, the standardised values, or on the raw indicator values etc. The rank assigned by the composite indicator to a given country, i.e. the Rank ( CI j ), is an output of the uncertainty/sensitivity analysis. It is interesting to explore the average shift in country ranking. This may be determined as the average of the absolute differences in countries ranks with respect to a reference ranking over the M countries: M Rs= Rangref ( CIj) Rang( CIj) (2) M j In the reference ranking, the investigation Rank (CIj) and R S is in fact the threshold of uncertainty and sensitivity analysis. The analysis itself is conducted as a single MONTE CARLO experiment, e.g. by exploring all uncertainty sources simultaneously to capture all possible synergistic effects among uncertain input factors. This involves the use of triggers, e.g. the use of uncertain input factors to make a decision which aggregation system and weighting scheme to adopt. A discrete uncertain factor, which can take integer values between and 3, is used for the aggregation system and similarly for the weighting scheme. Other trigger factors are generated to select those indicators to be omitted, the editing scheme, the normalisation scheme and so on, until a full set of input variables is available to calculate Rank (CI j ) and R S. Uncertainty analysis (UA) The components of a composite indicator (CI) can introduce uncertainty into the output variables, Rank (CI j ) and R S. In essence, the UA is based on simulations that are carried out on various equations that constitute 30 Romanian Statistical Review - Supplement nr. 2 / 204

3 the underlying model. The uncertainty elements are transferred into a set of scalar input factors, so that the resulting Rank (CI j ) and R S are nonlinear functions of the uncertain input factors and the estimated probability distribution of Rank (CI j ) and R S. Various methods are available for evaluating output uncertainty. In what follows, we shall present the MONTE CARLO approach, which is based on multiple evaluations of the model with k randomly selected model input factors. There are three steps in this procedure:. Assign a probability distribution to each input factor X i (i=,k). The first input factor X is used for the selection of the editing scheme. X Estimation of missing data. Use bivariate correlation to impute missing data. 2. Assign zero to missing data. The second input factor, X 2, is the trigger to select the normalisation method. X2 Normalisation Min-Max 2 Standardisation 3 None Both X and X 2 are supposed to be discrete random variables. In practice they are generated by drawing a random number ε, uniformly distributed between [0;]. Applying the so-called Russian roulette algorithm, e.g. for X select if ε ( [0;0,5] and 2 if ε [0,5;]. Uncertain factor X 3 is generated to select which individual indicator, if any, should be omitted. Revista Română de Statistică - Supliment nr. 2 / 204 3

4 ε X3 excluded individual indicator 0; Q + X 3 =0, all indicators are favoured, Q + Q + 2 X 3 = Q Q +, X 3 =Q In other words, with probability, no individual indicator will Q + be excluded, while with probability one of the Q individual Q + indicators will be excluded with equal probability. Obviously, the probability of X 3 =0 could have been made larger or smaller than and the values Q + X 3 =,2...,Q could have been sampled with equal probability. A sensitivity analysis is useful in order to track which indicator affects the output the most when excluded. Trigger X 4 is used to select the aggregation system: X4 Aggregation Scheme LIN (linear) 2 GEO (geometric) 3 MCR (multicriterial) When the LIN scheme is selected, the composite indicators are computed as: 32 Romanian Statistical Review - Supplement nr. 2 / 204

5 Q CI j= WSiI ij i, (3) while when the GEO scheme is selected, the relation is: CI j= a i= () Iij WSi (4) When MCR is used, the countries are ranked directly from the outscoring matrix. Factor X 5 ( to select the weighting scheme), and X 6 (to select the expert) etc. are used in a similar way. 2. Generate randomly N combinations of independent input factors, X l (l=, N )- A set of independent input factors, X l l l = [ X X X ] l, 2,... k is called a sample. For each trial sample X l the computational model can be evaluated, generating values for the scalar output variable Y, where Y is either Rank (C ij ), either RS. 3. Close the loop over l=, N and analyse the resulting output vector Y l (cu l=, N ). The generation of samples can be performed using various procedures (simple random sampling, stratified sampling, quasi-random sampling or othersetc). The sequence of Y l gives the random probability distribution of Y the output. The characteristics of the random probability distribution, such as the variance and higher order moments, can be estimated with an arbitrary level of precision related to the size of the simulation N. Sensitivity analysis using variance-based techniques A necessary step when designing a sensitivity analysis is to identify the output variables of interest. Ideally these should be relevant. It has been noted earlier that composite indicators may be considered as models. When several layers of uncertainty are present simultaneously, a composite indicator could become a non-linear, possibly non-additive model. Specialists in non-linear models argued that robust, model-free, variance-based techniques should be used. That is because it displays additional properties convenient in the analysis. Revista Română de Statistică - Supliment nr. 2 /

6 - They allow an exploration of the whole range of variation of the input factors, instead of just sampling factors over a limited number of values. - They are numerical, and can distinguish main effects (first order) from interaction effects (higher order), see also ANOVA. - They are easy to measure, to interpret and to explain. - They allow for a sensitivity analysis whereby uncertain input factors are treated in groups instead of being treated individually; To compute a variance-based sensitivity measure for a given input factor X i, tart from the fractional contribution to the model output variance, i.e. the variance of Y, where Y is either a country s rank Rank (CIj), or the overall shift in countries ranking (Rs), due to the uncertainty in X V i =V x (E x (Y/X i )) (5) * Fix factor X i e.g. to a specific value in its range X i and calculate the mean of the output Y averaging over all factors excluding factor X i : X i :E x-i (Y/X i = * X i ) (6) * In relation (6) the dependence on X i has been dropped. V i is a number between zero (when X does not make a contribution to Y, at the first order), and V(Y) the unconditional variance of Y when all factors other than X are non-influential at any order. Thus, V xi (E x-i (Y/X i )) + E xi (V x-i Y/x i ))=V(Y) (7) where the first term of the relation (7) is called the main effect, and the second, the residual. An important factor should have a small residual, e.g. a small value of E xi (V x-i (Y/xi)). A first-order sensitivity index is obtained through normalising the first- order term by the unconditional variance: S Vxi i= ( Ex i( Y / Xi) ) V ( Y ) Vi = (8) V (Y ) It is possible to calculate conditional variances corresponding to more than one factor. This means that for two factors X i and X j, the conditional variance is: V xi, xj (E x-ij (Y/x i,x j )), and the variance contribution of the secondorder term becomes: 34 Romanian Statistical Review - Supplement nr. 2 / 204

7 Vij=Vxixj(Ex-ij(Y/xi,xj))-Vxi(Ex-i(Y/xi))-Vxj(Ex-j(Ex-j(Y/xj)) (9) where V ij 0 only if Vxixj (E x-i (Y/X i,x j )) is larger than the sum of the first- order term relative to factors X i and X j. When all k factors are independent from one another, the sensitivity indices can be computed using the following decomposition formula for the total output variance (V(y)). V(Y)= Wi Vij Vijl Vi2... K i i ji i ji l j i ji i j li li K K Revista Română de Statistică - Supliment nr. 2 / (0) Terms above the first order in equation (0) are known as interactions. A model without interactions among its input factors is said to be additive. In this case, K i= V i = K V ( Y ) şi S i = () i= and the first-order conditional variances of equation (5) are all needed to decompose the total output variance. For a non- additive model, higher order sensitivity indices, responsible for interaction effects among sets of input factors, have to be determined, However, higher order sensitivity indices are usually not estimated, since in a model with k factors, the total number of indices (including Sis * ) to be estimated would be as high as 2 k -. Instead a more compact sensitivity measure can be used, which is able to concentrate on a single term for all the interactions, involving a given factor x. For example, for a model of k=3 independent factors, there would be three total sensitivity indices: V S T = Y V X 2 X 3 E xi Y / X 2 X 3 S S 2 S 3 S 23 (2) V ( Y ) Analogously: S T2 =S 2 +S 2 +S 23 +S 23 (3) S T3 =S 3 +S 3 +S 23 +S 23 (4) The conditional variance V X2X3 (E X (Y/ X2X3 ) written in general term is Vxi(Exi(Y/xi)). This is the total contribution to the variance of Y due to non-x i, i.e. to the k- remaining factors, so that K V(Y)-Vx-i(E xi (Y/x-i)) generally includes all terms., STi i=

8 IndiceleThe total effect sensitivity index can also be written as: V ( Y ) VX i S T = EXi Y / X i EX i VXiY X i / V Y V Y (5) For a given factor X i a significant difference between S ij and S i signals important interactions for that factor in Y. Highlighting interactions among input factors helps to improve our understanding of the model structure. Estimators for both S ij and S i are provided by a variety of methods applied to quasi-random sampling of the input factors The pair (S i, S Ti ) gives a good numeric description of sensitivity. (Si, STi) in the case of non- independent input factors, could also be interpreted as settings for sensitivity analysis. The two settings linked to S i and S Ti are: - Factor Prioritisation Setting with the highest S i ; - Factor Fixing Setting - in case of factors for which S Ti is zero. Bibliography. Andrei, T. - Statistică şi Econometrie, Editura Economică, Bucureşti, Cox, D.R., Donnelley, A.C. - Principies of Applied Statistics, Cambridge University Press, Dobre, I.; Mustaţă, F. - Simularea proceselor economice, Editura INFOREC, Bucureşti, Diday, E. et. col. - Elements d annalyse de donnes, Dunod, Paris, Droesbeke, J.J.; Fichet, B.; Tassi, Ph. - Models pour l annalyse des donnes multidimensionelles, Economica, Paris, Granger, C.; Tarasvirta, T. - Modeling Nonliniear Economic Time Series, Oxford University Press, Oxford, Iosifescu, M. - Lanţuri Markov finite şi aplicaţii, Edituta Tehnică, Bucureşti, Voineagu, V.; Furtună, F. ş.a. - Analiză factorială a fenomenelor social-economice în profil territorial, Editura Aramis, Bucureşti, Voineagu, V.; Tiţan, E. - Analiză factorială, Editura Andrei Şaguna, Constanţa, *** - Manual privind construirea indicatorilor compuşi, OECD, JRC European Comission, Paris, 200. *** - European Conference on Quality in Official Statistics, 3-6 May 200, Helsinki 2. *** - Statistics in transition, vol. şi 2, , Poland 36 Romanian Statistical Review - Supplement nr. 2 / 204

Lecturer Marian SFETCU PhD. ARTIFEX University of Bucharest Andreea-Ioana MARINESCU, PhD. Student Bucharest University of Economic Studies

USING THE LINEAR REGRESSION MODEL IN ORDER TO ANALYSE THE CORRELATION BETWEEN THE GROSS DOMESTIC PRODUCT AND THE HOUSEHOLD EFFECTIVE INDIVIDUAL FINAL CONSUMPTION Lecturer Marian SFETCU PhD. ARTIFEX University