Sensitivity Analysis When Model Outputs Are Functions

Transcription

1 LAUR Sensitivity Analysis When Model Outputs Are Functions Michael D. McKay and Katherine Campbell formerly of the Statistical Sciences Group Los Alamos National Laboratory Presented at the Fourth International Conference on Sensitivity Analysis of Model Output (SAMO), Santa Fe, NM, USA, March 8-11, 2004, and from a presentation at the International Conference on Probabilistic Safety Assessment and Management (PSAM 6), San Juan, Puerto Rico, June 23-28, 2002.

2 Main points 1. Functions--think of y(t)--are characterized by shapes and features which: are defined by relationships among neighboring points, e.g., values around a maximum. may be identified by inspection may be characterized (and identified) via a decomposition, e.g., into linear, quadratic, etc. terms. 2. Functional sensitivity analysis is not a blind procedure.

3 Outline 1. Point-by-point analysis with (1) environmental pathways model and (2) particle beam accelerator model. 2. Analysis of features from pathways model. 3. Introduce basis function decomposition. 4. Use data-adaptive principal components basis for analysis of (1) pathways model and (2) particle beam transport model. 5. Use fixed basis of (Legendre) polynomials for analysis of particle beam transport model.

4 Model of environmental pathways 0. COMPARTMENTS 1. Vegetation surface 2. Vegetation interior 3. Terrestrial invertebrates 4. Small herbivores 5. Large herbivores 6. Insectivores 7. Predators 8. Litter 84 MODEL INPUTS Input Lower Upper Nominal X X

5 Output of pathways model Compartment 3 when inputs range over plausible values Concentration Time in days

6 Importance indicator R 2 as a function of time X24 X63 R 2 Time in days

7 Output of proton beam transport model Particle density Position in meters

8 R 2 as a function of position R 2 Position in meters

9 Output of pathways model Compartment 3 when inputs range over plausible values Concentration Time in days

10 Scalar outputs: Candidates for analysis of pathways model output Equilibrium concentration (how high): Y 1 = C max Time to Equilibrium (how fast): Y 2 = t max = 0.9 C max Time dependent outputs: C(t) Normalized: C(t) / C max where C max = C(t max ) Standard time: C * (u) = C(u x t max ) / C max, 0 u 1

11 R 2 for equilibrium concentration R 2 Parameters ordered by R 2

12 R 2 for time to equilibrium R 2 Parameters ordered by R 2

13 Standardized output from pathways model Normalized concentration (color-coded by X63) Standardized time

14 Functional sensitivity analysis Given N model runs with functional output {y i (t), t = 1,, T; i = 1,, N}: (5) Expand each y i (t) in a set of basis functions: y i (t) ~ Y N T - K + ikϕk k= 1 y t h t ( ) ( ) Y for 1 i N or ~ H N K Φ K T (K T). (2) Perform sensitivity and importance analysis on the (new data) {h i1 },, {h ik },, {h ik }.

15 Options for basis functions Φ Fixed basis functions (Legendre) polynomials, trigonometric functions, Haar functions, wavelets,. Data-adaptive basis functions Principal components, partial least squares, canonical correlation,... I have found this website useful -- Eric Weisstein's World of Mathematics mathworld.wolfram.com

16 Data-adaptive basis: principal components Principal components (PCs) Rotation in T-dimensional space (i.e., an orthonormal transformation) Diagonalizes Var(Y) Maximizes variance in first n components; most of the significant effects captured by K<<T components

17 Expansion of C(t) for one run in the first 4 PCs C(t) is + and fit well by 4 terms Mean * first P.C. Mean value from all runs

18 Principal component functions

19 PCs as perturbations of mean function PC 1: 99.3% Mean + m*p.c. 2 PC 2: 0.73% Mean Mean m*p.c. 2 PC 3: 0.02% PC 4: %

20 R 2 for coefficients of PCs PC 1: 99.3% (equilibrium value) PC 2: 0.73% (~time to equilibrium) PC 3: 0.02% PC 4: %

21 Comparison of R 2 First principal component Feature: Equilibrium Concentration

22 Comparison of R 2 (cont) Second principal component Feature: Time to equilibrium

23 Recall the main points 1. Functions--think of y(t)--are characterized by shapes and features which: are defined by relationships among neighboring points, e.g., values around a maximum. may be identified by inspection may be characterized (and identified) via a decomposition, e.g., into linear, quadratic, etc. terms. 2. Functional sensitivity analysis is not a blind procedure.

24 Approximation of a beam profile Mean 0.012*PC1 Mean over all runs

25 Principal component functions PC 1: 69.4% PC 2: 27.2% PC 3: 2.1% PC 4: 1.3%

26 PCs as perturbations of mean function Mean m*p.c. 2 PC 1: 69.4% Mean + m*p.c. 2 PC 2: 27.2% Mean PC 3: 2.1% PC 4: 1.3%

27 R 2 for coefficients of PCs PC 1: 69.4% (beam width) PC 2: 27.2% (beam center) PC 3: 2.1% (tail weight) PC 4: 1.3% (skewness)

28 PC analysis for beam profiles 68.66% 95.59% 97.62% 98.9% 99.22% 99.39% Principal components perturbations of mean function dependence on pyfac dependence on yshift dependence on dy1 dependence on ptfac dependence on tfac

29 Fixed basis: Legendre polynomials Orthogonal polynomials in t on [-1,1] φ 0 (t) = 1 (1, 1, 1, 1, 1) for T = 5 φ 1 (t) = t (-1, -.5, 0,.5, 1) φ 2 (t) = ½ (3t 2 1) ½ (2, -.25, -1, -.25, 2) sometimes polynomials in sin(πt/2) may be ill suited for some problems

30 First four Legendre polynomials

31 Next four Legendre polynomials

32 Legendre polynomials in sin(x) Legendre analysis for beam profiles 5.24% 16.15% 33.08% 74.78% 79.37% 90.91% perturbations of mean function dependence on pyfac dependence on yshift dependence on dy1 dependence on ptfac dependence on tfac

33 Data-adaptive basis: partial least squares (PLS) Takes input variables in addition to output variables into account in choosing the basis functions. Can provide valuable insight. Resulting basis vectors (for computer model output from designed computer experiments) tend to be similar to principal components Does not form an orthonormal basis or provide a linear variance decomposition in coefficients {h ik }.

34 PLS analysis for beam profiles PLS components 68.57% 95.54% 96.44% 98.31% 98.78% 99.08% perturbations of mean function Dependence on pyfac Dependence on yshift Dependence on dy1 Dependence on ptfac Dependence on tfac

35 Comparison of basis functions Adaptive Usually concentrate information in first few terms. Respond to shape changes across problems. Variability affects both coefficients and shape of basis functions Fixed May spread description of simple features across many terms. Consistent fundamental shapes across problems. Variability only affects coefficients.

36 Two important points not covered Curve registration -- a monotonic transformation of time (translation/scale). One indication that it is needed is that y ( t ), the sample mean, is not representative the sample of curves. y t Penalty methods can be used to enforce a degree of smoothness of basis functions, which can make comparisons across problems easier. (See Ramsay and Silverman reference, Functional Data Analysis.) ( )

37 Summary Point-by-point analyses may work well for functions of time representing evolving systems. Manual selection of appropriate features can be useful. Adaptive basis decompositions may reveal dependencies not easily extracted by other means.

38 References Campbell, K. (2001). Functional Sensitivity Analysis of Computer Model Output, Proceedings of the Seventh Army Conference on Statistics, Santa Fe, NM. Frank, I. E. and Friedman, J. H. (1993). A Statistical View of Some Chemometrics Regression Tools, Technometrics 35, McKay, M. D. (1997). Nonparametric Variance-based Methods of Assessing Uncertainty Importance, Reliability Engineering and System Safety, 57, Ramsay, J. O. and Silverman, B. W. (1997). Functional Data Analysis, Springer, New York. Saltelli, A., Chan, K., Scott, E. M. (Eds.), Sensitivity Analysis, John Wiley and Sons, Ltd. (2000).

39 What are Principal Components? 1. Geometrical interpretation in 2 dimensions: coordinate rotations that maximize sample variance. 2. Algebraic description: eigenvectors of (sample) the covariance matrix. 3. Empirical (sample dependent) orthonormal basis functions that best fit functional data.

40 5 points in original coordinates (X 1, X 2 ) X 2 P 3 P 4 5 SS = ( X X ) 2 2i 2 i= 1 2 P 1 P 2 P 5 Rotation coming X 1 5 SS = ( X X ) 1 1i 1 i= 1 2

41 5 points in rotated coordinates (X 1*, X 2* ) X 2 * SS > SS * 1 1 P 4 X 1 * P 3 P 2 P 1 P 5 SS < SS * 2 2

42 Geometrical principal component The first principal component (PC) is the direction for which the orthogonal projection of the data points has the most spread, as measured by the sum of squares about the mean (SS * ). The second principal component is the direction orthogonal to the first PC for which the data points have the second most spread. And so forth for the third, fourth, etc.

43 Algebra of principal components Let X be an N x K matrix whose rows are vectors of k-dimensional data and whose columns have zero mean. For N = 3 points and K = 3 components, p1 x11 x12 x13 P = p = X = x x x p 3 x31 x32 x 33. Let β be a K x 1 vector of coefficients that defines a linear combination for the K components of a data vector. For the first point, the linear combination is x β = β x + β x + β x

44 More algebra of principal components The (unit length) β that maximizes the sample variance of the N values Xβ, max ' X' Xβ/ N, β β' β=1 satisfies X' Xβ=λβ. That β is the dominant eigenvector of X X, which is the first principal component.

45 Basis functions and principal components Consider N functions y i (t), with mean 0, defined at T points t j. Write y i (t) in terms of K T basis functions φ k (t) as K y ( t ) = h ϕ ( t). i ik k k = 1 The orthonormal functions φ k (t) that minimize N T K ( ) yi t j hik ϕk ( t j ) i= 1 j= 1 k = 1 are K eigenvectors/principal components of the N x T covariance matrix of the y i (t j ). 2