Validation Metrics. Kathryn Maupin. Laura Swiler. June 28, 2017

Size: px

Start display at page:

Download "Validation Metrics. Kathryn Maupin. Laura Swiler. June 28, 2017"

Hubert Hopkins
5 years ago
Views:

1 Validation Metrics Kathryn Maupin Laura Swiler June 28, 2017 Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc. for the U.S. Department of Energy s National Nuclear Security Administration under contract DE-NA SAND NO XXX

2 Overview Model Validation Data Classification Validation Metrics Examples Conclusions June 28,

3 Model Validation The comparison of experimental observations with model output Observed values contain uncertainties Experimental measurement error Limited/incomplete data Model form error Parameter uncertainty Approximation/discretization error Validation Metric: quantifies the difference between physical and simulation observations Observations may be considered with or without uncertainties June 28,

4 Data Types Type 1: Experimental and model values are treated without uncertainty Type 2: Experimental values are treated as uncertain Nominal value, given by experts Standard deviation, calculated using multiple experiments Type 3: Experimental and model values are treated as uncertain Uncertainty analysis June 28,

5 Data Types Type 1 (no uncertainties) June 28,

6 Data Types Type 1 (no uncertainties) Type 2 (experimental uncertainty) June 28,

7 Data Types Type 3 (experimental and model uncertainty) June 28,

8 Classification of Validation Metrics Metric Type 1 Type 2 Type 3 Root Mean Square Minkowski Distance Simple Cross Correlation Normalized Cross Correlation Normalized Zero-Mean Sum of Squared Distances Moravec Correlation Index of Agreement Sprague-Geers Metric Normalized Euclidean Metric Mahalanobis Distance Hellinger Metric Kolmogorov-Smirnoff Test Kullback-Leibler Divergence Symmetrized Divergence Jensen-Shannon Divergence Total Variation Distance June 28,

9 Example Validation Metrics Type 1 Data Minkowski Distance (l p Distance) ( ) p d = P i D i p i Type 2 Data Mahalanobis Distance d = (P D) T Σ 1 D (P D) Type 3 Data Kullback-Leibler Divergence D KL (N D N P ) = 1 [ ( )] tr(σ 1 P Σ D ) + (P D) T Σ 1 det ΣP P (P D) k + ln 2 det Σ D Kolmogorov-Smirnov Test D KS = sup F P (x) F D (x) June 28,

10 Example Validation Metrics Type 1 Data Minkowski Distance (l p Distance) ( ) p d = P i D i p i Type 2 Data Mahalanobis Distance d = (P D) T Σ 1 D (P D) Type 3 Data Kullback-Leibler Divergence D KL (N D N P ) = 1 [ ( )] tr(σ 1 P Σ D ) + (P D) T Σ 1 det ΣP P (P D) k + ln 2 det Σ D Kolmogorov-Smirnov Test D KS = sup F P (x) F D (x) June 28,

11 Example Validation Metrics Type 1 Data Minkowski Distance (l p Distance) ( ) p d = P i D i p i Type 2 Data Mahalanobis Distance d = (P D) T Σ 1 D (P D) Type 3 Data Kullback-Leibler Divergence D KL (N D N P ) = 1 [ ( )] tr(σ 1 P Σ D ) + (P D) T Σ 1 det ΣP P (P D) k + ln 2 det Σ D Kolmogorov-Smirnov Test D KS = sup F P (x) F D (x) June 28,

12 Example 1 Experiment f(x) = 1.1 log(10x) + ε Model g(x) = log(10x) with ε = measurement error x i = 1, 2, Type 1 Data Metric Value Min Max Root Mean Square Average Relative Minkowski Distance p = p = June 28,

13 Example 1 Type 2 Data June 28,

14 Example 1 Type 2 Data Metric Value (5%) Value (10%) Min Max Average Mahalanobis Distance June 28,

15 Example 1 Type 3 Data June 28,

16 Example 1 Type 3 Data Metric Value (5%) Value (10%) Min Max Kullback-Leibler Divergence Total Average per point June 28,

17 Example 1 Type 3 Data Metric Value (5%) Value (10%) Min Max Kolmogorov-Smirnov Test Maximum Average June 28,

Example 2 Experiment f(x) = sin 2 (x) + 5 + ε Model g(x) = sin 2 (x) + θ with ε = measurement error Type 1 Data Metric Value

18 Example 2 Experiment f(x) = sin 2 (x) ε Model g(x) = sin 2 (x) + θ with ε = measurement error Type 1 Data Metric Value Min Max Root Mean Square Average Relative Minkowski Distance p = p = June 28,

19 Example 2 Type 2 Data Metric Value (5%) Value (10%) Min Max Average Mahalanobis Distance June 28,

20 Example 1 Type 3 Data Metric Value (5%) Value (10%) Min Max Kullback-Leibler Divergence Total Average per point June 28,

21 Example 1 Type 3 Data Metric Value (5%) Value (10%) Min Max Kolmogorov-Smirnov Test Maximum Average June 28,

22 Conclusions Computational models require validation before they can be reliably used in prediction scenarios Metrics exist for deterministic data and probabilistic (uncertain) data Choice of metric is application and quantity of interest dependent Future work: develop a guide to determine choice of validation metric validation tolerance June 28,

New Approaches in Process Monitoring for Fuel Cycle Facilities

New Approaches in Process Monitoring for Fuel Cycle Facilities P R E S E N T E D B Y Ben Cipiti & Nathan Shoman SAND2018-4411C Sandia National Laboratories is a multimission laboratory managed and operated