Model verification / validation A distributions-oriented approach

Transcription

1 Model verification / validation A distributions-oriented approach Dr. Christian Ohlwein Hans-Ertel-Centre for Weather Research Meteorological Institute, University of Bonn, Germany Ringvorlesung: Quantitative Methods in the Social Sciences Universität Tübingen, Germany 3 July 2014

2 Background Model output f Statistical model (LM, GLM, BHM, Physical model Expert system Observations o Independent observations Cross validation (Block bootstrap * Quality management Verification Validation Confirmation Evaluation Common measures of model quality* RMSE, linear correlation, rank correlation, R 2,! ISO 9000 (? How to deal with more complex or probabilistic models? 2

3 Joint, marginal, and conditional distributions E(o f i Random Variables Model f i e.g., categories i =1,,I Observation o j e.g., categories j=1,,j p(o j f i Probabilities Joint probability p(f i,o j = P { F i O j } p(o j E(o Marginal probability p(o j = p(f i,o j i=1,,i p(f i,o j Conditional probability p(o j f i = p(f i,o j p(f i f 8 Expectation Expectation and conditional expectation E(o = o j p(o j, E(o f i = o j p(o j f i j=1,,j j=1,,j f 2 f 1 o 2 o 1 o 8 3

4 The verification problem è Most general objective Search for descriptions of p(f i,o j Categorical: contingency table Continuous: joint probability distributions f 1 f 8 f 2 o 2 o 1 o 8 è Standard approaches Dimension reduction Strong distributional assumptions Full description: 64 values in this example How to deal with more complex or probabilistic models? 4

5 Distributions-oriented approach Murphy-Winkler framework p(f i,o j = p(o i f j p(f j Calibration-refinement factorization Calibration Refinement (marginal distribution of forecasts p(f i,o j = p(f i o j p(o j Likelihood-base rate factorization Likelihood Base rate (marginal distribution of observations, sample climatology 5

6 Distributions-oriented approach p(f i,o j = p(o j f i p(f i Calibration p(f i,o j = p(f i o j p(o j Likelihood f 1 f 2 f 8 o 2 o 1 o 8 6

7 Statistical inference Example of binary events Model f i i =1,2 Observation o j j=1,2 finite sample (f i,o j k=1,,n Observation o 1 o 2 a: hits ˆp(f 1,o 1 = a n Model f 1 a b a+b f 2 c d c+d b: false alarms c: misses ˆp(f 1,o 2 = b n ˆp(f 2,o 1 = c n d: correct rejections ˆp(f 2,o 2 = d n a+c b+d n 2 2 contingency table for a sample of binary events p: ^ sample estimates 7

8 Statistical inference Relates to Murphy-Winkler framework Conditional distributions Marginal distributions Observation o 1 o 2 Calibration-refinement ˆp(o 1 f 1 = ˆp(f 1,o 1 ˆp(f 1 ˆp(o 1 f 2 = ˆp(o 2 f 1 =! = a a + b! Model f 1 a b a+b f 2 c d c+d a+c b+d n 2 2 contingency table for a sample of binary events Likelihood-base rate ˆp(f 1 o 1 = ˆp(f 1,o 1 ˆp(o 1 ˆp(f 1 o 2 = ˆp(f 2 o 1 =! = a a + c 8

9 Attributes of forecast performance How to describe forecast performance? Verification problem fully described by Murphy-Winkler framework Need to know the full I J-dimensional contingency table Data reduction required Attributes General characteristics of the verification problem Mostly in terms of scalar attributes à relates to scores and skill scores, accuracy, bias, reliability, resolution, discrimination, sharpness, All of them describe some aspect of forecast performance 9

10 Accuracy Average correspondence between individual forecasts and the events they predict Meant to summarize the overall quality Note: by far not the full information Common measures of accuracy Mean squared error (MSE Proportion correct (PC Critical success index (CSI Correlation coefficient? Following attributes: often components or aspects of accuracy 10

11 Accuracy Average correspondence between individual forecasts and the events they predict Meant to summarize the overall quality Note: by far not the full information Common measures of accuracy (Log-odds ratio Observation o 1 o 2 θ = ad bc = ˆp(f 1 o 1 ˆp(f 2 o 1 ˆp(f 1 o 2 ˆp(f 2 o 2 Model f 1 a b a+b f 2 c d c+d log(θ = a+c b+d n 11

12 Bias Correspondence between average forecast and average observation Aims at marginal distributions Does not care about individual pairs? Also named conditional or systematic bias Common measures of bias Bias (continuous Bias (frequency 12

13 Reliability Relationship of the forecast to the average observation for specific values of the forecast Conditional distributions of the observations given a specific forecast Aims at p(o j f i of the calibration-refinement factorization? Also named calibration or conditional bias 13

14 Reliability Relationship of the forecast to the average observation for specific values of the forecast Conditional distributions of the observations given a specific forecast Aims at p(o j f i of the calibration-refinement factorization Also named calibration or conditional bias Common measures of reliability False alarm ratio (FAR FAR = b a + b = ˆp(o 2 f 1 Model Observation o 1 o 2 f 1 a b a+b f 2 c d c+d a+c b+d n 14

15 Resolution Differences between the conditional averages of the observations for different forecasts? Similar to reliability Aims at p(o j f i of the calibration-refinement factorization, but Pertains to the differences between the conditional average (c.f. reliability compares the conditional averages of the observations with the forecast values themselves Common measures of resolution False alarm ratio (FAR 15

16 Discrimination Differences between the conditional averages of the forecasts for different observations Converse of resolution Aims at p(f i o j in the likelihood-base rate factorization Ability of the forecasting system to produce different forecasts in case of different observations? 16

17 Discrimination Differences between the conditional averages of the forecasts for different observations Converse of resolution Aims at p(f i o j in the likelihood-base rate factorization Ability of the forecasting system to produce different forecasts in case of different observations Common measures of discrimination Hit rate (H H = a a + c = ˆp(f 1 o 1 False alarm rate (F F = b b + d = ˆp(f 1 o 2 Model Observation o 1 o 2 f 1 a b a+b f 2 c d c+d a+c b+d n 17

18 Sharpness Characterizes the unconditional distribution of the model Attribute of the model alone Low sharpness: models that rarely deviate much (e.g., from climate Sharp model output: tendency to stick their neck out Relates to p(f i in the calibration-refinement factorization Practical applications Relevant for ensemble simulations Aim to produce forecasts that are sharp and reliable? 18

19 Review: attributes and scores Scores Murphy- Winkler Describe the Attributes Summarize the Which score? Binary events: three scores fully parameterize the contingency table (e.g., B,H,F Continuous or multi-categorical events: always a (severe loss of information No single answer 19

20 Verification of probabilistic models Deterministic model Probabilistic model, bias, discrimination, sharpness, 20

21 Verification of probabilistic models Murphy-Winkler framework, bias, discrimination, sharpness,, bias, reliability, resolution, 21

22 Summary è Distributions-oriented point of view Aims at the joint probability of model and observations Decomposition by factorizations Calibration-refinement Likelihood-base rate è Attributes of model performance Accuracy, bias, reliability, resolution, discrimination, sharpness, Ways to reduce ore summarize the verification problem Tell you how to improve your forecast model (!! è Practical applications Verification of complex statistical, physical, models Verification of probabilistic models 22

23 Model verification / validation A distributions-oriented approach Christian Ohlwein Hans-Ertel-Centre for Weather Research Meteorological Institute, University of Bonn, Germany Ringvorlesung: Quantitative Methods in the Social Sciences Universität Tübingen, Germany 3 July 2014