Validation of Forecasts (Forecast Verification) Overview. Ian Jolliffe

Transcription

1 Validation of Forecasts (Forecast Verification) Overview Ian Jolliffe 1

2 Outline 1. Introduction and history (4) 2. Types of forecast (2) 3. Properties of forecasts (3) verification measures (2) 4. Terminology (1) 5. Types of forecast more detail Binary forecasts (8) Probability forecasts (6) Forecasts of extremes (1) Forecasts of quantiles (1) Spatial forecasts (2) 6. Comparing forecasts of different types (1) 7. Omissions (1) 2

3 Validation vs. verification Treat as synonymous, though some disciplines (e.g software engineering) give different meanings. Sometimes distinguished as: Validation = detailed assessment of the forecast of one specific event Verification = assurance of the overall quality of all the forecasts 3

4 Why verify/evaluate/assess/ validate forecasts? Decisions are based on past data but also on forecasts of data not yet observed A look back at the accuracy of forecasts is necessary to determine whether current forecasting methods should be continued, abandoned or modified 4

5 Two (very different) relevant books I T Jolliffe and D B Stephenson (eds.) (2003; 2 nd edition 2011) Forecast verification A practitioner s guide in atmospheric science. Wiley. M S Pepe (2003) The statistical evaluation of medical tests for classification and prediction. Oxford. Quite different approaches in the two literatures Weather/climate. Lots of measures used. Literature on desirable properties, but often ignored. Inference (tests, confidence intervals, power) seldom considered Medical. Far fewer measures. Little discussion of desirable properties. More inference: confidence intervals, complex models for ROC curves Other literatures too e.g. economics 5

6 History of verification in atmospheric science The first burst of activity in forecast verification followed Finley s (1884) assessment of tornado forecasts Finley used the phrase verification of predictions and verification has stuck, though rarely used in other disciplines Not a great amount of activity in the first half of the 20 th century but an increasing amount since Work has concentrated on weather forecasts. Relevant to climate forecasts, but there is the problem of where the verifying observations come from. 6

7 Types of forecast Binary (frost/no frost in an orchard) More than two categories (above/near/below normal seasonal precipitation) Discrete (number of land-falling hurricanes in a season) Continuous (maximum daily temperature in a city) Multivariate (spatial sea level pressure fields) 7

8 Types of forecast II A more fundamental division is deterministic vs probabilistic Information about a quantity to be forecast will (almost) always lead to a probability distribution for that quantity but for various reasons a deterministic forecast may be required Whereas an individual deterministic forecast can often be judged right or wrong, this is impossible for probabilistic forecasts Different types of forecast require different verification techniques 8

9 Distributions vs. measures A distributions-oriented approach to forecast verification involves looking at the joint probability distribution of forecasts and verification observations Often this joint distribution is replaced by one or more scalar measures of the quality of a set of forecasts, giving a measures-oriented approach. These measures attempt to quantify desirable properties of a set of forecasts. One measure is rarely sufficient to satisfactorily describe the joint distribution. 9

10 Some desirable properties (attributes) of forecasts Accuracy. How close are individual forecasts to corresponding observations? Bias. Do forecasts tend to over- or underestimate observations on average? Reliability. Conditional unbiasedness. Given a forecast value, does the expected (average) value of corresponding observations equal that forecast value. Resolution. The sensitivity of the expected (average) values of the observations to different forecast values (or more generally the sensitivity of the conditional distribution as a whole). 10

11 More attributes Discrimination. The sensitivity of the conditional distribution of forecasts, given observations, to the value of the observation. Sharpness. Measures spread of marginal distribution of forecasts. The attributes described come from the distributions-oriented approach to verification They are relevant to several different types of forecast: continuous, categorical, probabilistic The reliability diagram (see later) illustrates some of them. 11

12 Desirable properties of measures Often a variety of measures is available for a set of forecasts. To choose between them, desirable properties of the measures themselves have been defined. This is called metaverification. Several of these properties address the problem of hedging, which is when a forecaster issues a forecast other than his/her true belief A measure should ideally be such that it can t be improved by hedging However, if missing an extreme event is far more costly than a false alarm, hedging by over-forecasting is actually desirable. This is because the concept of value comes into play. 12

13 Choice of measures Could be done by trying to explicitly measure desirable attributes of forecasts achieve desirable properties of measures Perhaps more often they are chosen for more intuitive reasons and their properties only looked at later 13

14 Terminology Differing terminology and notation can make reading the verification literature very difficult at times - beware This is partly, but not entirely, due to different measures being discovered in different disciplines, but also rediscovered within the same discipline Examples False alarm rate vs. false alarm ratio Hit rate is used for two different things Peirce skill score = Kuiper s performance index = Hansen and Kuiper s score = true skill score = Youden index = Gerrity s score 14

15 Binary deterministic forecasts Such forecasts might be Whether temperature will fall below a threshold, damaging crops or forming ice on roads Whether maximum river flow will exceed a threshold, causing floods Whether a tornado will occur in a specified area The classic Finley Tornado example (next 2 slides) illustrates that assessing such forecasts is more subtle than it looks There are many possible verification measures most have some poor properties 15

16 Forecasting tornados Tornado Observed Tornado not observed Total Tornado Forecast Tornado not forecast Total

17 Tornado forecasts One possible measure of how good is the forecasting system is Proportion (or Percent) Correct = 2708/2803 = 96.6% But we could achieve a better value, 2752/2803 = 98.2% by always forecasting No Tornado It s easy to forecast No Tornado, and get it right but more difficult to forecast when tornadoes will occur, and the forecasting system does seem have some skill PC is a measure that can be hedged Tornado Forecast Tornado not forecast Tornado Observed Tornado not observed Total Total

18 Tornado forecasts - comments It will be recognised that we have a (2x2) contingency table, and we are concerned with measuring the relationship between the variables that define its margins Many literatures have substantial contributions (statistics, ecology, medicine, ) to this topic, which is remarkably subtle, both practically and philosophically 18

19 Forecast/observed contingency Event Forecast Event not forecast Event Observed a (Hits) c (Misses) table Event not observed b (False alarms) d (Correct rejections) Total a + c b + d n Total a + b c + d 19

20 Some verification measures for (2 x 2) tables (far from exhaustive see Hogan and Mason, 2011) a/(a+c) Hit rate (H) = true positive fraction = sensitivity = probability of detection b/(b+d) False alarm rate (F) = 1- specificity = probability of false detection b/(a+b) False alarm ratio = 1 positive predictive value d/(c+d) Negative predictive value (a+d)/n Proportion correct (PC) (a+b)/(a+c) Frequency bias Event Forecast Event not forecast Event Observed Event not observed Total a b a + b c d c + d Total a + c b + d n 20

21 Skill scores A skill score is a verification measure adjusted to show improvement over some unskilful baseline, typically a forecast of climatology or long-run average behaviour a random forecast a forecast of persistence Often adjustment gives zero value for the baseline and unity for a perfect forecast 21

22 Some skill scores (PC E)/(1- E), where E is the expected value of PC assuming no skill the Heidke (1926) skill score; also Doolittle (1885), and a special case of Cohen s (1960) kappa H-F - Peirce s (1884) skill score (ad bc)/(ad +bc) Yule s Q (1900). A skill score version of the odds ratio ad/bc. 22

23 Probability forecasts Could be produced by various means: Ensembles Logistic regression Could be: Probabilities of falling in one or more categories Estimate of whole probability density function 23

24 Probability forecasts II Measures: Binary: Brier score (average squared difference between probability and observed 0-1) Brier skill score Several ordered categories Rank(ed) probability (skill) score extension of Brier score, based on cumulative probabilities Continuous rank(ed) probability (skill) score integral of squared differences between cumulative probability distributions 24

25 Probability forecasts III Ignorance score At its simplest = log(p(a)) where p(a) is the forecast probability of the event A which occurs Can be defined for binary forecasts, ordered categories, continuous variables e.g for binary forecasts it is the summation over j of -log(p j (A)) for forecasts in which the event occurred and log(1-p j (A)) for forecasts in which the event did not occur 25

26 Probability forecasts IV The ignorance score (Tödter and Ahrens, 2012) and the Brier score and its extensions (Ferro and Fricker, 2012) can be decomposed into contributions due to Reliability - measures conditional bias; should be small; can be removed by calibration Resolution should be large Uncertainty uncontrollable random variability 26

27 A reliability diagram For a probability forecast of an event based on 850hPa temperature. Lots of grid points, so lots of forecasts (16380). Plots observed proportion of event occurrence for each forecast probability vs. forecast probability (solid line) forecasts are over-confident Forecast probability takes only 17 possible values (0, 1/16, 2/16, 15/16, 1) because forecast is based on proportion of an ensemble of 16 forecasts that predict the event Because of the nature of the forecast event, 0 or 1 are forecast most of the time (see inset sharpness diagram) 27

28 # fcsts Obs. frequency Obs. frequency Observed frequency Interpretation of Reliability Diagram Reliability: Proximity to diagonal Resolution: Variation about horizontal (climatology) line No skill line: Where reliability and resolution are equal Brier skill score goes to 0 (from Laurie Wilson) 1 skill Forecast probability clim Forecast probability climatology Reliability Resolution 0 0 P fcst 1 Forecast probability

29 Extremes For binary events (e.g. precipitation, temperature, windspeed above some extreme threshold) can use measures discussed earlier, but many go to trivial non-informative limits as event becomes more extreme For recent work on improved measures, specifically defined for rare events, see Ferro and Stephenson (2011) 29

30 Quantiles Given a forecast f of a quantile τ and a corresponding observation x, calculate τ (x-f) if x>f and (1- τ)(f-x) if f>x. Then sum over all forecasts to give the quantile verifications score (QVS). A skill score version can also be constructed (Friederichs and Hense, 2007). A more informal approach is to compare the observed proportion of observations above/below the forecast α- quantile with α.this is equivalent to what is done in the reliability diagram except that here probability α is fixed and the corresponding value of the variable (quantile) is forecast, whereas in the reliability diagram a value of the variable is given and the corresponding probability is forecast 30

31 Spatial forecasts There has been a lot of recent work on verification for spatial data (Brown et al., 2011) Much has concentrated on seeing how forecast spatial objects, such as precipitation areas, correspond to observed objects 31

32 Verification of precipitation forecast over USA from Beth Ebert Forecast is on the left; verification radar data on the right 1. What is the location error of the forecast? 2. How do the forecast and observed rain areas compare? Average values? Maximum values? 3. How do the displacement, volume, and pattern errors contribute to the total error? 32 Mini-course JOCLAD April 6th 2011

33 Comparison of forecasts of different types A possible approach here is the generalized discrimination score (Mason & Weigel, 2009) Look at every pair of (forecast, observation) pairs, say (f i,o i ), (f j,o j ). Calculate the proportion of all such pairs for which {f i <f j AND o i <o j } OR {f i >f j AND o i >o j } i.e. for which forecasts are ordered in the same way as the observations. 33

34 Concluding remarks We can say more about these topics in the discussion that follows, though I may have to consult my verification bible but I forgot to bring it There are other topics not covered, for example: Value Software ROC curves 34

35 Additional References Brown BG, Gilleland E & Ebert EE (2011) Forecasts of spatial fields, In: Forecast Verification: A Practitioner s Guide 2 nd edition (eds: IT Jolliffe & DB Stephenson). Chichester: Wiley, pp Cohen J (1960) A coefficient of agreement for nominal scales. Educ. Psychol. Meas., 20, Doolittle MH (1885) The verification of predictions. Bull. Phil. Soc. Washington, 7, Ferro CAT & Fricker TE (2012) A bias-corrected decomposition of the Brier score. Q. J. Roy. Met. Soc., 138, In Press. Ferro CAT & Stephenson DB (2011) Deterministic forecasts of extreme events and warnings, In: Forecast Verification: A Practitioner s Guide 2 nd edition (eds: IT Jolliffe & DB Stephenson). Chichester: Wiley, pp Finley JP (1884) Tornado predictions. Amer. Met. J., 1, Friederichs P & Hense A (2007) Statistical downscaling of extreme precipitation events using censored quantile regression. Mon. Wea. Rev., 135,

36 References II Heidke P (1926) Berechnung der erfolges und der gute der windstarkevorhersagenim sturmwarnungdienst. Geogr. Ann., 8, Hogan RJ & Mason IB (2011) Deterministic forecasts of binary events, In: Forecast Verification: A Practitioner s Guide 2 nd edition (eds: IT Jolliffe & DB Stephenson). Chichester: Wiley, pp Mason SJ & Weigel AP (2009) A generic forecast verification framework for administrative purposes. Mon. Wea. Rev., 137, Peirce CS (1884) The numerical measure of the success of predictions. Science, 4, Tödter J & Ahrens B (2012) Generalization of the ignorance score: continuous ranked version and its decomposition. Mon. Wea. Rev., 140, Yule GU (1900) On the association of attributes in statistics. Philos. Trans. Roy. Soc. London, 194A,