Verification of ensemble and probability forecasts

Transcription

1 Verification of ensemble and probability forecasts Barbara Brown NCAR, USA Collaborators: Tara Jensen (NCAR), Eric Gilleland (NCAR), Ed Tollerud (NOAA/ESRL), Beth Ebert (CAWCR), Laurence Wilson (EC, Canada) Asia THORPEX Science Workshop 1 November 2012

2 Overview Verification measure and method overview what do they tell us? Some examples of new approaches Spatial methods

3 Approaches for ensemble evaluation Treat ensemble as deterministic forecast (individual members, mean forecast etc.) Use traditional approaches for evaluation of deterministic forecasts (contingency table, scores for continuous forecasts) Can learn about differences among members (e.g., if members are defined uniquely) Create probability forecasts Post-process ensemble to obtain probabilities Evaluation considers both the ensemble and the postprocessing Whole distribution Allows direct evaluation of the ensemble generation process

4 Measure Attribute evaluated Comments Probability forecasts Brier score Accuracy Based on squared error Resolution Reliability Resolution (resolving different categories) Calibration Compares forecast category climatologies to overall climatology Skill score Skill Skill involves comparison of forecasts Sharpness measure Sharpness Only considers distribution of forecasts ROC Discrimination Ignores calibration C/L Value Value Ignores calibration Ensemble distribution Rank histogram Calibration Can be misleading Spread-skill Calibration Difficult to achieve CRPS log p score ( Ignorance ) Accuracy Accuracy Squared difference between forecast and observed distributions Analogous to MAE in limit Local score, rewards for correct category; infinite if observed category has 0 density

5 Components of the Brier Score Reliability Measures how well the conditional relative frequency of events matches the forecast Resolution Measures how well the forecasts distinguish situations with different frequencies of occurrence Uncertainty Measures the variability in the observations (i.e., the difficulty of the forecast situations) 1 I n i = 1 1 I n i = 1 N ( f x ) i i i N ( x x) i i x(1 x) 2 2 Looking at Brier Score components is critical to understand forecast performance

6 Diagrams for evaluating probability and ensemble forecasts Probability forecasts Reliability (or attributes) diagram Sharpness Discrimination ROC Ensemble forecasts Rank histogram (Spread-skill) frequency observed observed non-events events forecast Workshop on Verification Methods, Beijing, China, October 2012

7 Ignorance score (for multicategory or ensemble forecasts) A local score k * () t 1 n ) IS = log ( p ) n i = 1 2 * tk, ( t) is the category that actually was observed at time t Based on information theory Only rewards forecasts with some probability in correct category Is receiving some attention as the score to use

8 Measure Attribute evaluated Comments Probability forecasts Brier score Accuracy Based on squared error Resolution Resolution (resolving different categories) Compares forecast category climatologies to overall climatology Reliability Calibration Skill involves comparison Verification Skill score is a high-dimensional Skill problem. Ensembles have of forecasts even higher dimensions than typical verification Only considers problems... distribution Sharpness measure Sharpness of forecasts ROC Discrimination Ignores calibration Each measure has its own benefits and issues. C/L Value Value Ignores calibration Ensemble distribution Rank histogram Calibration Can be misleading One score will not do the job! Multiple attributes are needed to define a good forecast. Spread-skill Calibration Difficult to achieve CRPS log p score Accuracy Accuracy Squared difference between forecast and observed distributions Analogous to MAE in limit Local score, rewards for correct category; infinite if observed category has 0 density Question: Are we measuring what we need to measure? Do we need different/additional measures?

9 Multivariate approaches Minimum spanning tree Analogous to Rank Histogram for multivariate ensemble predictions Ex: For TCs, treat track location and intensity as a 3-dimensional vector Great Circle and Euclidean distances Bias correction and Scaling recommended (Wilks) Multivariate energy score (Gneiting) Multivariate generalization of CRPS From Wilks (2004) Multivariate approaches allow optimization on the basis of more than one variable Bearing and Wind speed errors

10 Challenge: High resolution forecasts Mesoscale model (5 km) 21 Mar 2004 Which rain forecast is best? Global model (100 km) 21 Mar 2004 Observed 24h rain Sydney Sydney RMS=13.0 RMS=4.6 Smooth forecasts generally Win according to traditional verification approaches. From E. Ebert

11 Spatial method motivation POD = 0.40 FAR = 0.56 CSI = 0.27 Forecast Observed Traditional approaches ignore spatial structure in many (most?) forecasts Spatial correlations Small errors lead to poor scores (squared errors smooth forecasts are rewarded) Methods for evaluation are not diagnostic Same issues exist for ensemble forecasts

12 Example: Probability forecasts Example Example 2 Forecast probability 4 3 Example 5 Observed precipitation Example 3 Example 4

13 Poor skill Greatest overlap Conclusion: Calibration and skill are highly dependent on displacement Large displacemen t Poor skill

14 50% Prob(APCP_06>25.4 mm) vs. QPE_06 >25.4 mm Traditional Metrics Brier Score: 0.07 Area Under ROC: 0.62

15 Approaches for applying spatial methods for ensemble forecasts Treat as deterministic forecast (evaluate mean or other representative forecast) using spatial methods Evaluate spatial probability field OBS Transform ensemble gridpoint values to probabilities Compare probability field to observed field using spatial approach(es) Characterize distributions of spatial OBS attributes for ensemble Prob > 0.5 Mean precip

16 Applying spatial methods to ensembles As probabilities: Areas do not have shape of precipitation areas; may spread the area As mean: Area is not equivalent to any of the underlying ensemble members

17 Treatment of Spatial Ensemble Forecasts Alternative: Consider ensembles of attributes Evaluate distributions of attribute errors

18 New Spatial Verification Approaches Neighborhood Successive smoothing of forecasts/obs Object- and feature-based Evaluate attributes of identifiable features Scale separation Measure scale-dependent error Field deformation Measure distortion and displacement (phase error) for whole field Web site:

19 Object-based methods CRA: Contiguous Rain Area (Ebert, Gallus, McBride) MODE: Method for Object-based Diagnostic Evaluation (Davis, Brown, Bullock)

20 Contiguous Rain Area (CRA) Approach Ebert and McBride, J. Hydrol., 2000 Define entities using threshold (Contiguous Rain Areas) Horizontally translate the forecast until a pattern matching criterion is met: minimum total squared error between forecast and observations maximum correlation maximum overlap The displacement is the vector difference between the original and final locations of the forecast. Observed Forecast

21 Contiguous Rain Area (CRA) CRA example (Ebert and Gallus) The CRA method measures displacement and estimates error due to displacement, pattern, and volume

22 MODE Method for Object-based Diagnostic Evaluation (Davis et al., MWR, 2006, 2011) Object identification Two parameters: 1. Convolution radius 2. Threshold MODE Steps: (1) Identify objects; (2) Measure attributes (area, intensity, etc) (3) Match forecast and observed objects (4) Compare attributes (area, location, intensity, etc)

23 Object-based example 1 June Area ratios (1) 1.3 (2) 1.2 (3) In contrast: POD = 0.40 FAR = 0.56 CSI = Location errors (1) Too far West (2) Too far South (3) Too far North Median intensity ratio (1) 1.3 (2) 0.7 (3) th intensity ratio (1) 1.8 (2) 2.9 (3) 1.1

24 Ebert application to CRAs 24

25 MODE application to HWT ensembles Observed CAPS PM Mean Radar Echo Tops (RETOP) RETOP

26 50% Prob(APCP_06>25.4 mm) vs. QPE_06 >25.4 mm Good Forecast with Displacement Error? Traditional Metrics Brier Score: 0.07 Area Under ROC: Spatial Metrics Centroid Distance: Obj1) 200 km Obj2) 88km Area Ratio: Obj1) 0.69 Obj2) 0.65 Obj PODY: 0.72 Obj FAR: Median Of Max Interest: 0.77

27 Ensemble of QPF Objects Area of Research QPF Fields QPF Fields Member 1 Member 2 What does an Ensemble of Objects look like? Member 5 Member 6 Example from HMT-West season Courtesy of DTC/HMT Collaboration Member 3 Member 7

28 Area of active research within NCAR/RAL JNT and DTC Ensemble of MODE Objects Ensemble Probability Most Overlap Observed Object Highest Probability Consider and compare various attributes, such as: Area Location Intensity distribution Summarize distributions of attributes and differences Shape / Orientation Overlap with obs Measure of overall fit to obs Example from HMT- West season

29 Example: MODE application to HMT ensemble members Systematic microphysics impacts 3 Thompson Scheme members (circled) are: Less intense Larger areas Note Heavy tails Non-symmetric distributions for both size and intensity (medians vs. averages) 90 th percentile intensity Object area >6.35 >25,4 Threshold

30 Not just for precipitation Wind speed

31 Alternative approach to ensemble evaluation Translate ensemble info into user-relevant information Evaluate on the basis of the impact variable Ideal: User-specific info for many users; more general, user-relevant info for others Predicted chance of 30% capacity loss in E-W direction 9 h ahead Steiner: Translate convective ensembles into probability maps of aircraft capacity

32 Resources Jolliffe and Stephenson new edition: New chapter on spatial methods Intercomparison Project (ICP) web site: /index.html NCAR/DTC MET package Eric Gilleland spatial verification package in R

33 Summary Evaluation of high-impact weather is moving toward use of spatial verification methods Ensemble and probability forecasts are not immune to the same issues associated with verification of spatial fields for deterministic forecasts Initial efforts are in place to bring these methods forward for ensemble evaluation

34 Thank you!!

35 MODE: Method for Object-based Diagnostic Evaluation Matched Object 1 Matched Object 2 Unmatched Object FCST Radius=5 Thresh>0.25 OBS Radius=5 Thresh>0.25 Merging Merging No false alarms Misses Matching HMT: APCP_06h Valid: Z (F18)