Overview of Verification Methods

Transcription

1 Overview of Verification Methods Joint Working Group on Forecast Verification Research (JWGFVR) Greg Smith on behalf of Barbara Casati, ECCC

2 Existing Verification Techniques Traditional (point-by-point) methods: 1. graphical summary (scatter-plot, box-plot) 2. Continuous scores (MSE, correlation) 3. Categorical scores from contingency tables (FBI,HSS,PC). 4. Probabilistic verification (Brier, CRPS, rank histograms) Extreme dependency scores : Ferro and Stephenson (2011) Wea&For (EDI,SEDI) Spatial methods: 1. Scale-separation 2. Neighbourhood 3. Field-deformation 4. Feature-based 5. Distance metrics for binary images There is no single technique which fully describes the complex observationforecast relationship Key Q1: what do we wish to know from our verification? (end-user / verification purpose / questions addressed / attributes of interest) Key Q2: What are the (statistical) characteristics of the variable and forecast verified? What are the available obs?

3 Exploratory methods: joint distribution Scatter-plot: inform on bias, outliers, error magnitude, linear association, peculiar behaviours in extremes, misses and false alarms (can be linked to contingency table) outlier

4 Exploratory methods: marginal distributions Box-plots and quantile-quantile (qq) plots compare the forecast vs obs empirical marginal distributions (regardless of the forecast-obs pairing) => assess (solely) the bias q 0.75 Joint distribution (e.g. scatter-plots): assess also the accuracy

5 Exploratory methods: conditional distributions Conditional histogram and conditional box-plot 9/29/2016 5

6 Exploratory methods: conditional box-plot and qq-plot 9/29/2016 6

7 Continuous Scores 1 lin e a r b ia s = M E = y x i i = Y X n 1 n i= 1 2 M S E = y x i n 1 n i n i= 1 y y x x i= 1 r = = XY n n n n i y y x x i n i= 1 i= 1 i i c o v(y, X ) s s Y X bias accuracy linear association Suitable for continuous Gaussian variables (e.g. temperature) Not suitable for mixed, right-skewed, episodic discontinuous variables (e.g. precipitation)

8 Continuous scores are related to each other: s M S E = Y X + s + s s r 2 M S E = M E + v a r(y X ) Y X Y X X Y This mathematical relation can be exploited to display multiple scores in a single summary diagram (Taylor, 2001, JGR). A skill score compares the accuracy of the forecast versus that of a reference (often trivial) forecast (e.g. persistence, climatology, random): M S E M S E ref M S E S S = = 1 M S E M S E M S E M S E p e rf ref ref Example: Reduction of Variance, ref=sample climatology, MSE clim = s 2 X

9 Predicted Contingency table and joint distribution Murphy and Winkler (1987): A general framework for forecast verification. Mon. Wea. Rev., 115 Contingency table entries / total counts n = hits+misses+false alarms+nils Provides an estimate of the joint distribution: 1/n O Observed O F P(O,F) P(O,F) P(F) F P(O,F) P(O,F) P(F) P(O) P(O) 1 Categorical scores are (functions of) marginal, joint and conditional probabilities. Examples: FBI = P(F)/P(O) PC=P(O,F)+ P(O,F)

10 Weather: chaotic system, solution might diverge due to small changes in initial state. Ensembles: enable to reproduce many possible weather scenarios (associate a probability to each weather scenario, informs on the uncertainty associated with weather forecasts). Does the ensemble always encompass the observation? Is the ensemble underdispersive of overdispersive? How well does the ensemble spread of the forecast represent the true variability (uncertainty) of the observations? Rank Histogram: histogram of the obs rank within the (ordered) ensemble member values under-dispersive over-dispersive overdispersive underdispersive Spread-error relationship: compare the ensemble spread (e.g. standard deviation of ensemble values) with the ensemble-mean (root mean squared) error.

11 Probabilities and ensemble verification Probabilistic forecast: Pr(event) Example: Pr(precipitation>0.2 mm) = 70% Ensemble forecast: distribution of values Example: {1, 1.2, 1.1, 0.1, 0.1, 1.4, 1.2, 0, 1, 1.2} I can define a Pr(event) from a distributuion of values, but NOT viceversa! The ensemble forecast is more informative (e.g. spread of forecast, clusters). Verification of ensembles can be performed with more informative methods. Brier Score: pi = forecast probability of ith event oi = observation = Heaviside = { 0 no-event 1 event Continuous Ranked Probability Score: Where Pf, Po are the cdf of forecast and obs. Both BS and CRPS can be decomposed into reliability - resolution + uncertainty A skill score (against sample climatology) is defined for both BS and CRPS. BS verifies probabilities, CRPS verifies the whole distribution (suitable for ensembles).

12 Reliability: measures the conditional bias frequency(obs event) forecast Pr(event) Reliability Diagram

13 Summary of verification basic concepts Verification pertains to summarizing the properties of the forecast-obs joint, marginal and conditional distributions Some attributes assessed by different verification scores: bias, accuracy, assocuiation, skill Exploratory (graphical) methods: simple and informative! Continuous scores: suitable for Gaussian variables Contingency tables and categorical scores: Murphy and Winkler (1987): A general framework for forecast verification. Mon. Wea. Rev., 115 Ensembles: verify the whole distribution. Key References: Jolliffe and Stephenson (2012); chapter 7 in Wilks (2011); von Storch and Zwiers (1999).

14 Spatial Verification Spatial verification techniques: neighbourhood, scale-separation, feature-based, field deformation. neighbourhood scale-separation Most of these techniques need (high resolution, possibly model independent) gridded obs! feature-based field deformation The Spatial Verification Inter-Comparison Project (ICP), is entering its second phase (MesoVICT). Includes an impressive list of references for spatial verification studies (>200). Review article: Gilleland et al (2009): Inter-comparison of Spatial Forecast Verification Methods. Wea&For, 24 Spatial verification techniques aim to: account for field spatial structure and the presence of features provide information on error in physical terms (meaningful verification) Assess location and timing errors (separate from intensity error) account for small time-space uncertainties (avoid double-penalty issues)

15 1. Scale-separation approaches Briggs and Levine (1997), wavelet cont (MSE, corr); Casati et al. (2004), Casati (2010), wavelet cat (HSS, FBI, scale structure) Zepeda-Arce et al. (2000), Harris et al. (2001), Tustison et al. (2003), scale invariants parameters; Casati and Wilson (2007), wavelet prob (BSS=BSSres-BSSrel, En2 bias, scale structure); Jung and Leutbecher (2008), spherical harmonics, prob (EPS spread-error, BSS, RPSS); Denis et al. (2002,2003), De Elia et al. (2002), discrete cosine transform, taylor diag; Livina et al (2008), wavelet coefficient score. De Sales and Xue (2010) 1.Decompose forecast and observation fields into the sum of spatial components on different scales (wavelets, Fourier, DCT) 2.Perform verification on different scale components, separately (cont. scores; categ. approaches; probability verif. scores) Assess scale structure from Jung and Leutbecher (2008) Bias, error and skill on different scales Scale dependency of forecast predictability (no-skill to skill transition scale)

16 2. Neighbourhood verification 1. Define neighbourhood of grid-points: relax requirements for exact positioning (mitigate double penalty: suitable for high resolution models); account for forecast and obs time-space uncertainty. t - 1 t t + 1 observation forecast observation forecast Frequency Forecast value Rainfall 2. Perform verification over neighbourhoods of different sizes: verify deterministic forecast with probabilistic approach Frequency Yates (2006), upscaling, cont&cat scores; Roberts and Lean (2008) Fraction Skill Score; Theis et al (2005); pragmatical approach; Atger (2001), spatial multi-event ROC curve; Marsigli et al (2005, 2006) probabilistic approach. random

17 3. Field-deformation approaches Hoffmann et al (1995); Hoffman and Grassotti (1996), Nehrkorn et al. (2003); Germann and Zawadzki (2002, 2004); Keil and Craig (2007, 2009) DAS; Marzbar and Sandgathe (2010) optical flow; Alexander et al (1999), Gilleland et al (2010) image warping 1.Use a vector (wind) field to deform the forecast field towards the obs field 2.Use an amplitude field to correct intensities of (deformed) forecast field to those of the obs field Vector and amplitude fields provide physically meaningful diagnostic information: feedback for data assimilation and now-casting. Error decomposition is performed on different spectral components: directly inform about small scales uncertainty versus large scale errors.

18 4. Feature-based techniques Ebert and McBride (2000), Grams et al (2006), Ebert and Gallus (2009): CRA Davis, Brown, Bullok (2006) I and II, Davis et al (2009): MODE Wernli, Paulat, Frei (2008): SAL score Nachamkin (2004, 2005): composites Marzban and Sandgathe (2006): cluster Lack et al (2010): procrustes Observed Forecast 1. Identify and isolate (precipitation) features in forecast and observation fields (thresholding, image processing, composites, cluster analysis) 2. assess displacement and amount (extent and intensity) error for each pairs of obs and forecast features; identify and verify attributes of object pairs (e.g. intensity, area, centroid location); evaluate distance-based contingency tables and categorical scores; perform verification as function of feature size (scale); add time dimension for the assessement of the timing error of precipitation systems.

19 Verification Resources Forecast verification FAQ: web-page maintained by the WMO Joint Working Group on Forecast Verification Research (JWGFVR). Includes verification basic concepts, overview traditional and spatial verification approaches, links to other verification pages and verification software, key verification references. Web page of the Spatial Verification Inter-Comparison Project (ICP), which now is entering its second phase (MesoVIC). Includes an impressive list of references for spatial verification studies. Review article: Gilleland, E., D. Ahijevych, B.G. Brown, B. Casati, and E.E. Ebert, 2009: Intercomparison of Spatial Forecast Verification Methods. Wea. Forecasting, 24 (5),

20 Thank you!