Forecast verification at Deltares

Transcription

1 Forecast verification at Deltares Dr Jan Verkade October 26, 2016

2 Brief introduction: Jan Verkade Hydrologist; expert in Real-time hydrological forecasting Member of the Rijkswaterstaat River Forecasting Service (Guest) researcher at Delft University of Technology, Faculty of TPM Research interests: uncertainty analyses probabilistic forecasting forecast verification forecast use

3 Deltares and forecast verification Introduction: why verify? Our approach to forecast verification Verification tools Selected experience Training Challenges

4 Why verify?

5 What is forecast verification? Verification is the assessment and quantification of the relationship between a matched set of forecasts and observations. (Stanski et al., 1989) Verification is the posterior assessment of the skill and value of the forecasts It ties into the question: What is a good forecast? Quality Value Consistency (Murphy, 1993) Murphy, Allan H. What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting. Weather and Forecasting 8, no. 2 (1993): Stanski, Henry R., Laurence J. Wilson, and William R. Burrows. Survey of Common Verification Methods in Meteorology. World Meteorological Organization Geneva,

6 Why verify? 1. Administrative reasons Provision of the rationale for (additional) investments in forecasts 2. Scientific reasons: Where can the forecasts be improved? 3. Economic reasons: What is the value to an end user? (Jolliffe and Stephenson, 2012; Brier & Allen, 1951; Stanski et al., 1989)

7 Forecast quality versus forecast value Quality: high correlation between forecasts and observations Value: degree to which an end user can make better decisioons Classic example: forecast of a sunny day over Sahara desert Quality? Value? Source: Bertrand Devouard / Florence Devouard

8 Users of verification information Model developers and forecasting system designers Operational forecasters Forecast users System administrators / finance providers these may require different verification information and may have different ability to ingest this, and use it for their purpose model development forecast decision response observation verification

9 Verification at Deltares: approach Dr Jan Verkade October 24, 2016

10 Verification at Deltares: approach Prepare data record including hind-casting if required Qualitative eyeball verification : take a look at forecasts and observations Summary metrics: Graphical verification measures Numerical: metrics and skill scores

11 Step 0: prepare data record Operational record: Deltares OpenArchive if you have it (designed for this type of activity) Hindcasting: offline activity using Delft-FEWS in standalone application The Delft-FEWS forecast production system

12 Step 1: Visual inspection: hydrographs Example: River IJssel water levels at Kampen

13 Step 1: Visual inspection: scatters (forecast v observation) All available fcst, obs pairs in a single figure Separate plots for separate leadtimes Horizontal axis: forecast Vertical axis: observation Where would we like to see the points? Ensemble: multiple forecasts for every observation Lot of points are plotted on top of one another transparency helps to identify this complicates interpretation nonetheless

14 Step 1: Visual inspection: scatters (ensemble mean v obs)

15 Step 1: Visual inspection: scatters ( error versus observation) What do you notice? Can these forecasts be bias-corrected?

16 Step 1: Visual inspection: scatters ( error v ensemble mean) What do you notice? Can these forecasts be bias-corrected?

17 Step 2: Graphical measures of forecast quality Many possible graphs, including Talagrand plots (or Rank Histograms) Reliability plots ROC plots

18 Step 2: Talagrand diagrams, or Rank Histograms How often does the observation land below the lowest member? between the lowest and second-lowest member? the second-lowest and third-lowest? the third-lowest and fourth-lowest? above the highest member?

19 Step 2: Reliability plots Reliability: correspondence of predicted probabilities with observed relative frequencies Graphical measure: reliability plots Horizontal axis: event probabilities Vertical axis: observed relative frequencies Important! How many verification pairs were used to determine the points on the graph? Observed relative frequency, o 1 Flooding yes/no, 1-hour forecast No skill No resolution Forecast probability, y i

20 hit rate [-] Step 2: Relative Operating Characteristics Contingency table hit rate and false alarm rate ROC plot Relative Operating Characteristic hits 28 hit rate = 0.55 observed events 51 false alarms 72 false alarm rate = 0.03 observed non-events false alarm rate [-]

21 Relative economic value V [-] Step 2: Relative Economic Value plots What is the value of a forecast for a user that is characterized by his cost-to-loss ratio? Cost-loss ratio r [-]

22 Step 3: Numerical measures of forecast quality Many, many, many metrics available Deterministic Probabilistic: Brier s probability score, Ranked Probability Skill Score Metrics and skills Some metrics can be de-composed into multiple other metrics (example: Brier score = uncertainty + reliability - resolution) Conditional verification, using sub-sets of the data record (for example, only consider forecasts that are paired with top 10% of observations)

23 Step 3: Brier s probability score Average squared error of a probability forecast Example: P =.80 Event occurs (1) or does not occur (0) Non-occurrence: (.8 0)^2 =.64 Occurrence: (.8 1)^2 =.04 Do this for every forecast, then average: PS 1 N N i 1 f i o i 2 Best possible score: 0 Worst possible score: 1

24 Metric #3: Brier s Probality Skill Score

25 Tools for verification Dr Jan Verkade

26 Ensemble Verification System Development started in 2008 by US National Weather Service, Office for Hydrologic Dev ment Main developer: James Brown. Has since moved to UK based Hydrologic Solutions, Ltd from which he continues to develop the EVS Open source, available for download via (have a look, if anything, the documentation can be very useful) Contrary to what the name suggests, EVS can be used to quality-assess deterministic forecasts and simulations (not characterised by lead time) also!

27 Available scores, metrics and skill scores Pretty much all commonly used verification metrics for both single-valued (deterministic) and ensemble forecast, including, where applicable, decompositions: correlation, mean error, RME, RMSE, MAE Brier Score, CRPS, ROC diagram and score, Reliability diagram, rank histogram Boot-strap for estimation of uncertainty in metric values

28 forecasts: validtime, leadtime, forecast(s) Pairing pairs: validtime, leadtime, obs, forecast(s) observations: validtime, observation allowed input formats include PI.xml, netcdf, text files

29 Conditional verification: how good are my forecasts that coincide with the top 10% of observations? that are in the top 10% of forecast? that were produced in summer? that were produced when antecedent precipitation exceeded 80mm in 3 days? that were produced when the temperatures were below 0?

30 EVS user interface Java based platform independent Tabs show workflow: set up verification parameters aggregate if required explore outputs All options are saved in a.evs file (which is really an XML file)

31 EVS as repository of verification knowlege Comprises accessible explanations of available metrics (see right) Wealth of options will trigger operator to consider use thereof in her verification exercise This is amplified by extensive documentation

32 EVS through command line.evs file may be created in EVS GUI, then manipulated through XML editor can be executed through command line. example: java -jar EVS.jar myverificationexercise.evs this allows for running multiple verification units without manual interference

33 EVS output Option to output as.xml and/or.png XML can be useful for post-processing data EVS comes with scripts that import EVS output xml into R scripting language and subsequently present verification results in pretty much whatever way you like

34 Verification results as post-processed through R (Note the uncertainty bands, acquired through boot-strapping.)

35 EVS and the Deltares OpenArchive Verification is typically done using a long record of forecasts and observations The Deltares OpenArchive is designed to hold such records The new version of EVS will be able to directly read from the Deltares OpenArchive EVS Delft-FEWS PI web service Deltares OpenArchive Source data netcdf

36 Some additional verification tools Verification package in R (UCAR) + vignette MET: Model Evaluation Tools (UCAR) verif package for Python (UBC) Flooding yes/no, 1-hour forecast Observed relative frequency, o 1 No skill No resolution Forecast probability, y i

37 Selected experience Dr Jan Verkade

38 Verification of ECMWF precipitation reforecasts over Rhine basin Purpose: explore potential for post-processing weather forecasts Both raw and post-processed forecasts were postprocessed (temperature and precipitation) Conclusions: post-processing of weather forecasts adds some skill to streamflow forecasts but a lot less than was expected a priori post-processing techniques deemed inadequate for use in hydro forecasting! we need techniques that take into account temporal and spatial relationships = ongoing research

39 Verification of interpolated hourly precipitation over the Rhine Purpose: explore quality of the algorithm for interpolation of precip observations - Comparison of historic REGNIE hourly climatologies against other products (HYRAS,EOBS,EFAS) - Cross-validation of interpolation (leave-one-out) - Focus on difference between hourly and daily temporal resolution - i.e. average length of precipitation events Location overview of meteorological gauges in the data set Future activities: - Effect of the above on streamflow simulations - Data assimilation as verification tool

40 Verification of pilot ensemble water level forecasts at Lake IJssel Decision at hand: whether or not to inflate the barrier at Ramspol. Rijkswaterstaat initiated pilot ensemble water level forecasts in 2014: COSMO LEPS wind forecasts hydrological model chain water level forecasts Main Q: can the ensemble water level prediction be interpreted as a probabilistic forecast?

41 Verification of pilot ensemble water level forecasts at Lake IJssel Main conclusion: too many observations are outside of the ensemble classic example of underdispersion or overconfident forecasts Next steps: explore quality of underlying wind fcst explore uncertainties other than those in wind forecasts

42 Verification Analyst Tool suggestion to BoM BoM is moving towards verifying flood warnings Warnings are distributed through a tool external to the Delft-FEWS-based HyFS: HyFS forecast warning tool Verification Analyst Tool Warnings will be re-imported into HyFS and paired with their verifying observation prior to being verified timing errors magnitude errors

43 Verification of precipitation forecasts over Rhine basin (planned!) QA of various precipitation forecast products used within the RWsOS Rivers application: ECMWF products (EPS and DET) COSMO-LEPS DWD products (LM/Icon and GM) KNMI product (HIRLAM) Verification at HBV sub-basin level (= inputs to the hydrological model) observations spatially aggregated from points to polygons forecasts spatial aggregation and temporal dis-aggregation Should contribute to the answer to the question which weather forecast should I use?

44 Training Dr Jan Verkade

45 Training on verification Verification module in the probabilistic forecasting course ( rationale for verification available tools and techniques exercises in deriving reliability plots and Brier s probability score At request, a more elaborate verification course can be delivered Most verification projects will include a short refresher on verification

46 Challenges Dr Jan Verkade

47 Some open challenges (1) Verification can be a little scary as it explicitly shows that the forecasts are uncertain Benchmarking: RMSE in my headwater basins for 1:10y events is approx. 4ft; is that good or bad? little exchange of forecast quality indicators between forecasting agencies How to communicate quality metrics that can be quite abstract to non-experts?

48 Some open challenges (2) OK I have verified. Now what? Werner, Kevin, Jan S Verkade, and Thomas C Pagano. Application of Hydrological Forecast Verification Information. In Handbook of Hydrometeorological Ensemble Forecasting, edited by Qingyun Duan, Florian Pappenberger, Jutta Thielen, Andy Wood, Hannah L. Cloke, and John C. Schaake, Berlin, Heidelberg: Springer Berlin Heidelberg,

49 Who to approach re verification? Jan Verkade Prof. Albrecht Weerts Edwin Welles (D-USA) (PhD in forecast verification!) André Grijze (OpenArchive EVS)

50 Forecast Verification: Delft-FEWS, the OpenArchive and the EVS 1. The EVS-to-OpenArchive link 2. EVS exercise: verification of COSMO-LEPS ensemble forecasts over the Meuse basin 3. Additional potential applications of the OpenArchive Hands-on session, this afternoon. Two identical sessions. (14:30 15:30 and 16:00 17:00) Image credits: and

51 Thank you for your attention! Jan Verkade