Concepts of Forecast Verification FROST-14

Transcription

1 Concepts of Forecast Verification FROST-14 Forecast & Research Olympics Sochi Testbed Pertti Nurmi WWRP JWGFVR Finnish Meteorological Institute FROST-14 Verification concepts

2 WMO JWGFVR Draft verification plan WG-1 (NC), WG-2 (NWP), potentially WG-4 (End Forecast, if available) Participate in practical verification efforts Participate in analyzing verification results + publication of results FMI (Meteorological Research Unit) Contribute to the development of localized Road Weather forecasting systems E.g. slipperiness (aka friction) forecasting Depends on potential availability of RWS information (dedicated AWSs along the critical road networks) to enable model tuning. pertti.nurmi@fmi.fi FROST-14 Verification concepts

3 FROST-14 Verification concepts

4 FROST-14 Verification concepts

5 FROST-14 Verification concepts

6 FROST-14 Verification concepts

7 Model contributors #1 FROST-14 Verification concepts

8 Model contributors #2 FROST-14 Verification concepts

9 Verification planning #1 User-oriented pointwise verification of all forecasts, tuned to the decision points of the decision maker Activities take place at points Two separate complementary approaches (like SNOW-V10) 1. High temporal resolution Point verification User needs Users require thresholds for their decisions Categorical approach 2. Research-oriented verification Model comparisons Verification of nowcasts and regional model output Collect information of available forecast products from all contributors Already started Pre-requisite for detailed verification planning FROST-14 Verification concepts

10 Verification planning #2 Listing of available/planned models/systems Already there (?) Available forecast lead times and projections Already there (?) List available/planned observational network To be confirmed Importance/relevance of site-specific observations Define requirements to enhance obs network (30 new expected AWS?) Establish standardized formats for fc & obs presentation and archiving Rich datasets serve both user- and research-oriented verification, when becoming available ( SNOW-V10 analysis apparently only recently started? ) Which : Retrospective or real-time verification (or both)? Comprehensive post verification is possible when all fc-obs archived (in standardized manner) and QC d pertti.nurmi@fmi.fi FROST-14 Verification concepts

11 Verification planning #3 Pointwise verification of all forecasts at all forecast=decision points a) Baseline quality from a coarse model (GME,ECMWF?) as reference b) Hi-res models at various (time/space) resolutions c) Statistical adaptations Available? d) End forecasts produced by the Olympics forecast team Available? e) Comparison between, and added value, if any, (a) to (d) Use of verification software tools: R, MET, VerSUS (COSMO)? Preliminary simulation/testing phase during pre-winter(s), starting Definition of required manpower and/or how much is expected to be available (Roshydromet vs. external) Lessons learnt from SNOW-V10 pertti.nurmi@fmi.fi FROST-14 Verification concepts

12 Verification planning #4 1. To consider: New categorical verification measures, SEDS, EDI,... Real-time verification (RTFV) Visualization of fc-obs time series Timing (error) of events onset, duration, cessation Verification wrt observation uncertainty Spatial verification (CRA, SAL, MODE, FSS, wavelets, neighborhood, FSS,...) Applicability of remotely sensed data Requires lotsa efforts Spatial verification of ensembles Requires even more efforts? User-oriented probability forecast verification Are prob fcs provided to users or Olympics forecast team, Sochi decision-makers or end-users? Road weather forecasting and verification ( FMI interests ) pertti.nurmi@fmi.fi FROST-14 Verification concepts

13 SNOW-V10: Model output variables and thresholds FROST-14 Verification concepts

14 SNOW-V10: Suggested weather parameter categories Table 5 (2nd Revised Suggestion for SNOW-V10 Verification) Variable Cat 1 Cat 2 Cat 3 Cat 4 Cat 5 Cat 6 Cat 7 Cat 8 Cat 9 Temperature ( C) -25 < -25 T< T<-4C -4 T<-2-2 T< 0 0 T< T< RH (%) < 30% 30 RH< 65% 65 RH< 90% 90 RH< 94% 94 RH< 98% 98% Winds (m/s) < 3 3 w < 4 4 w < 5 5 w < 7 7 w < w < w < w < Wind Gust (m/s) Wind Direction < 3 3 w < 4 4 w < 5 5 w < 7 7 w < w < w < w < d 339 & d < 24º (N) 24 d < 69º (NE) 69 d < 114º (E) Visibility (m) v < v < v < d < 159º (SE) 200 v < d < 204º (S) 300 v < d < 249º (SW) 249 d < 294º (W) 294 d < 339º (NW) Ceiling (m) c < c< c< c< c< 3000 c Precip Rate (mm/hr) r = 0 (None) 0 < r 0.2 (Trace) 0.2 < r 2.5 (Light) 2.5 < r 7.5 (Moderate) r > 7.5 (Heavy) Precip Type No Precip Liquid Freezing Frozen Mixed (w/liquid) Unknown pertti.nurmi@fmi.fi FROST-14 Verification concepts

15 Model output variables (based on Kiktev, ) FROST-14 Verification concepts

16 Model output thresholds (based on Kiktev, ) FROST-14 Verification concepts

17 How the forecast fields are to be verified? Deterministic forecasts For such parameters as Temperature, Horizontal winds, Humidity, Surface pressure, Cloud base height, Cloudiness, Height of the zero isotherm, the Bias and RMS scores seem to be sufficient. For Precipitation rate, snowfall, accumulated snow, Wind gusts, Visibility, the scores are to be specified. A unified radar analysis (or radar QPE) with horizontal resolution about 2 km will be used for verification of gridded precipitation products from participating systems. Radar based verification techniques applied to precipitation fields, or in some way defined rain areas, are to be specified. Probabilistic forecasts A set of critical thresholds will be used to transform continuous variables into dichotomous events (Annex 2). These thresholds must be adjusted for the SOCHI area with particular considerations to sporting needs of the Olympic Games. Traditional probabilistic measures (ROC, Brier Score, Reliability Diagram and Frequency Histograms) are to be applied. Model verification (based on Kiktev, ) This needs to be planned in detail asap pertti.nurmi@fmi.fi FROST-14 Verification concepts

18 Real-Time Forecast Verification (RTFV) system Existed in B08FDP, but not in V10 How about Sochi-2014? > Requires huge efforts Decision needs to be made soon Fast qualitative and quantitative feedback on forecast system performance in real-time Verification products generated whenever new observations arrive Ability to inter-compare forecast systems on the fly Three levels of complexity Visual (quick look) Statistics (quantitative) Diagnostic (more information) FROST-14 Verification concepts

19 Advertisement XXII Olympic Winter Games Grid-to-Point version of Road Weather model En-route RW forecasts Olympic Tampere sites Sochi Helsinki sea coast FROST-14 Verification concepts

20 Advertisement Next Conference Helsinki Exhibition and Conference Centre May 2012 (Wed-Fri) to be organized jointly by Finnish Transport Agency (FTA) Finnish Meteorological Institute (FMI) Welcome! FROST-14 Verification concepts