Concepts of Forecast Verification FROST-14 Forecast & Research Olympics Sochi Testbed Pertti Nurmi WWRP JWGFVR Finnish Meteorological Institute pertti.nurmi@fmi.fi pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 1
WMO JWGFVR Draft verification plan WG-1 (NC), WG-2 (NWP), potentially WG-4 (End Forecast, if available) Participate in practical verification efforts Participate in analyzing verification results + publication of results FMI (Meteorological Research Unit) Contribute to the development of localized Road Weather forecasting systems E.g. slipperiness (aka friction) forecasting Depends on potential availability of RWS information (dedicated AWSs along the critical road networks) to enable model tuning. pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 2
pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 3
pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 4
pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 5
pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 6
Model contributors #1 pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 7
Model contributors #2 pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 8
Verification planning #1 User-oriented pointwise verification of all forecasts, tuned to the decision points of the decision maker Activities take place at points Two separate complementary approaches (like SNOW-V10) 1. High temporal resolution Point verification User needs Users require thresholds for their decisions Categorical approach 2. Research-oriented verification Model comparisons Verification of nowcasts and regional model output Collect information of available forecast products from all contributors Already started Pre-requisite for detailed verification planning pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 9
Verification planning #2 Listing of available/planned models/systems Already there (?) Available forecast lead times and projections Already there (?) List available/planned observational network To be confirmed Importance/relevance of site-specific observations Define requirements to enhance obs network (30 new expected AWS?) Establish standardized formats for fc & obs presentation and archiving Rich datasets serve both user- and research-oriented verification, when becoming available ( SNOW-V10 analysis apparently only recently started? ) Which : Retrospective or real-time verification (or both)? Comprehensive post verification is possible when all fc-obs archived (in standardized manner) and QC d pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 10
Verification planning #3 Pointwise verification of all forecasts at all forecast=decision points a) Baseline quality from a coarse model (GME,ECMWF?) as reference b) Hi-res models at various (time/space) resolutions c) Statistical adaptations Available? d) End forecasts produced by the Olympics forecast team Available? e) Comparison between, and added value, if any, (a) to (d) Use of verification software tools: R, MET, VerSUS (COSMO)? Preliminary simulation/testing phase during pre-winter(s), starting 2011-12 Definition of required manpower and/or how much is expected to be available (Roshydromet vs. external) Lessons learnt from SNOW-V10 pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 11
Verification planning #4 1. To consider: New categorical verification measures, SEDS, EDI,... Real-time verification (RTFV) Visualization of fc-obs time series Timing (error) of events onset, duration, cessation Verification wrt observation uncertainty Spatial verification (CRA, SAL, MODE, FSS, wavelets, neighborhood, FSS,...) Applicability of remotely sensed data Requires lotsa efforts Spatial verification of ensembles Requires even more efforts? User-oriented probability forecast verification Are prob fcs provided to users or Olympics forecast team, Sochi decision-makers or end-users? Road weather forecasting and verification ( FMI interests ) pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 12
SNOW-V10: Model output variables and thresholds pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 13
SNOW-V10: Suggested weather parameter categories Table 5 (2nd Revised Suggestion for SNOW-V10 Verification) Variable Cat 1 Cat 2 Cat 3 Cat 4 Cat 5 Cat 6 Cat 7 Cat 8 Cat 9 Temperature ( C) -25 < -25 T<-20-20 T<-4C -4 T<-2-2 T< 0 0 T< +2 +2 T< +4 +4 RH (%) < 30% 30 RH< 65% 65 RH< 90% 90 RH< 94% 94 RH< 98% 98% Winds (m/s) < 3 3 w < 4 4 w < 5 5 w < 7 7 w < 11 11 w < 13 13 w < 15 15 w < 17 17 Wind Gust (m/s) Wind Direction < 3 3 w < 4 4 w < 5 5 w < 7 7 w < 11 11 w < 13 13 w < 15 15 w < 17 17 d 339 & d < 24º (N) 24 d < 69º (NE) 69 d < 114º (E) Visibility (m) v < 30 30 v < 50 50 v < 200 114 d < 159º (SE) 200 v < 300 159 d < 204º (S) 300 v < 500 204 d < 249º (SW) 249 d < 294º (W) 294 d < 339º (NW) 500 - - Ceiling (m) c < 50 50 c< 120 120 c< 300 300 c< 750 750 c< 3000 c 3000 - - - Precip Rate (mm/hr) r = 0 (None) 0 < r 0.2 (Trace) 0.2 < r 2.5 (Light) 2.5 < r 7.5 (Moderate) r > 7.5 (Heavy) - - - - Precip Type No Precip Liquid Freezing Frozen Mixed (w/liquid) Unknown - - - pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 14
Model output variables (based on Kiktev, 24.8.11) pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 15
Model output thresholds (based on Kiktev, 24.8.11) pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 16
How the forecast fields are to be verified? Deterministic forecasts For such parameters as Temperature, Horizontal winds, Humidity, Surface pressure, Cloud base height, Cloudiness, Height of the zero isotherm, the Bias and RMS scores seem to be sufficient. For Precipitation rate, snowfall, accumulated snow, Wind gusts, Visibility, the scores are to be specified. A unified radar analysis (or radar QPE) with horizontal resolution about 2 km will be used for verification of gridded precipitation products from participating systems. Radar based verification techniques applied to precipitation fields, or in some way defined rain areas, are to be specified. Probabilistic forecasts A set of critical thresholds will be used to transform continuous variables into dichotomous events (Annex 2). These thresholds must be adjusted for the SOCHI area with particular considerations to sporting needs of the Olympic Games. Traditional probabilistic measures (ROC, Brier Score, Reliability Diagram and Frequency Histograms) are to be applied. Model verification (based on Kiktev, 24.8.11) This needs to be planned in detail asap pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 17
Real-Time Forecast Verification (RTFV) system Existed in B08FDP, but not in V10 How about Sochi-2014? > Requires huge efforts Decision needs to be made soon Fast qualitative and quantitative feedback on forecast system performance in real-time Verification products generated whenever new observations arrive Ability to inter-compare forecast systems on the fly Three levels of complexity Visual (quick look) Statistics (quantitative) Diagnostic (more information) pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 20
Advertisement XXII Olympic Winter Games Grid-to-Point version of Road Weather model En-route RW forecasts Olympic Tampere sites Sochi Helsinki sea coast pertti.nurmi@fmi.fi FROST-14 Verification concepts 18.10.2011 22
Advertisement www.sirwec.org Next Conference Helsinki Exhibition and Conference Centre 23-25 May 2012 (Wed-Fri) to be organized jointly by Finnish Transport Agency (FTA) Finnish Meteorological Institute (FMI) pertti.nurmi@fmi.fi Welcome! FROST-14 Verification concepts