Perfomance of the WWRP project FROST-2014 forecasting systems: Preliminary assessments (FROST = Forecast and Research in the Olympic Sochi Testbed)

Transcription

1 Perfomance of the WWRP project FROST-2014 forecasting systems: Preliminary assessments (FROST = Forecast and Research in the Olympic Sochi Testbed) D.Kiktev, E.Astakhova, A.Muravyev, M.Tsyrulnikov WWOSC-2014, August 2014

2 Goals of WMO WWRP RDP/FDP FROST-2014: To develop a comprehensive information resource of alpine winter weather observations; To improve and exploit: high-resolution deterministic mesoscale forecasts of meteorological conditions in winter complex terrain environment; regional meso-scale ensemble forecast products in winter complex terrain environment; nowcast systems of high impact weather phenomena (wind, precipitation type and intensity, visibility, etc.) in complex terrain. To improve the understanding of physics of high impact weather phenomena in the region; To deliver deterministic and probabilistic forecasts in real time to Olympic weather forecasters and decision makers. To assess benefits of forecast improvement (verification and societal impacts)

3 International participants of the FROST-2014 project COSMO, EC, FMI, HIRLAM, KMA, NOAA, ZAMG 3 rd meeting of the project participants (10-12 April 2013) under supervision of the WWRP WGs on Nowcasting, Mesoscale Forecasting, Verification Research

4 Observational network - About 50 AMS; - C-band Doppler Radar WRM200; - Temperature/Humidity profiler HATPRO; - Wind Scintec-3000 Radar Wind Profiler; - Two Micro Rain vertically pointing Radars (MRR-2); - 4 times/day upper air hi-vertical-res. sounding on site

5 Forecasting systems participating in RDP/FDP FROST-2014 Deterministic NWP: COSMO-RU with grid spacing 1km, 2.2 km, 7km; GEM with grid spacing 2.5 km, 1 km, 0.25 km; NMMB 1 km; HARMONIE 1 km; INCA 1 km Ensemble NWP: COSMO-S14-EPS (7km), Aladin LAEF (11km), GLAMEPS (11km), NNMB-EPS (7km ), COSMO-RU2-EPS (2.2km), HARMON-EPS (2.5km) Nowcasting: ABOM, CARDS, INCA, INTW, MeteoExpert, Joint (Multi-system forecast integration)

6 FROST-2014 Online Monitoring of Forecast Quality: Role of resolution - GEM-2.5 km vs GEM-1 km vs GEM-250 m Forecast mean absolute errors as a function of forecast lead time. Location: Mountain skiing finish (Roza-Khutor-7 station). Period: 15 Jan 15 March Effect is not straightforward. It depends on meteorological variable, location, lead time etc.

7 Some examples of diagnostic verification: Role of spatial resolution. COSMO-S14-EPS (7km grid spacing) vs COSMO-RU2-EPS (2km grid spacing) Parameter: T2m, Location: Biathlon Stadium (1075m), Verification Period: , Verification approach: Nearest point COSMO-S14-EPS (7km grid spacing) COSMO-S14-EPS (7km grid spacing) COSMO-RU2-EPS (2km grid spacing) COSMO-RU2-EPS (2km grid spacing) Hi-res ensemble forecasts: better pdfs, higher variability but poorer ensemble mean scores.

8 Role of spatial resolution for ensemble forecasts continued COSMO-S14-EPS (7km grid spacing) vs COSMO-RU2-EPS (2km grid spacing) Verifications for T2m ensemble mean Verification Period: Station BIAS (for 6/12/18hr lead time) Mean Absolute Error (for 6/12/18hr lead time) COSMO-S14-EPS COSMO-RU2-EPS COSMO-S14-EPS COSMO-RU2-EPS Sledge (~700m) Freestyle (~1000m) Biathlon Stadium (~1500m) Mountain Skiing(start) (~2000m) -1.3 / -2.0/ / -1.9 / / 2.2 / / 3.5 / / -1.8 / / -0.7 / / 2.0 / / 2.4 / / -1.3 / / 0.0 / / 1.8 / / 2.6 / / 2.2 / / 0.2 / / 3.1 / / 2.2 / 2.6 T2m: Some positive effect of downscaling from 7 to 2 km resolution. == Wind Speed: No positive effect of dynamical downscaling was found.

9 FROST-2014 experience demonstrates that direct forecast of visibility is a serious challenge. However, some results were encouraging. Example: 17 February 2014, 11:00-12:00 UTC (Biathlon venue) Forecast of time slot for competitions during the 3-days period with low visibility. Forecast of wind direction and relative humidity (as proxy of visibility) by COSMO-Ru1 (1km grid spacing) Camera shots from Gornaya Carousel :00 UTC 11:30 UTC 12:00 UTC RH at 2m: Forecast and observations Wind and RH at 850 hpa. Forecast from 12 UTC :00 UTC 12:00 UTC 13:00 UTC Biathlon Stadium Biathlon Stadium Biathlon Stadium

10 How was the window of good visibility on 17 February predicted by various systems?

11 Some ensemble verifications: ROC Area, Brier Skill Score, and Brier Score for Precip > 0.01 mm/3h COSMO-S14-EPS red COSMO-RU2-EPS orange LAEF-EPS brown NMMB-EPS black HARMON-EPS blue GLAMEPS green Verification approach: 13 stations in the area of Krasnaya Polyana were clustered for matching to forecasts. ROCA BSS BS Lead time COSMO-S14-EPS, NMMB-EPS and COSMO-RU2-EPS look most informative.

12 ROCA, BSS, and BS scores for Precip > 5 mm/3h COSMO-S14-EPS red COSMO-RU2-EPS orange LAEF-EPS brown NMMB-EPS black HARMON-EPS blue GLAMEPS green ROCA BSS BS For higher Precip threshold (w.r.t. the lower threshold): = COSMO-S14-EPS, NMMB, and HARMON-EPS become worse. = In contrast, LAEF and GLAMEPS become better.

13 It was not simple for forecasters to deal with such an amount of information under the operational time constraints => Integrated Forecast F. Woodcock and C. Engel: Operational Consensus Forecasts, Weather and Forecasting, 2005; L.X. Huang and G.A. Isaac: Integrating NWP Forecasts and Observation Data to Improve Nowcasting Accuracy, Weather and Forecasting, 2012 F( t) ( t) O (1 ( t)) ( t) ( f N i i i ( t) b F(t) integrated forecast (t forecast time); O last available observation; f i (t) forecast of i-th participating forecasting system; α(t), β i (t) - weights; b i (t) - bias for i-th forecasting system i ( t))

14 Web-editor was developed for forecasters for correction of objective forecasts. FROST-2014 weather data feed for the Olympic information system Integrated objective multi-model forecasts served as a first guess for preparation of the official forecasts for the Olympic information system.

15 Subjective Evaluation of forecast technologies by Sochi forecasters: survey (3 best models are shown in red) Model Grid mesh size COSMO- Ru7 7 km COSMO- Ru2 2.2 km COSMO- Ru1 1.1 km COSMO- S14-EPS 7 km NMMB 1 km NMMB- EPS 7 km GEM km GEM-1 1 km GEM m Ove rall usef ulne ss T Forecast accuracy: Prec ip Win d Gust s Vis Visu aliza tion (app eara nce) Timel iness and reliab ility Comments The basic model for the forecasters. Reasonable precip fact. Overestimated precip intensity. Tmin, Tmax poor. Wind poor. dt/dt OK The basic model for the forecasters. In general better than Cosmo-Ru Comments are contradictory. The majority of forecasters considered COSMO-Ru2 to be more useful than COSMO- Ru1. Some forecasters preferred Cosmo-Ru1 (helpful wind, humidity). Overestimates precip intensity Precip reasonable. Good tendencies. Wind poor. Was available well before the Olympics that was helpful to get used to this information Good in T and Precip. Informative visibility Nice. Informative visibility. Precip reasonable. Tmin, Tmax poor Good precip, humidity Good precip, humidity Good precip, humidity. Very detailed maps.

16 Subjective Evaluation of forecast technologies by Sochi forecasters: survey (3 best models are shown in red) - continued Model Grid mesh size GLAMEPS 11 km GLAMEPS calibr.,freque nt update 11 km HarmonEPS 2.5 km Harmonie 1 km ALADIN LAEF 11 km WRF 600 m COSMO- Ru2-EPS 2.2 km Over all usefu lness T Forecast accuracy: Preci p Win d Gust s Vis Visua lizati on (appe aran ce) Time lines s and relia bility Comments Informative tendencies. Issues with absolute values Interesting and helpful In general good in T and Precip, but there were problems with T in anticyclones and Foehn Good T, Precip Good Wind, including Vmax. Nice plots Useful but late Experimental

17 Model Subjective Evaluation: Nowcasting technologies: (two best systems in each column marked by red) Ove rall usef uln ess T Forecast accuracy: Pre c Wi nd Gus ts Vis Vis uali zati on (ap pea ran ce) Tim elin ess and reli abil ity Comments Joint (an aggregate of obs., INCA, COSMO, etc.) Good T, including Tmin, Tmax. OK on average. ABOM Graphics was not convenient. CARDS Very good, but with some drop-offs (precip trapped in the mountains). Surprisingly informative graphics. INTW Informative, but graphics was not convenient. INCA minutes Precip became available in mid-january 2014 and were not presented at the project site. Meteo-expert Too few locations with nowcasts, but Meteo-Expert site was useful

18 Some forecasters comments = A lot of forecasts available: helpful but too much info in the operational situation. = Models performed more or less in a similar way: temperature, precip (tendencies, onset/end of precip) more useful, wind, gusts, visibility - poor. = Synoptic-scale forecasts very useful (despite the presence of mesoscale forecasts): NCEP/GFS, UKMO/UM, ECMWF/IFS. = Nowcasts helpful, especially for precipitation. Visibility nowcasts had useful skill, but there were serious failures. The forecasters are very grateful indeed to all the FROST forecast/nowcast providers!

19 Further steps Enhanced quality control of the project observations archive. Additional diagnostic tools and export facilities (inclu. TIGGE-style archiving) on the project web-site Open access for international research community. Validation and intercomparison of the participating forecasting systems, case studies and numerical experiments, assessments of predictability of various weather elements Continuation of developed technologies and transfer of positive experience into operational practice.

20 Project Social and Economic Impacts Socially significant project application areas: Education Transfer of technologies Practical forecasting first guess for operational official forecasts. Informal exchange of ideas, experience etc. Meetings, Blog... Spirit of the project..

21 Thank you! Gratitude to all the participants!

22 Along with traditional verification measures some new scores were implemented. EDI - Extremal Dependence Index EDI = (logf logh) / (logf+logh) EDI is especially recommended for low base-rate thresholds, but it will give a good comparative estimate of accuracy for all thresholds ( Suggested methods for the verification of precipitation forecasts against high resolution limited area observations by the JWGFVR (Laurie Wilson, Beth Ebert et al.) NOTES: - Pictures will refer to thresholds 0.01 and 3, and the last threshold at which any of the three EDI curves remains not interrupted in the 0-36h interval - The base rate has the following approximate values: P(0.01mm/3h)=0.3; P(1mm/3h)=0.2; P(2mm/3h)=0.15; P(3mm/3h)=0.1; P(4mm/3h)=0.055;P(5mm/3h)=0.05

23 COSMO-S14-EPS Highest threshold: 8 mm/3h Lower decision-making level Blue: EDI for 50% probability threshold Green: for 66% Red: for 90%

24 Conclusions, EDI Extremal Dependence Index, EDI, can be used for decision making, especially for rare events when other scores, such as PSS, approach zero. Constructing EDI for different probability decision levels (50, 66, and 90%) showed that the participated EPSs demonstrate skill for all these levels up to the following precipitation thresholds: COSMO-S14-EPS and NMMB-EPS informative up to 8mm/3h; COSMO-RU2-EPS, Harmon-EPS, ALADIN LAEF - informative up to 6mm/3h; GLAMEPS informative up to 4mm/3h. Sampling effects are evident for all the models, especially for higher thresholds of variables. It is not possible to single out the best ensemble producing system, but still some conclusions can be drawn.