Probabilistic Weather Forecasting and the EPS at ECMWF

Probabilistic Weather Forecasting and the EPS at ECMWF Renate Hagedorn European Centre for Medium-Range Weather Forecasts 30 January 2009: Ensemble Prediction at ECMWF 1/ 30

Questions What is an Ensemble Prediction System (EPS)? multiple forecasts from slightly different initial conditions enables probabilistic forecasts Why do we need an EPS and probabilistic forecasts? to account for uncertainties in initial conditions and model error to support user specific decision-making processes How do we use probabilistic forecasts? products (stamp maps, probability maps, EFI, EPSgrams) important verification aspects 30 January 2009: Ensemble Prediction at ECMWF 2/ 30

Deterministic Forecasting Temperature Is this forecast correct? Initial Uncertainty Initial condition Forecast time Forecast Model Error 30 January 2009: Ensemble Prediction at ECMWF 3/ 30

Ensemble Forecasting Temperature Initial condition Forecast time Forecast Complete description of weather prediction in terms of a Probability Density Function (PDF) 30 January 2009: Ensemble Prediction at ECMWF 4/ 30

Flow dependence of forecast errors 26 th June 1995 26 th June 1994 ECMWF ensemble forecast - Air temperature Date: 26/06/1995 London Lat: 51.5 Long: 0 ECMWF ensemble forecast - Air temperature Date: 26/06/1994 London Lat: 51.5 Long: 0 Control Analysis Ensemble Control Analysis Ensemble 30 34 28 30 26 28 24 26 22 24 20 22 18 20 16 18 14 16 12 14 10 8 UK 12 10 UK 0 1 2 3 4 5 6 7 8 9 10 Forecast day 0 1 2 3 4 5 6 7 8 9 10 Forecast day If the forecasts are coherent (small spread) the atmosphere is in a more predictable state than if the forecasts diverge (large spread) 30 January 2009: Ensemble Prediction at ECMWF 5/ 30

Multi-Model Ensemble Forecasting model 1 verification climate system phase space t=t t=0 model 3 Model 2 model 2 30 January 2009: Ensemble Prediction at ECMWF 6/ 30

Multi-Model Ensemble Forecasting verification climate system phase space t=t t=0 multi-model ensemble 30 January 2009: Ensemble Prediction at ECMWF 7/ 30

THORPEX Interactive Grand Global Ensemble A key component of THORPEX (to accelerate the improvements in the 1-day to 2-week high-impact weather forecasts for the benefit of humanity) Enhance international collaboration on ensemble prediction for severe weather Develop theory and practice of multi-model ensembles Develop the concept of a Global Interactive Forecasting System (GIFS), responding dynamically to changing uncertainty TIGGE archive: Global operational ensemble forecasts from 10 centres: BMRC (Australia), CMA (China), CPTEC (Brazil), ECMWF (Europe), JMA (Japan), KMA (Korea), Météo-France (France), Met Office (UK), MSC (Canada), NCEP (USA) 30 January 2009: Ensemble Prediction at ECMWF 8/ 30

SM vs. MM results T-2m, 250 European stations 2008060100 2008073000 (60 cases) 0.4 0.2 Multi-Model ECMWF Met Office NCEP CRPSS 0.0-0.2 Solid: no BC 0 2 4 6 8 10 Lead time / days 30 January 2009: Ensemble Prediction at ECMWF 9/ 30

SM vs. MM results T-2m, 250 European stations 2008060100 2008073000 (60 cases) 0.4 0.2 Multi-Model ECMWF Met Office NCEP CRPSS 0.0-0.2 Dotted: 30d-BC Solid: no BC 0 2 4 6 8 10 Lead time / days 30 January 2009: Ensemble Prediction at ECMWF 10 / 30

SM vs. MM results T-2m, 250 European stations 2008060100 2008073000 (60 cases) 0.4 0.2 Multi-Model ECMWF Met Office NCEP CRPSS 0.0-0.2 Dashed: REFC-NGR Dotted: 30d-BC Solid: no BC 0 2 4 6 8 10 Lead time / days 30 January 2009: Ensemble Prediction at ECMWF 11 / 30

Goal of Ensemble Prediction Represent uncertainty of prediction Ensemble Spread should o capture truth (spread ~ RMS error) o indicate range of uncertainty Move from deterministic to probabilistic forecast 30 January 2009: Ensemble Prediction at ECMWF 12 / 30

Why Probabilities? Energy contract scenario: penalty for non-delivery: 100 (if ff>25m/s), buying in energy: 20 weather forecast: 30% probability for ff>25m/s what would you do? Test the system for 100 cases of forecasts predicting 30% probability for the event ff>25m/s to occur Note: statistically, 30 times the event should occur, 70 times not occur buy extra energy in these 100 cases -> 100 x 20 = 2000 do not buy extra energy in these 100 cases -> 30 x 100 = 3000 Buying in extra energy beforehand (spending 20 ) is beneficial when probability for ff>25m/s is greater 20% The higher/lower the cost loss ratio, the higher/lower probabilities are needed in order to benefit from action on forecast 30 January 2009: Ensemble Prediction at ECMWF 14 / 30

Defining the cost-loss ratio Defining the specific costs and losses associated to your application Can prove to be a non-trivial problem Should be done in consultation with economic experts Examples for research and applications in this area exist, e.g.: http://www.iiasa.ac.at/research/rav/projects/gov-fair.html?sb=4 The Decisions and Governance theme investigates how the presence of risk and uncertainty influences the design of successful policies in the areas of environmental management and climate change. In Europe, the group has identified successful ways of involving citizens in the design of programmes to reduce flood risk, and is now applying these lessons to similar efforts in China and Japan 30 January 2009: Ensemble Prediction at ECMWF 15 / 30

Ensemble Prediction System 1 control run + 50 perturbed runs (T L 399 L62) added dimension of ensemble members f(x,y,z,t,e) How do we deal with added dimension when interpreting, verifying and diagnosing EPS output? Transition from deterministic (yes/no) to probabilistic 30 January 2009: Ensemble Prediction at ECMWF 16 / 30

EPS products: stamp maps AN 19871016, 06GMT 956 EPS Cont FC +66 h 979 Mean sea level pressure (5hPa) VT: 16 Oct 1987, 6 UTC TL399 ensemble, TL95 moist SVs with t_opt=24h - mem no. 1 of 51 +66 h - mem no. 2 of 51 +66 h - mem no. 3 of 51 +66 h - mem no. 4 of 51 +66 h - mem no. 5 of 51 +66 h - mem no. 6 of 51 +66 h - mem no. 7 of 51 +66 h - mem no. 8 of 51 +66 h - mem no. 9 of 51 +66 h - mem no. 10 of 51 +66 h 984 963 968 978 981 979 962 988 966 969 981 984 - mem no. 11 of 51 +66 h - mem no. 12 of 51 +66 h - mem no. 13 of 51 +66 h - mem no. 14 of 51 +66 h 979 - mem no. 15 of 51 +66 h - mem no. 16 of 51 +66 h - mem no. 17 of 51 +66 h - mem no. 18 of 51 +66 h - mem no. 19 of 51 +66 h 984 - mem no. 20 of 51 +66 h 964 965 990 965 976 970 962 - mem no. 21 of 51 +66 h - mem no. 22 of 51 +66 h - mem no. 23 of 51 +66 h - mem no. 24 of 51 +66 h - mem no. 25 of 51 +66 h - mem no. 26 of 51 +66 h - mem no. 27 of 51 +66 h - mem no. 28 of 51 +66 h - mem no. 29 of 51 +66 h - mem no. 30 of 51 +66 h 979 982 979 967 961 966 970 975 964 - mem no. 31 of 51 +66 h 980 - mem no. 32 of 51 +66 h - mem no. 33 of 51 +66 h - mem no. 34 of 51 +66 h - mem no. 35 of 51 +66 h - mem no. 36 of 51 +66 h - mem no. 37 of 51 +66 h 972 - mem no. 38 of 51 +66 h - mem no. 39 of 51 +66 h 978 - mem no. 40 of 51 +66 h 964 983 980 974 972 981 964 988 978 960 - mem no. 41 of 51 +66 h - mem no. 42 of 51 +66 h - mem no. 43 of 51 +66 h - mem no. 44 of 51 +66 h - mem no. 45 of 51 +66 h - mem no. 46 of 51 +66 h - mem no. 47 of 51 +66 h - mem no. 48 of 51 +66 h - mem no. 49 of 51 +66 h - mem no. 50 of 51 +66 h 988 985 980 977 986 979 976 980 958 968 987 963 989 30 January 2009: Ensemble Prediction at ECMWF 17 / 30

5 Probability of 10m windspeed 15 m/s Wednesday 21 January 2009 00UTC ECMWF Forecast probability t+084 VT: Saturday 24 January 2009 12UTC Surface: 10 metre Wind speed of at least 15 m/s 60 W 40 W 20 W 0 20 E 40 E 60 E 100 65 5 65 5 95 60 N 5 65 5 50 N 65 5 65 5 5 40 N 35 30 N 5 60 W 40 W 20 W 0 20 E 40 E 60 E 30 January 2009: Ensemble Prediction at ECMWF 18 / 30

Extreme Forecast Index Wednesday 21 January 2009 00UTC ECMWF Extreme forecast index t+072-096 VT: Saturday 24 January 2009 00UTC - Sunday 25 January 2009 00UTC Surface: 10 metre speed index 60 W 40 W 2000 1000 0.3 0.4 0.5 0.6 0.7 20 W 0 70 N 20 E 40 E 0.3 60 E 60 E 1 0.3 0.9 0.5 0.4 60 N 0.6 0.3 0.5 0.3 0.4 0.8 0.7 0.6 0.4 0.3 50 N 0.4 1000 0.4 0.5 0.6 0.3 1000 1000 0.3 0.5 40 E 0.7 0.5 0.4 30 N 0.3 0.4 40 N 0.6 0.4 0.6 20 W 0 0.8 0.6 0.5 0.4 0.3 0.4 20 E 0.3 30 N 0.5 30 January 2009: Ensemble Prediction at ECMWF 19 / 30

EPS products: EPSgram EPS Meteogram Reading (48m) 51.46 N 1.33 W Deterministic Forecast and EPS Distribution Wednesday 23 January 2008 00 UTC EPS Meteogram Reading (48m) 51.58 N 1 W Extended Range Forecast based on EPS Distribution Wednesday 23 January 2008 00 UTC Total Cloud Cover (okta) 8 Daily mean of Total Cloud Cover (okta) 8 6 6 4 4 2 2 0 Total Precipitation (mm/6h) 8 6 4 2 0 10m Wind Speed (m/s) 15 12 9 6 3 2m Temperature reduced to station height ( C) 115m (T799) 105m (T399) 12 9 6 3 0 Total Precipitation (mm/24h) 24 21 18 15 12 9 6 3 0 Daily distribution of 10m Wind Direction Daily mean of 10m Wind Speed (m/s) 12 10 8 6 4 2 2m min/max temperature (C) reduced to station height 90m (T255) 12 9 6 3 0% 25% 50% 75% 100% -3 Wed 23 Thu 24 Fri 25 Sat 26 Sun 27 Mon 28 Tue 29 Wed 30 Thu 31 Fri 1 max 90% January 2008 75% median T799 OPS T399 CTRL 25% 10% min Magics++ 2.4.0 0-3 Wed 23 Thu 24 Fri 25 Sat 26 Sun 27 Mon 28 Tue 29 Wed 30 Thu 31 Fri 1 Sat 2 Sun 3 Mon 4 Tue 5 Wed 6 max 90% 75% January 2008 February 2008 median 25% 10% min Magics++ 2.4.0 30 January 2009: Ensemble Prediction at ECMWF 20 / 30

Assessing the quality of a forecast The forecast indicated 10% probability for rain It did rain on the day Was it a good forecast? Yes No I don t know Single probabilistic forecasts are never completely wrong or right (unless they give 0% or 100% probabilities) To evaluate a forecast system we need to look at a (large) number of forecast observation pairs 30 January 2009: Ensemble Prediction at ECMWF 21 / 30

Assessing the quality of a forecast system Characteristics of a forecast system: Consistency*: Do the observations statistically belong to the distributions of the forecast ensembles? (consistent degree of ensemble dispersion) Reliability: Can I trust the probabilities to mean what they say? Sharpness: How much do the forecasts differ from the climatological mean probabilities of the event? Resolution: How much do the forecasts differ from the climatological mean probabilities of the event, and the systems gets it right? Skill: Are the forecasts better than my reference system (chance, climatology, persistence, )? * Note that terms like consistency, reliability etc. are not always well defined in verification theory and can be used with different meanings in other contexts 30 January 2009: Ensemble Prediction at ECMWF 22 / 30

Rank Histogram Rank Histograms asses whether the ensemble spread is consistent with the assumption that the observations are statistically just another member of the forecast distribution Check whether observations are equally distributed amongst predicted ensemble Sort ensemble members in increasing order and determine where the observation lies with respect to the ensemble members Rank 1 case Rank 4 case Temperature -> Temperature -> 30 January 2009: Ensemble Prediction at ECMWF 23 / 30

Rank Histograms OBS is indistinguishable from any other ensemble member OBS is too often below the ensemble members (biased forecast) OBS is too often outside the ensemble spread A uniform rank histogram is a necessary but not sufficient criterion for determining that the ensemble is reliable (see also: T. Hamill, 2001, MWR) 30 January 2009: Ensemble Prediction at ECMWF 24 / 30

Reliability A forecast system is reliable if: statistically the predicted probabilities agree with the observed frequencies, i.e. taking all cases in which the event is predicted to occur with a probability of x%, that event should occur exactly in x% of these cases; not more and not less. A reliability diagram displays whether a forecast system is reliable (unbiased) or produces over-confident / underconfident probability forecasts A reliability diagram also gives information on the resolution (and sharpness) of a forecast system Forecast PDF Climatological PDF 30 January 2009: Ensemble Prediction at ECMWF 25 / 30

Reliability Diagram Take a sample of probabilistic forecasts: e.g. 30 days x 2200 GP = 66000 forecasts How often was event (T > 25) forecasted with X probability? FC Prob. # FC perfect FC OBS-Freq. real OBS-Freq. 25 100% 8000 8000 (100%) 7200 (90%) 90% 5000 4500 ( 90%) 4000 (80%) 80% 4500 3600 ( 80%) 3000 (66%).... 25........ 10% 5500 550 ( 10%) 800 (15%) 25 0% 7000 0 ( 0%) 700 (10%) 30 January 2009: Ensemble Prediction at ECMWF 26 / 30

Reliability Diagram Take a sample of probabilistic forecasts: e.g. 30 days x 2200 GP = 66000 forecasts How often was event (T > 25) forecasted with X probability? 100 OBS-Frequency 0 0 FC-Probability 100 FC Prob. # FC perfect FC OBS-Freq. real OBS-Freq. 100% 8000 8000 (100%) 7200 (90%) 90% 5000 4500 ( 90%) 4000 (80%) 80% 4500 3600 ( 80%) 3000 (66%)............ 10% 5500 550 ( 10%) 800 (15%) 0% 7000 0 ( 0%) 700 (10%) 30 January 2009: Ensemble Prediction at ECMWF 27 / 30

Reliability Diagram over-confident model perfect model 30 January 2009: Ensemble Prediction at ECMWF 28 / 30

Reliability Diagram under-confident model perfect model 30 January 2009: Ensemble Prediction at ECMWF 29 / 30