The value of probabilistic prediction

Transcription

1 ATMOSPHERIC SCIENCE LETTERS Atmos. Sci. Let. 9: (2008) Published online 15 April 2008 in Wiley InterScience ( The value of probabilistic prediction Roberto Buizza* European Centre for Medium-range Weather Forecasts, UK *Correspondence to: Roberto Buizza, ECMWF, Shienfield Park, Reading, RG2-9AX, United Kingdom, UK. Received: 28 September 2007 Revised: 19 December 2007 Accepted: 30 January 2008 Abstract Probabilistic forecasts designed to estimate the probability density function of forecast states are potentially more valuable than single forecasts because (1) they can predict not only the most likely outcome but also the probability of occurrence of extreme and rare events and (2) probabilistic forecasts issued on consecutive days are usually more consistent than corresponding single forecasts. In this communication, the potential economic value of probabilistic and single forecasts are compared, and single and probabilistic forecasts of precipitation for a flooding event are analyzed to illustrate the potential value of consistency between successive forecasts. Copyright 2008 Royal Meteorological Society Keywords: forecast value; probabilistic prediction; ensemble prediction 1. Single- and probabilistic-based approaches to the prediction problem Ensemble systems, capable of estimating the probability distribution of forecast states can be used not only to identify the most likely outcome as was done in the past using single, high-resolution forecasts but also to assess the probability of occurrence of weather event of interest. In particular, a probabilistic forecast can be used to assess the probability of occurrence of events linked with maximum acceptable losses, and can thus be more valuable than single forecasts in weatherrisk management. Following the operational implementation of global ensemble predictions systems in the 1990s at the European Centre for Medium-range Weather Forecasts (ECMWF, Molteni et al., 1996), the National Centres for Environmental Predictions (NCEP, Tracton and Kalnay, 1993) and the Meteorological Service of Canada (MSC, Houtekamer et al., 1996), other centres implemented ensembles systems in the 2000s. Today, ten operational centres (based in Australia, Brazil, Canada, China, England, Japan, Korea, and the USA) run global ensemble prediction systems for up to 10 days. Following the availability of ensemble-based probabilistic weather forecasts and indications of their values, the past decade saw the development and implementation of hydrological probabilistic forecasting systems (e.g. Gouweleeuw et al., 2005; Schaake et al., 2006). A key role in the development and promotion of the use of a probabilistic approach in ensemble has been played by HEPEX (the Hydrological Ensemble Prediction Experiment) established in 2004, it has fostered the communication between meteorologists and hydrologists, and promoted the establishment of test-bed projects to assess the potential value of ensemble prediction in hydrological applications (see the HEPEX web page for more details at scope of this communication is to discuss two of the advantages of ensemble-based probabilistic forecasts compared to single ones. The first advantage, already mentioned above, is linked to the fact that ensemble systems allow to predict not only the most likely scenario but also the probability of occurrence of any event, as can be quantified using the potential economic value (Richardson, 2000; Buizza, 2001). The second advantage is linked to the fact that ensemble systems can provide forecasters with more consistent (i.e. less changeable) successive forecasts: this fact is illustrated considering one case study. After this introduction, a brief review of the ECMWF approach to ensemble meteorological prediction and of the use of the ECMWF ensemble forecasts in hydrological ensemble prediction systems is discussed in Section 2. The seasonal average potential economic value of single and ensemble forecasts of meteorological values are compared in Section 3. The importance of having consistent successive forecasts is discussed in Section 4, considering one case study. Finally, some conclusions are drawn in Section The meteorological and hydrological ensemble approaches to probabilistic prediction In meteorology, ensemble prediction systems based on a finite number of deterministic integrations appear to be, so far, the only feasible method to predict this probability density function beyond the range of linear error growth. These systems have been designed to simulate the effect on initial and model uncertainties on forecast states. When first implemented in 1992 (note that at the time of writing, ECMWF is celebrating 15 years of operational ensemble prediction), the ECMWF Ensemble Prediction System (EPS, Molteni et al., 1996) was based on 33 forecasts produced with a T63L19 (spectral triangular truncation Copyright 2008 Royal Meteorological Society

2 The value of probabilistic prediction 37 T63 with 19 vertical levels) resolution version of the ECMWF model. The initial uncertainties were simulated by starting 32 members from perturbed initial conditions defined by the fastest growing perturbations (the singular vectors of the tangent forward model version, Buizza and Palmer, 1995). Between December 1992 and September 2006, the EPS was upgraded several times, benefiting both from changes of the ECMWF data assimilation and forecasting system, and from modifications of the EPS configuration designed to improve the simulation of initial and model uncertainties. Among these changes, it is worth reminding that the introduction, in 1998, of a stochastic scheme to simulate the effect of model uncertainties on the forecast probability; the resolution increases in 1996 and 2000, and the increase in membership from 33 (32 perturbed plus a control integration) to 51 in More recently, in 2006 the EPS was upgraded to the new variable resolution EPS (variable resolution ensemble prediction system (VAREPS), Buizza et al., 2007), characterized by a T L 399L62 (spectral triangular truncation T L 399 with linear grid and 62 vertical levels) resolution between forecast day 0 and 10, and T L 255L62 between day 10 and 15. Since September 2006, VAREPS has been providing ECMWF users with 15-day global ensemble forecasts twice a-day, with initial times 0000UTC and 1200UTC (at the time of writing, September 2007, this is still the operational system). Hydrological ensemble prediction models have been developed either to use directly global meteorological ensemble forecasts as input, or to use dynamically downscaled meteorological ensemble forecasts, e.g. generated by meso-scale ensemble prediction system nested in global ensembles. In hydrological ensemble systems, observations are used to estimate the initial states, and an ensemble of weather forecasts (of variables such as precipitation, surface temperature, soil moisture, and snow) is used to predict the future hydrological conditions. For example, the European Flood Alert System (EFAS), developed and run in preoperation mode at the Joint Research Centre of Ispra (De Roo et al., 2003; Thielen et al., 2006), uses the ECMWF global ensemble forecasts to generate an ensemble of river discharge forecasts for all European river catchments larger than 4000 km 2. EFAS forecasts are used as a prealert to allow the receiving authorities to be aware of the possibility of a flood to take place. The reader is referred, e.g. Diomede et al. (2007); Grossi et al. (2007); Hou et al. (2007); Voisin et al. (2007) and Pietroniro et al. (2007) for a review of similar hydrological systems. 3. The Potential Economic Value : a metric of forecast value One way to quantify the potential value of a forecasting system is to use the concept of potential economic value, estimated using simple cost/loss models (Richardson, 2000; Buizza, 2001). Consider a user who has access to a forecast, and who can take a protection action of cost C to avoid a loss L (note that L denotes only the avoidable losses not all the losses that a user can incur). This forecast can lead to four possible outcomes (Table I). The forecast system can produce two right decisions, a hit and an inverse hit: in these cases, an event (or no event) which is forecast is also observed (or not observed). Alternatively, the system can produce two false decisions, a false alarm and a failure. A failure means that no event has been forecast, but the event occurs. According to this simple cost/loss decision model the potential economic value of a forecast system can be calculated by combining the outcomes of the decision-making process with an economic decision model like the static cost/loss model approach. In this cost/loss model, a hit and a false alarm are associated with a cost C, since an alarm causes the user to protect his environment against the event at a cost C. By contrast, if no alarm is given, no protective action is done: if the event is not observed, no loss occurs, but if the event occurs the user faces alossl. Considering Table I, the mean expense E f that users with a cost-loss ratio C /L face can be calculated in the following way: E f = a + b C + c n n L (1) where a, b and c are defined in Table I. This average expense can be compared with the average expense of a reference forecast E c : E c = min(e a, E n ) (2) where E c is the minimum expense of the two following decisions: either (1) the user always protects if the climatological base rate of an event s is smaller than the cost-loss ratio E a = C, thus incurring an expense E a, or (2) the user never protects if s is greater than the cost-loss ratio E n = s L, thus incurring an expense E n. Since for some variables, e.g. for synoptic scale meteorological variables or for hydrological discharges, the forecasts are highly auto-correlated, a persistence forecast might be successful in predicting an event, especially for the next forecast time-step. In Table I. Possible Outcomes (hit a, false alarm b, failure c and inverse hit d) and expense matrix (protection cost C, lossl) of a decision-making process for a decision maker that takes a protective action or not. A hit and a false alarm a related with acostc, whereas a failure of the system causes a loss L Cost/Loss model Yes Observation Forecast Yes a Cost C b Cost C No c Cost L d Cost 0 No

3 38 R. Buizza this case, the average expense of a persistence forecast E p can be used as reference: E p = a p + b p C + c p L (3) n p n p where a p is the number of hits, b p is number of false alarms, c p is the number of failures and n p is the number of forecasts. In this case, the reference expense can be defined in the following way: E 0 = min(e a, E n, E p ) (4) Note that the average expense sustained by using a perfect forecast system E 1 is given by: E 1 = s C (5) The average reference expense associated with the forecast E f can be transformed into its potential economic value PEV f using the average expenses of the reference forecast E c and of the perfect forecast E 1 : PEV f = E 0 E f. (6) E 0 E 1 PEV f ranges between minus infinity to 1: a forecast that is better than the reference has positive PEV, and a perfect forecast has PEV f = 1. Equation (6) is very similar to the formulation of Richardson (2000), the only difference being that the reference forecast is extended by the persistence forecast. Figure 1. Average potential economic value of the ECMWF operational t +144-hour forecasts given by the Ensemble Prediction System (blue-dashed lines) and the single EPS-control (solid red lines) for the prediction of precipitation in excess of 5 (top-left panel, (a)) and 20 (top-right panel (b)) mm/day, and of 850-hPa temperature anomalies warmer/colder than 4 C (bottom-left/bottom-right panels (c,d)). Averages have been taken for May June July 2007 over Europe. Precipitation forecasts have been verified against 24-hour high-resolution forecasts, and 850-hPa temperature forecasts against analysis. The abscissa reports the cost/loss ratios from 0 to 1, and the ordinates report the potential economic value, from 0 to 1 for a perfect forecast.

4 The value of probabilistic prediction Figure 1 shows the potential economic value of single EPS-controlled forecasts (i.e. forecasts given by the single EPS member started from the unperturbed analysis) and of EPS probabilistic predictions for four events: total precipitation in excess of 5 and 20 mm/day, and 850-hPa temperature anomalies warmer/colder than 4 C (these variables were selected because precipitation and temperature are two key drivers of hydrological discharge models). Figure 1 shows that for all four predictions the potential economic value of EPS probabilistic forecasts is higher. As discussed in Richardson (2000) and Buizza (2001), this is due to the fact that the information content of a probabilistic forecast is higher than the information 39 content of a single (yes/no) forecast. The end result is that users can manage weather-related risks better if they are given access to the whole probability distribution function. 4. On the consistency of successive single and ensemble forecasts: the case of the flood of Northern Italy of October 2000 Consistency between forecasts issued on consecutive days is a desirable property of a forecasting system. Figures 2 and 3 illustrate the importance of consistency for the prediction of 48-hour precipitation accumulated between 14 and 16 October During this Figure 2. Top-left panel: EPS-control 144-to-192 hour forecast of 48-hour accumulated precipitation issued on 8 October and valid for October Top-right panel: as top-left but for the 96-to-144 hour forecast issued on 10 October and valid for the same period. Bottom-left panel: as top-left but for the 48-to-96 hour forecast issued on 12 October and valid for the same period. Bottom-right panel: 48-hour accumulated precipitation observed between 14 and 16 October Contour isolines: 2, 25, 50, and 100 mm. Copyright 2008 Royal Meteorological Society Atmos. Sci. Let. 9: (2008)

5 40 period, intense precipitation caused the Po river to flood parts of Northern Italy, causing death, the evacuation of people from flooded areas, and a lot of damages. This event was the most severe flood of the Po river after the terrible events of 1951, which lead to an even higher death toll and to the evacuation of about people from flooded areas. Single control forecasts (Figure 2) of 48-hour precipitation issued on 8, 10 and 12 of October and valid from 14 to 16 October were rather inconsistent. The 144-to-192 hour forecast issued on the 8th predicted more than 50 mm of rain over Central Italy and the Northern Adriatic Sea rather than over Northwestern Italy where 50 mm (actually up to 100 mm, see R. Buizza Figure 2 bottom-right panel) of rain was observed (the observed map has been constructed interpolating data from synop stations on the regular 0.5 degree resolution grid where forecasts have been generated). The 96-to-144 hour forecast issued on the 10th moved the area with more than 50 mm of precipitation over the whole of Northern Italy, more in agreement with the observed pattern. The 48-to-96 hour forecast kept the area with more than 25 mm of precipitation over the whole of Northern Italy, but reduced the area with more than 50 mm to a single point. Making a prediction using these successive forecasts would have been rather difficult, due to these changes in the area and the amount of predicted rainfall. Figure 3. Top-left panel: EPS 144-to-192 hour probability of more than 50 mm of rain forecast issued on 8 October and valid for October Top-right panel: as top-left but for the 96-to-144 hour probability forecast issued on 10 October and valid for the same period. Bottom-left panel: as top-left but for the 48-to-96 hour probability forecast issued on 12 October and valid for the same period. Bottom-right panel: 48-hour accumulated precipitation observed between 14 and 16 October Contour isolines for probabilities: 2, 10, 20, 30, 60%; contour isolines for observed precipitation: 2, 25, 50, and 100 mm. Copyright 2008 Royal Meteorological Society Atmos. Sci. Let. 9: (2008)

6 The value of probabilistic prediction 41 By contrast, EPS probabilistic forecasts of precipitation in excess of 50 mm (Figure 3) issued on the same dates were more consistent. The 144-to- 196 forecast gave a 2 10% probability of more than 50mm of rain would fall over Northwestern Italy. The 96-to-144 hour forecast gave a higher probability of 10 20% over Northern Italy, with a 20 30% probability over Liguria (the strip around 44 N latitude and 7 10 E longitude) and a 30 60% probability over Northeastern Italy. Finally, the 48-to-96 hour forecast gave a 30 50% probability over Northwestern Italy, in the region where more than 50 mm of rainfall was observed, and reduced the probability over Northeastern Italy to 10 20%. Figure 3 indicates that there is a more consistent gradual refinement of the area with higher probabilities of intense precipitation towards the area where intense precipitation was observed, with probabilities consistently increasing. It is also interesting to point out that the combined ensemble-based probabilistic forecasts and single forecasts can help the forecasters to judge the potential forecast skill of the single forecast: regions with smaller probabilities are regions where the ensemble members differ most, and thus are the ones more uncertain and less predictable. Disagreements between probabilistic and single forecasts can be used as an indication of potentially low predictability. For example, the disagreement between the 144-to-196 hour probability and the EPS-control forecast that more than 50 mm of rain would hit Central Italy and the Northern Adriatic Sea can be used as an indication that the control forecast has a high chance to be wrong. By contrast, the agreement between the 96- to-144 hour probability and EPS-control forecast can be used as an indication that the Northwestward shift of the area of intense precipitation towards Northern Italy has a high chance to be right. Furthermore, the agreement between the 48-to-96 hour and the 96-to- 144 hour probabilities can be used as an indication that there is a high chance that Northern rather than Central Italy was going to be hit by intense precipitation. By contrast, the disagreement between the 48-to-96 hour probability of precipitation in excess of 50 mm and the EPS-control forecast can be used as an indication that the 48-to-96 hour control forecast was most likely underestimating the precipitation amount. 5. Conclusions, and ongoing research at ECMWF Two of the main advantages of ensemble-based probabilistic forecasts compared to single forecasts have been discussed: first, the fact that an ensemble system predicts not only the most likely scenario but also the probability of occurrence of any event, and second the fact that an ensemble system can provide more consistent (i.e. less changeable) successive forecasts. The first point has been discussed comparing the average potential economic value of four types of forecasts given either by the single EPS-control forecast or EPS probabilities. The second point has been illustrated discussing precipitation forecasts for the case of the flood of the Po river of October 2000 (although the skill of an ensemble system can only be judged considering a large sample of cases, and thus no statistically significant conclusions can be drawn, this case illustrates the advantage of having more consistent, ensemble-based forecasts). Despite all the recent progresss in probabilistic prediction (see Palmer et al., 2007 for a review of the status of the ECMWF ensemble system), it should be stated that the current operational ensemble prediction systems still suffer from some limitations. One of them is that their spatial resolution is still less then the one of single high-resolution systems (at ECMWF the two systems have a spatial resolution equivalent to 50 and 25 km, but the difference in resolution is even higher for other centres), a fact that makes it difficult to use ensemble forecasts to complement single higher-resolution forecasts. Another limitation, which is also shared by single high-resolution forecasts, is that forecasts of weather variables that drive hydrological models (e.g. rainfall, surface temperature) are still problematic, often not quantitatively accurate enough, requiring the post-processing and calibration of weather forecasts before using them in hydrological systems. On this second point, ECMWF is planning for the beginning of 2008 the operational implementation of a reforecast suite, which should help users to calibrate ensemble weather forecasts before they are used to drive hydrological models. In fact, as Hamill et al. (2004) has shown, reforecast datasets can be used to estimate in a more accurate way the ensemble model climatology, and can thus reduce the problem. At ECMWF, work is also progressing in two other areas to further improve the performance of the ensemble system: in the simulation of initial uncertainties, by exploring the use of ensemble data assimilation methods combined with singular vectors and by improving the tangent forward and adjoint physics in the singular vector computation and in the simulation of model errors, by testing new stochastic schemes. These ongoing works should make it easier to integrate meteorological and hydrological systems, and provide users with better, more integrated metrological/hydrological forecast information. References Buizza R Accuracy and economic value of categorical and probabilistic forecasts of discrete events. Monthly Weather Review 129: Buizza R, Palmer TN The singular-vector structure of the atmospheric global circulation. Journal of the Atmospheric Sciences 52: Buizza R, Bidlot J-R, Wedi N, Fuentes M, Hamrud M, Holt G, Vitart F The new ECMWF VAREPS. Quarterly Journal of the Royal Meteorological Society 133: De Roo A, Gouweleeuw B, Thielen J, Bartholmes J, Bongiovanni- Cerlini P, Todini E, Bates PD, Horritt M, Hunter N, Beven K,

7 42 R. Buizza Pappenberger F, Heise E, Rivin G, Hils M, Hollingsworth A, Holst B, Kwadjik J, Reggiani P, Van Dijk M, Sattler K, Sprokkereff E Development of a European flood forecasting system. International Journal of River Basin Management 1(1): Diomede T, Marsigli C, Montani A, Paccagnella T Discharge ensemble forecasts based on the COSMO-LEPS quantitative precipitation. In Book of Abstracts of the 3 rd HEPEX Workshop, Stresa, Italy, June 2007, European Commission EUR22861EN, (available from JRC, Ispra). Gouweleeuw BT, Thielen J, Franchello G, De Roo APJ, Buizza R Flood forecasting using medium-range probabilistic weather prediction. Hydrology and Earth System Sciences (HESS) 9: Grossi G, Bacchi B, Buizza R, Ranzi R Hydrological ensemble prediction systems: the 1966 flood experiment. In Book of Abstracts of the 3 rd HEPEX Workshop, Stresa, June 2007, European Commission EUR22861EN, (available from JRC, Ispra). Hamill TM, Whitaker JS, Wei X Ensemble re-forecasting: improving medium-range forecast using retrospective forecasts. Monthly Weather Review 132: Hou D, Mitchell K, Toth Z, Lohman D, Wei H Ensemble streamflow forecasting with the coupled GFS-NOAH modelling system. In Book of Abstracts of the 3 rd HEPEX Workshop, Stresa, June 2007, European Commission EUR22861EN, 65 69, (available from JRC, Ispra). Houtekamer PL, Derome J, Ritchie H, Mitchell HL A system simulation approach to ensemble prediction. Monthly Weather Review 124: Molteni F, Buizza R, Palmer TN, Petroliagis T The ECMWF ensemble prediction system: methodology and validation. Quarterly Journal of the Royal Meteorological Society 122: Palmer TN, Buizza R, Leutbecher M, Hagedorn R, Jung T, Rodwell M, Vitart F, Berner J, Hagel E, Lawrence A, Pappenberger F, Park Y-Y, van Bremen L, Gilmour I, Smith L The ECMWF Ensemble Prediction System: recent and on-going developments. In A paper presented at the 36th Session of the ECMWF Scientific Advisory Committee. ECMWF Research Department Technical Memorandum n. 540, available from ECMWF, Shinfield Park, Reading RG2-9AX. Pietroniro A, Fortin V, Kouwen N, Neal C, Turcotte R, Davison B, Verseghy D, Soulis ED, Caldwell R, Evora N, Pellerin P Using the MESH modelling system for hydrological ensemble forecasting of the Laurentian Great Lakes at the regional scale. Hydrology And Earth System Sciences 11: Richardson DS Skill and economic value of the ECMWF Ensemble Prediction System. Quarterly Journal of the Royal Meteorological Society 126: Schaake J, Franz K, Bradley A, Buizza R The hydrological ensemble prediction experiment (HEPEX). Hydrology and Earth System Sciences Discussion 3: Thielen J, Bartholmes J, Ramos M-H, Kalas M, van der Knijff J, De Roo A Added value of ensemble prediction system products for medium-range flood forecasting on European scale, In Proceedings of the workshop Ensemble Predictions and Uncertainties in Flood Forecasting, International Commission for the Hydrology of the Rhine Basin (CHR), Bern Switzerland, March 2006, Tracton MS, Kalnay ES Operational ensemble prediction at the National Meteorological Center: practical aspects. Weather and Forecasting 8: Voisin N, Woods AW, Lettenmaier DP Downscaling global medium-range meteorological predictions for flood predictions. In Book of Abstracts of the 3 rd HEPEX Workshop, Stresa, June 2007, European Commission EUR22861EN, 134, (available from JRC, Ispra).