David Demeritt, Sébastien Nobert, Hannah Cloke. Seminar presentation to European Centre for Medium Range Weather Forecasting, 18 February 2010

Transcription

1 Communicating Ensemble Forecasts: Reflections on the European Experience David Demeritt, Sébastien Nobert, Hannah Cloke King s College London, Department of Geography Seminar presentation to European Centre for Medium Range Weather Forecasting, 18 February 2010

2 Outline of Talk Today 1. Background to study 2. Flood forecaster perceptions of EPS & EFAS 3. Understandings of uncertainty and use of EPS in flood forecasting 4. Communicating ensembles to non-experts 5. Does EPS improve operational flood warning & management? 6. Summary and conclusions

3 1. Background: Preliminary findings from a larger study Europeanizing flood forecasting and the geographies of risk and science in the EU (RES ) Project Aims: 1. Explain the institutional politics of flood and disaster management in Europe; 2. Document the beliefs and practices involved in flood forecasting and explore their implications for managing floods in the face of uncertainty; 3. Enrich the understanding and communication of ensemble predictions (EPs) and advise EFAS about improving its specific role in flood risk assessment.

4 Promises of Ensemble Forecasting the use of meteorological l ensembles to produce sets of hydrological predictions increased the capability to issue flood warnings (Balint et al., 2006, p.67) The hydrological ensemble predictions have greater skill than deterministic ones. (Roulin, 2007) The use of EPS in hydrological forecasting proved to be of great added value to a flood early warning system,, as the EPS-based forecasts showed in general higher skill than the deterministic-based ones. (Bartholmes et al., 2008) Are these technical promises realized in operational practice?

5 Research Design and Evidence Base 55 interviews i with forecasting experts, water authorities and CPAs in 17 European countries involved in flood forecasting, warning and emergency response Participant observation at forecasting workshops & conferences and during visits to forecasting centres All interview quotations are anonymized what we call our data are really our constructions of other people s constructions of what they are up to (Geertz 1973: 9)

6 Methodology: Qualitative Social Science Not an experimental science in search of laws; an interpretative one in search of understanding, or Verstehen Different objects of study, different methodologies Indifferent kinds (ie. rocks and other unconscious objects of study) Interactive kinds (people: objects who are also conscious subjects) Natural science Social science

7 Accessing End User Understandings Solicited it for broad interviews i about operational flood warning and response Promised anonymity Enthusiastic to show us around positionality : scientific novice, institutional independence informants can express ignorance without losing face in expert community Offer more frank assessment of individuals and institutions in world of hydro-met forecasting

8 Interview Protocol & Questions Semi-structured Flexibility to adapt to flow of conversation Interviewees use own words and can steer conversation Broad set of topics & prompts Topics Describe your flood forecasting, warning and emergency response system forecasting and data systems used Warning protocols and communication CPA responsibilities and responses Attitudes to uncertainty t and EPS Knowledge & Experience EFAS

9 1. Background to Study Outline of Talk Today 2. Flood forecaster perceptions of EPS & EFAS 3. Understandings of uncertainty and use of EPS in flood forecasting 4. Communicating ensembles to non-experts 5. Does EPS improve operational flood warning & management? 6. Summary and conclusions

10 2. Flood forecaster perceptions of EPS Some FFs very knowledgeable Others had less experience Well, the EPS is rather, um, is rather rare, from what I ve heard. Most countries really work with deterministic (FF, Germany) I've seen products from EFAS several times at meetings, at workshops. I don t get it regularly. So I can't comment on the operational use or something like that... I would be interested in having this product and looking at that... that they can give a more profound... understanding (FF Switzerland) Some surprising ignorance, even among MoU signatories: EPS, what does that mean? Issue of the model ensembles is it makes some average number of when you have few different models and this is the resulting model. Is it right? Some average? (Director of forecasting center, Eastern Europe)

11 Probabilistic EFAS alerts welcomed pre-alert for national FF centres: Yeah. We also get EFAS reports. For us, it's a useful pre information, but normally 2 or 3 or 4 days before a flood, we normally know from the weather forecast that we should be aware of this... (FF, Hungary) complements national capacities: It is useful because these warnings came to our office let's say 4 or 5 days before.... it's useful for us to raise the preparedness p of our staff a few days before. (FF, Slovenia) Only policymakers saw EFAS leading directly to earlier warnings: it s definitely very useful, the direct operational relevance is for the moment in terms of civil protection. (Policymaker, European Commission)

12 Broad patterns of response: new accession states most consistently enthusiastic i very important I think is this initiative of EFAS project. It's useful one. And once they improve their models, it will be [even better]... So anyway, we are a partner in this cooperation and we receive their product. It's another source of information of possibility of having flood events [here] (FF, Serbia) For scientific reasons... This is quite comfortable for us because they [EFAS] have very good information (FF Romania) Indirect access to higher quality ECMWF ensemble products For internal organizational reasons... Participating in EFAS also about being European Provides leverage for securing additional resources Access to international scientific networks

13 Old Europe less enthusiastic about EFAS EFAS one of many, many sources of information It is useful. But we could easily live without it. Sometimes think local systems superior. (FF, Germany) They coordinate I think from Ispra from Italy. Yeah, I've heard of it. But the colleagues es...said it's not so good than our model. (FF, Germany) Developing own national ensemble flood forecasting capacity (e.g. UK, the Netherlands Switzerland, France, Sweden, Germany, Austria)

14 EPS seen to work best for big rivers... But forecasters and CPAs not especially troubled by forecasting those risks They are testing this in our ensemble forecast. But normally these forecasts are normally only useful for bigger catchments. Our catchments are not so big. So we use ensemble forecasts only [a little]. (FF, Germany) CPA: Danube, it's very well-known. So it's very slow from rising and falling. Interviewer: But for small rivers though it must be more difficult? CPA: The problems there are really dangerous. You can't see... may be you have small scale thunderstorms. It's not possible to give a prognosis in about 30 minutes. (CPA, Austria) (Courtesy of VITUKI)

15 Flash flooding bigger challenge, but EPS less help The problem [here] is these rivers are alpine type, very small scale and [EFAS], it's good information.... but not for detailed flood forecasting... It's a rough idea for someone sitting in Brussels and in London. But not really useful for someone sitting [here]. This is just a problem of scale (FF Austria)

16 1. Background to Study Outline of Talk Today 2. Flood forecaster perceptions of EPS & EFAS 3. Understandings of uncertainty and use of EPS in flood forecasting 4. Communicating ensembles to non-experts 5. Does EPS improve operational flood warning & management? 6. Summary and conclusions

17 3. Understandings of uncertainty in flood forecasting B. uncertainty about runoff processes? In hydrology, we are obviously conscious of our model uncertainty. What is more uncertain than hydrological models? I don t really know frankly, but it is obvious that hydrological models are as uncertain as meteorological models. This is because the representation of river basin is so simplified in comparison with reality (FF, France) C. Quantifying total uncertainty? I said if a forecast is 5 spaghetti lines above our threshold, it doesn't mean that the probability is 10%...it might be 5% or 50%. I don t know. I can't estimate it. (FF Germany) A. uncertainty about rainfall? What we would like to see is a better precision in rainfall forecasts. This is because the main source of uncertainty in forecasting floods comes from the rain scenarios (FF Slovenia) Time

18 Informational Value of EPS Contested Ensemble mean= the best guess forecast? But generally, it's better to have these ensembles. Even if you don t use the probabilistic information, just use the ensemble mean. This is basically what we do for some customers. We give them ensemble mean because it's better than the deterministic one. (Meteorologist, Austria) Ensemble mean=meaningless statistic This is still an important debate. So there are some people around the world trying to use the mean. If you use the mean, it's not too much value. (FF, Romania)

19 Ensemble spread as heuristic measure of forecast uncertainty [EPS] gives us some impression about the uncertainty that is in the weather forecast. So if there s a big spread, there s a lot of uncertainty and we would not be able to give [flood] forecasts with big, big lead-times. And if there s a very small spread then we are a little bit more certain about the forecast and we might enlarge the, the lead-time of the forecast for this, for this occasion. (FF, Netherlands) And then if it's lot of forecast that's almost the same, then you have... but if it's a huge spread, there's going to be a lot of uncertainties. (CPA, Sweden)

20 Using EPS as a check on local deterministic i i model Forecasters sometimes looked to EFAS as a check on their own deterministic models: Forecaster: We are linked, in the EFAS family and so we compare our data. Interviewer: With their predications? Forecaster: Yeah, we use EFAS... we usually check it if we see the situation is dangerous or might be dangerous by our own judgement... (FF, Hungary) Interviewer: Let's say you're using your model and then you look at what EFAS is predicting... Forecaster: Yeah, we can compare and we can decide and we can strictly express that it's true or not... [whether] our statements are right or wrong (FF, Slovakia) EPS not designed as validation tool

21 Can EPS be converted to a quantitative probability distribution? Some flood forecasters very enthusiastic: ti What I would like though is something that gives you the spaghetti plot which is like the raw output, but also then sort of says right, you've got a threshold in the middle of that spaghetti plume, this is the probability that this spaghetti plot is telling you that this output is... rather than having to sit there and count how many of the plots go above the line, it does some sort of post processing to give you a probability. (FF, United Kingdom) Others deeply skeptical: if you make a spaghetti plot and 5 of the 51 lines are higher than the threshold value, you can't say the probability is 10%. It may be it's a special case. It's a special event. May be it's higher or lower than 10%. I said if a forecast is 5 spaghetti lines above our threshold, it doesn't mean that the probability is 10%...it might be 5% or 50%. I don t know. I can't estimate it. (FF, Germany)

22 1. Background to Study Outline of Talk Today 2. Flood forecaster perceptions of EPS & EFAS 3. Understandings of uncertainty and use of EPS in flood forecasting 4. Communicating ensembles to non-experts 5. Does EPS Improve operational flood warning & management? 6. Summary and conclusions

23 4. Communicating ensembles to nonexperts Flood forecasters see communication as a major obstacle to using EPS: Forecaster: We're going to start using ensemble forecast. We did a lot of work to prepare our models for this ensemble within the preview project. Now we're able to use this ensemble forecast. But at the moment we're still... Interviewer: Not operational yet. Forecaster: We can do it operationally. But we do not because we're not sure how to handle all of this spread It's a problem of communication. (FF, Germany)

24 Spaghetti plots of ensemble hd hydrographsh Used by forecasters themselves, but difficult to communicate: These are spaghetti plots. These are different realisations of weather forecasts. That's going to happen if we will have this water level and it's not going to happen when we have this. That s our environment of flood forecasting. But the disaster managers don t want to look by themselves.... They want the expert sitting next to them explaining what's going to happen. (FF, Germany) But we're not sure we really can publish these pictures to our customers and if they really can work with it or if they really can understand. (FF, Germany)

25 Summarising ensembles for non scientists we never deliver a forecast as it is. Even if we run the EPS and have the spaghetti plot, we will never send to civil protection a spaghetti plot. We will make either an easier to understand picture or a text that we describe into the text t how the situation ti is or it will be (FF,Sweden)

26 Showing 75% confidence intervals: an example from Lower Austria Flood forecaster 1: [ ] The first step was just to give an information about the discharge of the water level can be in between this range at this time. So this is the information we now have. The next step is to scientifically write and the next step will be to say we'll have this discharge in this... Interviewer: Bracket let's say? Flood forecaster 2: Range. Flood forecaster 1: In between this range at this time with 75%. This would be the next step. But this is too much information I think that the public cannot use. (FF, Austria)

27 EFAS alerts: summary tables of ensemble members exceeding threshold h ldl levelsl Hydrologists also wanted raw ensemble hydrographs: If they would give a hydrograph, h forecast with hydrograph, that would be fantastic! (FF, Serbia) Difficulties relating EFAS thresholds to local ones: The problem with EFAS particularly is that they do not show hydrographs, but they only tell you that there is a chance of serious flooding But they don t say what the level will be. They only say there is a chance of serious flooding or extreme flooding, but you don t know what it means. (FF, Netherlands) Number of EFAS ensemble members exceeding the high (HAL) and serious (SAL) alert levels

28 1. Background to Study Outline of Talk Today 2. Flood forecaster perceptions of EPS & EFAS 3. Understandings of uncertainty and use of EPS in flood forecasting 4. Communicating ensembles to non-experts 5. Does EPS improve operational flood warning & management? 6. Summary and conclusions

29 5. Does EPS improve operational flood warning & management? Some welcome ensembles: it's presented in percentage... That s really good because...i can see, Well, I have just this amount of man or equipment and so on, where shall I put it? If I see it like a likelihood, then I can take my own decision. (CPA, Sweden) Others complain CPAs can t cope with uncertainty: People cannot deal with uncertainties, it is too complicated. To go for a walk, to know whether we go to the picnic or not, we could cope with it. However, when it is time to decide whether we evacuate or not, it is another story. (FF, France)

30 Forecasters fear ensembles too easily Potential for information overload: misunderstood But this [EPS] is too much information, I think, that the public cannot use (FF, Austria) But these people simply don t understand. They don t need this information. I don t care what the probability is. Give me exact figure!! [they say]. It really doesn t operate on uncertainties. I said, there is uncertainty of 10%. What does it, what do you mean, 10% uncertainty? Give me the figure. I want exact forecast. [Laughs] (FF, Serbia) EPS may lead to false confidence and over-optimization : But even if you have a wide range, you can't be sure that really the water level is outside the range. I think this is really a problem. People think, I'm safe, my threshold isn't reached. But the threshold can be reached, can be the higher water level...you can't be sure that the water level is in this range. Therefore the benefit [of EPS] is not very high h (FF, Germany)

31 Donald MacKenzie s uncertainty trough in perceived accuracy of technical systems high low forecasters CPAs & other users Those estranged from the of EPS outputs for institutions and aims to policy which models applied Training and better EPS visualizations can help, but...

32 Conflicting institutional demands Greater certainty t We believe that there must be a step change in the quality of flood warnings The public and emergency responders must be able to rely on this information with greater certainty than last year (Pitt 2008: vii) And (?!?) Lower thresholds h The Met Office and the Environment Agency should issue warnings against a lower threshold of probability to increase preparation lead times (Recommendation 34: Pitt review)

33 EPS changes relationship between forecasters and recipients Shifts responsibility for coping with forecast uncertainty onto forecast recipients You're putting the onus on the people thatt receive thatt probabilistic bili warning to make a decision what to do with it themselves. (FF, United Kingdom) EPS also means dumping responsibility onto forecast users. By forcing forecasters to provide deterministic predictions, the accountability remains entirely on the shoulders of forecasters. If a forecaster provides a probabilistic forecast, they give the import for the decision to forecasts users.... Asking for a deterministic prediction is also a way for the person in charge oftaking a decision to avoid decisional problems and blame. (FF, France) Is EPS more honest or an abdication of professional responsibility?

34 You can t take chances with public safety we are the hydrologists and we are the ones that are responsible for this forecast...[eps] is not so good because this is like putting the responsibility down to them and saying we have no idea, make what you want of the spaghetti plots. This shouldn't happen I think. (FF, Germany) The Environment Agency is primarily concerned with issuing short-notice flood warnings, you know 2 hours, with as a high level of certainty about that as they can manage so normally, unless they are absolutely certain that there is going to be a flood, there are not going to issue a warning, even if there is a fair chance of flooding. And this is because their primary customer for flood warnings is the general public. So they think that s what they have to do (Meteorologist, United Kingdom)

35 Blame management and the institutional appeals of ensembles Before now we said yes or no, you are going to flood or not. If you say yes, they're going to do something about it. If we say 40% chance, then it's up to them what they do with it. To me, that s where the Met Office have always coped better with things. In the Met Office, all their weather warnings always come out as probabilistic. So when it doesn't happen, they never have any complaints because they always say we only said it was a 60% chance and so it hasn't happened. In the Agency, our flood warning services have been built on yes, no and if we get it wrong, then that s got higher consequences of the Met Office getting it wrong in producing a probabilistic forecast. So I think for our own reputation as well, to go to probabilistic forecasting would be quite useful because it almost gives us, not an excuse exactly, but it gives us a reason, it quantifies our uncertainty and it means we won't necessarily get criticised as much. (FF, United Kingdom)

36 1. Background to Study Outline of Talk Today 2. Flood forecaster perceptions of EPS & EFAS 3. Understandings of uncertainty and use of EPS in flood forecasting 4. Communicating ensembles to non-experts 5. Does EPS improve operational flood warning & management? 6. Summary and conclusions

37 6. Summary and conclusions Operational value of EPS not self-evident Generally welcomed by flood forecasters, but... informational contested not always used as proponents imagine it should be Difficulties of communicating EPS and uncertainty EPS changes relationship between forecasters and recipients Challenges are political and institutional as well as technical and communicative