End user understandings of uncertainty, ensemble forecasts, and EFAS

End user understandings of uncertainty, ensemble forecasts, and EFAS Sebastien Nobert, David Demeritt, Hannah Cloke Department of Geography

Outline of Talk Today 1. Background to Study 2. General understandings of ensembles and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Challenges to using EPS 5. Conclusions & recommendations

1. Background: Preliminary findings from a larger study Europeanizing flood forecasting and the geographies of risk and science in the EU (RES-062-23-0913) Project Aims: 1. Explain the institutional politics of flood and disaster management in Europe; 2. Document the beliefs and practices involved in flood forecasting and explore their implications for managing floods in the face of uncertainty; 3. Enrich the understanding and communication of ensemble predictions (EPs) and advise EFAS about improving its specific role in flood risk assessment.

Promises of Ensemble Flood Forecasts the use of meteorological ensembles to produce sets of hydrological predictions increased the capability to issue flood warnings (Balint et al., 2006, p.67) The hydrological ensemble predictions have greater skill than deterministic ones. (Roulin, 2007) The use of EPS in hydrological forecasting proved to be of great added value to a flood early warning system, as the EPS-based forecasts showed in general higher skill than the deterministic-based ones. (Bartholmes et al., 2008) Are these technical promises realized in operational practice?

Research Design and Evidence Base 45+ interviews with forecasting experts, water authorities and CPAs in 14 European countries involved in flood forecasting, warning and emergency response Participant observation at forecasting workshops & conferences and during visits to forecasting centres All interview quotations are anonymized what we call our data are really our constructions of other people s constructions of what they are up to (Geertz 1973: 9)

Methodology: Qualitative Social Science Not an experimental science in search of laws; an interpretative one in search of understanding, or Verstehen Different objects of study, different methodologies Indifferent kinds (ie. rocks and other unconscious objects of study) Interactive kinds (people: objects who are also conscious subjects) Natural science Social science

Seeking a Thick Description of Meaningful Behaviour and Belief Consider two boys rapidly contracting the eyelids of their right eyes. In one this is an involuntary twitch; in the other a conspiratorial signal to a friend. The two are, as movements, identical... Yet the difference, however, unphotographable, between a twitch and wink is vast; as anyone unfortunate enough to have had the first taken for the second knows [T]he point is that between the thin description of what the winker/twitcher is doing ( rapidly contracting his right eyelid ) and the thick description of what he is doing ( practicing a burlesque of a friend faking a wink to deceive an innocent into thinking a conspiracy is in motion ) lies the object of ethnography: a stratified hierarchy of meaningful structures in terms of which twitches, winks, fake-winks, parodies, rehearsals of parodies are produced, perceived, and interpreted. Geertz (1973)

Accessing End User Understandings Solicited for broad interviews about operational flood warning and response EFAS not specifically mentioned Promised anonymity Enthusiastic to show me around My positionality : independent of JRC: allows more frank assessment of EFAS Scientific novice: requires informants to explain

Interview Protocol & Questions Semi-structured Broad set of topics & prompts Flexibility to adapt to flow of conversation Interviewees use own words and can steer conversation Topics Describe your flood forecasting, warning and emergency response system forecasting and data systems used Warning protocols and communication CPA responsibilities and responses Attitudes to uncertainty and EPS Knowledge & Experience EFAS Asking about EFAS at the start of the interview would have resulted in very different answers

Outline of Talk Today 1. Background to Study 2. General understandings of ensembles and uncertainty in flood forecasting 3. Responses to EFAS in particular 4. Communicating EPS and uncertainty 5. Conclusions & recommendations

2. EPS: an often unfamiliar product Differing levels of awareness Well, the EPS is rather, um, is rather rare, from what I ve heard. Most countries really work with deterministic I've seen products from EFAS several times at meetings, at workshops. I don t get it regularly. So I can't comment on the operational use or something like that... I would be interested in having this product and looking at that... that they can give a more profound... understanding Some surprising ignorance, even among MoU signatories: EPS, what does that mean? Issue of the model ensembles is it makes some average number of when you have few different models and this is the resulting model. Is it right? Some average?

Informational Value of EPS Contested Ensemble mean= the best guess forecast? But generally, it's better to have these ensembles. Even if you don t use the probabilistic information, just use the ensemble mean. This is basically what we do for some customers. We give them ensemble mean because it's better than the deterministic one. Ensemble mean=meaningless statistic This is still an important debate. So there are some people around the world trying to use the mean. If you use the mean, it's not too much value.

Using EPS as a check on local deterministic model? Forecasters sometimes looked to EFAS as a check on their own deterministic models: Forecaster: We are linked, in the EFAS family and so we compare our data. Interviewer: With their predications? Forecaster: Yeah, we use EFAS... we usually check it if we see the situation is dangerous or might be dangerous by our own judgement... Interviewer: Let's say you're using your model and then you look at what EFAS is predicting... Forecaster: Yeah, we can compare and we can decide and we can strictly express that it's true or not... [whether] our statements are right or wrong EPS not designed as validation tool

Can EPS improve flood risk management? Forecasters complain CPAs can t tolerate uncertainty or error: At the very end, I think everything must be a kind of deterministic forecast... Does lake exceed this threshold, yes or no? It's always... at the very end, it's always a yes or no decision. Others welcome probabilistic information: it's presented in percentage... That s really good because...i can see, Well, I have just this amount of man or equipment and so on, where shall I put it? If I see it like a likelihood, then I can take my own decision.

Dispute over critical forecasting uncertainties to resolve 2. uncertainty about runoff processes? In hydrology, we are obviously conscious of our model uncertainty. What is more uncertain than hydrological models? I don t really know frankly, but it is obvious that hydrological models are as uncertain as meteorological models. This is because the representation of river basin is so simplified in comparison with reality 3. Quantifying total uncertainty? I said if a forecast is 5 spaghetti lines above our threshold, it doesn't mean that the probability is 10%...it might be 5% or 50%. I don t know. I can't estimate it. 1. uncertainty about rainfall? What we would like to see is a better precision in rainfall forecasts. This is because the main source of uncertainty in forecasting floods comes from the rain scenarios Time

Broad patterns in understandings of EPS and flood forecast uncertainty Still working to quantify open-ended interview data No consistent differences in attitudes between meteorologists and hydrologists Clear contrasts between old EU states and new accession & EU candidate countries Wealthy EU states often have or developing own EPS capacity for NWP and/or flood forecasting New accession states have fewer resources & other development priorities: better instrumentation, more computing, training, etc Meteorological and hydrological forecasting cascaded or state of art prediction model for select profiles. This is our dream. This is our big... how to say? We want very much to do something like that, but have some obstacles... You need very good people, very good in mathematics, very good in modelling and very good in programming. But those people are very expensive. Salary in our institute is not very attractive for those people.

3. Responses to EFAS in particular Generally welcomed by forecasters as a useful pre-alert: Yeah. We also get EFAS reports. For us, it's a useful pre information, but normally 2 or 3 or 4 days before a flood, we normally know from the weather forecast that we should be aware of this... Summary tables a good way to show ensemble But I have the impression with the presentation of these boxes which shift every new round is quite a good way of presenting uncertainty in the forecasts Widespread desire to see hydrographs as part of alerts If they would give a hydrograph, forecast with hydrograph, that would be fantastic!

EFAS complements national capacities Medium term warnings operationally useful to forecasting centres It is useful because these warnings came to our office let's say 4 or 5 days before.... it's useful for us to raise the preparedness of our staff a few days before. A few saw EFAS pre-alerts as useful for allowing earlier warnings to CPAs it s definitely very useful, the direct operational relevance is for the moment in terms of civil protection.

EFAS works best on big rivers Forecasters and CPAs are not especially troubled by forecasting those risks The other thing is it's very, very dependent on the size of the catchment, the quality of the forecasts. If you do forecast for large rivers, it's more easy than if you have some torrents. If you have in the alpine region, you can't do forecasting there. CPA: Danube, it's very well-known. So it's very slow from rising and falling. Interviewer: But for small rivers though it must be more difficult? CPA: The problems there are really dangerous. You can't see... may be you have small scale thunderstorms. It's not possible to give a prognosis in about 30 minutes.

Flash flooding perceived as bigger threat, but EFAS less help The problem [here] is these rivers are alpine type, very small scale and [EFAS], it's good information.... but not for detailed flood forecasting... It's a rough idea for someone sitting in Brussels and in London. But not really useful for someone sitting [here]. This is just a problem of scale

Concerns: what do EFAS warnings mean? Hydrographs of river flow or water levels the traditional measure of flood risk in hydrology EFAS alerts based on # ensemble members exceeding unspecified thresholds The problem with EFAS particularly is that they do not show hydrographs, but they only tell you that there is a chance of serious flooding But they don t say what the level will be. They only say there is a chance of serious flooding or extreme flooding, but you don t know what it means. Number of EFAS ensemble members exceeding the high (HAL) and serious (SAL) alert levels

Thresholds: too low, confusion about how they are calculated [...] the question is how EFAS thresholds are established. This is our big problem since the thresholds of our two models are tailored to our data bank. [...] They have simply taken the 20 years of flow simulation since 1995-98. Thus, because these years were relatively dry, their thresholds are very very low. Forecaster: So far, most alerts have been a little bit too early. Interviewer: Do you think it s because of EPS system, or Forecaster: No, because their warning levels are too low. Interviewer: Do you know what they use as thresholds? Forecaster: They use observations, they look at the data set that they have in some cases and this is a very limited data set of about five six years. And they look at the maximum level that has occurred in these periods.

Broad patterns of response: new accession states most consistently enthusiastic very important I think is this initiative of EFAS project. It's useful one. And once they improve their models, it will be [even better]... So anyway, we are a partner in this cooperation and we receive their product. It's another source of getting information of possibility of having flood events [here] For scientific reasons... This is quite comfortable for us because they [EFAS] have very good information Back door access to higher quality ECMWF ensemble products For internal organizational reasons... Participating in EFAS also about being European Provides leverage for securing additional resources Access to international scientific networks

Old Europe less enthusiastic EFAS one of many, many sources of information It is useful. But we could easily live without it. Sometimes think local systems superior They coordinate I think from Ispra from Italy. Yeah, I've heard of it. But the colleagues...said it's not so good than our model. Developing own national ensemble flood forecasting capacity (e.g. UK, Switzerland, France, Sweden, Germany)

EFAS inspires innovation at national level EFAS proves ensemble flood forecasting can be done operationally EFAS precedent often referred to in R&D strategies Interviewer: Was the idea sort of influenced by EFAS or it had something else to do with it? Forecaster: I have seen presentations by Jens Bartholmes and Jutta Thielen. This was an inspiration of course.

4. Challenges to using EPS Displaying full ensemble may lead to: confused inaction: Because we can't predict exactly... The problem is how to show this uncertainty. But this [EPS] is too much information, I think, that the public cannot use false confidence and over-optimization of response: But even if you have a wide range, you can't be sure that really the water level is outside the range. I think this is really a problem. People think, I'm safe, my threshold isn't reached. But the threshold can be reached, can be the higher water level...you can't be sure that the water level is in this range. Therefore the benefit [of EPS] is not very high

Communicating Probabilistic Forecasts to Civil Protection Forecaster doubts over ability of CPAs to understand probabilistic warnings: But these people simply don t understand. They don t need this information. I don t care what the probability is. Give me exact figure!! [they say]. It really doesn t operate on uncertainties. I said, there is uncertainty of 10%. What does it, what do you mean, 10% uncertainty? Give me the figure. I want exact forecast. [Laughs] People cannot deal with uncertainties, it is too complicated. The problem is that to live, to go for a walk, to know whether we go to the picnic or nor, we could cope with it. However, when it is time to decide whether we evacuate or not, it is another story. Training and better EPS visualizations can help, but...

Conflicting institutional demands Greater certainty We believe that there must be a step change in the quality of flood warnings The public and emergency responders must be able to rely on this information with greater certainty than last year (Pitt 2008: vii) And Earlier warnings The Met Office and the Environment Agency should issue warnings against a lower threshold of probability to increase preparation lead times (Recommendation 34: Pitt review)

Accountability concerns over EPS Re-distributing responsibility for flood risk management decisions EPS also means dumping responsibility onto forecast users. By forcing forecasters to provide deterministic predictions, the accountability remains entirely on the shoulders of forecasters. If a forecaster provides a probabilistic forecast, they give the import for the decision to forecasts users.... Asking for a deterministic prediction is also a way for the person in charge of taking a decision to avoid decisional problems and blame. Concerns over Europeanization and competencies If the [national river forecasting centre] starts talking about weather, they get heavily criticised. If we mention the word flooding,... we got heavily criticised...if you have a European Agency [EFAS] with an unclear mandate that does not have a national responsibility, providing such a service, then we would be very nervous about this.

Conclusions Informational value of EPS not self-evident understood in different ways by operational flood forecasters Ensembles not always used as proponents imagine they should be Meaning and value of EFAS thresholds unclear Substantial institutional challenges to ensemble flood forecasting in general and to EFAS in particular

Recommendations for further improvements Improve calibration of thresholds Tailor threshold levels to needs of different end users Clarify and communicate how thresholds are calculated Experiment with providing hydrographs to some cooperating water authorities Visit forecasting centres and provide more training in situ