National Oceanography Centre. Research & Consultancy Report No. 38

Transcription

1 National Oceanography Centre Research & Consultancy Report No. 38 Evaluation and comparison of the operational Bristol Channel Model storm surge suite J A Williams & K J Horsburgh 2013 Revised September 2013 National Oceanography Centre, Liverpool 6 Brownlow Street Liverpool L3 5DA UK Author contact details: Tel: +44 (0) jane@noc.ac.uk

2 National Oceanography Centre, 2013

3 AUTHOR WILLIAMS, J A & HORSBURGH, K J DOCUMENT DATA SHEET TITLE Evaluation and comparison of the operational Bristol Channel Model storm surge suite. REFERENCE Southampton, UK: National Oceanography Centre, 37pp. (National Oceanography Centre Research and Consultancy Report, No. 38 ABSTRACT PUBLICATION DATE 2013 Due to its exceptional tidal range, complex geometry, and exposure to flood risk the operational storm surge modelling system for the Bristol Channel comprises a number of nested hydrodynamic models. Forecasts for the region are available from the shelf-wide storm surge model, CS3X, as well as from two finer-scale models of the Bristol Channel itself (the Bristol Channel model, BCM, and the Severn River Model, SRM), and finally a model that provides total water level for the region. This report provides for the first time a systematic comparison of the accuracy of the different models when compared against tide gauge observations, for five significant storm surge events in the model archive. As forecasters wish to know total water level as accurately as possible, the most accurate determination of the tide is critical. The best method of deriving total water levels in the region is to make the empirical correction to tidal predictions (Hibbert et al., in preparation) and then add the appropriate model residual at high water. The best forecasting technique would be a port-by-port local ensemble that adds the relevant modelled high-water residual to the empirically corrected tides. For Avonmouth, Hinkley Point and Newport the two fine resolution models (BCM, SRM) have comparable skill. For the outer estuary ports of Ilfracombe and Mumbles, the best approach would be to use modelled residuals from CS3X and BCM. The best understanding of the large scale uncertainty in the weather systems is still provided by the full MOGREPS-driven surge ensemble based on CS3. The predictive skill of the finer resolution models indicates that these should be retained within the present system. The Total Level model does not provide any additional forecasting skill once empirical tidal corrections are available; at that time this tuned model could be disabled, or removed from future developments of the operational system. Any future development of the surge model suite for UKCMF should ensure that model performance in the Bristol Channel delivers comparable accuracy to the values presented here. The most accurate performing model overall was the Bristol Channel Model (BCM) at approximately 4km horizontal resolution; future development of the surge model suite should therefore aim for comparable resolution at the shelf-scale. KEYWORDS: ISSUING ORGANISATION National Oceanography Centre University of Southampton Waterfront Campus European Way Southampton SO14 3ZH UK Not generally distributed - please refer to author

4 This page is intentionally left blank

5 Evaluation and comparison of the operational Bristol Channel Model storm surge suite Jane A. Williams and Kevin J. Horsburgh Executive summary Due to its exceptional tidal range, complex geometry, and exposure to flood risk the operational storm surge modelling system for the Bristol Channel comprises a number of nested hydrodynamic models. Forecasts for the region are available from the shelf-wide storm surge model, CS3X, as well as from two finer-scale models of the Bristol Channel itself (the Bristol Channel model, BCM, and the Severn River Model, SRM), and finally a model that provides total water level for the region. This report provides for the first time a systematic comparison of the accuracy of the different models when compared against tide gauge observations, for five significant storm surge events in the model archive. As forecasters wish to know total water level as accurately as possible, the most accurate determination of the tide is critical. The best method of deriving total water levels in the region is to make the empirical correction to tidal predictions (Hibbert et al., in preparation) and then add the appropriate model residual at high water. The best forecasting technique would be a port-by-port local ensemble that adds the relevant modelled high-water residual to the empirically corrected tides. For Avonmouth, Hinkley Point and Newport the two fine resolution models (BCM, SRM) have comparable skill. For the outer estuary ports of Ilfracombe and Mumbles, the best approach would be to use modelled residuals from CS3X and BCM. The best understanding of the large scale uncertainty in the weather systems is still provided by the full MOGREPS-driven surge ensemble based on CS3. The predictive skill of the finer resolution models indicates that these should be retained within the present system. The Total Level model does not provide any additional forecasting skill once empirical tidal corrections are available; at that time this tuned model could be disabled, or removed from future developments of the operational system. Any future development of the surge model suite for UKCMF should ensure that model performance in the Bristol Channel delivers comparable accuracy to the values presented here. The most accurate performing model overall was the Bristol Channel Model (BCM) at approximately 4km horizontal resolution; future development of the surge model suite should therefore aim for comparable resolution at the shelf-scale. 5

6 Evaluation and comparison of the operational Bristol Channel Model storm surge suite Jane A. Williams and Kevin J. Horsburgh 1. Background The Bristol Channel is an area of complex hydrodynamics which include a very large tidal range, strong currents, extensive inter-tidal areas and river inputs, all of which contribute to make predicting storm surges difficult. The 12km shelf-scale surge model, CS3X provides surge forecasts at Class A tide gauge locations including those in the Bristol Channel. Additionally there are two high resolution local models covering the Bristol Channel and Severn Estuary. The Bristol Channel model (BCM) is a ~4km resolution model of the Bristol Channel east of 5 W and is driven at the boundary by 26 tidal harmonics plussurge components from CS3X. The Severn Estuary model +1D River Severn model (SRM) has resolution ~1.3km and is driven at the boundary by tide and surge data from BCM. All models are forced by NAE atmospheric model data. The models are shown in Figure 1. Figure 1: High resolution models for Bristol Channel and Severn Estuary. The tide gauge locations for the region are also indicated. 6

7 Forecast data (in the form of residual elevations at each tide gauge location) from CS3X, BCM and SRM are archived routinely at the Met Office after each operational run (at 0000, 0600, 1200 and 1800UTC) and transferred to NOC each month. Routinely each month, the performance of CS3X is evaluated by comparison to observed residuals derived from the observations from the A Class tide gauge data. Although archived, the forecast data from BCM and SRM are not part of the monthly validation. To produce water level forecasts from these models, the modelled non-tidal residual is added to harmonic tidal predictions at each tide gauge location. A further operational forecasting tool (which is only available for the Bristol Channel region) is a set of re-tuned total water levels derived from BCM and SRM (Williams and Horsburgh, 2009). These models are re-calibrated effectively increasing the modelled tide by statistically regressing modelled against observed sea levels for the year In this report we refer to this as the Total Level model. This report firstly evaluates the forecast non-tidal residual data from the BCM and SRM with respect to CS3X and observations in order to determine whether model resolution produces a more accurate residual forecast in any model. We then go on to compare the total sea level obtained from all approaches (i.e. combining the three model residual outputs with tidal predictions and also extracting the Total Level information, in order to assess the multiple forecasting tools and recommend a preferred forecasting approach for this region. 2. Analysis of non-tidal residuals A typical monthly time series plot of hourly model versus observed residuals for locations in the Bristol Channel will show noise, particularly in the form of semidiurnal oscillations, in the observed residuals. Because of the large tidal range, only small timing errors in the predicted tide or observed water level are required to cause these (Horsburgh and Wilson, 2007). Or if they are real, then processes not included in the surge model could account for them e.g. the interaction of wind waves. Figure 2 shows a time series of model (CS3X) v observations for a typical quiet month at Avonmouth. The model is represented by the continuous line, the observations are blue crosses and the red diamonds mark the hour closest to model high water. It can be seen from Figure 2 that the oscillatory nature of the observations makes a meaningful comparison with the model output difficult. The first decision was to restrict the analysis to six-months of winter for each year where the signal to noise ratio would be higher. We chose September 2011 to February 2012 as the first period to analyse. This was chosen as it was prior to the removal of the gauge at Avonmouth in April Residual elevations from hours t+6 to t+11 of each of the four daily archived forecasts were extracted to compile a continuous time series of forecast data at each tide gauge location for each month of the analysis period. Data from Avonmouth, 7

8 Ilfracombe, Hinkley Point, Mumbles and Newport were extracted from BCM, and data from Avonmouth, Hinkley Point and Newport were extracted from SRM. We compared each month s data at each location against observed residuals and also CS3X. Plots at all locations still showed large oscillations in the observed residual compared to all three models except where there was a significant surge event,in which case the magnitude of the model and observed residual would be considerably greater than the background oscillations. Analysis of this period showed only one such significant event, January It was therefore decided to analyse the three previous winters (September to February) of forecast data in order to identify other significant surge events which could provide the basis for validation. This data was extracted in the same way and analysis of the resulting time series gave four additional significant events in the Bristol Channel. 8

9 Figure 2: Time series of hourly model forecast (CS3X) v observed residual elevations at Avonmouth for July

10 We identified five significant Bristol Channel surge events to analyse in detail: 1. 4 th December th January th November th November rd January 2012 We will analyse these events in turn, comparing performanceof the three models (where available) against observations. Event 1: 4 th December 2008 Figure 3 shows the meteorological conditions for 4-5 th December The passage of a low pressure system across Scotland with central pressure of 965HPa produced strong south westerly to westerly winds giving a maximum observed surge of 1.15m at Avonmouth and 1.03m at Newport at approximately 0600/0700UTC. Figure 3: Analysis charts for 0000UTC 4 th and 0z 5 th December Figures 4 8 show the time series of model v observed residual elevation at each of the Class A tide gauge locations in the Bristol Channel. In the outer estuary there are only data available from CS3X and BCM as the domain of the SRM does not cover this area. A qualitative look at the plots shows that the BCM and SRM models are generally better at predicting the magnitude of the peak than CS3X. Table 1 quantifies this by tabulating the observed maximum residual against the maximum modelled value around the time of the maximum observation. This will then discount any small timing errors in the models. The closest model values to the observations are highlighted in green. Table 1 shows that where available the SRM produces the most accurate values. At Ilfracombe and Mumbles, where no values for SRM exist, then the BCM forecasts are closer to the observations than CS3X for this event. Interestingly, the plots show that the model reproduces the semidiurnal oscillations in the residual: this suggests a genuine dynamical mechanism (e.g. weather influencing the tide) which the model can simulate. 10

11 Figure 4: CS3X, BCM and SRM residual elevations at Avonmouth for Event 1. Figure 5: CS3X and BCM residual elevations at Ilfracombe for Event 1. 11

12 Figure 6: CS3X, BCM and SRM residual elevations at Hinkley Point for Event 1. Figure 7: CS3X and BCM residual elevations at Mumbles for Event 1. 12

13 Figure 8: CS3X, BCM and SRM residual elevations at Newport for Event 1. PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 1: Maximum model v observed residual 4 th December

14 Event 2: 17 th January 2009 Figure 9: Analysis charts for 0000UTC 17 th and 0z 18 th January Figure 9 shows the analysis charts at 0000UTC for 17 th and 18 th January A low pressure of 959HPa intensifies over the course of 17 th January to 946HPa at 0000UTC 18 th January off the North West coast of Scotland producing strong westerly winds in the Bristol Channel. This feature caused a peak surge at ~2000UTC of 1.50m at Avonmouth and 1.76m at Newport. Figures show the output from the three models at the tide gauge locations. It can be clearly seen in Figure 10 that for this event the BCM and SRM over-predict the peak considerably. At Avonmouth, CS3X gives the best forecast (an underprediction of 0.14m). However the finer resolution models do better at the other four locations. At Newport, CS3X underestimates the peak by 0.38m, whereas BCM only underestimates by 0.06m. Table 2 summarises the forecasts from the models compared with observations. 14

15 Figure 10: CS3X, BCM and SRM residual elevations at Avonmouth for Event 2. Figure 11: CS3X and BCM residual elevations at Ilfracombe for Event 2. 15

16 Figure 12: CS3X, BCM and SRM residual elevations at Hinkley Point for Event 2. 16

17 Figure 13: CS3X and BCM residual elevations at Mumbles for Event 2. Figure 14: CS3X, BCM and SRM residual elevations at Newport for Event 2. 17

18 PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 2: Maximum model v observed residual 17 th January Event 3: 14 th November 2009 Figure 15:Analysis charts for 0000UTC 14 th and 0000UTC 15 th November Figure 15 shows the meteorological situation for event 3. A low pressure system of central pressure 969HPa off the South West coast of Ireland tracked North East producing South Westerly winds in the Bristol Channel. This event resulted in a sustained event of approximately 12 hours duration producing two peaks. The second peak is the maximum of the two (1.56m at Avonmouth and 1.42m at Newport at approximately 1200UTC). Figures show time series of model v observations. For this event no data were available from the tide gauge at Ilfracombe. The data in Table 3 show that at Avonmouth, SRM correctly forecast the maximum peak with BCM only underpredicting by 0.01m. However, CS3X under-predicted the peak by 0.43m.At Newport, all three models under predicted the peak of this event; however BCM gave the closest value with an under-prediction of 0.13m. 18

19 Figure 16: CS3X, BCM and SRM residual elevations at Avonmouth for Event 3. 19

20 Figure 17: CS3X, BCM and SRM residual elevations at Hinkley Point for Event 3. Figure 18: CS3X and BCM residual elevations at Mumbles for Event 3. 20

21 Figure 19: CS3X, BCM and SRM residual elevations at Newport for Event 3. PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a n/a n/a n/a Hinkley Point Mumbles n/a Newport Table 3: Maximum model v observed residual 14 th November

22 Event 4: 11 th November 2010 Figure 20:Analysis charts for 0z 11 th and 0z 12 th November Figure 20 shows the analysis charts for event 4. A low pressure system passed over northern Scotland producing South Westerly to Westerly winds. The resulting observed maximum surge was a similar double peak to Event 3. The second of these peaks was the largest so these values were used in our analysis. At Avonmouth, the fine resolution models over-predicted the peak, however they underpredicted the peak at Ilfracombe. Table 4 summarises the results. Figure 21: CS3X, BCM and SRM residual elevations at Avonmouth for Event 4. 22

23 Figure 22: CS3X and BCM residual elevations at Ilfracombe for Event 4. Figure 23: CS3X, BCM and SRM residual elevations at Hinkley Point for Event 4. 23

24 Figure 24: CS3X and BCM residual elevations at Mumbles for Event 4. Figure 25: CS3X, BCM and SRM residual elevations at Newport for Event 4. 24

25 PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 4: Maximum model v observed residual 11 th November Event 5: 3 rd January 2012 Figure 26:Analysis charts for 0000UTC 3 rd and 0000UTC 4 th January Figure 26 shows the meteorological charts for Event 5. A low pressure system with central pressure 968HPa was situated off North West Ireland producing South Westerly winds giving a peak surge of 1.39m at Avonmouth and 1.52m at Newport at approximately 0600UTC. Table 5 summarises the model results. In this case, BCM produced the best results at all five tide gauge locations. At Avonmouth CS3X under predicted by 0.45m whereas BCM under predicted by 0.06m. Similarly at Newport, CS3X under predicted 0.64m whereas BCM under predicted by 0.28m. Conversely, at Ilfracombe CS3X over predicted by 0.07m whereas BCM under predicted by 0.01m. 25

26 Figure 27: CS3X, BCM and SRM residual elevations at Avonmouth for Event 5. Figure 28: CS3X and BCM residual elevations at Ilfracombe for Event 5. 26

27 Figure 29: CS3X, BCM and SRM residual elevations at Hinkley Point for Event 5. Figure 30: CS3X and BCM residual elevations at Mumbles for Event 5. 27

28 Figure 31: CS3X, BCM and SRM residual elevations at Newport for Event 5. PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 5: Maximum model v observed residual 3 rd January The data from the individual events (Tables 1-5) are reproduced in Annex 1 so that all the data can be seen simultaneously: the highlighted green cells designate the model residual that is closest to the observations. Considering the data in Annex 1, there is a great deal of variability in terms of which model produces the best result, in terms of the non-tidal residual. For Event 5 the BCM gave the most accurate predictions at all locations, whereas for Event 4 the CS3X model was most accurate at the majority of sites. It is possible to inter-compare the residuals from the three models in a quantitative way if the data are further analysed. The simplest approach is to simply count the number of occasions where each model is most accurate (i.e. count the green cells in Annex 1). The results of this clear winner approach are shown in Table 6 below. However, this simple method devalues the predictions of 28

29 the SRM which are not available at Ilfracombe and Mumbles. Table 7 repeats the count of clear winners but only where an SRM forecast is available. CS3X 5 BCM 12 SRM 7 Table 6: Number of most accurate forecasts (all events, all locations) CS3X 2 BCM 6 SRM 7 Table 7: Number of most accurate forecasts (only where SRM forecast is available) An alternative analysis is to assign 3 points for the best forecast, 2 points for the next closest, and 1 point for the poorest forecast (an Olympic medal table approach). The outcome from this evaluation restricted to those occasions where an SRM forecast is available is shown in Table 8. CS3X 23 BCM 30 SRM 33 Table 8: Cumulative skill score (1 st = 3; 2 nd = 2; 3 rd = 1) where SRM forecast is available The results in Tables 7 and 8 confirm that the finer resolution models better simulate the residual in a quantitative sense but that CS3X still has useful skill. There are dynamical reasons why one might expect BCM and SRM to generally provide more accurate residuals: they have the spatial resolution that can better accommodate both seiching modes (i.e. resonant waves at the cross-channel scale) as well as the higher tidal harmonics which are large in the upper reaches of the estuary. It is also possible to consider model residual accuracy for each tide gauge location in turn. This is shown in Table 9 where we repeat the cumulative skill score for each model at each tide gauge. RMS errors (m) of the modelled residual compared to that observed are also given. CS3X BCM SRM Score RMS Score RMS Score RMS Avonmouth Ilfracombe Hinkley Point Mumbles Newport Table 9: Cumulative skill score (1 st = 3; 2 nd = 2; 3 rd = 1 OR 1 st = 2; 2 nd =1 if SRM not available) broken down by tide gauge location and RMS error (m) of residual. 29

30 If the accuracy of non-tidal residual was the primary goal of operational forecasting then Table 9 would suggest adopting an ensemble approach for combining the three models. Model performance is similar at Avonmouth: although SRM more often has the best forecasts, the RMS error for CS3X is better. At both Ilfracombe and Mumbles, BCM is more frequently the closer forecast but has a poorer RMS error than CS3X. Only at Newport is there a clearly preferred model, with BCM giving the highest number of best forecasts and having the least RMS error. However, of far more use to forecasters is the accurate prediction of the total water level. The next section repeats our detailed analysis but based on the total forecast water level taking into account tide levels as well as surge. 3. Analysis of total water levels We now analyse the total forecast sea level. The predicted high water at each location is that obtained from the NOC harmonic constant library, as used by EA operationally in the region. The model values chosen to add to this harmonically predicted tide are the respective forecast residuals (from CS3X, BCM, SRM) at the time of predicted HW. We also show the maximum forecast level from the Total Level model. For information, the tuned outputs from the Total Level model are based on the SRM for Avonmouth, Hinkley Point and Newport and the BCM for Ilfracombe and Mumbles. The Total Level model can provide actual turning points of high water. However, we do not show these as the greatest possible difference of high water from the nearest 15 minute value is of the order 0.2% of the tidal amplitude. For Avonmouth, this would result in an error of approximately 2cm on the largest spring tides. The tables below present total sea level thus derived and compare each against observations for the five storm surge events. All values in the tables are given to Chart Datum and are in metres. Predicted Max Obs. Level (10z) CS3X + BCM + SRM + Total Level Avonmouth (11z) Ilfracombe 7.58 (10z) (09z) Hinkley 9.74 (10z) Point (10z) Mumbles 7.93 (10z) (10z) Newport 9.75 (10z) (10z) Table 10: Comparison of model forecast output for Event 1: 4 th December

31 Predicted Max Obs. Level CS3X + BCM + SRM + Total Level Avonmouth (0z 18 th ) (0z 18 th ) Ilfracombe (23z) (22z) Hinkley A Point (23z) (23z) Mumbles (23z) (23z) Newport (0z 18 th ) (0z 18 th ) Table 11: Comparison of model forecast output for Event 2: 17 th January Predicted Max Obs. Level (17z) CS3X + BCM + SRM + Total Level Avonmouth (17z) Ilfracombe Hinkley Point (17z) (17z) Mumbles (16z) (16z) Newport (17z) (17z) Table 12: Comparison of model forecast output for Event 3: 14 th November 2009 Predicted Max Obs. Level (22z) 8.06 (21z) (22z) 8.35 (22z) (22z) CS3X + BCM + SRM + Total Level Avonmouth (22z) B Ilfracombe (21z) Hinkley Point (22z) Mumbles (22z) Newport C (22z) Table 13: Comparison of model forecast output for Event 4: 11 th November

32 Predicted Max Obs. Level (14z) 7.07 (13z) 8.90 (14z) 7.21 (13z) 9.09 (14z) CS3X + BCM + SRM + Total Level Avonmouth 9.82 (14z) Ilfracombe (13z) Hinkley Point (13z) Mumbles (13z) Newport (14z) Table 14: Comparison of model forecast output for Event 5: 3 rd January 2012 The clear winner approach for total water level is repeated in Tables 15 and 16 below. CS3X 5 BCM 10 SRM 4 Total Level 5 Table 15: Number of most accurate forecasts (all events, all locations) CS3X 1 BCM 6 SRM 4 Total Level 4 Table 16: Number of most accurate forecasts (only where SRM forecast is available) On closer analysis there are many occasions where the Total Level model fails to correlate with the results from the fine resolution models. Since the Total Level model is an empirically scaled version of the SRM and BCM output then any apparent benefit from the Total Level model may simply be due to the empirical correction of the tide. Since the production of the tuned Total Level model (Williams and Horsburgh, 2009), a superior method of improving tidal predictions has been developed based on correlating differences from previously observed tides using real-time tide gauge data (Hibbert et al., in preparation). The Total Level model is only closest to observation in 5 out of 24 cases; only in three cases (marked A, B, C in Tables above) does it improve the prediction by 10cm or better when compared to BCM or SRM. In case A, all three alternative models over-predict the storm surge and the largest actual storm surge is in any 32

33 case only 16 cm; this was not really an event at all, save that the weather caused a significant change in the tide and thus the residuals (Figs 10-14). For cases B and C, the closest alternative model to the Total Level is CS3X plus the tide; this is not physically plausible and suggests that the empirical correction to the tide is responsible for the results. In conclusion, any apparent forecasting skill is entirely due to correcting the tidal predictions where there now exists a more robust method based on combining tide tables with real-time tide gauge data. We therefore repeat the Olympic medal table approach to determine the skill of different means of obtaining the total sea level, ignoring the Total Level model for the reasons above; scoring methods are defined in the table captions. The outcome from this evaluation restricted to those occasions where an SRM forecast is available is shown in Table 17. We perform a similar analysis restricted to where the SRM is not available in Table 8. CS3X 8 BCM 21 SRM 16 Table 17: Cumulative skill score (1 st = 2; 2 nd = 1; 3 rd = 0) only where SRM forecast is available (i.e. for Avonmouth, Hinkley Point and Newport) CS3X 6 BCM 6 Table 18: Cumulative skill score (1 st = 1; 2 nd = 0)where SRM forecast is NOT available (i.e. for Ilfracombe and Mumbles) For Table 20 below, we isolate the analysis for each tide gauge location using the same scoring system and also calculate the RMS error in the total sea level. CS3X BCM SRM Score RMS Score RMS Score RMS Avonmouth Ilfracombe Hinkley Point Mumbles Newport Table 19: Cumulative skill score (1 st = 2; 2 nd = 1; 3 rd = 0 OR 1 st = 1; 2 nd = 0 if SRM not available) broken down by tide gauge location and RMS error (m) of total sea level (predicted tide plus modelled residual at predicted HW). In the table above, the model value added to the harmonically predicted tide is the forecast residual at the time of harmonically predicted HW; although model high water normally coincides with the harmonically predicted high water, there are occasions where it differs. The more physically realistic summation would be the 33

34 model residual at the time of model high water. We repeat the port-by-port analysis in Table 20 below, this time deriving the total sea level by adding the residual at model high water to the predicted tide. CS3X BCM SRM Score RMS Score RMS Score RMS Avonmouth Ilfracombe Hinkley Point Mumbles Newport Table 20: Cumulative skill score broken down by tide gauge location and RMS error (m) of total sea level. As Table 19 but deriving total sea level by adding the model residual at model high water to the predicted tide. Using the residual at model high water does not change the overall pattern of results, with the BCM still the best all-round performing model. The main change to the results is to improve the RMS error of CS3X, specifically at Ilfracombe and Newport. 4. Conclusions We have identified and analysed the five largest storm surge events which occurred during the last five years in the Bristol Channel. In general, a careful analysis of significant events is more meaningful than comparisons based on monthly RMS errors (where inaccuracy of the tidal prediction dominates the statistics). Firstly, forecast residual data from the three Bristol Channel operational models - CS3X, BCM and SRM were compared with observations. Next, we compared the total predicted sea level at each of five tide gauge locations by combining modelled residual with the harmonically predicted tide. The physically correct combination is to add the residual at model high water to the predicted tide (even if the times of high water differ). Another point worth noting is that these analyses are the statistics of a small number of events: a sixth weather event would doubtless change these findings in some way. The Bristol Channel has the third highest tidal range in the world (a spring tidal range of approximately 16m in places). As forecasters wish to know total water level as accurately as possible, then having the most accurate determination of the tide is critical. In 2010 the National Oceanography Centre was commissioned by the Environment Agency to devise a method (Hibbert et al., in preparation) to improve tidal predictions by correlating with real-time tide gauge measurements from the previous three days; this approach is able to provide significant reductions to harmonic prediction errors and is superior to the ad-hoc tuning of model levels based on a single year of comparisons (Williams and Horsburgh, 2009). The results presented here therefore supersede any recommendations in Williams and 34

35 Horsburgh (2009). The best method of deriving total water levels in the region is to make the empirical correction to tidal predictions (Hibbert et al., in preparation) and then add the high water model residual. When the empirical tidal corrections were applied to the events studied here, the error in predicted tide was reduced in 17 out of 24 cases. Applying the tidal correction did not significantly change the comparative model performance shown in Table 20. Our analysis of non-tidal residuals (Section 2) suggests that the finer resolution models (BCM, SRM) generally provide more accurate residuals. This is not surprising since these models have the spatial resolution to simulate local dynamic mechanisms that generate shallow water waves on the same spatial scale as the estuary dimensions. The analysis of predicted total water levels (Section 3) delivers some important conclusions. Table 17 suggests that the shelf-wide CS3X model does not provide as useful a high water residual as the two finer resolution models. Table 18 implies that for the two outer estuary locations (Ilfracombe and Mumbles) the CS3X model and BCM both have skill. This would suggest that the best forecasting technique would be a port-by-port local ensemble that takes the relevant model outputs into account. The port-by-port analyses of Tables 19 and 20 are remarkably robust given the small number of events available. In every case the skill score correlates with the RMS error (i.e. the model that is most frequently the most accurate also has the smallest RMS error). On the basis of this table, forecasts for the different locations should consider outputs from a particular combination of models. For Avonmouth, Hinkley Point and Newport the two fine resolution models (BCM, SRM) have comparable skill with CS3X generally performing less well (except for Newport where it exhibits some skill). For the outer estuary ports of Ilfracombe and Mumbles, the best approach would be to focus on CS3X and BCM (with CS3X the better predictor for Mumbles albeit with this limited sample). These recommendations do not imply that these optimum local approaches outweigh the requirement to understand the larger scale uncertainty obtained from the full MOGREPS-driven surge ensemble based on CS3: differences in the full ensemble provide essential information about uncertainty in the forcing weather systems. To put it another way, whilst BCM and SRM give the most useful high water surge predictions for Avonmouth, the spread in CS3 derived from the full ensemble remains the best measure of the meteorological uncertainty. It is therefore recommended that forecasting methods for the Bristol Channel area should adopt an appropriate combination of surge model outputs combined with tidal predictions modified empirically using real-time tide gauge data. The predictive skill of the finer resolution models revealed by this study indicates that these should be retained within the present system. The Total Level model does not provide any additional forecasting skill once the Hibbert et al. (in preparation) tidal corrections are available, and at that time this tuned model could be disabled or removed from future developments of the operational system. 35

36 Any future development of the surge model suite for UKCMF should ensure that model performance in the Bristol Channel delivers comparable accuracy to the values presented in this report. The most compelling reason for development of a new surge model suite is robustness of the software in the context of supercomputer upgrades, but other factors include the desirability of fully coupling the operational storm surge and wave models. On the basis of these results, the most accurate performing model overall was the Bristol Channel Model (BCM) at approximately 4km horizontal resolution. Any future development of the surge model suite should therefore aim for comparable resolution at the shelf-scale; this would maintain the level of skill reported herein and also facilitate an improved operational ensemble whilst remaining compatible with atmospheric model resolution (e.g. UKV) and wave model requirements. References Hibbert, A., Royston, S., and Horsburgh, K.J. (2013) An empirical approach to improving tidal predictions using recent real-time tide gauge data (in preparation) Horsburgh, K.J. and Wilson, C. (2007) -surge interaction and its role in the distribution of surge residuals in the North Sea. Journal of Geophysical Research Oceans, 112, C08003, doi: /2006jc Williams, J.A. and Horsburgh, K.J. (2009) Evaluation and re-tuning of the local surge models for the Bristol Channel and Severn Estuary. POL Internal Report No. 196, 28pp, Proudman Oceanographic Laboratory, November

37 Annex 1 Composite of Tables 1-5 PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 1: Maximum model v observed residual 4 th December PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 2: Maximum model v observed residual 17 th January PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a n/a n/a n/a Hinkley Point Mumbles n/a Newport Table 3: Maximum model v observed residual 14 th November PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 4: Maximum model v observed residual 11 th November PORT OBS CS3X BCM SRM Avonmouth Ilfracombe n/a Hinkley Point Mumbles n/a Newport Table 5: Maximum model v observed residual 3 rd January