Data assimilation and adaptive forecasting of water levels in the river Severn catchment, United Kingdom

Transcription

1 WATER RESOURCES RESEARCH, VOL. 42, W06407, doi: /2005wr004373, 2006 Data assimilation and adaptive forecasting of water levels in the river Severn catchment, United Kingdom Renata J. Romanowicz, 1 Peter C. Young, 1,2 and Keith J. Beven 1 Received 18 June 2005; revised 6 December 2005; accepted 24 February 2006; published 14 June [1] This paper describes data assimilation (DA) and adaptive forecasting techniques for flood forecasting and their application to forecasting water levels at various locations along a 120 km reach of the river Severn, United Kingdom. The methodology exploits the top-down, data-based mechanistic (DBM) approach to the modeling of environmental processes, concentrating on the identification and estimation of those dominant modes of dynamic behavior that are most important for flood prediction. In particular, hydrological processes active in the catchment are modeled using the state-dependent parameter (SDP) method of estimating a nonlinear, effective rainfall transformation together with a linear stochastic transfer function (STF) method for characterizing both the effective rainfall river level behavior and the river level routing processes. The complete model consists of these lumped parameter, linear and nonlinear stochastic, dynamic elements connected in a quasi-distributed manner that represents the physical structure of the catchment. The adaptive forecasting system then utilizes a state-space form of the complete catchment model, including allowance for heteroscedasticity in the errors, as the basis for data assimilation and forecasting using a Kalman filter forecasting engine. Here the predicted model states (water levels) and adaptive parameters are updated recursively in response to input data received in real time from sensors in the catchment. Direct water level forecasting is considered, rather than flow, because this removes the need to transform the level measurement through the rating curve and tends to decrease the forecasting errors. Citation: Romanowicz, R. J., P. C. Young, and K. J. Beven (2006), Data assimilation and adaptive forecasting of water levels in the river Severn catchment, United Kingdom, Water Resour. Res., 42, W06407, doi: /2005wr Introduction [2] In this paper, flood forecasting is understood in a specific sense: namely the derivation of real-time updated, online forecasts of the flood level at certain strategic locations along the river, over a specified time horizon into the future, based on the information about the rainfall and the behavior of the flood wave upstream. Depending on the length of the river reach and the slope of the river bed, a realistic forecast lead time, obtained in this manner, may range from hours to days. The information upstream can include the observations of river levels and/or rainfall measurements. In the situation where meteorological ensemble forecasts are available, they can be used to further extend the forecast lead times, as in the approach presented by Krzysztofowicz [1999, 2002a, 2002b] and Pappenberger et al. [2005], but we have not utilized such information in this paper. [3] The flow forecasting procedures described in the paper are incorporated within a two-step data assimilation (DA) procedure based on data-based mechanistic (DBM) 1 Institute of Environmental and Natural Sciences, Lancaster University, Lancaster, UK. 2 Also at Centre for Resource and Environmental Studies, Australian National University, Canberra, ACT, Australia. Copyright 2006 by the American Geophysical Union /06/2005WR W06407 models [e.g., Young, 2002a, 2002b], formulated within a stochastic state space setting for the purposes of recursive state estimation and forecasting. In the first step, available observations of rainfall and river levels at different locations along the river are sequentially assimilated into the forecasting algorithm, based on the statistically identified and estimated DBM dynamic models, in order to derive the multistep-ahead forecasts. These forecasts are then updated in real time, using a Kalman filter based approach [Kalman, 1960], when new data become available. The incorporation of new observations into the dynamic model via DA is performed online for every time step of the forecasting procedure. [4] DA techniques have found wide application in the fields of meteorology and oceanography and an extensive review of sequential DA techniques, together with examples of their application in oceanography is presented by Bertino et al. [2003]. The problems described there involve the integration of multidimensional, spatiotemporal observations into fully distributed numerical ocean models and thus differ from the flood forecasting systems, which have much smaller spatial dimensionality (these are usually interconnected quasi-distributed systems of lumped rainfall-flow and flow routing models), but require longer lead times for the forecasts. However, certain aspects of the problem remain the same and this has led to recent applications of the ensemble Kalman filter (ENKF) in a flow forecasting context. 1of12

2 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 [5] The ENKF was developed by Evensen [1994] as an alternative to the extended Kalman filter (EKF) approach. The ENKF is the adaptation of Kalman filter (KF) model to nonlinear systems using Monte Carlo sampling in the prediction (or propagation) step and linear updating in the correction (or analysis) step. It has been applied to rainfallflow modeling by Vrugt et al. [2005] and Moradkhani et al. [2005b]. In these works, the authors exploit sequential, ENKF-based estimation of both the hydrological model parameters and the associated state variables for a single rainfall-flow model (in contrast to the present paper which considers a much more complicated, quasi-distributed catchment model, involving a number of interconnected rainfall-level and routing models). Moradkhani et al., [2005a] also apply a particle filter (PF) algorithm to implement sequential hydrologic data assimilation. Yet another approach to data assimilation is presented by Madsen et al. [2003] and Madsen and Skotner [2005], who apply updating of the modeling error of the distributed Mike-11 flow forecasts at the observation sites using a constant-in-time, proportional gain depending on the river chainage. Essentially, this gain is included to adjust for a hydrological model bias. [6] One problem with both PF and ENKF is that they are computationally intensive approaches to DA and forecasting, requiring many Monte Carlo realizations at each propagation step. Moreover, due to the high complexity of these approaches, there may be questions about the identifiability of parameters involved in the different aspects of the applied routines (e.g., the estimation of the variance hyperparameters associated with stochastic inputs during the state and parameter estimation processes). In order to reduce the computational burden of ENKF and other Kalman filter based schemes, regularization may be introduced, as discussed by Sørensen et al. [2004]. However, as we shall see in this paper, the relatively simple nature of the rainfall-flow and flow routing processes means that there are simpler and computationally much less intensive alternatives to DA that are able to provide comparable forecasting performance. [7] There are a number of simplified, conventional approaches to flow routing that have been applied to flow forecasting. Among others, for example, these include the Muskingham model with multiple inputs [Khan, 1993], multiple regression (MR) models [Holder, 1985], or autoregressive (AR) models [Thirumalaiah and Deo, 2000]. The Muskingum model is deterministic and does not give the required uncertainty bounds for the forecasts. Moreover, all of these more conventional models have a completely linear structure and so do not perform as well as nonlinear alternatives within a rainfall-flow context. In contrast, a considerable amount of research has been published recently on the application of nonlinear methods in flood forecasting. Among others, Porporato and Ridolfi [2001] present the application of a nonlinear prediction approach to multivariate flow routing and compare it successfully with ARMAX model forecasts. However, this approach does not have a recursive form and requires online automatic optimization when applied to online forecasting. Another nonlinear approach is the application of neural networks (NN) for flood forecasting [e.g., Thirumalaiah and Deo, 2000; Park et al., 2005]. As discussed in these papers, NN models can yield better forecasts than conventional linear models and, if designed appropriately, they also allow online data assimilation. However, the NN based models normally have an overly complex nonlinear structure and so can provide overparameterized representations of the fairly simple nonlinearity that characterizes the rainfall-flow process (see section 2.1, equation (1a)). They are also the epitome of the black box model and provide very little information on the underlying physical nature of the rainfall-flow process: information that can provide confidence in the model and allows for better implementation of the forecasting algorithm within a recursive estimation context. [8] In this paper, we consider another, newer approach to simplified modeling that utilizes statistical estimation to identify the special, serially connected nature of the rainfall input nonlinearity in the rainfall-water level process and then exploits this in order to develop a computationally efficient forecasting algorithm. This is based on statistically estimated, stochastic-dynamic DBM models of the rainfalllevel and level routing components of the system, which are then integrated into an adaptive version of the standard recursive Kalman filter (KF) state estimation algorithm that generates both the state variable forecasts and their 95% confidence bounds. Here, the state variables are defined as the fast and slow flow levels (see later, section 2.2), that together characterize the main inferred variables in the identified DBM rainfall-level models. The serial input nonlinearity in the rainfall-flow model is obtained via a nonlinear transformation of rainfall, the nature of which is estimated directly from the data using a method of statedependent parameter (SDP) estimation for nonlinear stochastic systems [e.g., Young, 2000, 2001]. In contrast to the model predictions produced by a fully distributed parameter model, the water level forecasts in this case are made only at the location of measurements, in accordance with the goal of the flood forecasting system under consideration. Of course, the resulting water level forecasts could be used as an input to a fully distributed flood forecasting model [Romanowicz and Beven, 1998; Romanowicz et al., 2004a]; or they may be used to derive the risk maps at the gauging sites along the river, if the necessary elevation data are available. [9] In regard to model and forecast updating, Refsgaard [1997] gives a review of different updating techniques used in real time flood forecasting systems, as well as the comparison of two different updating procedures applied to a conceptual hydrological model of the catchment, including rainfall-flow and flow routing models. In relation to his classification, the methods developed in the present paper utilize both recursive parameter and state updating. However, unlike approaches such as the EKF algorithm used by Refsgaard, where the updating is carried out within a single, nonlinear, state space setting, with the parameters considered as adjoined state variables, our state and parameter estimation procedures are carried out separately but concurrently, employing coordinated recursive estimation algorithms, as used by Young [2002a, 2002b]. This avoids the well known deficiencies of the EKF (such as problems with covariance estimation and convergence for multidimensional, nonlinear models) and yields more statistically efficient estimates of the model parameters. On the basis of the results of previous research [Romanowicz et al., 2004b], 2of12

3 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 the present study also considers an implementation based on the modeling and forecasting of water level (stage) rather than flow. This approach avoids the errors introduced by conversion of levels to flow and yields directly the forecasts of water levels that are normally required for flood forecasting and warning. [10] The methodology used in this study is presented in the next section. It exploits the top-down, data-based mechanistic (DBM) approach to the stochastic-dynamic modeling of environmental processes, concentrating on the identification and estimation of those physically interpretable, dominant modes of dynamic behavior that are most important for flood prediction [e.g., Young, 2002a, 2002b]. In particular, hydrological processes active in the catchment are modeled using the SDP method of estimating the location and nature of significant nonlinearities in the system (here the effective rainfall input nonlinearity), together with a stochastic transfer function (STF) method for characterizing both the linear effective rainfall water level behavior and the water level routing processes. The complete model consists of these linear and nonlinear, stochastic-dynamic elements connected in a manner that represents the physical structure of the Severn River catchment. [11] The adaptive forecasting system utilizes a state space form of the complete catchment model, including allowance for heteroscedastic errors, as the basis for data assimilation and forecasting using a standard Kalman filter forecasting engine. Here, both the predicted model states (water levels including the inferred fast and slow components: see section 2.2), as well as adaptive parameters that allow for unpredictable changes in the catchment behavior and state-dependent heteroscedasticity (changing variance) associated with the model residuals, are updated recursively in response to input data received in real time from remote sensors in the catchment. It is also worth noting that this methodology may be used when backwater effects are not negligible (e.g., from tidal effects). Such effects can be considered as a second (downstream) input to the STF model and so that a multi-input, single-output (MISO) model can be incorporated within the same methodological framework. This latter approach was not necessary in the present case study but it is currently under investigation in other applications. 2. Methodology [12] In this paper, we develop further an earlier approach to forecasting system design for a single component rainfallflow model [Young, 2002a, 2002b] so that it can be used with a complete, multicomponent, quasi-distributed model of the river Severn catchment in the United Kingdom. An early version of this system has been outlined by Romanowicz et al. [2004b]. The present paper describes, in some detail, a considerably enhanced version, where water levels are forecast instead of flows and the online derivation of gain and variance of the forecasts is modified. The changes introduced into the forecasting system presented here were incorporated to facilitate its application within the National Flood Forecasting System (NFFS) developed for the UK Environment Agency by Delft Hydraulics [Beven et al., 2006]. The extension also includes the application of a larger number of rainfall gauging stations; the development of an enhanced method of dealing with the rainfall-water level nonlinearity; and a simpler, more robust method of accounting for the heteroscedastic variance. As a result of these changes, the forecasts have smaller bias and longer lead times. [13] The use of water levels, instead of flows, enables much better utilization of existing water level observations, including those stations for which rating curves do not exist or are not reliable. All of the river gauging stations provide water level measurements which, as usual, could be transformed into flow using the rating curve specific to each station. However, variations in flow velocity and the way that losses of energy due to friction change with water level and gradient, mean that this transformation is not very well defined, particularly for high flows. Moreover, it is normally based on historic calibration that may well have become out of date or could have been changed during an extreme flood event. Hence, by using level measurements instead of the flow, we bypass these uncertainties related to level-flow conversion. Additionally and conveniently, we also tend to decrease the heteroscedasticity of the prediction errors, as explained below. As water levels are usually measured relative to a base level (i.e., historically justified minimum water level at a given gauging station) a correction for this reference datum is introduced for each gauging station Nonlinear Rainfall-Water Level Model [14] At any level measurement location, it is assumed that the rainfall and water level measurements above the base level, r(t) and y(t), are sampled uniformly in time at a sampling interval of Dt time units (here hours) and that these discrete time, sampled measurements are denoted by r k and y k. It has been shown [Young, 1993, 2001, 2002a, 2002b, 2003; Young and Beven, 1994; Young and Tomlin, 2000; Romanowicz et al., 2004b] that the nonlinearities arising from the relationship between measured rainfall r k and transformed (effective) rainfall, denoted here by u k, can be approximated using gauged flow or water levels as a surrogate measure of the antecedent wetness or soil water storage in the catchment. In particular, the scalar function describing the nonlinearity between the rainfall and the soil moisture surrogate y k is initially identified nonparametrically using SDP estimation. As in the case of the flow modeling and forecasting studies considered previously and cited above, this shows that the nonlinearity can often be parameterized to reasonable accuracy by a power law relationship. In the present application we modified this relation to take into account the flattening of the flood wave due to the overbank flooding. It should be mentioned that this flattening effect is visible both for the flows and water levels. The relation takes the form u k ¼ c 0 cy ð k Þy g k r k; cy ð k Þ ¼ 1 for y k < y 0 c p for y k y 0 ð1aþ where u k denotes the transformed (effective) rainfall; r k denotes measured rainfall; c 0 is a scaling constant; c p (0 < c p 1) is a constant describing the degree of flattening of the flood wave; and y 0 is related to the bankfull water level. The power law exponent g and constants c p and y 0 are estimated by a special optimization procedure that includes the concurrent optimal instrumental variable estimation (see section 2.2 and Young [1984]) of the following linear, stochastic transfer 3of12

4 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 function (STF) model between the delayed, transformed rainfall u k d and the water level y k. y k ¼ Bðz 1 Þ Aðz 1 Þ u k d þ x k ð1bþ These decomposed components, as obtained by partial fraction expansion [e.g., Young, 2006], take the following form: Fast component Here d is a pure, advective time delay of ddt time units, while A(z 1 ) and B(z 1 ) are polynomials in the backward shift operator z 1 : (i.e., z 1 y k = y k i ) of the following form: Slow component b 1 y 1;k ¼ 1 þ a 1 z 1 u k d ð3þ Aðz 1 Þ¼1þa 1 z 1 þ a 2 z 2 þþa n z n ; Bðz 1 Þ¼b 0 þ b 1 z 1 þ b 2 z 2 þþb m z m Combining equations (1a) and (1b) and incorporating the scaling constant c 0 into c(y k ), the complete rainfall-flow model can be written as: y k ¼ b 0 þ b 1 z 1 1 þ a 1 z 1 þ a 2 z 2 u k d þ x k u k ¼fcðy k Þy g k gr k x k ¼ Nð0; s 2 k Þ For simplicity, the additive noise term x k in (2) is assumed here to be a zero mean, normally distributed, white noise sequence (i.e., uncorrelated in time), although an extension of the model to incorporate colored noise is straightforward [Young, 2003]. It is also assumed that x k, which accounts for all the uncertainty associated with the inputs affecting the model, including measurement noise, unmeasured inputs, and uncertainty in the model, is heteroscedastic (i.e., its variance s k 2 changes over time) and that it is not significantly correlated with the rainfall measurement. The orders of the polynomials n and m are identified from the data during the estimation process and are normally in the range 1 3. In the following analysis, the triad [n md] is used to characterize this model structure. [15] The STF model (1b) can be written in the alternative discrete time equation form, y k ¼ a 1 y k 1 a 2 y k 2 a n y k n þ b 0 u k d þ b 1 u k d 1 þþb m u k d m þ h k where h k = A(z 1 )x k is the transformed noise input. This shows that the river level at the kth hour is dependent on level and effective rainfall measurements made over previous hours, as well as the uncertainty h k arising from all sources. Note that although the STF relationship between effective rainfall and water level is linear, the noise h k is dependent on the model parameters, thus precluding linear estimation, and the complete model between measured rainfall and water level is quite heavily nonlinear because of the effective rainfall nonlinearity in (1a) State Space Formulation of Rainfall-Water Level Model and Parameter Updating [16] The mechanistic interpretation of the model (2) follows from the decomposition of the linear STF part of this model in (1b) into fast and slow components that, in broad terms, reflect the fast and slow physical processes operative in the catchment (see the previous references). ð2þ b 2 y 2;k ¼ 1 þ a 2 z 1 u k d where a 1, a 2, b 1, b 2 are parameters derived from (1b) and the total gauged water level is the sum of these two components and a model error, i.e. y k = y 1k + y 2k + x k. The associated residence times (time constants), T 1, T 2, steady state gains, G 1, G 2, and partition percentages, P 1, P 2, are given by the following expressions: T i ¼ Dt log e ða i Þ ; i ¼ 1; 2 G i ¼ b i ; i ¼ 1; 2 1 þ a i P i ¼ 100G i ; i ¼ 1; 2 G 1 þ G 2 The parameters of this model are derived from statistical model identification and estimation analysis based on the observed rainfall-water level data. Here, the statistically optimal RIV (refined instrumental variable) algorithm from the CAPTAIN toolbox ( captain/) for Matlab TM and the associated DBM statistical modeling concepts are used to identify the order of the STF model (the triad [n md]) and to estimate the associated parameters (see previous references). In this manner, the DBM model efficiently reflects the information content of the data, so that the possibility of overparameterization and associated poor identifiability is avoided. A comprehensive description of the DBM modeling and physical interpretation of the DBM models is given by Young [2000, 2002a, 2002b, 2003]. [17] For forecasting purposes, the DBM model (2), with the linear STF component (1b) considered in its decomposed form (3), is converted to a stochastic state space model. This then allows for the solution of the resulting state equations within a KF framework and the online, real time estimation of both water level components, as well as the optimization of the hyperparameters (variance/covariance of the stochastic inputs) associated with the state space model, as discussed later. The state space form of the decomposed DBM model can be written as follows: where F ¼ a a 2 x k ¼ x 1;k x 2;k x k ¼ Fx k 1 þ Gu k d þ z k y k ¼ h T x k þ x k G ¼ b " 1 z k ¼ z # 1;k b 2 z 2;k ðþ i ðiiþ h T ¼½1 1Š ð4þ ð5þ 4of12

5 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 The state vector corresponds to the slow and fast component of the rainfall water level process. Here the system noise variables z i,k, i = 1, 2, are introduced to allow for unmeasurable stochastic inputs to the system. For simplicity, it is assumed that these are zero mean, serially uncorrelated and statistically independent random variables, with diagonal covariance matrix Q = diag(q 1, q 2 ). This covariance matrix is optimized together with the heteroscedastic variance s k 2 of the observational noise x k. Note that we are able to use the linear KF here because the effective rainfall nonlinearity in our model resides only at the input to the model (a so-called Hammerstein process in systems terminology) and the variance of the stochastic inputs is allowed to change because the KF was developed for such nonstationary processes. [18] The transformation of rainfall in equation (1a) accounts for the changes in soil water storage and gives a much better explanation of high water level changes, as well as reducing the correlation in the prediction errors. The methodology presented here could allow the residual correlation to be modeled as an AR or ARMA process and introduced into the state space model [Young, 2003], but this was not incorporated in the present study in order to reduce the complexity of the final forecasting system, as required by the design objectives. [19] As regards the heteroscedasticity of the observational noise, Sorooshian and Dracup [1980] present an approach to deal with correlated and heteroscedastic errors of flow measurements based on a maximum likelihood approach and an alternative procedure of either transforming the observation errors using an AR(1) model to obtain uncorrelated error, or introducing a Box Cox transformation [Box and Cox, 1964] to deal with heteroscedasticity. Their methodology follows an en bloc estimation procedure with hydrological model and error transformation parameters treated as deterministic variables. Vrugt et al. [2005], on the other hand, apply a nonparametric, local differencebased estimator of the observation error variance, based on the work by Hall et al. [1990], to model the error heteroscedasticity in their ENKF solution of the rainfall-flow model. [20] In order to account for the heteroscedasticity in the present study, a new approach is used, where the variance s k 2 of x k is assumed to be a known state-dependent function of a simulated output, taking the following form: s 2 k ¼ l 0 þ l 1^y 2 k where l 0, l 1 are the parameters optimized, together with the NVR hyperparameter that controls the gain updating procedure discussed below. The estimate bs k 2 of s k 2 is fed back to the recursive Kalman filter engine as an estimate of the observational variance. The N-step-ahead prediction variance is then given by: varðby kþnjk Þ¼bs 2 k þ ht P kþnjk h where P k+njk is the error covariance matrix estimate associated with the N-step-ahead prediction of the state estimates and h denotes the observation vector (equation (4)). This estimate of the N-step ahead prediction variance is used to derive approximate 95% confidence bounds for the forecasts, under the assumption that the prediction error can be characterized as a nonstationary Gaussian process (i.e., ð6þ ð7þ plus or minus twice the square root of the variance at each time step is used to define the 95% confidence region). [21] It is interesting to note here that an analysis of the heteroscedastic characteristics of the noise, in the above manner, has revealed the advantage of modeling in terms of the water level variable instead of the flow. The transformation of the water levels, through the rating curve, into the flow variable S k may be written in the form: S k ¼ f ðy k Þ According to delta method calculations [Box and Cox, 1964], for a smooth function f(), the asymptotic variance AV(S k ) of the flow S k as y! y k will have the form: where AV ðs k Þ¼ _ f ðy k Þ 2 varðy k Þ _f ðy k Þ¼ df dy j y¼y k Here the function f(y k ) denotes the rating curve relationship which, quite often, can be approximated by a power function of water levels, with the power greater than 1. Hence the variance of the flow is increased in comparison with the water level variance, particularly for larger flows. In this regard, it is interesting to note that Sorooshian and Dracup [1980] regarded the error in the rating curve transformation of water level observations as a main source of heteroscedasticity of the flow data. [22] Another important adaptive element in the forecasting system is gain updating. Here unpredictable, small changes of the system response over time are handled by an online gain updating procedure that is built into the system using real-time recursive estimation [Young, 1984, 2002a, 2002b]. The changes of system response are accounted for, in the simplest possible manner, by introducing an adaptive update of a time-variable gain factor g k in the expression: y k ¼ g k ^y k þ x k Here the variable ^y k denotes the simulated, deterministic output of the second-order STF model (2), x k is the noise term that represents the error in the estimation (lack of fit). For the purposes of time variable parameter estimation, g k is assumed to vary stochastically as a random walk (RW) process with q g defining the noise variance ratio (NVR) hyperparameter associated with the stochastic input to the RW model (NVR is sometimes termed the signal to noise ratio: it is defined as the ratio of the variance of the white noise input into the RW model to the variance of the noise x k ). With this assumption, the gain g k can be estimated in real time using the following scalar recursive least squares (RLS) algorithm [Young, 1984]: p kjk 1 ¼ p k 1 þ q g p k ¼ p kjk 1 p2 kjk 1^y2 k 1þp kjk 1^y 2 k ^g k ¼ ^g k 1 þ p k^y k fy k ^g k 1^y k g where ^g k is the estimate of g k. ð8þ ð9þ ð10þ 5of12

6 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 Figure 1. Part of the river Severn catchment with marked rain and water level gauging stations. [23] The combination of a Kalman filter solution of the state space model with the above variance and gain updating (equations (6), (7), (9), and (10)), provides a tool for online data assimilation. The NVR hyperparameters associated with the stochastic inputs to the state space equation (4) are estimated in the first place by maximum likelihood based on prediction error decomposition [Schweppe, 1973]. Subsequently, in line with the current forecasting objectives, the hyperparameters minimizing the variance of the N-stepahead prediction errors for the maximum water levels are estimated using an optimization routine from the Matlab 1 toolbox (based on the simplex direct search method of Nelder and Mead [1965]). The optimization is performed only during the calibration stage, the optimal values obtained in this manner are then used during the application of the online forecasting scheme. [24] In summary, the procedure for developing a single module of the forecasting system can be summarized as follows. [25] 1. Estimate the complete nonlinear STF model, including the effective rainfall nonlinearity, using the RIV and SDP estimation algorithms in the CAPTAIN toolbox, based on the available input-output data and ensuring that the chosen model is physically meaningful in accordance with DBM modeling requirements. [26] 2. Formulate the state space model and optimize the hyperparameters, without online updating, using maximum likelihood estimation based on prediction error decomposition. [27] 3. Optimize the updating and heteroscedastic variance parameters using a set of optimization criteria based on the N-step-ahead forecast error. [28] Three criteria are used during the identification of the model structure and estimation of its parameters. The first is a coefficient of determination associated with the level predictions, R 2 T =(1 s 2 inn /s 2 y ), presented as a percentage, with s 2 inn, s 2 y denoting the variances of N-step prediction error and observations, respectively. This is a multistepahead forecasting efficiency measure, similar in motivation to the Nash Sutcliffe efficiency measure used for model calibration [Nash and Sutcliffe, 1970]. The others are the YIC and AIC [Akaike, 1974] criteria, which are model order identification (identifiability) statistics: a low relative measure of these criteria ensures that the chosen model has a dynamic structure that reflects the information content of the data, so ensuring well identified parameters and no overparameterization. The R 2 T and YIC measures are discussed by Young [1989] Complete Forecasting System for the Whole Catchment [29] Each rainfall-level forecasting subsystem, as described in section 2.2, together with other such subsystems associated with the linear routing models, constitute modules in the complete forecasting system for the entire catchment. For the lead times not exceeding the natural subreach delay, these modules are effectively independent from each other. However, for larger lead times, when the forecasts of the rainfall-flow and routing modules are introduced instead of observed input, the modules become coupled. The influence of the correlation between the modules on the forecast uncertainty is the subject of a separate study. 3. Case Study [30] The methodology outlined in the previous section has been used to derive an online, adaptive forecasting system for the river Severn in the U.K, as far downstream as the gauge at Buildwas shown in Figure 1. The rain gauge stations are shown as black rectangles in Figure 1. Larger markers represent water level gauging stations. The river Severn above Buildwas has one major tributary, the river Vyrnwy, which enters the main river upstream of Shrewsbury at the Welsh Bridge gauging station. The observations used in this study consist of hourly measurements of rainfall at the Dollyd, Cefn Coch, Vyrnwy Llanfyllin and Pen y Coed gauges in the Upper Severn and Vyrnwy catchments; and water level time series at Meifod on the Vyrnwy, and Abermule, Welsh Bridge (Shrewsbury) and Buildwas on the river Severn. The Severn catchment area at Abermule is 580 km 2, at Welsh Bridge 2325 km 2 and at Buildwas 3717 km 2. The Vyrnwy catchment area at Meifod is 675 km 2. The length of the river Severn between Abermule and Welsh Bridge is about 80 km and the distance between Welsh Bridge and Buildwas is about 37 km. We use the October 1998 year flood event data for the calibration of the models. The 6of12

7 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 Figure 2. Schematic representation of the rainfall water level water level forecasting system for the Abermule Buildwas reach of river Severn, United Kingdom, incorporating the river Vyrnwy tributary of the river Severn above Welsh Bridge. validation event starts at 9:00 on 24 October 2000 for all the stations under consideration. [31] The aim of the forecasting system design is the development of a relatively simple, robust, online forecasting system, with acceptable accuracy and the longest possible lead time. In order to maximize the lead times, rainfall measurements are used as an input to the rainfallwater level predictions in the upper part of the Severn catchment. The sequential structure of the forecasting model is shown in Figure 2. In the following subsections, we present the rainfall-water level and water level routing models for the river Severn from Abermule to Welsh Bridge and Welsh Bridge to Buildwas. The water level routing model for Welsh Bridge uses the averaged water levels from Abermule (Severn) and Meifod (Vyrnwy) as a single input variable: the use of these observations as two separate input variables was not possible due to their high cross correlation and the associated poor identifiability of the model parameters (multicollinearity). All the STF models in this network are estimated using the RIV estimation procedures from the CAPTAIN toolbox (see previous section). The results from the RIV/SDP analysis of the STF model structures for Abermule, Meifod, Welsh Bridge and Buildwas, as well as their diagnostic performance measures are summarized in Table Rainfall Water Level Model for Abermule [32] The rainfall-water level model for Abermule uses rainfall measurements from three gauging stations in the Upper Severn catchment, Cefn Coch, Pen y Coed and Dolydd, as inputs. The weights for the rainfall measurements are derived through the optimization of the least square difference between the observed and modeled water levels at Abermule. The nonlinearity between the rainfall and water levels is identified initially using the nonparametric SDP estimation tool from the CAPTAIN toolbox. This nonparametric (graphical) relation is then parameterized using a step-wise power law parameterization of the form given in (2) and the coefficients c p,y 0 and g are optimized simultaneously with the estimation of the other model parameters. The model, after the decomposition into a slow and fast component and the power law transformation of the rainfall, has the form 0:0038 y k ¼ 1 0:9955z 1 u 0:134 k 5 þ 1 0:9304z 1 u k 5 þ x k ; u k ¼ 1 y k < 3:36 y 0:7 k r k ð11þ 0:68 y k 3:36 The identified residence times for the fast and slow flow components of this model are 13 hours and 9 days, respectively. This model explains 93% of the variance of the observations, with the variance of the simulation modeling errors equal to m 2. The prediction error series shows both autocorrelation and heteroscedasticity, even though the latter is noticeably smaller than in the case of the model based on flow rather than water level measurements. In the case of the autocorrelation, an AR (3) model for the noise is identified based on the AIC criterion. The noise model could be incorporated into the forecasting model, as mentioned previously. However, in order to make the whole catchment model as simple as possible, in compliance with the design objectives, it was decided not to do this but to retain the adaptive estimation of heteroscedastic variance of the predictions, with the option of introducing a noise model if the performance needed improvement and the added complexity could be justified at a later stage. Although the standard errors on the parameters of the TF model are generated during the RIV/SDP estimation, the standard error estimates on these decomposed model Table 1. Summary of the Results of the Adaptive Flood Forecasting System for Abermule, Meifod, Welsh Bridge, and Buildwas Model Structure Forecast Lead, h R T 2 Calibration, % R T 2 Validation, % Abermule [2 2 5 ] Meifod [2 2 5 ] Welsh Bridge [1 1 23] Buildwas [2 2 7] of12

8 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 Figure 3. Validation stage of adaptive rainfall water level model for Abermule, river Severn. The solid line denotes the 5-hour-ahead forecasts; the dotted line denotes the observations; the shaded area denotes 95% confidence limits; time is after start of the event (24 October 2000, 9:00). parameters would need to be obtained by Monte Carlo simulation [e.g., Young, 2002b] and this was not thought necessary in this paper. [33] On the basis of these initial results, heteroscedastic variance and the online updating procedures (equations (6), (7), (9), and (10)) were applied to the rainfall-water level model (11). At the calibration stage, the resulting 5-stepahead forecasts of water levels explained 95.4% of the observational variance. The subsequent validation stage was carried out for the year 2000 floods and the results are shown in Figure 3, where the 5-hour-ahead forecast explains 94.5% of the variance of observations, only a little less than that obtained during calibration. The bankfull level at this site is at about 3.7 meters. [34] The results obtained so far, of which those shown in Figures 3 6 are typical, suggest that the adaptive forecasting system predicts the high water levels well, which is the principal objective of the forecasting system design. The magnitude and timing of the lower peaks is not captured quite so well. This may be due to a number of factors such as the changes in the catchment residence times arising from a dependence on the catchment wetness. These possibilities have not been investigated so far, however, because the forecasting performance is considered satisfactory in relation to the current objectives Rainfall Water Level Model for Meifod [35] The same procedure used in the last section was applied to derive the DBM model for the Meifod gauging station on river Vyrnwy. Among the rainfall gauging stations on the Vyrnwy catchment, three were chosen: Pen y Coed, Llanfyllin and Vyrnwy. The weights for the rainfall measurements were optimized, together with the parameters of the DBM model. The best rainfall-water level model, without any adaptive updating, has the same form as for Figure 4. Validation stage of adaptive rainfall water level model for Meifod, river Vyrnwy. The solid line denotes the 5-hour-ahead forecasts; the dotted line denotes the observations; the shaded area denotes the 95% confidence limits; time is after start of the event (24 October 2000, 9:00). 8of12

9 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 Figure 5. Validation stage of adaptive 28-hour-ahead forecasts for Welsh Bridge, river Severn. The solid line denotes the online updated 28-hour-ahead water level forecasts; the dotted line denotes the observations; the shaded area denotes the 95% confidence limits; time is after start of the event (24 October 2000, 9:00). Abermule, [2 2 5], and explains 92% of the variance of the observations. The full model equation, with the TF decomposition and the power transformation of the rainfall is 0:0247 y k ¼ 1 0:9711z 1 u 0:0424 k 5 þ 1 0:8491z 1 u k 5þx k ; u k ¼ 1 y k < 2:1 y 0:1 k r k ð12þ 0:72 y k 2:1 The identified residence times for the fast and slow flow components of this model are 6 hours and 34 hours, respectively. [36] The online updated 5-hour-ahead forecast for Meifod, over the calibration period in October 1998, explains 97% of the variance of observations. The validation was performed on the same time period in the year 2000 as the Abermule model and the results are shown in Figure 4. Here, the model explains 95.3% of the variance of observations. As in the case of Abermule, the results show some nonlinearity in timing. However, we do not have any information about the bankfull level for this site Water Level Routing Model for Welsh Bridge [37] Although, as we shall see, there are signs of nonlinearity in the routing dynamics, the water level routing modeling is restricted here to linear TF models in order to simplify the nature of the overall catchment model. The best results for the water level data between Abermule, Meifod and Welsh Bridge were obtained when the TF model was calibrated on the water level data with the minimum water level at Welsh Bridge removed for identification and estimation purposes. It was then reintroduced for implementa- Figure 6. Validation stage of adaptive 35-hour-ahead forecasts for Buildwas, river Severn. Solid line denotes the online updated 35-hour-ahead water level forecasts; dotted line denotes the observations; shaded area denotes 95% confidence limits; time is after start of the event (24 October 2000, 9:00). 9of12

10 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 tion of the model in the forecasting engine. The first-order model estimated in this manner is given below: it has a time delay of 23 hours and a residence time of about 28 hours. 0:0692 y t ¼ 1 0:9650z 1 u k 23 þ x k ð13þ As mentioned previously, the input u k is the average of the water level measurements at Meifod and Abermule. The 5-hour-ahead forecasts of water levels for Abermule and Meifod enable the Welsh Bridge water level forecasts to be extended to 28 hours, which should be more than adequate for flood warning purposes at Shrewsbury. The resulting, adaptively updated, water level forecast at Welsh Bridge explains 94.7% of the output variation. [38] Validation of the model at Welsh Bridge was performed using all of the upstream models: i.e., rainfall-water level models for Meifod and Abermule and water level routing model for Welsh Bridge, with summed water level observations/forecasts for Abermule and Meifod. The resulting online updated 28-hour-ahead forecasts are shown in Figure 5, validated on year 2000 floods. These forecasts explain 94.8% of the observed water levels variance at Welsh Bridge, a little better than the calibration performance. [39] The above results suggest that there are some nonlinearities in the water level routing process. It is interesting to notice that the rating curve for Welsh Bridge is nearly linear, which means that any nonlinearities in water routing probably results from the overbank flow at the input rather than at Welsh Bridge. An attempt to improve the forecast performance by introducing an SDP estimated nonlinearity transform at the input of the model improved the explanation the data but this did not yield better results during the validation period for this long lead (28 hours ahead) forecast. Nevertheless, the SDP model appears promising and further research on this is proceeding Water Level Routing Model for Welsh-Bridge-Buildwas [40] Calibration of the water level routing model for Welsh Bridge Buildwas, without adaptive updating, resulted in a [2 2 7] model, which explains 98% of the variance of observations with the variance of simulation errors equal to 0.03 m 2. This model has the following decomposed TF form: 0:0034 y t ¼ 1 0:9866z 1 u 0:6456 k 7 þ 1 0:4669z 1 u k 7 þ x k ð14þ where u k is now the recorded water level at Welsh Bridge. Here, the residence times are about 1 hour for the fast component and about 3 days for the slow component. The bankfull level at this site is at about 6 meters. Finally, when the adaptive updating procedures were applied to the model and the 28-hour-ahead forecasts were used to extend the total forecast lead time of the final model to 35 hours, the forecast explains nearly 97% of the variance of observed water levels at Buildwas. [41] The validation of the model was performed on the November floods from the year 2000 and the results are shown in Figure 6. The model explains 96% of the variance of the water level observations at Buildwas. The high peak values are predicted with reasonable accuracy, while the lower water level changes are overpredicted and there is a visible time difference between the simulated and observed water levels for the smaller peaks. [42] The forecasting results described in this and the previous sections show that the forecasting and DA system is already acceptable for most flood warning purposes. However, there is evidence in these results of nonlinearities in the low-flow behavior that are not being fully accounted for by the rainfall-level and level routing models that are the basis for forecasting. A more sophisticated SDP model, allowing for nonlinearly induced changes in the advective time delay, the residence time and the input nonlinearity parameters should be able to correct these residual deficiencies and is the subject of current research. However, as pointed out in the Introduction, one aim of the present study is to produce a forecasting system that is as simple and robust as possible, with good potential for practical application and providing maximum possible leads time. We believe the present system satisfies these objectives. Of course, as the accuracy of the forecasts improves with the decrease of the lead time, the same online forecasting system may be used for the shorter lead times, thus providing much superior short-time forecasts than the long lead time forecasts presented above. 4. Summary and Conclusions [43] The main aim of the study described in this paper is the derivation of a relatively simple and robust, online system for the forecasting of water levels in the river Severn system, with the maximum possible lead time and good forecasting performance under high water level conditions. As a result, it was decided to base the analysis and forecasting system design on the water levels, rather than the more conventional flow variables, in order to obtain superior long-term forecasting performance. This removes the need to introduce the nonlinear water level-flow transformation, thus reducing the reliance on the prior calibration of this relationship and conveniently reducing the magnitude of the heteroscedasticity in the model residuals. In addition, improved forecasting performance is obtained by the introduction of real-time adaptive mechanisms that account for changes in both the system gain and the variance of the forecasting errors (heteroscedasticity). [44] The data assimilation approach developed and used in this paper performs the updating sequentially, at each measurement location. The forecast uncertainty of this decomposed model will be different from the uncertainty of the entire system, unless the correlation between the observations is negligible. As this is not the case, the predicted uncertainty is probably a little smaller than it should be. However, we decided to use the decomposed system for the derivation of model parameters due to much better identifiability of this decomposed form and the associated higher robustness of the resulting model. Further work is being undertaken on the propagation of uncertainty in the entire forecasting system and initial results are presented by Romanowicz et al. [2006]. However, in order to check the estimates of the uncertainty bands of the forecasts, we applied an empirical approach based on the observed behavior of the past forecast [Gilchrist, 1978], which confirmed that they are reasonable. 10 of 12

11 W06407 ROMANOWICZ ET AL.: ADAPTIVE FORECASTING SYSTEM W06407 [45] An important contribution of the paper is the development of an adaptive forecasting system for a complete catchment based on a statistically estimated stochasticdynamic, nonlinear model composed of interconnected rainfall-level and level routing modules. Although this adaptive forecasting system design is generic and could be applied to a wide variety of catchment systems, it has been developed specifically for the river Severn between Abermule and Buildwas, including the effects of the Vyrnwy tributary. As such, it is a quasi-distributed system: i.e., spatially distributed inputs but lumped parameter component models that consist of four main submodels: two rainfall-water level models and two water level routing models for the two river gauging stations along the Severn at Welsh Bridge and Buildwas. The forecasting lead times at Buildwas are extended for up to 35 hours ahead by exploiting the lead times obtained from the combined rainfall-water level models at Abermule and Meifod (5 hours) and the water level routing model at Welsh Bridge (23 hours). [46] The forecasting system exploits a development of the online updating methods developed by Young [2002a, 2002b]. Forecasting is achieved through a sequential data assimilation procedure. Here, the available observations of rainfall and water levels at different locations along the river are sequentially assimilated into the Kalman filter based forecasting engine, based on the off-line calibrated models, in order to derive the forecasts. These forecasts and the adaptive gains are updated recursively, in real time, as the new data become available from the remote sensors, so providing an adaptive capability that is better able to handle unforeseen changes in the catchment dynamics or lateral input effects. [47] Without introducing external rainfall forecasts, the maximum length of the forecasting lead time that can be handled successfully by this adaptive forecasting system depends on the advective delays that are inherent in the rainfall-water level dynamics; delays that depend on factors such as the changing speed (celerity) as the flood wave moves along the river and the flood starts to inundate the flood plain. The gain adaptation can only offset this nonlinearity in the effective time delay partially, but this still results in acceptable flood peak predictions. The result is a state/parameter updating system, where the parameter updating is limited to the minimum that is necessary to achieve acceptable long-term forecasting performance, combined with good potential for practical robustness and reliability. In this sense, the system represents a sophisticated development of the flood warning system for Dumfries [Lees et al., 1994] that has been operative since the early 1990s. [48] In order to achieve the main aim of the study, namely acceptable long-horizon flood forecasting, the research reported here has concentrated on a design that achieves good forecast accuracy at high water levels for long lead times. Our results show that the changes of water level dynamics for these high peak values can be predicted adequately using the recursively updated adaptive gain and variance parameters. As a flood event continues, the same methodology can be used to produce accurate forecasts with much smaller prediction variance for shorter lead times. However, there remain some noticeable differences in the forecast responses for low-flow values. Consequently, research is continuing in order to reproduce the full nonlinear flow dynamics (i.e., nonlinearly defined advective time delays and residence times) in the forecasts and so achieve good forecasting performance simultaneously for both high and low river water levels. [49] Two other aspects of the proposed adaptive forecasting system design are currently receiving attention. First, the effect of introducing AR or ARMA models for the observational noise at each spatial location, in order to allow for autocorrelation in the prediction errors. Second, nonlinearity in the level routing parts of the system is being investigated using state-dependent parameter modeling. Although, theoretically, both of these innovations should improve forecasting performance, initial results suggest that the added complexity results in less robustness and may not be justified in practical terms. [50] Acknowledgments. This work was done as a part of the EPSRC project Predicting the probability of flooding over long reaches (including real time applications) (GR/R66044/01) and the UK Flood Risk Management Research Consortium Research Priority Area 3 on Real-time Forecasting. Our colleagues from the UK Environment Agency are thanked for supplying the rainfall and flow data, which were used in this research. We would like to thank the anonymous reviewers for very thorough comments that helped in clarification of the paper. References Akaike, H. (1974), A new look at the statistical model identification, IEEE Trans. Autom. Control, 19, Bertino, L., G. Evensen, and H. Vackernagel (2003), Sequential data assimilation techniques in oceanography, Int. Stat. Rev., 71, Beven, K. J., R. J. Romanowicz, F. Pappenberger, P. C. Young, and M. Werner (2006), The uncertainty cascade in flood forecasting, in Innovation, Advances and Implementation of Flood Forecasting Technology: Proceedings of the ACTIF Meeting, Tromsø, edited by P. Balbanis, D. Lambroso, and P. Samuels, p. 33, HR Wallingford, Wallingford, U. K. Box, G. E. P., and D. R. Cox (1964), An analysis of transformations, J. R. Stat. Soc., Ser. B, 26, Evensen, G. (1994), Sequential data assimilation with a nonlinear quasigeostrophic model using Monte Carlo methods to forecast error statistics, J. Geophys. Res., 99, 10,143 10,162. Gilchrist, W. (1978), Statistical Forecasting, John Wiley, Hoboken, N. J. Hall, P., J. W. Kay, and D. M. Titterington (1990), Asymptotically optimal difference-based estimation of variance in nonparametric regression, Biometrika, 77, Holder, L. (1985), Multiple Regression in Hydrology, Inst. of Hydrol., Wallingford, U. K. Kalman, R. (1960), New approach to linear filtering and prediction problems, J. Basic Eng., 82-D, Khan, M. H. (1993), Muskingum flood routing model for multiple tributaries, Water Resour. Res., 29, Krzysztofowicz, R. (1999), Bayesian theory of probabilistic forecasting via deterministic hydrologic model, Water Resour. Res., 35, Krzysztofowicz, R. (2002a), Bayesian system for probabilistic river stage forecasting, J. Hydrol., 268, Krzysztofowicz, R. (2002b), Probabilistic floods forecast: Bounds and approximations, J. Hydrol., 268, Lees, M. J., P. C. Young, K. J. Beven, S. Ferguson, and J. Burns (1994), An adaptive flood warning system for the river Nith at Dumfries, in River Flood Hydraulics, edited by W. R. White and J. Watts, pp., 65 77, John Wiley, Hoboken, N. J. Madsen, H., and C. Skotner (2005), Adaptive state updating in real-time river flow forecasting A combined filtering and error forecasting procedure, J. Hydrol., 308, Madsen, H., D. Rosbjerg, J. Damgård, and F. S. Hansen (2003), Data assimilation in the MIKE 11 flood forecasting system using Kalman filtering, in Water Resources Systems Hydrological Risk, Management and Development, Proceedings of Symposium HS02b Held During IUGG2003 at Sapporo, July 2003, IAHS Publ., 281, Moradkhani, H., K.-L. Hsu, H. Gupta, and S. Sorooshian (2005a), Uncertainty assessment of hydrologic model states and parameters: Sequential 11 of 12