DATA ASSIMILATION FOR FLOOD FORECASTING

DATA ASSIMILATION FOR FLOOD FORECASTING Arnold Heemin Delft University of Technology 09/16/14 1

Data assimilation is the incorporation of measurement into a numerical model to improve the model results Applications: Weather forecasting using numerical models Storm surge forecasting Reconstructing air pollution emissions Ground water flow and transport problems Reservoir models Soil mechanics 09/16/14 2

Some real life applications of data assimilation 09/16/14 3

Grid of storm surge forecasting model 09/16/14 4

Example of a flow pattern 09/16/14 5

Example of a water level forecast 09/16/14 6

09/16/14 7 Grid of Coastal model

Data locations 09/16/14 8

HF radar data 09/16/14 9

Measurements of the vertical velocity profile 09/16/14 10

State space model The (non linear) physics: X = f( X, p, 1) + G ( 1) W 1 1 where X is the state, p is vector of uncertain parameters, f represents the (numerical) model, G is a noise input matrix and W is zero mean system noise with covariance Q The measurements: Z = M ( ) X + V where M is the measurement matrix and V is zero mean measurement noise with covariance R 09/16/14 11

Formulation of the wea constraint data assimilation problem It is desired to combine the data with the stochastic model in order to obtain an optimal estimate of the state and parameters of the system. We define the criterion: J( px, ) = Z MX ( ) K = 1 K R = 1 2 1 2 2 + X f( X, p, 1) T + α p p 1 ( GQG ) 0 C 1 1 09/16/14 12

Overview Ensemble Kalman filtering Classical Kalman filtering Deterministic ensemble Kalman filter for large scale systems: Reduced Ran Square Root filter Stochastic filter: Ensemble Kalman filter Hybrid filters: Square Root Ensemble Kalman filter Two sample filter with application to storm surge forecasting Variational data assimilation Basic idea of variational data assimilation Model reduced variational data assimilation with application to the calibration of a storm surge model 09/16/14 13

Kalman filtering: Linear dynamics F() and constant parameters State space model: The linear physics: X = F ( 1) X + G ( 1) W 1 1 where X is the state, F represents the (numerical) model, G is a noise input matrix and W is zero mean system noise with covariance Q The measurements: Z = M ( ) X + V where M is the measurement matrix and V is zero mean measurement noise with covariance R 09/16/14 14

Linear dynamics F() and constant parameters: The criterion in this case becomes: J( X ) = Z M( ) X R = 1 K = 1 K 2 2 1 + X F ( 1) X T 1 ( GQG ) 1 09/16/14 15

Linear dynamics F() and constant parameters: State estimation using Kalman filtering A recursive algorithm for =1,2, to determine : X P X P f f a a Optimal estimate of the state at time using measurements up to and including -1 Covariance matrix of the estimation error Optimal estimate of the state at time using measurements up to and including Covariance matrix of the estimation error 09/16/14 16

Kalman filter algorithm f X = F ( 1) X, a 1 P = F( 1) P F( 1) + G( 1) Q( 1) G( 1) f a T T 1 a f f X = X + K ( )[ Z MX ( ) ] a P = [ I KM ( ) ( )] P f T f T K( ) = P M( ) [ M( ) P M( ) + R( )] f 1 09/16/14 17

09/16/14 18 Extended Kalman filter algorithm for nonlinear systems 1, 1 1 )] ( ) ( ) ( [ ) ( ) ( )] ( ) ( [ ] ) ( )[ ( 1) ( : 1) ( 1) ( 1) ( 1) ( 1) ( ) ( 1 + = = + = = + = = R M P M M P K P M K I P X M Z K X X x f F w here G Q G F P F P X f X T f T f f a f f a X j i j i T T a f a f a

Large scale models Problems with the applications of the standard Kalman filter to large scale problems: -hugh computational burden -P very ill conditioned -tangent linear system is required 09/16/14 19

SQRT formulation of the covariance P Define S according to P=SS And rewrite the algorithm in terms of S Advantages: -SS always positive definite -S is less ill conditioned (than P) -S can be approximated by a matrix with reduced number of columns Ensemble algorithms are of the square root type and in addition do not require the tangent linear model 09/16/14 20

Reduced Ran square root filtering The square root matrix S is defined according to P=SS where S are the q leading EOF s of P: ξ 0 = xˆ ξ = xˆ + εs i i S is generally of very low ran: q<<n 09/16/14 21

Reduced Ran Square Root filter Representing a q dimensional Gaussian distribution using q+1 ensembles, first order accurate for nonlinear systems Ensemble member Variance of the model uncertainty 9/7/2016 22

Uncented Kalman filter Representing a p dimensional Gaussian distribution using p+2 ensembles, second order accurate for nonlinear systems Ensemble member Variance of the model uncertainty 9/7/2016 23

Summary Reduced Ran filters Each ensemble member is propagated using the original (non linear) model, no tangent linear model is required Errors are caused by truncation of the eigenvectors The algorithm is sensitive to filter divergence problems and, therefore q has to be chosen sufficiently large Computational effort required is approximately q+1 model simulations + eigenvalue decomposition (~q³) 09/16/14 24

Classical Ensemble Kalman filter (EnKF) To represent the probability density of the state estimate N ensemble members are chosen randomly: 1 ˆ N x S = = [... ξ ξ i i N xˆ 1...] 09/16/14 25

Ensemble Kalman filter Representing multi dimensional Gaussian distribution using N random ensembles Ensemble member Variance of the model uncertainty 9/7/2016 26

Summary Ensemble Kalman filter Each ensemble member is propagated using the original (non linear) model, no tangent linear model is required Errors are of statistical nature Errors decrease very slowly with large sample size Computational effort required is approximately N model simulations 09/16/14 27

28 09/16/14 Semi-deterministic schemes: Ensemble Square Root Filters (ESRF) An alternative way to solve the measurement update step is: The general solution is given by: where T is an ensemble transform matrix. R H S H S R S M R M S I S S S P T f f T f f T f f T a a a + = = = ) ( ) ] ( ) ( [ ) ( 1 ) ( ) ( [ 1 f T f T f a M R M S I T T T S S = =

The symmetric ESRF (with symmetric T) Unbiased updated ensemble mean Minimum analysis increment: a X X f 09/16/14 29

Two sample Kalman filter with application to storm surge forecasting (based on J.H. Sumihar, M Verlaan, A. W. Heemin, Monthly Weather Review, 2008) 09/16/14 30

Storm surge forecasting model: A numerical model based on 2D shallow water equations DCSM 09/16/14 31

Storm surge forecasting system with steady state Kalman filter to improve the model results using real-time data wind DCSM X H - y delay + K X: model state, containing values of water level and flow velocities at all grid points. y: water-level observation data. 09/16/14 32

K depends on the model error statistics The success of a Kalman filter depends very much on the specification of model error statistics. 09/16/14 33

Spatial correlation of the system noise of the operational Kalman filter (isotropic) windu windv 09/16/14 34

Two-sample Kalman filter algorithm The well-nown ensemble Kalman filter computes the statistics of the model uncertainty from an ensemble of model simulations at every time when a measurement is available. The two sample Kalman filter computes the statistics of the model uncertainty using only two samples by averaging over time (Sumihar et al., Monthly Weather Review, 2008). 09/16/14 35

Two-sample Kalman filter using two different meteorological models to generate the forecasts: UKMO and HiRLAM KNMI-HiRLAM UKMO Δt 1 The difference between UKMO and HiRLAM is used as proxy to unnown error. Error statistics are computed by averaging the difference over time. 09/16/14 36

Model error spatial correlation computed from UKMO-HiRLAM difference 09/16/14 37

Forecast performance 09/16/14 38

Conclusion The Two sample Kalman filter with two different forecasts of two different meteorological models (UKMO and HIRLAM) improves short term storm surge forecast because of more realistic model error statistics. Next step: Multi sample Kalman filter 09/16/14 39

Overview Ensemble Kalman filtering Classical Kalman filtering Deterministic ensemble Kalman filter for large scale systems: Reduced Ran Square Root filter Stochastic filter: Ensemble Kalman filter Hybrid filters: Square Root Ensemble Kalman filter Two sample filter with application to storm surge forecasting Variational data assimilation Basic idea of variational data assimilation Model reduced variational data assimilation with application to the calibration of a storm surge model 09/16/14 40

The wea constraint data assimilation problem revisited It is desired to combine the data with the stochastic model in order to obtain an optimal estimate of the state and parameters of the system. We define the criterion: K J( px, ) = Z MX ( ) K R = 1 2 1 2 2 + X f( X, p, 1) T + α p p 1 ( GQG ) 0 C 1 1 09/16/14 41

Non linear deterministic systems. Strong constraint variational data assimilation (4Dvar): Parameter estimation For Q=0 the error criterion reduces to: J ( p) Z M ( ) X 1 + α p p 1 = Introducing the adjoint state we can rewrite the criterion: K = 1 2 2 J( p) = Z M( ) X + α p p K R 0 C + v ( X F( X, p, 1)) K = 1 T 1 1 1 09/16/14 42 2 R 0 2 C

Non linear deterministic systems. Strong constraint variational data assimilation (4Dvar): Parameter estimation If we solve the uncoupled system: X = f( X, p, 1) 1 T T 1 1 = ( 1) + ( ) ( ( ) ) v F v M R Z M X X = x, v = 0 0 0 K where F() is the tangent linear model. The gradient of the criterion can be computed by: J p = = K = 0 v T f αc p 1 ( p p0) Very efficient in combination with a gradient-based optimization scheme. 09/16/14 BUT: we need the adjoint implementation! 43

Model reduction with application to variational data assimilation (based on M.U. Altaf, M Verlaan, A.W. Heemin, International Journal on Multi Scale Computational Engineering, 2009) 09/16/14 44

Motivation Adjoint based methods are attractive for parameter estimation problems, but the development of the adjoint is usually too much wor. Non-intrusive model reduction for nonlinear systems is based on an ensemble of model simulations and can be used in data assimialtion to avoid the coding of the adjoint 45 9/7/2016

A POD model reduction approach to data assimilation: The linear case Consider the q dimensional sub space: P = [... p j...] And project the original model onto this sub space: r + 1 Z = [ P T F( ) P] r = [ M ( ) P] r We now have an explicit low dimensional (approximate) system description of the model variations including its adjoint! The sub space can be determined by computing the EOF (Empirical Orthogonal Functions) of an ensemble of model simulations 09/16/14 46 + v

Nonlinear systems We now have to determine: T f i r+ 1 = [ P Pr ] x j For every column l of P we have f x i j p ( f( X + ε p ) f( X )) 1 a a l ε 1 l 1 This is a first order accurate approximation. It is easy to extend this idea to higher order approximations We do not need the tangent linear approximation! 09/16/14 47

MRVDA Re-parameterization Initial Parameters High-order Model Simulation Objective Function Calculation Converged? Done Snapshots Simulation Building of The Low-order Model Low-order Model Simulation Low-order Adjoint Model Simulation Gradient Calculation Parameters Update Reduced Objective Function Calculation Converged? Sup Optimal Parameters 09/16/14 48

Some remars Very efficient in case the simulation period of the ensemble of model simulation is very small compared to the calibration period For many iterations the reduced model can be the same and only the residuals have to be updated 09/16/14 49

Application to the calibration of a numerical tidal model 09/16/14 50

Based on shallow water equations Grid size: 1.5 by 1.0 (~2 m) Grid dimensions: 1120 x 1260 cells Active Grid Points: 869544 Time step: 2 minutes 8 main constituents 09/16/14

DCSM(Water level time series) 09/16/14

09/16/14 53

09/16/14 54

09/16/14 55

09/16/14 56

Experiment with field data Parameter: Depth Calibration run: 28 Dec 2006 to 30 Jan 2007 Measurement data: 01 Jan 2007 to 30 Jan 2007 Includes two spring-neap cycles Assimilation Stations: 35 Validation Stations: 15 Ensemble of forward model simulations for a period of four days (01 Jan 2007 to 04 Jan 2007) 09/16/14

DCSM Divide model area in 4 sub domains + 1 overall parameter No. of snapshots: 132 (Every three hours) 24 POD modes are required to capture 97% energy Same POD modes are used in 2 rd iteration Initial RMS: 25.7 cm After 2 rd iteration: 14.9 cm Improvement : 42% 09/16/14

09/16/14 59

DCSM(Validation results) Similar improvement as in the case of assimilation stations 09/16/14

Computational Cost Estimation 5 parameters, calibration period 1 month: Number of simulations of 1 month: 4.7, reduction criterion 42% (2 iterations, no model update in second iteration) Estimation 20 parameters (4 bottom friction and 16 depth values), calibration period 1 month: Number of simulations of 1 month: 11, reduction criterion 50% (5 iterations, no model update in second and fourth iteration)

Conclusions Model reduced variational data assimilation For the estimation of constant parameters in large scale numerical models the variational methods are generally the most accurate The adjoint implementation can be avoided using model reduction: MRVAR Efficiency MRVAR is very problem dependent More analysis is required 09/16/14 62