Comparison of Ensemble Data Assimilation methods for the shallow water equations model in the presence of nonlinear observation operators

Transcription

1 Comparison of Ensemble Data Assimilation methods for the shallow water equations model in the presence of nonlinear observation operators M. Jardak Laboratoire de Métórologie Dynamique École Normale Supérieure, Paris, France I. M. Navon Department of Scientific Computing, Florida State University Tallahassee, FL , USA M. Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University, 1375 Campus Deliver Fort Collins, CO , USA March 24, 213 submitted to Tellus A Corresponding author: mjardak@lmd.ens.fr 1

2 Abstract A new comparison of three frequently used sequential data assimilation methods illuminating their strengths and weaknesses in the presence of linear and nonlinear observation operators is presented. The ensemble Kalman filter (EnKF), the particle filter (PF) and the maximum likelihood ensemble filter (MLEF) methods were implemented and the spectral shallow water equations model in spherical geometry model was employed using the Rossby-Haurwitz Wave no. 4 test case as initial condition. Numerical tests conducted reveal that all three methods perform satisfactorily in the presence of linear observation operator for 1 to 15 days model integration, whereas the EnKF, even with the Evensen fixture [Evensen 3] for the nonlinear observation operator failed in all tested metrics. The particle filter PF and the hybrid filter MLEF both performed satisfactorily in the presence of highly nonlinear observation operators with a slight advantage in terms of CPU time to the MLEF method. 1 Introduction Sequential data assimilation fuses observations of the current (and possibly, past) state of a system with predictions from a mathematical model (the forecast) to produce an analysis, providing the best estimate of the current state of the system. Central to the concept of sequential estimation data assimilation is the propagation of flow dependent probability density function (pdf) given an estimate of the initial pdf. In sequential estimation, the analysis and forecasts can be viewed as probability distributions. The analysis step is an application of the Bayes theorem. Advancing the probability distribution in time, for the general case is done by the Chapman-Kolmogorov equation, but since it is unrealistically expensive, various approximations operating on representations of the probability distributions are used instead. If the probability distributions are normal, they can be represented by their mean and covariance, which gives rise to the Kalman filter (KF). However, due to the high computational and storage overheads it requires, an approximation based on Monte-Carlo ensemble calculations has been proposed by [Evensen 1994], [Evensen and Van Leeuwen 1996], [Burgers et al. 1998] and [Houtekamer and Mitchell 1998]. The method is essentially a Monte-Carlo approximation of the Kalman filter which avoids evolving the covariance matrix of the state vector. A second type of EnKF filter consists of the class of square root filters of [Anderson and Anderson 23] see also [Bishop et al. 21]. A number of variants of the standard EnKF have been proposed, such as Kalman ensemble square root filters (KSRF), the singular evolutive extended Kalman (SEEK) filter, and the less common singular evolutive interpolated Kalman (SEIK) filter can be found in the paper of [Tippett et al. 23], see also the paper of [Nerger et al. 25] where the aforementioned filters were reviewed and M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 2

3 compared. However, they will not be studied here, as the purpose is rather to investigate differences between EnKF, PF and MLEF in the presence of nonlinear observation operators. The PF methods are also known as sequential Monte-Carlo (SMC) methods or Bayesian filters. The SMC methods are an efficient means for tracking and forecasting dynamical systems subject to both process and observation noise. Applications include robot tracking, video or audio analysis, and general time series analysis see [Doucet et al. 21]. These methods utilize a large number N of random samples named particles to represent the posterior probability distributions. The particles are propagated over time using a combination of sequential importance sampling and resampling steps. Resampling for PF is used to avoid the problem of degeneracy of this algorithm that is, avoiding the situation where all but one of the importance weights are close to zero. The performance of the PF algorithm can be crucially affected by a judicious choice of a resampling method. See [Arulampalam et al. 22] for a listing of the most used resampling algorithms. The PF suffers from the curse of dimensionality requiring computations that increase exponentially with dimension as pointed out by the recent work of [Bengtsson et al. 28] and [Bickel et al. 28] and finally explicitly quantified by [Synder et al. 28]. They indicated that unless the ensemble size is greater than exp(τ 2 /2), where τ 2 is the variance of the observation log-likelihood, the PF update suffers from a collapse in which with high probability only few members are assigned a posterior weight close to one while all other members have vanishing small weights. Recently, it has been proven by Van Leeuwen [[Van Leeuwen 21],[Van Leeuwen 211]] that the above claim of [Bengtsson et al. 28],[Bickel et al. 28] and [Snyder et al. 28], are in error when the proposal densities are taken into consideration. The MLEF filter of [Zupanski 25]; [Zupanski and Zupanski 26] is a hybrid between the 3-D variational method and the EnKF. It maximizes the likelihood of a posterior probability distribution, thus its name. The MLEF belongs to the class of deterministic ensemble filters, since no perturbed observations are employed. As in variational and ensemble data assimilation methods, the cost function is derived using Gaussian probability density function framework. Like other ensemble data assimilation algorithms, the MLEF produces an estimate of the analysis uncertainty ( e.g. analysis error covariance) In this paper data assimilation experiments are performed and compared using the ensemble Kalman filter EnKF as a two-parameter mean and covariance estimation method, the particle filter PF as a full PDF estimation method and the Maximum Likelihood Ensemble Filter MLEF as a two-parameter PDF mode and covariance estimation method. Therefore, the filters used in this study cover the fundamental characteristics of most of the filters present in the literature. These methods were tested on a spectral shallow water equations model using the M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 3

4 Rossby-Haurwitz wave no 4 test case for both linear and nonlinear observation operators. The spectral shallow water equations model in spherical geometry of Williamson [Williamson 1992] and [Jakob et al.1995] is used to generate the true solution and forecasts. To improve EnKF analysis errors and avoid ensemble errors that generate spurious corrections, a covariance localization investigated by [Houtekamer and Mitchell 1998,21] is incorporated. To improve both the analysis and the forecast results, the forecast ensemble solutions are inflated from the mean as suggested in [Anderson and Anderson 1999] and reported in [Hamill et al. 21]. Since the resampling is a crucial step for (PF) method, the systematic, multinomial and the residual resampling methods [Arulampalam et al. 22],[Doucet et al. 21] and [Nakano and al. 27] were tested. The paper is structured as follows. Section 2 presents the spectral shallow-water equations in spherical coordinates model and the numerical methods used for its resolution. In section 3 we present algorithmic details of each of the data assimilation methods considered. In section 4 we present the numerical results obtained and discuss them for both the linear and nonlinear observation operators in several commonly used error metrics. In particular the Talagrand diagrams ( skill or score test) see [Wilks 25], the early work of Murphy [ Murphy 1971 and 1993] and [Talagrand et al ] and the interpretation of [Hamill 21], and the root mean square error (RMSE) are employed to detect ensemble members spread, while the RMSE are used to track filter divergence [Houtekamer 25] Finally section 5 is reserved for summary and conclusions. 2 Shallow-Water equations in spherical geometry The shallow water equations are a set of hyperbolic partial differential equations that describe the flow below a pressure surface in a fluid. The equations are derived from depth-integrating the Navier-Stokes equations, they rely primarily on the assumptions of constant density and hydrostatic balance. The shallowwater equations in spherical geometry are given by u t + u u a cos θ λ + v u a v t + u v a cos θ λ + v v a h t + θ tan θ a θ + tan θ a u h a cos θ λ + v h a θ + h a cos θ g h vu fv = a cos θ λ u2 + fu = g h a θ [ u (cos θ) + λ θ ] = M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 4

5 where V = u i + v j is the horizontal velocity vector ( with respect to the surface of the sphere), gh is the free surface geopotential, h is the free surface height, g is the gravity acceleration. f = 2Ω sin θ is the Coriolis parameter, Ω is the angular velocity of the earth. θ denotes the angle of latitude, µ = sin θ is the longitude. λ the longitude,and a is the radius of the earth. One of the major advances in meteorology was the use by [Rossby 1939] of the barotropic vorticity equation with the β -plane approximation to the spherical of the Earth, and the deduction of solutions reminiscent of some large scale waves in the atmosphere. These solutions have become known as Rossby waves see [Haurwitz 194 ] While the shallow water equations do not have corresponding analytic solutions they are expected to evolve in a similar way as the above R-H equations which explains why they have been widely used to test shallow water numerical models since the seminal paper of [Phillips 1959]. Following the research work of [Hoskins 1973] Rossby -Haurwitz waves with zonal wave numbers less or equal to 5 are believed to be stable. This makes the R-H zonal wave no 4 a suitable candidate for assessing accuracy of numerical schemes as was evident from its being chosen as a test case by [Williamson et al. 1992] and by a multitude of other authors. It has been numerically shown that the R-H wave no 4 breaks down into more turbulent behavior after long term numerical integration as recently discovered by [Thuburn and Li 2] and also by [Smith and Dritschel 26]. The initial velocity field for the Rossby-Haurwitz wave is defined as The initial height field is defined as, u = aω cos φ + ak cos r 1 φ(r sin 2 φ cos 2 φ) cos(rλ) v = akr cos r 1 φ sin φ sin(rλ) h = h + a2 [A(φ) + B(φ) cos(rλ) + C(φ) cos(2rλ)] (2) g (1) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 5

6 where the variables A(φ), B(φ), C(φ) are given by A(φ) = ω 2 (2Ω + ω) cos2 φ k2 cos 2r φ[(r + 1) cos 2 φ + (2r 2 2r 2) 2r 2 cos 2 φ] 2(Ω + ω)k B(φ) = (r + 1)(r + 2) cos r φ[(r 2 + 2r + 2) (r + 1) 2 cos 2 φ] C(φ) = 1 4 k2 cos 2r φ[(r + 1) cos 2 φ (r + 2)] In here, r represents the wave number,h is the height at the poles. The strength of the underlying zonal wind from west to east is given by ω and k controls the amplitude of the wave. 3 Sequential Bayesian Filter: theoretical setting This section applies to the three sequential data assimilation methods discussed herein. One also can see the survey of Diard [Diard et al. 23] and the overview of [Clappe et al. 27]. The sequential Bayesian filter consists in a large number N of random samples advanced in time by a stochastic evolution equation, to approximate the probability densities. In order to analyze and make inference about the dynamic system at least a model equation along with an observation operator are required. Generically, stochastic filtering problem is a dynamic system that assumes the form ẋ t = f(t, x t 1, v t ) (3) y t = h(t, x t, n t ) The first equation of (3) is the state equation or the system model, the second represents the observation equation. The vectors x t and y t are respectively the state and the observation vectors. The state and observation noises are represented by v t and n t respectively. The discrete-time counterpart of the system (3) reads ẋ k = f k (x k 1, v k 1 ) (4) y k = h k (x k, n k ) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 6

7 The deterministic mapping f k : R nx R n d R n x is a possibly non-linear function of the state x k 1, {v k, k N} is an independent identically distributed (i.i.d) process noise sequence, n x, n d are dimensions of the state and the process noise vectors, respectively. Likewise, the deterministic mapping h k : R nx R nn R nz is a possibly non-linear function, {n k, k N} is an i.i.d. observation noise sequence, and n x, n n are dimensions of the state and observation noise vectors, respectively. Let y 1:k denote the set of all available observations y i up to time t = k, (i.e. y 1:k = {y i i = 1,, k}). From a Bayesian point of view, the problem is to recursively calculate some degree of belief in the state x k at time t = k, taking different values, given the data y 1:k up to the time t = k. Equivalently, the Bayesian solution would be to calculate the pdf p(x k y 1:k ). This density will encapsulate all the information about the state vector x k that is contained in the observations y 1:k and the prior distribution for x k. Assume the required pdf p(x k 1 y 1:k 1 ) at time k 1 is available. The prediction stage uses the state equation (4) to obtain the prior pdf of the state variable at time k via the Chapman-Kolmogorov equation p(x k y 1:k 1 ) = p(x k x k 1 )p(x k 1 y 1:k 1 )dx k 1, (5) where the probabilistic model of the state evolution, p(x k x k 1 ) is readily obtained from the state equation (4) and the known statistics of v k 1. As an observation y k becomes available at time t = k, the prior pdf could be updated via the Bayes rule p(x k y 1:k ) = p(y k x k )p(x k y 1:k 1 ), (6) p(y k y 1:k 1 ) where the normalizing constant p(y k y 1:k 1 ) = p(y k x k )p(x k y 1:k 1 )dx k. (7) depends on the likelihood function p(y k x k ), defined by the measurement or observation equation (4) and the known statistics of n k. The relations (5) and (6) form the basis for the optimal Bayesian solution. This recursive propagation of the posterior density is only a conceptual solution. One cannot generally obtain an analytical solution. Solutions exist only in a very restrictive set of cases like that of the Kalman filters for instance (namely, if f k and h k are linear and both v k and n k are Gaussian). We turn now to describe in detail each of the three used filters. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 7

8 3.1 Particle Filters Particle filters see [Doucet et al. 2], [Doucet et al. 21], [Arulampalam et al. 22] [Berliner and Wikle 27A] and recently the review paper of [Van Leeuwen 29] approximate the posterior densities by population of states. These states are called particles. Each of the particles has an assigned weight, and the posterior distribution can then be approximated by a discrete distribution which has support on each of the particles. The probability assigned to each particle is proportional to its weight. See for instance [Metropolis and Ulam 1944], [Gordon et al., 1993] [Doucet et al. 21] and [Stuart 21]. The different (PF) algorithms differ in the way that the population of particles evolves and assimilates the incoming observations. A major drawback of particle filters is that they suffer from sample degeneracy after a few filtering steps. Recently, it has been proven by Van Leeuwen [[Van Leeuwen 21],[Van Leeuwen 211]] that such a sample degeneracy does not take place when the proposal densities are taken into consideration. In this 2-D plus time problem the common remedy is to resample the prior pdf whenever the weights focus on few members of the ensemble. Here we use several strategies such as Systematic Resampling (SR), Residual Resampling (RR) and the Multinomial Resampling (MR). For complete details on the resampling techniques see Gordon et al. [Gordon 1993] and also [Berliner and Wikle 27A and 27B]. The three resampling methods were tested, all yielding similar results but only the SR method was retained since it exhibited the fastest performance. The SR algorithm generates a population of equally weighted particles to approximate the posterior at some time k. This population of particles is assumed to be an approximate sample from the true posterior at that time instant. The PF algorithm proceeds as follows: Initialization: The filter is initialized by drawing a sample of size N from the prior pdf. Filtering: Preliminaries: Assume that {x i k 1 } i=1,,n is a population of N particles, approximately distributed as an independent sample from p(x k 1 y 1:k 1 ) Prediction: Sample N values, {qk 1,, qn k }, from the distribution of v k. Use these to generate a new population of particles, {x 1 k k 1, x2 k k 1,, xn k k 1 } via the equation x i k k 1 = f k (x i k 1, vk) i (8) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 8

9 Filtering: Assign each x i k k 1, a weight qi k. This weight is calculated by q i k = p(y k x i k k 1 ) N j=1 p(y k x j k k 1 ) (9) This defines a discrete distribution which, for i {1, 2,, N}, assigns probability mass q i k to element xi k k 1 Resampling: Resample independently N times, with replacement, from the distribution obtained in the filtering stage. The resulting particles, {x i k } i=1,,n, form an approximate sample from p(x k y 1:k ). The method outlined above can be justified as follows. If the particles at time t = k 1 were an i.i.d sample from the posterior at time t = k 1, then the predictive stage just produces an i.i.d. sample from the prior at time t = k. The filtering stage can be viewed as an importance sampling approach to generate an empirical distribution which approximates the posterior. The proposal density is just the prior p(x k y 1:k 1 ), and as a result of Bayes formula, we obtain p(x k y 1:k 1, y k ) p(x k y 1:k 1 )p(y k x k ), (1) and the weights are proportional to the likelihood p(y k x k ). As N tends to infinity, the discrete distribution which has probability mass q i at point x i k k 1, converges weakly to the true posterior. The resampling step is crucial to particle filters. It is used to generate equally weighted particles aimed at avoiding the problem of degeneracy of the algorithm, that is, avoiding the situation that all but one of the weights are close to zero. The resampling step modifies the weighted approximate density p(x k y k ) to an unweighted density ˆp(x k y k ) by eliminating particles having low importance weights and by multiplying particles having highly importance weights. Formally: is replaced by ˆp(x k y k ) = p(x k y k ) = N i=1 N q i δ(x k x i k ) (11) i=1 1 N δ(x k x k ) = N i=1 n i N δ(x k x k i ) (12) where n i is the number of copies of particle x k i in the new set of particles {x k }. For extensive review of particle filtering see[van Leeuwen 9]. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 9

10 3.2 The Ensemble Kalman Filter The ensemble Kalman filter (EnKF) was first proposed by Evensen [Evensen 1994] and further developed by [Burgers et al. 1998] and [Evensen 23, Evensen 27]. For the current status and the potential of the EnKF we refer to [Kalnay 29]. It is related to particle filters in the context that a particle is identical to an ensemble member. EnKF is a sequential filter method, which means that the model is integrated forward in time and, whenever observations are available, these are used to reinitialize the model before the integration continues. The EnKF originated as a version of the Extended Kalman Filter (EKF) of Jazwinski [Jazwinski 197] and Bucy[Bucy 1965] for large problems. The classical KF [Kalman 6] method is optimal in the sense of minimizing the variance only for linear systems and Gaussian statistics. Similar to the particle filter method, the EnKF stems from a Monte Carlo integration of the Fokker-Planck equation governing the evolution of the pdf that describes the prior, forecast, and error statistics. In the analysis step, each ensemble member is updated according to the KF scheme and replaces the covariance matrix by the sample covariance computed from the ensemble. However, the EnKF presents two potential problems namely: 1) The EnKF assumes that that error is a linear function of the innovation. 2) The EnKF assumes that it can correctly estimate the first two moments of the posterior distribution. However, if assumption 1) is wrong then so is 2) Let p(x) denote the Gaussian prior probability density distribution of the state vector x with mean µ and covariance P ( p(x) exp 1 ) 2 (x µ)t P 1 (x µ) We assume the data y to have a Gaussian pdf with covariance R and mean Hx, where H is the so-called the observation matrix, is related to H of equation (4), and where the value Hx assumes what the value of the data y would be in absence of observation errors. Then p(y x) is the probability (likelihood) of the observations y given that the truth is given by the model state x and is of the form ( p(y x) exp 1 ) 2 (y Hx)T R 1 (y Hx). According to the Bayes theorem the posterior probability density follows from the relation p(x y) p(y x)p(x). (13) There are many variants of implementing the EnKF of various computational efficiency and in what follows we employ standard formulation of the EnKF for linear and nonlinear M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 1

11 observation operators with covariance localization. See [Evensen 1994, Burgers et al. 1998, Mandel 26, Mandel 27 and Lewis et al. 26], also see [Nerger et al. 25] and Sakov and Oke [Sakov 28] The implementation of the standard EnKF may be divided into three steps, as follows: Setting and matching Define the ensemble X = [x 1,, x N ] (14) be an n x N matrix whose columns are a sample from the prior distribution. N being the number of the ensemble members. Form the ensemble mean X = 1 N X 1 N, (15) where 1 N R N N is the matrix where each element is equal to 1. Define the ensemble perturbation matrix X and set the R nx nx ensemble covariance matrix C X = X X, (16) Sampling Generate C = X X T N 1, (17) Y = [y 1,, y N ] (18) an n y N matrix whose columns are a replica of the measurement vector y plus a random vector from the normal distribution N (, R). Form the R ny ny measurement error covariance R = Y Y T N 1, (19) Updating Obtain the posterior X p by the linear combinations of members of the prior ensemble X p = X + CH T (HCH T + R) 1 (Y HX ). (2) The matrix K = CH T (HCH T + R) 1 (21) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 11

12 is the Kalman gain matrix. Since R is always positive definite( i.e. as it is a covariance matrix), the inverse (HCH T + R) 1 exists. The covariance of the posterior obeys C p = C K [ HCH T + R ] K T, (22) In the case of nonlinear observation operators, a modification to the above algorithm is advised. As advised by Evensen [Evensen 23] and following his notation, let ˆx be the augmented state vector made of the state vector and the predicted observation vector (nonlinear in this case). ˆx = ( x H(x) ). (23) Define the linear observation operator Ĥ ( by ) x Ĥ = y (24) y and carry out the steps of the EnKF formulation in augmented state space ˆx and Ĥ instead of x and H. Superficially, this technique appears to reduce the nonlinear problem to the previous linear observation operator case. However, whilst the augmented problem, involving linear observation problem, is a reasonable way of formulating the EnKF, it is not as well-founded as the linear case, which can be justified as an approximation to the exact and optimal KF. It has been argued by Kepert [Kepert 24] that the above EnKF modification has peculiar properties which render it far more useful than other forms. Kepert s major concern related to the performance of the EnKF when (a) the number of ensemble members was less than the number of observations. (b) the approximation to the observation error covariance R given by (19) rather than R itself was used in the gain matrix. Houtekamer s EnKF avoids this problem by using the best available estimate of R rather than equation (19). To prevent the occurrence of filter divergence usually due to the background-error covariance estimates from small number of ensemble members as pointed out in [Houtekamer and Mitchell 1998], the use of covariance localization was suggested. Mathematically, the covariance localization increases the effective rank of the background error covariances. See the work of [ Gaspari and Cohn 1999] also [Hamill and Snyder 2, 26] and [Ehrendorfer 27]. The covariance localization consists of multiplying point by point the covariance estimate from the ensemble with a correlation function that is 1. at the observation location and zero beyond some prescribed distance. Mathematically, to apply covariance localization, the Kalman gain K = CH T (HCH T + R) 1 M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 12

13 is replaced by a modified gain ˆK = [ρ C] H T (H [ρ C] H T + R) 1 (25) where ρ denotes the Schur product ( The Schur product of matrices A and B is a matrix D of the same dimension, where d ij = a ij b ij ) of a matrix S with local support with the covariance model generated by the ensemble. Various correlation matrices have been used. For the present model, we used the usual Gaussian correlation function ρ(d) = exp[ [ D l ]2 ], (26) where l is the correlation length here l = 2 length units. In addition, the additive covariance inflation of [Anderson and Anderson 1999] with the inflation factor r = 1.1 has been employed. 3.3 The Maximum Likelihood Ensemble Filter The Maximum Likelihood Ensemble Filter (MLEF) proposed by Zupanski [Zupanski 25], and Zupanski and Zupanski [Zupanski 26] is a hybrid filter combining the 3- D variational method with the EnKF. It maximizes the likelihood of posterior probability distribution which justifies its name. The MLEF does not use the tangent linear model. It uses the full non-linear model. This is one of the major advantages of the MLEF method. The method comprises three steps, a forecast step that is concerned with the evolution of the forecast error covariances, an analysis step based on solving a non-linear cost function and an updating step. Forecasting: In the absence of model errors, the forecasting step consists of evolving the square root analysis error covariance matrix through the ensembles. The starting point is from the evolution equation of the discrete Kalman filter described in Jazwinski [Jazwinski 197] P k f = M k 1,k P k a M T k 1,k, (27) where P (k) f is the forecast error covariance matrix at time k, M k 1,k is the linearized forecast model (e.g., Jacobian) from time k 1 to time k. Since Pa k 1 is a positive matrix for any k, equation (27) could be factorized and written as P k f = (P k f )1/2 {}}{ ( Mk 1,k (P k a ) 1/2) ( M k 1,k (P k a ) 1/2) T. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 13

14 where (P k a ) 1/2 is of the form (P k a ) 1/2 = (P k f ) 1/2 = p k,1 1 p k,2 1 p k,n 1 p k,1 2 p k,2 2 p k,n 2 p k,1 n p k,2 n p k,n n, (28) as usual N is the number of ensemble members and n the number of state variables. The lower case p k,i j are obtained by calculating the square root of (Pa k ). Using equation(28), the square root forecast error covariance matrix (Pf k)1/2 can then be expressed as p k,1 1 p k,2 1 p k,n 1 p k,1 2 p k,2 2 p k,n 2 p k,1 n p k,2 n p k,n n where for a given k and for each 1 i N they satisfy x k,i 1 x k x k x k,i 1 + p k,i 1 i = 2. = M x k 2 + p k,i 2 k 1,k. M k 1,k x k,i n x k n + p k,i n, (29) The vector x k = ( x k 1x k 2 x k n) T is the analysis state from the previous assimilation cycle. which is found from the posterior analysis pdf as presented in [Lorenc 1986]. Analyzing: The analysis step for the MLEF involves solving a non-linear minimization problem. As in Lorenc [Lorenc 1986], in the absence of model error, the associated cost function is defined in terms of the forecast error covariance matrix and is given as J (x) = 1 2 (x x b) T ( P k f ) 1 (x xb )+ 1 [y 2 H(x)]T R 1 [y H(x)] where y is the vector of observations, h is the non-linear observation operator, R is the observational error covariance matrix and x b is a background state given by x k 1 x k 2. x k n. (3) x b = M k 1,k (x k 1 ). (31) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 14

15 Through a Hessian preconditioner we introduce the change of variable (x x b ) = (P k f ) 1/2 (I + O) T/2 ξ (32) where ξ is vector of control variables, O is referred to as the observation information matrix and I is the identity matrix. The matrix O is provided by O = (P k f )T/2 HT R 1 H(P k f ) T/2 = (R 1/2 H(P k f ) 1/2 ) T (R 1/2 H(P k f ) 1/2 ) here H is the Jacobian matrix of the non-linear observation operator H evaluated at the background state x b. Let Z the matrix defined by and let z i assumes the form Z = z i = From the following approximations and z i 1 z i 2. z i n z1 1 z1 2 z1 N z1 1 z2 2 z2 N z 1 n z 2 n z N n, = R 1/2 k H x i. z i R 1/2 [ H(x + x k i ) H(x) ], (33) O ZZ T. (34) one can use an eigenvalue decomposition of the of symmetric positive definite matrix I + O to calculate the inverse square root matrix necessary to the updating step. It is worth mentioning that the approximation (33) is not necessary and a derivation of the MLEF not involving (33) has been recently developed in Zupanski et al. [Zupanski 8] Updating: The final point about MLEF is to update the square root analysis error covariance matrix. In order to estimate the analysis error covariance at the optimal point, the optimal state x opt minimizing the cost function J given by (3) is substituted (P k a ) T/2 = (P k f ) T/2 (I + O(x opt )) T/2. (35) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 15

16 3.4 Differences between the EnKF and MLEF The differences between the EnKF and the MLEF are that the latter uses the central forecast as best estimate, and defined the covariance around that state while the EnKF uses the ensemble mean. The second difference is that the MLEF uses a 3D-var method for the update, while the EnKF uses some linearisation. More differences between MLEF and other EnKF methods have been presented in the work of Carrassi et al. [Carrassi 29] in an application to chaotic system. Moreover, to avoid any confusion or ambiguity as to any equivalence between the ensemble transform Kalman filter (ETKF) of Bishop et al. [Bishop 21] and the MLEF we list the following major differences between them. Let p k,i is the i th the column of the square root of the analysis error covariance matrix. Further, suppose that the sum of the column vectors p k,i is zero. In this case, the corresponding p k,i column of the square root of the forecast error covariance matrix in the EnKF is given by p k,i = where as the MLEF it is given by 1 [M k 1,k (x a + p k,i ] N 1) M k 1,k (x a ) N 1 (36) p k,i = [ M k 1,k (x a + p k,i ) M k 1,k (x a ) ] (37) The MLEF employs a control deterministic forecast (as variational methods), while ETKF employs an ensemble forecast mean. Although they are same for assumed Gaussian PDF, there is no proof yet that they yield identical results in a reduced-rank (small ensemble) situation, or with nonlinear prediction models. The analysis error covariance is different, and this constitutes the most important difference. In ETKF the forecast error covariance is calculated as a statistical sample of 1 ensemble perturbations, implying the factor. In MLEF there is no such assumption, and the formula follows the idea from the Kalman Filter forecast step. The impact of N 1 1 is not trivial since it implies that analysis error covariance is completely different. N 1 For instance, let assume for a moment that both ETKF and MLEF yield the same ensemble forecast perturbations p f i = x f i xf for i = 1,, N (38) M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 16

17 where index i represents ensemble member and x f is a control forecast or ensemble mean. Whilst ETKF and MLEF forecast error covariances are ET KF Pf = 1 N 1 P MLEF f = 1 N 1 the ETKF analysis error covariance is N p f i (pf i )T i=1 N p f i (pf i )T i=1 ET KF Pa = 1 [ ET KF (Pf ) 1/2 I + O ] 1 ET KF (Pf ) T/2 (39) N 1 N 1 and the MLEF analysis error covariance is P MLEF a = (Pf MLEF ) 1/2 [I + O] 1 (Pf MLEF ) T/2, (4) where O has been defined earlier in equation (34). This is the most important difference since square-root P a is used to define initial ensemble perturbations for next cycle, and the difference amplifies after several data assimilation cycles. There is no empirical multiplication factor that can make the two identical, unless the forecast error covariance from MLEF is used in ETKF. The ETKF is a minimum variance method, while the MLEF is a maximum likelihood of a posterior PDF method. This itself is a fundamental difference. It should be obvious for non-gaussian PDFs (especially a skewed PDFs such as log-normal PDF), but it could be relevant for nonlinear problems even with (incorrectly) assuming Gaussian PDF. Minimization. In the case of nonlinear observation operator, one minimization iteration of MLEF is not identical to the ETKF one. In fact, the MLEF uses line-search (which typically produces a value much smaller than 1), while ETKF implies using a step-length equal to 1 (quadratic cost function). This produces a different analysis increment, and thus can also impact the dynamical balance of the analysis. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 17

18 4 Numerical results 4.1 Model set-up As in [Williamson 1992,1997,27] and [Jakob et al. 1995], the grid representation for any arbitrary variable φ is related to the following spectral decomposition φ(λ, µ) = M N(m) m= M n= m φ m,n P m,n (µ)e imλ, (41) where P m,n (µ)e imλ are the spherical harmonic functions [Boyd 1]. P m,n (µ) stands for the Legendre polynomial. M is the highest Fourier wavenumber included in the east-west representation, N(m) is the highest degree of the associated Legendre polynomials for longitudinal wavenumber m. The coefficients of the spectral representation (41) are determined by 1 1 2π φ m,n = φ(λ, µ)e imλ P m,n (µ)dλdµ (42) 2π 1 The inner integral represents a Fourier transform, φ m (µ) = 1 2π φ(λ, µ)e imλ dλ (43) 2π which is evaluated using a fast Fourier transform routine. The outer integer is evaluated using Gaussian quadrature on the transform grid. φ m,n = J φ m (µ)p m,n (µ)ω j, (44) j=1 where µ j denotes the Gaussian grid points in the meridional direction and ω j is the Gaussian weight at point µ j. The meridional grid points are located at the Gaussian latitudes θ j, which are the J roots of the Legendre polynomial P j (sin θ j ) =. The number of grid points in the longitudinal and meridional directions are determined so as to allow the unaliased representation of quadratic terms, I 3M + 1 J 3N M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 18

19 where N is the highest wavenumber retained in the latitudinal Legendre representation N = max N(m) = M. The pseudo spectral method, also known as the spectral transform method in the geophysical community, of [Orszag 1969, 197] and [Eliassen et al. 197] has been used to tackle the nonlinearity. In conjunction with the spatial discretization described before, the time discretization, two semi-implicit time steps have been used for the initialization. Because of the hyperbolic type of the shallow water equations, the centered leapfrog scheme φ k+1 m,n φ k 1 m,n 2 t = F(φ k m,n) (45) has been invoked for the subsequent time steps. After the leapfrog time-differencing scheme is used to obtain the solution at t = (k + 1) t, a slight time smoothing is applied to the solution at time k t (Asselin filter). φ k m,n = φ k m,n + α [ ] φ k+1 m,n 2φ k k 1 m,n + φ m,n (46) replacing the solution at time k. It reduces the amplitude of different frequencies ν by a factor 1 4α sin 2 ( ν t). 2 Even-tough the flow considered in this study is very simplified and has nearly linear dynamics, the shallow-water flows on the sphere provide a baseline against which to evaluate the effectiveness of the assimilation and give some insight into the behavior of some data assimilation schemes. Starting from an ensemble of 5 initial conditions generated from a 1% random perturbation around the mean of the velocity, geopotential, divergence and vorticity fields. In figure 2 one ensemble member of the ensemble geopotential fields is depicted. The ensemble initial conditions is then evolved forward (with no data assimilation). It results after 18 hours of time integration in a pattern breakdown of the R-H wave no 4 test that can be easily noticed from the expected velocity and geopotential fields presented in figure 3. As a consequence, the estimated error variance would grow unreasonably rapidly. This is the motivation behind to use of the shallow-water flows on the sphere as a testbed for some data assimilation schemes. 4.2 EnKF set-up and numerical results In all our numerical experiments a time step of t = 6 sec has been employed. The observations were provided at a frequency consisting of one set of observations every 36 t. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 19

20 4.2.1 Impact of the number of observations and the number of ensemble members The impact of the number of observations follows the findings of [ Fletcher and Zupanski 28]. In fact, there exists a number of observations threshold beyond which the RMSE for all the fields ( namely the components of the velocity and geopotential) coincide. In our test the number of the observations threshold was attained at around 912 observations at each observation time. The observational grid is a subset of the rhomboidal spectral truncation (48x38) and consists of observations distributed at every grid point in the longitude and every 2 points in the latitude, as presented in figure 1. The impact of the number of ensemble members on the results has also been examined. We have conducted several experiments with different numbers of ensemble members, namely, 5, 1, 2 and 3 ensemble members, respectively. Our results reveal that employing only 1 ensemble members was sufficient to successfully perform the data assimilation and that using higher number of ensemble members had no impact on the ensuing results Linear observation operator case - description of the EnKF results In figure 2 we present a 1% random perturbation around the mean of the geopotential field. This perturbed field along with the perturbed velocity divergence and vorticity fields will serve as initialization for the observations. In figure 4 we present an overview of the results obtained using 1 days EnKF data assimilation time integration. The linear observation operator H(u) = u has been employed, and 1 ensemble members were used with 1% random perturbation around the mean accounting for the standard deviation for the model error covariance matrix. In the presence of 1% and 3% random perturbations around the mean respectively, the RH wave no. 4 test case preserves its global characteristic shape for more than 3 days of data assimilation time integration. This shows the resilience of the RH wave no. 4 test case to random perturbation in the presence of linear observation operator Nonlinear observation operator case - description of the EnKF results Here we employ H(u) = u 2 as a representative of a nonlinear observation operator (we have also tested H(u) = u 4 ; results not shown). We use 912 observations points, their distribution in space and time are identical to those used in section The EnKF for the nonlinear observation operator H(u) = u 2 begins to deteriorate at around 5 days of data assimilation time integration. By 1 days of data assimilation time integration a full filter divergence takes place as it can be seen in the RMSE and Talagrand diagrams. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 2

21 We should point out that in the sequel of this paper,when producing the rank scores, you compare truth with ensemble forecast members. Figure 5 displays the corresponding Talagrand diagram also known as the rank score. Contrary to the linear observation operator Talagrand diagram where a uniform ensemble repartition is observed, the nonlinear case displays a weakly U-shaped ensemble repartition. This result characterizes the onset of filter divergence. In figure 6 we present the true, the EnKF analysis for one ensemble member and the expected analysis geopotential fields for 5 days data assimilation time integration. This illustration confirms the validity of the Talagrand diagram. In fact, a deterioration of the typical shape of the RH wave no.4 test case and the asymmetry(i.e. the first and the last bin of the Talagrand diagram are overpopulated are noticeable) Figure 7 regroups the Talagrand diagrams and RMSE corresponding to different values of the random perturbation around the mean, namely 1%, 3% and 6% perturbations. The Talagrand diagrams now display a non-symmetrical U shape characteristic of filter divergence as well as bias. An increase in the values of the frequencies follows the increase in the perturbation percentage. The RMSE of the geopotential field, presented in the same figure, increase accordingly with the perturbation. In figures 8 and 9 we present the true state, the EnKF analysis for one ensemble member and the expected analysis geopotential forecast and analysis geopotential fields for 1 days EnKF window of data assimilation with 1% and 3% perturbation around the mean. The same nonlinear observation operator H(u) = u 2 has been employed. The results obtained point to a total loss of shape and symmetry deterioration of the RH wave no. 4 test case. We also used the nonlinear observation operator H(u) = sign(u)u 2, such an operator is known not to lead to bimodal observation likelihoods. Again, the filter divergence is plausible as it can be easily seen from figure 1, where the expected velocity and geopotential fields are depicted. We should point out the difference between our setup and those used in the work of Buehner et al. [Buehner 21A],[Buehner 21B] and also in the work of Hamill et al. [Hamill 211], where the EnKF appears to perform reasonably well. All our observations have the same degree of nonlinearity, while in aformentionned references observation operators are mostly weakly nonlinear. Satellite radiances are included only for almost linear clear-sky radiances (done in all operational weather centers, with only one, quite recent exception at the ECMWF). The nonlinearity comes mainly from scattering that is included in cloudy radiances only, so true nonlinearity as used in our work has not been employed in [[Buehner 21A],[Buehner 21B] and [Hamill 211]]. New efforts to mitigate the problem of high nonlinearity in EnKF have been undertaken. The recent work of Kalnay and collaborators, [Kalnay priv. comm. 28], [Kalnay and Yang 28] and [Kalnay and Yang 21] suggests an approach involving outer loops of minimization, borrowed from variational data assimilation practice. They introduce M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 21

22 an empirical convergence criterion within a nonstandard iterative minimization scheme. Common to both mentioned algorithms is that they treat each iteration of minimization as a quadratic sub-problem and thus rely on the use of traditional Kalman filter linear analysis update equation in each iteration. 4.3 PF set-up and numerical results As in the previous (EnKF) section, all our numerical experiments employ a time step of t = 6 sec. The observations are sampled once every 6 hours i.e. at a frequency consisting of one observation every 36 time steps The impact of number of particles and of the resampling strategy In view of recent results in the literature such as ([Van Leeuwen 29][Snyder et al. 28]) the issue of choice of an adequate number of particles is of paramount importance. Since no proposal densities are considered and in order to avoid filter degeneracy we follow the suggestion of Van-Leeuwen [Van Leeuwen 29] regarding the choise of suitable number of particles, namely, that the number of particles should be not less than the degrees of freedom of the simulated system. We tested three cases, namely, 1, 15 and 19 particle filters as compared to modes in the latitude-longitude directions for the spherical-spectral shallow-water model. We tested three resampling techniques, namely, residual, systematic and multinomial resampling, respectively. All of these methods performed equally well with a slight edge to the systematic resampling technique as far as computational efficiency is concerned see ([Doucet et al. 21] [Arulampalam et al. 22][Van Leeuwen 29]). A Talagrand like diagram is introduced here, where instead of taking the ensemble members we took the particles. We then iterated over the number of bins (19 particles, 51 bins). Similar to the ensemble framework, this diagram could be a useful tool to detect systematic flaws of a particle filter prediction system Linear observation operator case The linear observation operator assuming the form H(u) = u has been employed for the particle filter case and we do not show the related results. However, we observe almost identical results to those obtained for the EnKF linear observation operator case; in fact the R-H wave no 4 test case preserves its global characteristic shapes with minor changes occurring in the center of the recirculation zones at around 1 days of data assimilation time integration. M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 22

23 4.3.3 Non-linear observation operator case We consider the impact of nonlinear observation operator for the particle filter (PF) when H(u) = u 2. The results are displayed in figure 11, where the true state, PF analysis for one particle geopotential fields for 1% and 3% perturbation around the mean are depicted. After 1 days data assimilation time integration,the R-H wave no 4 test case preserves its global characteristic shape, and no distortion nor cell pattern breakdown can be detected. These results should be contrasted to our EnKF findings of filter divergence for the same data assimilation time integration. Apart from the bias ( i.e. the first and last bins) of the U-shaped diagrams shown in figure 12, we see uniformity of the frequencies repartition indicating that the probability distribution has been well sampled. The rank score also shows under dispersive ensembles, this means that the number of particles is too small even if the mean converged. In figure 13 we present the time evolution of the RMSE after 1 days of data assimilation time integration of PF for 1% and 3% perturbation around the mean. The striking result is that the RMSE variation remains relatively small in contrast to the EnKF case where the corresponding RMSE increases quasi-exponentially. The effect of degree of nonlinearity of the observation operator in the PF case has been also examined. As figure 14 shows that a stronger nonlinearity (i.e H(u) = exp(u) ) has no substantial effect on the RMSE. Indeed, the RMSE envelope is not affected despite the presence of oscillations characteristic of the PF filter. 4.4 MLEF set-up and results Only 3 ensemble members were sufficient for carrying out the MLEF runs, while the number of cycles was set to 7. The iterative nonlinear minimization method employed for the cost function minimization was the Flecher-Reeves nonlinear conjugate gradient algorithm. In each of the MLEF data assimilation cycles, 5 iterations are performed to obtain the analysis. The standard deviation of each of the Gaussian state and observation noise was taken to be 1% random perturbation around the mean. No localization nor inflation were used within the ensemble part of this method. We observe similar results to those obtained for the EnKF and PF in the linear observation operator case, namely that the global characteristic shapes of the filtered solution are preserved. The striking resemblance between the PF and MLEF results demonstrates the successful implementation of both filters. This result is further confirmed as the degree of nonlinearity of the observation operators was increased from H(u) = u 2 to H(u) = e u. Our numerical experiments are presented in in figure 15. As presented in figure 16, we observe that apart from the bias ( i.e. the first and last bins) of the U-shaped diagram, the rank score also shows under dispersive ensembles, this means that the number of ensembles is M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 23

24 small even if the mean converged. The RMSE presented in figure 17 are bounded and the nonlinearity of the observation operator has no effect on the RMSE and consequently the MLEF filter predicts well the true state solution independently of the linearity of the observation operator. 5 Summary and conclusions In this paper, a comparison was carried out for the three ensemble data assimilation methods (EnsDA) for a spherical nonlinear shallow water equations model discretized via a spectral model and tested for the Rossby Haurwitz wave no. 4. The Monte Carlo version of EnKF, the particle filter PF with several resampling strategies and finally, as a representative of hybrid filters, the maximum likelihood filter MLEF were tested in the presence of both linear and highly nonlinear observation operators. The nonlinear observation operators replicate and stand as a proxy for more realistic processes such as cloud, aerosol and precipitation processes, as well as remote sensing (e.g., satellite and radar) observations. While the typical EnsDA analysis equation assumes linearity and Gaussianity for EnKF, (thus fundamentally preventing this EnsDA from extracting maximum information from nonlinear observations) we have also tested a Bayesian filter, the particle filter, whose analysis equation is nonlinear, as well as the MLEF hybrid filter, which accommodates nonlinear observation operators as well as non-gaussianity. The results for the linear observation operator for three above mentioned sequential data assimilation filters leads to the conclusion, that the aforementioned methods are comparable and yield satisfactory results in this case. Amongst the different error metrics used to assess filter divergence, we chose to retain the Talgrand diagram also know as the rank score and the usual RMSE statistics. Results of the Talagrand diagram for EnKF in the presence of nonlinear operators show that filter divergence onset occurs at about 5 days of data assimilation using the R-H wave no.4 test case. We conclude that EnKF with Evensen s fixture for nonlinearity fails to converge for nonlinear observation operators. In contrast, the other two filters (PF and MLEF) handled the issue of nonlinearity of the observation operators as well as non-gaussianity well. These results are of course a function of our particular implementation of the EnKF, PF and MLEF, respectively, and are a function of the number of particles/ensemble members, the number of observations, as well as the particular model tested. Regarding computational efficiency, the MLEF was found to be faster than the EnKF, while the PF filter proved to be the most time consuming. The EnKF version for the nonlinear observation operator H(u) = u 2 displays a signifi- M. Jardak, I.M. Navon & M. Zupanski, submitted to Tellus A 24