Optimal solution error covariance in highly nonlinear problems of variational data assimilation

Transcription

1 Nonlin. Processes Geophys., 9, 77 84, 22 doi:.594/npg Authors) 22. CC Attriution 3. License. Nonlinear Processes in Geophysics Optimal solution error covariance in highly nonlinear prolems variational data assimilation V. Shutyaev, I. Gejadze 2, G. J. M. Copeland 2, and F.-X. Le Dimet 3 Institute Numerical Mamatics, Russian Academy Sciences, 9333 Gukina 8, Moscow, Russia 2 Department Civil Engineering, University Strathclyde, 7 Rottenrow, Glasgow, G4 ONG, UK 3 MOISE project CNRS, INRIA, UJF, INPG), LJK, Université de Grenole, BP 53, 384 Grenole, France Correspondence to: V. Shutyaev shutyaev@inm.ras.ru) Received: 6 July 2 Revised: 6 Feruary 22 Accepted: 2 Feruary 22 Pulished: 6 March 22 Astract. The prolem variational data assimilation DA) for a nonlinear evolution model is formulated as an optimal control prolem to find initial condition, oundary conditions and/or model parameters. The input data contain oservation and ackground errors, hence re is an error in optimal solution. For mildly nonlinear dynamics, covariance matrix optimal solution error can e approximated y inverse Hessian cost function. For prolems with strongly nonlinear dynamics, a new statistical method ased on computation a sample inverse Hessians is suggested. This method relies on efficient computation inverse Hessian y means iterative methods Lanczos and quasi-newton BFGS) with preconditioning. Numerical examples are presented for model governed y Burgers equation with a nonlinear viscous term. Introduction State and/or parameter estimation for dynamical geophysical flow models is an important prolem in meteorology and oceanography. Among few methods feasile for solving se non-stationary large-scale prolems, variational data assimilation DA) method, called 4D-Var, is preferred method implemented at some major operational centers e.g. Courtier et al., 994; Fisher et al., 29). From mamatical point view, se prolems can e formulated as optimal control prolems e.g. Lions, 986; Le Dimet and Talagrand, 986) to find unknown control variales in such a way that a cost function related to oservation and a priori data takes its minimum value. A necessary optimality condition leads to so-called optimality system, which contains all availale information and involves original and adjoint models. Due to input errors ackground and oservation errors), re is an error in optimal solution. Its statistical properties are very important for quantifying accuracy optimal solution which is necessary to evaluate quality forecast), for sequential variational state estimation and optimal design oservation schemes. Assuming that proaility density function p.d.f.) optimal solution error can e reasonaly approximated y normal Gaussian) distriution, optimal solution error covariance matrix referred to elow simply as covariance ) is its most important statistic to e estimated. If errors input data are random and normally distriuted, n for a linearized finite-dimensional error evolution model, covariance is given y inverse Hessian cost function e.g. Thacker, 989; Raier and Courtier, 992). This is an extension a well-known result from nonlinear regression Draper and Smith, 98) to case nonlinear dynamical systems. A similar result in continuous case was presented y Gejadze et al. 28). In terms continuous representation, it is said that covariance operator can e approximated y inverse Hessian auxiliary control prolem ased on tangent linear model TLM) constraints, if so-called tangent linear hyposis TLH) is valid. The TLH implies that error dynamics can e satisfactorily descried y TLM. It was demonstrated y Gejadze et al. 2, 2) that approximation covariance y inverse Hessian could e sometimes sufficiently accurate even though TLH is not valid. However, in case highly nonlinear dynamics such an approximation may not e valid at all see, for example, Pires et al., 996). In present paper, for case under consideration, we do following: a) present an argument that even in this case p.d.f. optimal solution error may still e represented y a normal distriution defined y covariance matrix; ) outline a new method for estimation covariance; c) discuss implementation potentially feasile for large-scale dynamical models. One ojectives this paper is to highlight concept Pulished y Copernicus Pulications on ehalf European Geosciences Union & American Geophysical Union.

2 78 V. Shutyaev et al.: Optimal solution error covariance Effective Inverse Hessian EIH), first introduced y Gejadze et al. 2), to geophysical research community. The closest concept to this is proaly Expected Fisher Information Matrix used in Bayesian estimation ory. 2 Statement prolem Consider mamatical model a physical process that is descried y evolution prolem { ϕ t = F ϕ), t,t ) ϕ ) t= = u, where ϕ = ϕt) is unknown function elonging for any t to a Hilert space X, u X, F is a nonlinear operator mapping X into X. Let Y = L 2,T ;X) e a space astract functions ϕt) with values in X, with norm T ϕ = ϕ 2 X dt)/2. Suppose that for a given u X re exists a unique solution ϕ Y to Eq. ). Let ū e exact initial state and ϕ solution to prolem Eq. ) with u = ū, i.e. exact state evolution. We define input data as follows: ackground function u X, u = ū + ξ and oservations y Y o, y = C ϕ + ξ o, where C : Y Y o is a linear ounded operator oservation operator) and Y o is a Hilert space oservation space), ξ X, ξ o Y o. In particular, Y o may e finite-dimensional oth in space and in time). The random variales ξ and ξ o may e regarded as ackground and oservation error, respectively. Assuming that se errors are normally distriuted, uniased and mutually uncorrelated, we define covariance operators V = E[,ξ ) X ξ ] and V o = E[,ξ o ) Yo ξ o ], where denotes an argument respective operator, and E is expectation. We suppose that V and V o are positive definite, hence invertile. Let us introduce a cost function J u) J u) = V 2 u u ),u u ) X + + Vo Cϕ y),cϕ y) Yo, 2) 2 and formulate following DA prolem optimal control prolem) with aim to identify initial condition: find u X and ϕ Y such that y satisfy Eq. ) and cost function J u) takes its minimum value. Furr we assume that optimal solution error δu = u ū is uniased, i.e. E[δu] =, with covariance operator V δu = E[,δu) X δu]. Let us introduce operator R : X Y o as follows Rv = Cψ, v X, 3) where ψ Y is solution tangent linear prolem { ψ t F ϕ)ψ =, t,t ), 4) ψ t= = v. Nonlin. Processes Geophys., 9, 77 84, 22 For a given v we solve prolem Eq. 4), and n find Rv y Eq. 3). The definition R involves ϕ = ϕ + δϕ dependent on u = ū+δu via Eq. ), thus we can write as follows: R = Rū,δu). It has een shown in Gejadze et al., 28) that optimal solution error δu = u ū and data errors ξ and ξ o are related via following exact operator equation V +R ū,δu)vo Rū,τ δu))δu = = V ξ +R ū,δu)vo ξ o, 5) where R is adjoint to R and τ [,] is a parameter chosen to make truncated Taylor series exact. Let H ū) = V +R ū,)vo Rū,) e Hessian linearized auxiliary) control prolem Gejadze et al., 28). Under hyposis that F is twice continuously Fréchet differentiale, error Eq. 5) is approximated y: H ū)δu = V ξ +R ū,)vo ξ o. 6) From Eq. 6) it is easy to see that V δu = [H ū)]. 7) This is a well-estalished result Courtier et al., 994; Raier and Courtier, 992; Thacker, 989), which is usually deduced without considering Eq. 5) y straightforwardly linearizing original nonlinear DA prolem Eqs. ) 2) under assumption that F ϕ) F ϕ) F ϕ)δϕ, 8) which is called tangent linear hyposis. It is said that V δu can e approximated y [H ū)] if TLH Eq. 8) is valid. That usually happens if nonlinearity is mild and/or error δu and, susequently, δϕ are small. We derive Eq. 7) via Eq. 5). From this derivation one can see that accuracy Eq. 7) depends on accuracy approximations Rū,τδu) Rū,) and R ū,δu) R ū,) in Eq. 5). Clearly, transition from Eq. 5) to Eq. 6) could still e valid even though Eq. 8) is not satisfied. As already mentioned, we can use formula Eq. 7) if TLH is valid and, in some cases eyond range its validity. In general case, however, one may not expect H ū) always to e a satisfactory approximation to V δu. In Fig. we present a specially designed example for evolution model governed y -D Burgers equation for details see Sect. 4). The difference etween reference value variance circles) and inverse Hessian ased value old solid line) can e clearly seen within ellipse. The reference variance is otained y a direct Monte Carlo simulation. Since R ū,) and H ū) in Eq. 6) are linear operators and we assume that errors ξ and ξ o are uniased and normally distriuted, n δu N,V δu ). Clearly, this result is valid as far as TLH and consequently Eq. 6) itself are satisfied. However, for highly nonlinear dynamical models TLH ten reaks down e.g. Pires et al., 996); thus, we have to

3 V. Shutyaev et al.: Optimal solution error covariance 79 V. Shutyaev et al.: Optimal solution error covariance one may conclude that δu from Eq. 5) is ound to remain 95 These conditions are met under certain general regularity asymptotically normal. In practice oservation window reference ens. requirements to operator R, which are incomparaly.2 H-variance [,T ] and time step dt are always finite implying finite ackground weaker than TLH and do not depend on magnitude numer i.i.d. oservations. Moreover, it is not easy to input errors. Clearly, as applied to 5), first assess how large numer oservations must e for.5 condition holds if ξ is normally distriuted. Since V is desired asymptotic properties to e reasonaly approximated. Some nonlinear 2 a constant matrix, second condition always holds as long as it holds for R least-square ū,δu)v prolems, in which o Rū,τδu). Therefore, one may. normality estimation error holds for practically relevant sample sizes, are said to exhiit a close-to-linear sta- conclude that δu from 5) is ound to remain asymptotically tistical normal. ehavior In practice Ratkowsky, oservation 983). The window method [,T] suggested and time.5 in step Ratkowsky, dt are always 983) to finite verify implying this ehavior is, finite essentially, numer a 25 normality i.i.d. oservations. test applied to Moreover, a generated it is sample not easy optimal to assess solutions, large which numer is hardly feasile oservations for large-scale must e applications. for desired how x Neverless, asymptotic properties for certaintohighly e reasonaly nonlinearapproximated. evolution models, Some itnonlinear is reasonale least-square to expectprolems that distriution in which normality δu might estimation error holds for practically relevant sample Fig.. Reference variance, variance y inverse Hessian and e reasonaly close to normal if numer i.i.d. oservations is significant in time and oservation network is 2 Fig.. Reference variance, variance y inverse Hessian and sizes are said to exhiit a close-to-linear statistical ehavior ackground variance. ackground variance. sufficiently Ratkowsky, dense 983). in space. The This method maysuggested happen in in assimilation Ratkowsky, 983) long time to verify seriesthis satellite ehavioroservations is, essentially, ocean a normality surfacetest answer following question: can p.d.f. δu still e elevation applied and to atemperature, generated sample for example. optimal solutions, which is approximated Since R ū,) y andnormal Hū) distriution? in 6) are linear If operators answer and is hardly feasile for large-scale applications. Neverless, for positive, we assume onethat should errors look ξ for andaξetter o are uniased approximation and normally 25 certain highly nonlinear evolution models it is reasonale to covariance distriuted, than nthat δu given N,V y Eq. δu ).7). Clearly, this result is valid 3 expect Effective that Inverse distriution Hessian EIH) δu might method e reasonaly close aslet far us as consider TLH andcost consequently function Eq. 6) 2), itself ut without are satisfied. to normal if numer i.i.d. oservations is significant ackground However, for term. highly Thenonlinear corresponding dynamical error equation models Eq. TLH 5) 3. in time General and consideration oservation network is sufficiently dense is ten nreaks as follows: down e.g., Pires et al., 996); thus, we have to in space. This may happen in assimilation long time answer following question: can p.d.f. δu still e Here series we present satellite a oservations new method for ocean estimating surface 22 elevation covariance V and R approximated ū,δu)vo Rū,τδu)δu = R y normal distriution? ū,δu)vo ξ o. 9) If answer is temperature, δu to e used in case highly nonlinear dynamics, for example. For positive, a univariate one should case, look classical for a etter result approximation see Jennrich, 969) when [H ū)] is not expected to e a good approximation is covariance that δu isthan asymptotically that given y normal 7). if ξ o is an independent V δu. Let us consider discretized nonlinear error equation 3 Eq. Effective 5) andinverse denote Hessian y H EIH) left-hand method side operator in identically Let us consider distriuted i.i.d.) costrandom function variale 2), ut withwithout E[ξ o ] = and ackground E[ξo 2] = σ term. 2 < The asymptotically corresponding means error equation that T 5) Eq. 5). Then we can write down expression for δu is given n as follows: finite oservation time step dt, or dt given 3. General consideration δu = H V finite R oservation ū,δu)vo window [,T ]). Rū,τδu)δu = R Let us stress ū,δu)vo that for ξ +R ū,δu)vo ξ o ), asymptotic normality δu, error ξ ξ o. 9) Here we present a new method for estimating covariance o is not required to e whereas for covariance V 225 V δu to e used in case δu we otain as follows: normal. highly nonlinear dynamics, For a univariate This original case, result has classical een generalized result see to Jennrich, multivariate when [Hū)] is not expected to e a good approximation [ 969)) case is that andδu to is case asymptotically dependent, normal yet identically if ξ o is distriuted nonlinear error ] [ an V δu := E δuδu T = E H V ξ ξ T V δu. Let us consider discretized V H ] + independent oservations identically White distriuted and Domowitz, i.i.d.) 984), randomwhereas variale an with even E[ξmore o ] = general and E[ξcase o 2] = is equation 5) and denote y H left-hand side operator in σ2 considered < asymptotically in Yuan andmeans Jennrich, 5). Then we can write down expression for δu: [ that T 998). Here given we consider finite oservation complete time cost step function +E H R ū,δu)vo ξ o ξo T dt, or V o Rū,δu)H ]. ) Eq. dt 2) and, given correspondingly, finite oservation error window Eq. 5), which [,T]). contains Let δu = H V ξ +R ū,δu)vo 23 ξ o ), us stress terms that related for to asymptotic ackground normality term. To δu analyze error a As a result a series simplifications descried in Gejadze possile impact se terms let us follow reasoning et al., 2) aove equation can e reduced to form ξ o is not required to e normal. This original result has whereas for covariance V δu we otain as follows: in Amemiya, 983), pp , where error equation een generalized to multivariate case and to case equivalent to Eq. 9) is derived in a slightly different form. V dependent, yet identically distriuted oservations White δu := E [ [ V δuδu T] = E [ H V ξ ξ T V H ] δu V = E [H ū+δu)] ], + ) It is concluded that error δu is asymptotically normal and Domowitz, 984), whereas an even more general case when: a) right-hand side error equation is normal; ) left-hand side matrix converges in proaility to sian linearized auxiliary) control prolem. The right- where is considered in Yuan and Jennrich, 998). Here we +E [ H ū+δu) R ū,δu)v = V o ξ o ξo T Rū,δu)H ] +R ū,δu)vo is. Hes- ) consider complete cost function 2) and, correspondingly, a non-random value. These conditions are met under certain hand As aside result Eq. a) series may e simplifications called effective descried inverse Gejadze Hessian et al., EIH), 2) hence aove name equation suggested can e reduced method. toin order form error equation 5), which contains terms related to general regularity requirements to operator R, which are 235 ackground term. To analyze a possile impact se incomparaly weaker than TLH and do not depend on to compute V directly terms let us follow reasoning in Amemiya, 983), V δu V = E [ using this magnitude input errors. Clearly, as applied to Eq. 5), [Hū+δu)] ] equation, expectation is sustituted y sample mean:, ) pp. first , condition where holds if error ξ equation equivalent to 9) is is normally distriuted. Since V derived in a slightly different form. It is concluded that where Hū + δu) = V + R ū,δu)vo Rū,δu) is is a constant matrix, second condition always holds error δu is asymptotically normal when: a) righthand side error equation is normal; ) left-hand right-hand Hessian linearized auxiliary) control prolem. The as long as it holds for R ū,δu)vo V = L [H ū+δu Rū,τ δu). Therefore, L l )]. 2) l= side ) may e called effective inverse side matrix converges in proaility to a non-random value. 24 Hessian EIH), hence name suggested method. Nonlin. Processes Geophys., 9, 77 84, 22 variance

4 8 V. Shutyaev et al.: Optimal solution error covariance The main difficulty with implementation is a need to compute a sample optimal solutions u l = ū + δu l. However, formula Eq. ) does not necessarily require u l to e an optimal solution. If we denote y q δu p.d.f. δu, n equation Eq. ) can e rewritten in form: V = + [H ū+v)] q δu v) dv. 3) If we assume that in our nonlinear case covariance matrix V descries meaningfully p.d.f. optimal solution error, n, with same level validity, we should also accept pdf q δu to e approximately normal with zero expectation and covariance V, in which case we otain + V = c [H ū+v)] exp ) 2 vt V v dv, 4) where c = 2π) M/2 V /2. Formula Eq. 2) gives V explicitly, ut requires a sample optimal solutions u l, l =,...,L to e computed. In contrast, latest expression is a nonlinear matrix integral equation with respect to V, while v is a dummy variale. This equation is actually solved using iterative process Eq. 9), as explained in following section. It is also interesting to notice that Eq. 4) is a deterministic equation. 3.2 Implementation remarks Remark. Preconditioning is used in variational DA to accelerate convergence conjugate gradient algorithm at stage inner iterations Gauss-Newton GN) method, ut it also can e used to accelerate formation inverse Hessian y Lanczos algorithm Fisher et al., 29) or y BFGS Gejadze et al., 2). Since H is selfadjoint, we must consider a projected Hessian in a symmetric form H = B ) H B, with some operator B : X X, defined in such a way that eigenspectrum projected Hessian H is clustered around, i.e. majority eigenvalues H are equal or close to. Since condition numer H is supposed to e much smaller than condition numer H, a sensile approximation H can usually e otained eir y Lanczos or BFGS) with a relatively small numer iterations. After that, having H, one can easily recover H using formula: H = B H B ). 5) Assuming that B does not depend on δu l, we sustitute Eq. 5) into Eq. 2) and otain version Eq. 2) with preconditioning: V = B L [ H ū+δu L l )] )B ). 6) l= Nonlin. Processes Geophys., 9, 77 84, 22 Similarly, assuming that B does not depend on variale integration, we sustitute Eq. 5) into Eq. 4) and otain version Eq. 4) with preconditioning: V = B Ṽ B ), + Ṽ = c [ H ū+v)] exp ) 2 vt V v dv. 7) Formulas Eq. 6) and Eq. 7) instead H involve H which is much less expensive to compute and store in memory. Let us mention here that EIH method would hardly e feasile for large-scale prolems without appropriate preconditioning. Remark 2. The nonlinear Eq. 7) can e solved, for example, y fixed point iterative process as follows V p+ = B Ṽ B ), + Ṽ = c p [ H ū+v)] exp ) 2 vt V p ) v dv, 8) for p =,,..., starting with V = [H ū)]. The iterative processes this type are expected to converge if V is a good initial approximation V, which is case in considered examples. The convergence Eq. 8) and or methods for solving equation Eq. 7) are sujects for future research. Remark 3. Different methods can e used for evaluation multidimensional integral in Eq. 8) such as quasi- Monte Carlo Neiderreiter, 992). Here, for simplicity, we use standard Monte Carlo method. This actually implies a return to formula Eq. 6). Taking into account Eq. 5), iterative process takes form V p+ = B L [ H ū+δu p )] )B ), 9) l L l= where δu p l N,V p ). For each l, we compute δu p l as follows δu p l = V p ) /2 ξ l, where ξ N,I) is an independent random series, I is identity matrix and V p ) /2 is square root V p. One can see that for each p last formula looks similar to Eq. 6) with one key difference: δu p l in Eq. 9) is not an optimal solution, ut a vector having statistical properties optimal solution. Remark 4. Let us notice that a few tens outer iterations y GN method may e required to otain one optimal solution, while an approximate evaluation H is equivalent in terms computational costs) to just one outer iteration GN method. One has to repeat se computations p times, however, only a few iterations on index p are required in practice. Therefore, one should expect an order magnitude reduction computational costs y method

5 V. Shutyaev et al.: Optimal solution error covariance Remark V. Shutyaev 4. etlet al.: us Optimal noticesolution that a error few tens covariance outer 8 iterations y GN method may e required to otain one optimal Eq. 9) solution, as compared while an to approximate Eq. 6) for evaluation same sample H size. is equivalent Clearly, forin realistic termslarge-scale computational models, costs) sample to just sizeone L is outer going iteration to e limited. GNProaly, method. One minimum has to repeat ensemle se size computations for this method p times, to work however is 2Lonly +, a few where iterations L is onaccepted index ϕ p are numer required leading practice. eigenvectors Therefore, Vone p inshould Eq. 9). expect an orderremark magnitude 5. In order reduction to implement computational process costs Eq. 9) y a sample method 9) vectors as compared ϕ l x,) = to δu6) p l must fore propagated same sample from size. t = Clearly, to t = for T realistic using large-scale nonlinear modelseq. ). sample Therefore, size L is for going each pto one e limited. gets a sample Proaly, final states minimum ϕ l x,t ensemle ) consistent sizewith for thiscurrent method approximation to work is 2L V+ p, which, where canl e is used to accepted evaluate numer forecast leading andeigenvectors forecast covariance. V p in Since 9). V p is Remark a etter approximation 5. In order to implement analysis errorprocess covariance 9) than a sample simply [H vectors ū)] ϕ, l x,) one should = δu p expect a etter quality l must e propagated from t = forecast to t = and T using covariance nonlinear as eingmodel consistent ). Therefore, with V p, rar for each than p one with gets [H ū)] a sample ). final states ϕ l x,t) consistent with current approximation V p, which can e used to evaluate forecast and forecast covariance. Since V 4 Numerical implementation p is Fig. 2. Field evolution. a etter approximation analysis error covariance than Fig. 2. Field evolution. simply 4. [Hū)] Numerical, one model should expect a etter quality forecast and covariance as eing consistent with V p, rar scheme allows µϕ) to e as small as.5 4 for M = thanaswith a model [Hū)] we use ). where D Burgers equation with a nonlinear 2 without i =,...,N noticeale is time oscillations). integration For index, eachhtime t = T/N step we is viscous term: perform time step. nonlinear The spatial iterations operator on iscoefficients discretized wϕ) on a uniform = ϕ and ϕ t + ϕ 2 ) 2 = µϕ) ϕ ) grid µϕ) h x inis form spatial discretization step, j =,...,M is 4 Numerical implementation 37 node numer, M is total numer grid nodes) using, 2) ϕ n i power ϕi n law + ) first-order scheme as descried in Patankar, 4. Numerical model 98), h t which yields 2 wϕi n quite a )ϕi n stale µϕi n discretization ) ϕi n =, scheme ϕ = ϕx,t), t,t ), x,), this As a model we use D Burgers equation with a nonlinear for scheme n =,2,..., allows assuming µϕ) to initially e asthat small µϕas.5 4 for i M = 2 without noticeale oscillations). For ) = each µϕi ) and time step viscous with term: Neumann oundary conditions wϕ i ϕ ϕ = ϕ t + ϕ 2 ) = 2) 2 = µϕ) ϕ ) we perform ) = ϕi, and keep iterating until Eq. 23) is satisfied 375 i.e. norm nonlinear iterations left-hand on side coefficients in Eq. 23) wϕ) ecomes = ϕ and, 2) smaller µϕ) than in form threshold ɛ = 2 M). In all com- i x= x= ϕputations n ϕ i presented n ϕ=ϕx,t), t,t), x,), + in this paper we use and viscosity coefficient h t 2 wϕi n )ϕ i n µϕ i n ) ϕi following ) parameters: oservation period T =.32, discretiza- =, n tion steps h t =.4, h x =.5, state vector dimension n = M,2,..., = 2, assuming and initially parameters that inµϕ Eq. i ) 22) = µϕ µ i = ) and 4, with Neumann oundary ) ϕ 2 conditions for ϕµϕ) = µ +µ, µ,µ = const >. 22) = ϕ wϕ µ i = ) = = ϕ 6 i., and keep iterating until 23) is satisfied i.e. 2) 38 norm A general property left-hand side in Burgers 23) ecomes solutions smaller is that thana The x= nonlinear x= diffusion term with µϕ) dependent on ϕ/ smooth threshold initialǫ state = evolves 2 M). into a In state allcharacterized computations y andis introduced viscositytocoefficient mimic eddy viscosity turulence), which presented areas severe in thisgradients paper we oruse even shocks following inviscid parameters: case). depends on field ) 2 gradients pressure, temperature), rar These oservation are precisely period T areas =.32, a strong discretization nonlinearity where steps µϕ)=µ than on +µ field ϕ value, µ itself.,µ = This const type>. µϕ) also allows 22) h t one =.4, mighxpect x =.5, violations state vector TLH dimension and, susequently, M = 2, us to formally qualify prolem Eqs. 2) 22) as strongly and invalidity parameters in Eq. 22) 7). µ For = numerical 4, µ = experiments we The nonlinear Fučik diffusion andterm Kufner, with 98). µϕ) dependent Let us mention ϕ/ that choose A general a certain property initial condition Burgers that stimulates solutions is that highly a is introduced Burgers equations to mimicare sometimes eddy viscosity considered turulence), in DAwhich context smooth nonlinear initial ehavior state evolves into system; a state thischaracterized given y y formula: severe gradients depends as a simple on model field gradients descriing pressure, atmospheric temperature), flow motion. rar areas or even shocks in inviscid case). than on We usefield implicit value itself. time discretization This type µϕ) as follows also allows These are precisely.5.5cos8πx), areas a strongnonlinearity x.4, where us ϕ to i ϕ formally i qualify + ) prolem 2)-22) h t 2 wϕi )ϕ i µϕ i ) ϕi as strongly 39 one ūx) might = ϕx,) expect = violations,.4 < x.6tlh and, susequently, nonlinear Fučik and Kufner, 98). Let us mention =, that 23) invalidity 7)..5cos4πx).5, For numerical experiments.6 < x we. choose Burgers equations are sometimes considered in DA context a certain initial condition which stimulates highly The resulting field evolution ϕx,t) is presented in Fig. 2. as awhere simple i = model,...,ndescriing is time integration atmospheric index, flowh motion. t = T /N is nonlinear ehavior system; this is given y formula We use time step. implicit The spatial time discretization operator discretized as follows: on a uniform 4.2 BFGS for grid h x is spatial discretization step, j =,...,M is ϕ i ϕ node i numer, + ) computing.5.5cos8πx), inverse Hessian x.4, and M is total numer grid nodes), using h t power law 2 wϕi )ϕ i µϕ i ) ϕi ūx)=ϕx,)= or details,.4 < x.6 =, 23) first-order scheme as.5cos4πx).5,.6 < x. descried in Patankar, The projected inverse Hessian H ū + δu) is computed as 98), which yields quite a stale discretization scheme this a collateral result BFGS iterations while solving Nonlin. Processes Geophys., 9, 77 84, 22

6 82 V. Shutyaev et al.: Optimal solution error covariance V. Shutyaev et al.: Optimal solution error covariance.2. or ted as ng 24) 25) 26) ptimal olem + ξ o, ample ich is cried red as ample = correlation x Fig. 3. Correlation function. Fig. 3. Correlation function. following memers auxiliary and intoda twenty prolem: five susets including L = memers. Let us denote y ˆV L sample covariance matrix δϕ otained t F for ϕ)δϕ a suset =, including t,t ) L memers. Then, relative error δϕ in t= = Bsample variance which is relative δu 24) sampling error) J δu) can = inf e defined as vector ˆε L with v J v), components where ˆε L ) i = ˆV L ) i,i /ˆV i,i, i =,...,M. The relative error in a certain approximation V is defined J δu) = V as a vector 2 ε with B δu ξ ),B δu ξ )) X + components olesky H. -rootursive FGS) ed in and odels del y ascoët adient puted olev. The les is nce is. The at ns are ε i = V i,i /ˆV i,i, i =,...,M. 27) + Vo Cδϕ ξ o ),Cδϕ ξ o ) Yo. 25) We2 compute this error with V in 27) eing estimated y one following methods: The ) y preconditioner inverse Hessian used in our method, method i.e. issimply using V δu = [Hū)] ; B2a) y = V /2 EIH [ H ū)] method /2. implemented in form 6), which 26) requires a sample optimal solutions δu l to e computed; In 2) order y toeih compute method[ H implemented ū)] /2 weasapply iterative Cholesky process factorization 9), which requires explicitly a sampleformed δu l, matrix ut does H not. require However, that δu it l isare important optimaltosolutions. note that square-root-vector productfor H /2 computation w can e computed V y using methods a recursive 2a or 2 procedure a sample ased δu l on is required, accumulated hence secant result pairs depends BFGS) on or eigenvalues/eigenvectors size L. The results Lanczos) otained as descried y in methods Tshimanga 2a and et al., 2) sample 28), presented without in this paper need toare form computed H andwith to factorize L =. it. Consistent methodtangent 2 welinear currently and allow adjoint enough modelsiterations have eenon generated index In from p for original iterativeforward process model 9) toyconverge Automatic in terms Differentiation distance tool etween TAPENADE successive Hascoët iterates. and Pascual, In practice, 24) and this checked requiresusing just a fewstandard iterations, gradient typically test The ackground errorin covariance upper V panel is computed in Fig.4assuming a set that one hundred ackground vectors error ˆε 25 iselongs presented to in dark Soolev lines, space and aw set 2 2 [,] twenty see Gejadze five vectors et al., ˆε 2, - in for details). overlaying Thewhite correlation lines. function These used plotsinreveal numerical envelopes examples for is as relative presented error in in Fig. 3, sample ackground variance error otained variance with is L = σ 2 25 = and.2, L = oservation, respectively. error variance The graphs is σ o 2 ε = are 3 presented. The oservation in lower scheme panel: consists line corresponds 4 sensorsto located method at points inverse ˆx k =.4, Hessian.45,.55, method,.6, see andalso oservations lines 2 and are availale 3 - to at methods each time 2ainstant. and 2 variants EIH Fig.), method). Nonlin. Processes Geophys., 9, 77 84, 22 5 Numerical results First we computed a large sample L = 25) optimal solutions u l y solving L times data assimilation prolem Eqs. ) 2) with pertured data u = ū+ξ and y = C ϕ+ξ o, where ξ N,V ) and ξ o N,σ 2 o I). This large sample was used to evaluate sample covariance matrix, which was furr processed to filter sampling error as descried in Gejadze et al., 2); outcome was considered as a reference value ˆV. Then, original large sample was partitioned into one hundred susets including L = 25 memers and into twenty five susets including L = memers. Let us denote y ˆV L sample covariance matrix otained for a suset including L memers. Then, relative error in sample variance which is relative sampling error) can e defined as vector ˆε L with components: ˆε L ) i = ˆV L ) i,i / ˆV i,i, i =,...,M. The relative error in a certain approximation V is defined as a vector ε with components: ε i = V i,i / ˆV i,i, i =,...,M. 27) We compute this error with V in Eq. 27) eing estimated y one following methods:. y inverse Hessian method, i.e. simply using V δu = [H ū)] ; 2a. y EIH method implemented in form Eq. 6), which requires a sample optimal solutions δu l to e computed; 2. y EIH method implemented as iterative process Eq. 9), which requires a sample δu l, ut does not require that δu l are optimal solutions. For computation V y methods 2a or 2 a sample δu l is required, hence, result depends on sample size L. The results otained y methods 2a and 2) presented in this paper are computed with L =. In method 2 we currently allow enough iterations on index p for iterative process Eq. 9) to converge in terms distance etween successive iterates. In practice, this requires just a few iterations, typically 2 3. In upper panel in Fig. 4, a set one hundred vectors ˆε 25 is presented in dark lines, and a set twenty five vectors ˆε - in overlaying white lines. These plots reveal envelopes for relative error in sample variance otained with L = 25 and L =, respectively. The graphs ε are presented in lower panel: line corresponds to method inverse Hessian method, see also Fig. ), lines 2 and 3 to methods 2a and 2 variants EIH method).

7 V. Shutyaev et al.: Optimal solution error error covariance ε^ x At 25. At same same time, time, relative relative error otained error otained y y meth- methods 2a and 2a2and is much 2 is smaller much smaller as compared as compared to error to in line error inand lineitwould and it largely would largely remain remain within within white white enve- envelope. The difference The difference etween etween estimates estimates y methods y 2a methods and 22a does andnot 2look does significant. not look The significant. est improvement The est can improvement e achieved canfor e achieved diagonal forelements diagonal V δu elements variance). V δu Thus, variance). covariance Thus, estimate covariance y estimate EIH method y is noticealy EIH method etter is noticealy than etter samplethan covariance sample otained covariance with otained equivalent withsample equivalent size. Thesample suggested size. algorithm The suggested is computationally algorithm isefficient computationally in termsefficient CPU in time) termsif cost CPU computing time) if cost inverse computing Hessian is much inverse lesshessian than is cost much computing less than one cost optimal computing solution. one In optimal example solution. presented In in this example paperpresented one limited-memory in this paperinverse one limited-memory Hessian is aout inverse 2 3 Hessian timesis less aout expensive 2-3 times than less one expensive optimal solution. than onethus, optimal on average, solution. Thus, algorithm on average, 2 works aout algorithm times 2 works fasteraout than times algorithm faster2a, than whereas algorithm results 2a, y whereas oth algorithms results y are othsimilar algorithms in terms areaccuracy. similar in terms accuracy. ε.5 6 Conclusions x Fig. 4. Up: sample relative error ˆε. Set ˆε for L = 25 - dark envelope Fig. 4. Up: and set sample ˆε for L relative = - error whiteˆε. envelope. Set Down: ˆε for L = relative darkerror envelope ε y andinverse set Hessian ˆε for L = - line, - and white yenvelope. EIH methods Down: 25 - with relative L = : error method ε y 2a -inverse line 2; method Hessian2 - line - line, 3. and y EIH methods with L=: method 2a - line 2; method 2 - line 3. Looking at Fig.4 4, we we oserve oserve that that relative relative error error in in sample sample variance variance ˆε 25 ˆε 25 dark dark envelope) exceeds 5% almost everywhere, which is certainly eyond reasonale margins, and ˆε ˆε white envelope) is is around 25% that is still fairly large). In Inorder order to to reduce reduce white white envelope envelope two two times, times, one would one would need need to useto usesample sample size L size = 4, L = etc. 4, One etc. should One also should keep alsoinkeep mind inthat mind that relative relative error error in in diagonal diagonal elements elements sample sample covariance covariance matrix matrix is is smallest smallest as compared as compared to itsto su-diagonals, its su-diagonals, i.e. i.e. envelopes envelopes for any sudiagonal any su-diagonal would ewould wider than e wider thosethan presented those in presented Fig.4up). in for Thus, Fig.4up). development Thus, development methods formethods estimating for estimating covariance covariance alternative alternative to direct to sampling directmethod) samplingismethod) an important important task. task. is Whereas method inverse Hessian method) gives an estimate V δu with a small relative error as compared to sample covariance) in areas mild nonlinearity, this error can e much larger in areas high nonlinearity. For example, if we imagine that lower panel in in Fig.4 4 is superposed over its upper panel, n one could oserve line jumping outside dark envelopeinin area area surrounding surrounding x = x.5, =.5, i.e. i.e. relative errory y inverse Hessian is significantly larger here than than sampling error error for for L = L 25. = Error propagation is a key point in modeling large-scale geophysical flows, with main difficulty eing linked to nonlinearity governing equations. In this paper we consider hind-cast initialization) DA prolem. From mamatical point view, view, this this is initial-value initial-value con- control prolem prolem for for a nonlinear a nonlinear evolution evolution model model governed governed y partial y partial differential differential equations. equations. Assuming Assuming so-called so-called tan- tangent linear linear hyposis hyposis TLH) TLH) holds, holds, covariance covariance is ten is approximated ten approximated y y inverse inverse Hessian Hessian ojective ojective func- function. In practice, In practice, same same approximation approximation could could e valid e valid even though even though TLH istlh clearly is clearly violated. violated. However, However, here deal here with we deal suchwith a highly suchnonlinear a highlydynamics nonlinear that dynamics inverse thathes- sian inverse approach Hessian is no approach longer valid. is no longer In this case, valid. a new In this method case for a new computing method for computing covariance matrix, covariance named matrix effective named inverse effective Hessian inverse method, Hessian can emethod used. This canmethod e used. yields This a significant method yields improvement a significant in improvement covariance in estimate covariance as compared estimate to as compared inverse Hessian. to The inverse method Hessian. is potentially The method feasile is potentially for large-scale feasileapplications for large-scale ecause applications it can e ecause used in it acan multiprocessor e used in a multiprocessor environment and environment operates in andterms operates in Hessian-vector terms Hessian-vector products. Theproducts. stware locks The stware neededlocks for its implementation needed for its implementation are standardare locks standard any existing locks4-d Var any system. existingall 4D Varresults system. this Allpaper results are consistent this paper with are assumption consistent with a close-to-normal assumption nature a close-to-normal optimal solution nature error. optimal This should solution eerror. expected, Thistaking shouldinto e expected account taking consistency into account and asymptotic consistency normality and asymptotic normality estimator and estimator fact that andoservation fact that window oservation in variational window DA is inusually variational quiteda large. is usually In this quite case large. covariance In thismatrix case is acovariance meaningful matrix representative is a meaningful p.d.f. representative The method suggested p.d.f. The maymethod ecomesuggested a valualemay option ecome for uncertainty a valuale option analy- sis forinuncertainty framework analysis in classical framework 4D-VAR approach classical when applied 4D-VAR toapproach highly nonlinear when applied DA prolems. to highly nonlinear DA prolems. Acknowledgements. The first author acknowledges Russian Foundation for Basic Research and Russian Federal Research Nonlin. Processes Geophys., 9, 77 84, 22

8 84 V. Shutyaev et al.: Optimal solution error covariance Acknowledgements. The first author acknowledges Russian Foundation for Basic Research and Russian Federal Research Program Kadry. The second author acknowledges funding through Glasgow Research Partnership in Engineering y Scottish Funding Council. All authors thank editor and anonymous reviewers for ir useful comments and suggestions. Edited y: O. Talagrand Reviewed y: three anonymous referees References Amemiya, T.: Handook econometrics,, North-Holland Pulishing Company, Amsterdam, 983. Courtier, P., Thépaut, J. N., and Hollingsworth, A.: A strategy for operational implementation 4D-Var, using an incremental approach, Q. J. Roy. Meteor. Soc., 2, , 994. Draper, N.R., Smith, H.: Applied regression analysis, 2nd ed., Wiley, New York, 98. Fisher, M., Nocedal, J., Trémolet, Y., and Wright, S. J.: Data assimilation in wear forecasting: a case study in PDE-constrained optimization, Optim. Eng.,, , 29. Fučik, S. and Kufner, A.: Nonlinear differential equations, Elsevier, Amsterdam, 98. Gejadze, I., Le Dimet, F.-X., and Shutyaev, V.: On analysis error covariances in variational data assimilation, SIAM J. Sci. Comput., 3, , 28. Gejadze, I., Le Dimet, F.-X., and Shutyaev, V.: On optimal solution error covariances in variational data assimilation prolems, J. Comput. Phys., 229, , 2. Gejadze, I. Yu., Copeland, G. J. M., Le Dimet, F.-X., and Shutyaev, V.: Computation analysis error covariance in variational data assimilation prolems with nonlinear dynamics, J. Comput. Phys., 23, , 2. Hascoët, L., Pascual, V.: TAPENADE 2. user s guide, INRIA Technical Report 3, 78 pp., 24. Jennrich, R. I.: Asymptotic properties nonlinear least square estimation, Ann. Math. Stat., 4, , 969. Le Dimet, F. X. and Talagrand, O.: Variational algorithms for analysis and assimilation meteorological oservations: oretical aspects, Tellus, 38A, 97, 986. Lions, J. L.: Contrôle optimal des systèmes gouvernés par des équations aux dérivées partielles, Dunod, Paris, 986. Liu, D. C. and Nocedal, J.: On limited memory BFGS method for large scale minimization, Math. Program., 45, , 989. Marchuk, G. I., Agoshkov, V. I., and Shutyaev, V. P.: Adjoint equations and perturation algorithms in nonlinear prolems, CRC Press Inc., New York, 996. Neiderreiter, H.: Random numer generation and quasi-monte Carlo methods, CBMS-NSF Regional Conference Series in Applied Math., 63, SIAM, Philadelphia, 992. Patankar, S. V.: Numerical heat transfer and fluid flow, Hemisphere Pulishing Corporation, New York, 98. Pires, C., Vautard, R., and Talagrand, O.: On extending limits variational assimilation in nonlinear chaotic systems, Tellus, 48A, 96 2, 996. Raier, F. and Courtier, P.: Four-dimensional assimilation in presence aroclinic instaility, Q. J. Roy. Meteor. Soc., 8, , 992. Ratkowsky, D. A.: Nonlinear regression modelling: a unified practical approach, Marcel Dekker, New York, 983. Thacker, W. C.: The role Hessian matrix in fitting models to measurements, J. Geophys. Res., 94, , 989. Tshimanga, J., Gratton, S., Weaver, A. T., and Sartenaer, A.: Limited-memory preconditioners, with application to incremental four-dimensional variational assimilation, Q. J. Roy. Meteor. Soc., 34, , 28. White, H. and Domowitz, I.: Nonlinear regression and dependent oservations, Econometrica, 52/, 43 62, 984. Yuan, K.-H. and Jennrich, R. I.: Asymptotics estimating equations under natural conditions, J. Multivariate Anal., 65, , 998. Nonlin. Processes Geophys., 9, 77 84, 22