Data assimilation : Basics and meteorology

Transcription

1 Data assimilation : Basics and meteorology Olivier Talagrand Laboratoire de Météorologie Dynamique, École Normale Supérieure, Paris, France Workshop on Coupled Climate-Economics Modelling and Data Analysis Paris 23 November

2 What is exactly Data Assimilation? The numerical algorithms. (Ensemble) Kalman Filter, Variational Assimilation, Particle Filters Assimilation vs Stability and Instability. 2

3

4 ECMWF, Technical Report 499, 2006

5 5

6 6

7 December 2007: Satellite data volumes used: around 18 millions per day quantity of satellite data used per day at ECMWF number of data used per day (millions) CONV+SAT WINDS TOTAL Year Value as of March 2010 : 25 millions per day 7

8 S. Louvel, Doctoral Dissertation, 1999

9 Physical laws governing the flow Conservation of mass Dρ/Dt + ρ divu = 0 Conservation of energy De/Dt - (p/ρ 2 ) Dρ/Dt = Q Conservation of momentum DU/Dt + (1/ρ) gradp - g + 2 Ω U = F Equation of state f(p, ρ, e) = 0 (p/ρ = rt, e = C v T) Conservation of mass of secondary components (water in the atmosphere, salt in the ocean, chemical species, ) Dq/Dt + q divu = S Physical laws available in practice in the form of a discretized (and necessarily imperfect) numerical model 9

10 European Centre for Medium-range Weather Forecasts (ECMWF, Reading, UK) Horizontal spherical harmonics triangular truncation T1279 (horizontal resolution 16 kilometres) 91 levels on the vertical (0-80 km) Dimension of state vector n Timestep = 10 minutes 10

11 Purpose of assimilation : reconstruct as accurately as possible the state of the atmospheric or oceanic flow, using all available appropriate information. The latter essentially consists of The observations proper, which vary in nature, resolution and accuracy, and are distributed more or less regularly in space and time. The physical laws governing the evolution of the flow, available in practice in the form of a discretized, and necessarily approximate, numerical model. Asymptotic properties of the flow, such as, e. g., geostrophic balance of middle latitudes. Although they basically are necessary consequences of the physical laws which govern the flow, these properties can usefully be explicitly introduced in the assimilation process. 11

12 Both observations and model are affected with some uncertainty uncertainty on the estimate. For some reason, uncertainty is conveniently described by probability distributions (don t know too well why, but it works). Assimilation is a problem in bayesian estimation. Determine the conditional probability distribution for the state of the system, knowing everything we know 12

13 Difficulties specific to assimilation of meteorological and oceanographical observations : - Very large numerical dimensions (n parameters to be estimated, p observations per 24-hour period). Difficulty aggravated in Numerical Weather Prediction by the need for the forecast to be ready in time. - Non-trivial underlying dynamics. 13

14 Bayesian estimation is however impossible in its general theoretical form in meteorological or oceanographical practice because It is impossible to explicitly describe a probability distribution in a space with dimension even as low as n 10 3, not to speak of the dimension n of present Numerical Weather Prediction models. Probability distribution of errors on data very poorly known (model errors in particular).

15 One has to restrict oneself to a much more modest goal. Two approaches exist at present Obtain some central estimate of the conditional probability distribution (expectation, mode, ), plus some estimate of the corresponding spread (standard deviations and a number of correlations). Produce an ensemble of estimates which are meant to sample the conditional probability distribution (dimension N O(10-100)).

16 Data of the form z = Γx + ζ, ζ N [µ, S] Known data vector z belongs to data space D, dimd = m, Unknown state vector x belongs to state space S, dims = n Γ known (mxn)-matrix, ζ unknown error Then conditional probability distribution is where P(x z) = N [x a, P a ] x a = (Γ T S -1 Γ) -1 Γ T S -1 [z µ] P a = (Γ T S -1 Γ) -1 Determinacy condition : rankγ = n. Requires m n. 16

17 Expressions x a = (Γ T S -1 Γ) -1 Γ T S -1 [z µ] P a = (Γ T S -1 Γ) -1 (2a) (2b) can also be obtained, from the knowledge of only µ and S, as defining the variance minimizing linear estimator of x from z, or Best Linear Unbiased Estimate (BLUE) of x from z (but significance of P a is now totally different) Expressions (2) are commonly used, and are useful, in many (especially, but non only, geophysical) applications where operator Γ is mildly nonlinear (and error ζ can be non-gaussian). In fact, going beyond (2) turns out to be very challenging in many practical problems. 17

18 Variational form. BLUE x a minimizes following scalar objective function, defined on state space S ξ S J(ξ) (1/2) [Γξ - (z-µ)] T S -1 [Γξ - (z-µ)] BLUE is invariant in any invertible linear change of coordinates, either in state or data space. 18

19 If data of the form (not restrictive) x b = x + ζ b, ζ b N [0, P b ] (3a) y = Hx + ε, ε N [0, R] (3b) ζ b and ε independent Eq. 2 become Variational form x a = x b + P b H T [HP b H T + R] -1 (y - Hx b ) P a = P b - P b H T [HP b H T + R] -1 HP b d y - Hx b is innovation vector ξ S J(ξ) = (1/2) (x b - ξ) T [P b ] -1 (x b - ξ) + (1/2) (y - Hξ) T R -1 (y - Hξ) = J b + J o 19

20 Time dimension Observation vector at time k y k = H k x k + ε k k = 0,, K E(ε k ) = 0 ; E(ε k ε jt ) R k δ kj Evolution equation x k+1 = M k x k + η k k = 0,, K-1 E(η k ) = 0 ; E(η k η jt ) Q k δ kj E(η k ε jt ) = 0 Background estimate at time 0 x b 0 = x 0 + ζb 0 E(ζ b 0 ) = 0 ; E(ζb 0 ζ b 0 T ) P b 0 E(ζ b 0ε kt ) = 0 ; E(ζ b 0η kt ) = 0 Errors uncorrelated in time 20

21 BLUE can be obtained through time-sequential algorithm called Kalman filter Background x b k and associated error covariance matrix Pb k known Analysis step x a k = xb k + Pb k H k T [H k P b k H k T + R k ] -1 (y k - H k x b k ) P a k = Pb k - Pb k H k T [H k P b k H k T + R k ] -1 H k P b k Forecast step x b k+1 = M k xa k P b k+1 = M k Pa k M k T + Q k 21

22 Costliest part of computation P b k+1 = M k P a k M k T + Q k Need for determining the temporal evolution of the uncertainty on the state of the system is the major difficulty in assimilation of meteorological and oceanographical observations 22

23 Analysis of 500-hPa geopotential for 1 December 1989, 00:00 UTC (ECMWF, spectral truncation T21, unit m. After F. Bouttier) 23

24 Temporal evolution of the 500-hPa geopotential autocorrelation with respect to point located at 45N, 35W. From top to bottom: initial time, 6- and 24-hour range. Contour interval 0.1. After F. Bouttier. 24

25 Most common approach at present is Ensemble Kalman Filter (EnKF) Uncertainty is represented, not by a covariance matrix, but by an ensemble of point estimates in state space which are meant to sample the conditional probability distribution for the state of the system (dimension N O(10-100)). Ensemble is evolved in time through the full model, which eliminates any need for linear hypothesis as to the temporal evolution. 25

26 How to update predicted ensemble with new observations? Predicted ensemble at time t : {x b i }, i = 1,, N Observation vector at same time : y = Hx + ε Gaussian approach Produce sample of probability distribution for real observed quantity Hx y i = y - ε i where ε i is distributed according to probability distribution for observation error ε. Then use Kalman formula to produce sample of analysed states x a i = xb i + Pb H T [HP b H T + R] -1 (y i - Hx b i ), i = 1,, N (2) where P b is covariance matrix of predicted ensemble {x b i }. Remark. If all errors were Gaussian, and if P b was exact covariance matrix of background error, (2) would achieve Bayesian estimation, in the sense that {x a i } would be a sample of conditional probability distribution for x, given all data up to time t. 26

27 Month-long Performance of EnKF vs. 3Dvar with WRF EnKF 3DVar (prior, solid; posterior, dotted) posterior Prior Better performance of EnKF than 3DVar also seen in both 12-h forecast and posterior analysis in terms of root-mean square difference averaged over the entire month 27 (Meng and Zhang 2007c, MWR, in review )

28 Ensemble Kalman Filter (EnKF), of which many variants have been defined and are actively studied, has now become one of the two major powerful algorithm for assimilation of meteorological and oceanographical observations. Ensemble Transform Kalman Filter (ETKF, Bishop and others) 28

29 Ensemble Kalman Filter is only exception so far, in real life assimilation, to linear (and gaussian) approach. Analysis step remains linear (and gaussian). Bayesian character of EnKF? - If everything is linear, distribution of ensemble produced by EnKF tends to underlying bayesian pdf when ensemble dimension tends to infinity (if errors are not gaussian, gaussianity is ensured by Central Limit Theorem). - If non-linearity, distribution of ensemble tends to a limit which is not the bayesian distribution (F. Le Gland) 29

30 Variational Assimilation Observation vector at time k y k = H k x k + ε k k = 0,, K E(ε k ) = 0 ; E(ε k ε jt ) R k δ kj Evolution equation x k+1 = M k x k + η k k = 0,, K-1 E(η k ) = 0 ; E(η k η jt ) Q k δ kj E(η k ε jt ) = 0 Background estimate at time 0 x b 0 = x 0 + ζb 0 E(ζ b 0 ) = 0 ; E(ζb 0 ζ b 0 T ) P b 0 E(ζ b 0ε kt ) = 0 ; E(ζ b 0η kt ) = 0 Errors uncorrelated in time 30

31 Minimise objective function (ξ 0, ξ 1,..., ξ K ) J(ξ 0, ξ 1,..., ξ K ) = (1/2) (x 0 b - ξ 0 ) T [P 0b ] -1 (x 0 b - ξ 0 ) + (1/2) Σ k=0,,k [y k - H k ξ k ] T R k -1 [y k - H k ξ k ] + (1/2) Σ k=0,,k-1 [ξ k+1 - M k ξ k ] T Q k -1 [ξ k+1 - M k ξ k ] Can include nonlinear M k and/or H k. 4D-Var. In the linear case, equivalent to Kalman Filter. 31

32 4D-Var used operationally in a number of major meteorological centres (European Centre for Medium-Range Weather Forecasts, Météo-France, UK Meteorological Office, Canadian Meteorological Centre, Japan Meteorological Agency, ). In most cases, model error is ignored. Strong constraint variational assimilation ξ 0 J(ξ 0 ) = (1/2) (x b 0 - ξ 0 )T [P b 0 ]-1 (x b 0 - ξ 0 ) + (1/2) Σ k=0,, K (y k - H k ξ k ) T R k -1 (y k - H k ξ k ) subject to ξ k+1 = M k ξ k, k = 0,, K-1 32

33 J(ξ 0 ) = (1/2) (x 0 b - ξ 0 ) T [P 0b ] -1 (x 0 b - ξ 0 ) + (1/2) Σ k [y k - H k ξ k ] T R k -1 [y k - H k ξ k ] Background is not necessary, if observations are in sufficient number to overdetermine the problem. Nor is strict linearity. How to minimize objective function with respect to initial state u = ξ 0 (u is called the control variable of the problem)? Use iterative minimization algorithm, each step of which requires the explicit knowledge of the local gradient u J ( J/ u i ) of J with respect to u. Gradient computed by adjoint method. 33

34 Courtier and Talagrand, QJRMS,

35 Courtier and Talagrand, QJRMS,

36 500-hPa geopotential field as determined by : (left) operational assimilation system of French Weather Service (3D, primitive equation) and (right) experimental variational system (2D, vorticity equation) Courtier and Talagrand, QJRMS,

37 Thépaut et al., 1993, Mon. Wea. Rev., 121,

38 Analysis increments in a 3D-Var corresponding to a height observation at the 250- hpa pressure level (no temporal evolution of background error covariance matrix) 38 Thépaut et al., 1993, Mon. Wea. Rev., 121,

39 Same as before, but at the end of a 24-hr 4D-Var Thépaut et al., 1993, Mon. Wea. Rev., 121,

40 Analysis increments in a 3D-Var corresponding to a u-component wind observation at the 1000-hPa pressure level (no temporal evolution of background error covariance matrix) Thépaut et al., 1993, Mon. Wea. Rev., 121,

41 Same as before, but at the end of a 24-hr 4D-Var Thépaut et al., 1993, Mon. Wea. Rev., 121,

42 ECMWF, Results on one FASTEX case (1997)

43 Buehner (2008) For the same numerical cost, and in meteorologically realistic situations, Ensemble Kalman Filter and Variational Assimilation produce results of similar quality. 43

44 Exact bayesian estimation? Particle filters Predicted ensemble at time t : {x b n, n = 1,, N }, each element with its own weight (probability) P(x b n) Observation vector at same time : y = Hx + ε Bayes formula P(x b n y) P(y xb n ) P(xb n ) Defines updating of weights 44

45 Bayes formula P(x b n y) P(y x b n) P(x b n) Defines updating of weights; particles are not modified. Asymptotically converges to bayesian pdf. Very easy to implement. Observed fact. For large state dimension, ensemble tends to collapse. 45

46 C. Snyder, Snyder.pdf 46

47 Problem originates in the curse of dimensionality Large dimension pdf s are very diffuse, so that very few particles (if any) are present in areas where conditional probability ( likelihood ) P(y x) is large. Bengtsson et al. (2008) and Snyder et al. (2008) evaluate that stability of filter requires the size of ensembles to increase exponentially with space dimension. 47

48 Alternative possibilities (review in van Leeuwen, 2009, Mon. Wea. Rev., ) Resampling. Define new ensemble. Simplest way. Draw new ensemble according to probability distribution defined by the updated weights. Give same weight to all particles. Particles are not modified, but particles with low weights are likely to be eliminated, while particles with large weights are likely to be drawn repeatedly. For multiple particles, add noise, either from the start, or in the form of model noise in ensuing temporal integration. Random character of the sampling introduces noise. Alternatives exist, such as residual sampling (Lui and Chen, 1998, van Leeuwen, 2003). Updated weights w n are multiplied by ensemble dimension N. Then p copies of each particle n are taken, where p is the integer part of Nw n. Remaining particles, if needed, are taken randomly from the resulting distribution. 48

49 Importance Sampling. Use a proposal density that is closer to the new observations than the density defined by the predicted particles (for instance the density defined by EnKF, after the latter has used the new observations). Independence between observations is then lost in the computation of likelihood P(y x). In particular, Guided Sequential Importance Sampling (van Leeuwen, 2002). Idea : use observations performed at time k to resample ensemble at some timestep anterior to k. Implicit Particle Filter. The evolution of each particle is guided by the observations. 49

50 van Leeuwen, 2003, Mon. Wea. Rev., 131,

51 Particle filters are actively studied (Snyder et al., van Leeuwen et al.). They are independent of any hypothesis of linearity or gaussianity. It is not clear however if their cost can be sufficiently reduced to make them usable in the large scale applications of geophysical flows. 51

52 Proportion of resources devoted to assimilation in Numerical Weather Prediction has steadily increased over time. At present at ECMWF, the cost of 24 hours of assimilation is half the global cost of the 10-day forecast (i. e., including the ensemble forecast). 52

53 What is exactly Data Assimilation? The numerical algorithms. (Ensemble) Kalman Filter, Variational Assimilation, Particle Filters Assimilation vs Stability and Instability 53

54 If there is uncertainty on the state of the system, and dynamics of the system is perfectly known, uncertainty on the state along stable modes decreases over time, while uncertainty along unstable modes increases. Stable (unstable) modes : perturbations to the basic state that decrease (increase) over time. 54

55 Observations between times -τ and 0, no background 55

56 Consequence : 4D-Var assimilation, which carries information both forward and backward in time, performed over time interval [t 0, t 1 ] over uniformly distributed noisy data. If assimilating model is perfect, estimation error is concentrated in stable modes at time t 0, and in unstable modes at time t 1. Error is smallest somewhere within interval [t 0, t 1 ]. Similar result holds true for Kalman filter (or more generally any form of sequential assimilation), in which estimation error is concentrated in unstable modes at any time. 56

57 Trevisan et al., 2010, Q. J. R. Meteorol. Soc. 57

58 Lorenz (1963) dx/dt = σ(y-x) dy/dt = ρx - y - xz dz/dt = -βz + xy with parameter values σ = 10, ρ = 28, β = 8/3 chaos 58

59

60 60

61 Pires et al., Tellus, 1996 ; Lorenz system (1963) 61

62 Pires et al., Tellus,

63 Since, after an assimilation has been performed over a period of time, uncertainty is likely to be concentrated in modes that have been unstable, it might be useful for the next assimilation, and at least in terms of cost efficiency, to concentrate corrections on the background in those modes. Actually, presence of residual noise in stable modes can be damageable for analysis and subsequent forecast. Assimilation in the Unstable Subspace (AUS) (Carrassi et al., 2007, 2008, for the case of 3D-Var) 63

64 Four-dimensional variational assimilation in the unstable subspace (4DVar-AUS) Trevisan et al., 2010, Four-dimensional variational assimilation in the unstable subspace and the optimal subspace dimension, Q. J. R. Meteorol. Soc., 136,

65 4D-Var-AUS Algorithmic implementation Define N perturbations to the current state, and evolve them according to the tangent linear model, with periodic reorthonormalization in order to avoid collapse onto the dominant Lyapunov vector (same algorithm as for computation of Lyapunov exponents). Cycle successive 4D-Var s, restricting at each cycle the modification to be made on the current state to the space spanned by the N perturbations emanating from the previous cycle (if N is the dimension of state space, that is identical with standard 4D-Var). 65

66 Experiments performed on the Lorenz (1996) model with value F = 8, which gives rise to chaos. Three values of I have been used, namely I = 40, 60, 80, which correspond to respectively N + = 13, 19 and 26 positive Lyapunov exponents. In all three cases, the largest Lyapunov exponent corresponds to a doubling time of about 2 days (with 1 day = 1/5 model time unit). Identical twin experiments (perfect model) 66

67 Observing system defined as in Fertig et al. (Tellus, 2007): At each observation time, one observation every four grid points (observation points shifted by one grid point at each observation time). Observation frequency : 1.5 hour Random gaussian observation errors with expectation 0 and standard deviation σ 0 = 0.2 ( climatological standard deviation 5.1). Sequences of variational assimilations have been cycled over windows with length τ = 1,, 5 days. Results are averaged over 5000 successive windows. 67

68 No explicit background term (i. e., with error covariance matrix) in objective function : information from past lies in the background to be updated, and in the N perturbations which define the subspace in which updating is to be made. Best performance for N slightly above number N + of positive Lyapunov exponents. 68

69 Different curves are almost identical on all three panels. Relative improvement obtained by decreasing subspace dimension N to its optimal value is largest for smaller window length τ.

70 70

71 Experiments have been performed in which an explicit background term was present, the associated error covariance matrix having been obtained as the average of a sequence of full 4D-Var s. The estimates are systematically improved, and more for full 4D-Var than for 4D-Var- AUS. But they remain qualitatively similar, with best performance for 4D-Var-AUS with N slightly above N +. 71

72 Minimum of objective function cannot be made smaller by reducing control space. Numerical tests show that minimum of objective function is smaller (by a few percent) for full 4D-Var than for 4D-Var-AUS. Full 4D-Var is closer to the noisy observations, but farther away from the truth. And tests also show that full 4D-Var performs best when observations are perfect (no noise). Results show that, if all degrees of freedom that are available to the model are used, the minimization process introduces components along the stable modes of the system, in which no error is present, in order to ensure a closer fit to the observations. This degrades the closeness of the fit to reality. The optimal choice is to restrict the assimilation to the unstable modes. 72

73 Conclusions on AUS Error concentrates in unstable modes at the end of assimilation window. It must therefore be sufficient, at the beginning of new assimilation cycle, to introduce increments only in the subspace spanned by those unstable modes. In the perfect model case (and with noisy observations), assimilation is most efficient when increments are introduced in a space with dimension slightly above the number of non-negative Lyapunov exponents. In the case of imperfect model (and of strong constraint assimilation), preliminary results lead to similar conclusions, with larger optimal subspace dimension, and less well marked optimality (not shown here). Further work necessary. In agreement with theoretical and experimental results obtained for Kalman Filter assimilation (Trevisan and Palatella, Nonlin. Processes Geophys., 2011, Ng et al., Tellus, 2011) 73

74 Lorenz (1963) dx/dt = σ(y-x) dy/dt = ρx - y - xz dz/dt = -βz + xy with parameter values σ = 10, ρ = 28, β = 8/3 chaos 74

75 Pires et al., Tellus, 1996 ; Lorenz system (1963) 75

76 Minima in the variations of objective function correspond to solutions that have bifurcated from the observed solution, and to different folds in state space. 76

77 Quasi-Static Variational Assimilation (QSVA). Increase progressively length of the assimilation window, starting each new assimilation from the result of the previous one. This should ensure, at least if observations are in a sense sufficiently dense in time, that current estimation of the system always lies in the attractive basin of the absolute minimum of objective function. 77

78 Pires et al., Tellus, 1996 ; Lorenz system (1963)

79 79

80 Swanson, Vautard and Pires, 1998, Tellus, 50A,

81 Conclusions on QSVA Quasi-Static Variational Assimilation, which consists in progressively increasing the length of the assimilation window, is capable of efficiently assimilating observations over time windows that are much longer than the length of time over which the tangent linear approximation is valid and the 4DVar objective function is quadratic. For a quasi-geostrophic model, assimilation can be usefully assimilated over periods as large as 10 days. That however requires the observations to be in a sense sufficiently dense in time. Do EnKF, EnKS, or particle filters, achieve the same accuracy? Effect of model errors? 81

82 Assimilation, which originated from the need of defining initial conditions for numerical weather forecasts, has progressively extended to many diverse applications Oceanography Atmospheric chemistry (both troposphere and stratosphere) Oceanic biogeochemistry Ground hydrology Terrestrial biosphere and vegetation cover Glaciology Magnetism (both planetary and stellar) Plate tectonics Planetary atmospheres (Mars, ) Reassimilation of past observations (mostly for climatological purposes, ECMWF, NCEP/NCAR) Identification of source of tracers Parameter identification A priori evaluation of anticipated new instruments Definition of observing systems (Observing Systems Simulation Experiments) Validation of models Sensitivity studies (adjoints) It has now become a major tool of numerical environmental science 82

83 Aubert and Fournier, Nonlin. Processes Geophys.,

84 A few of the (many) remaining problems Assimilation of images (much work being done ) Observability (what does one obtain by observing what? Data are noisy, dynamics is chaotic!) More accurate identification and quantification of errors on data, especially on assimilating model (will always require independent hypotheses) 84

85