The Hybrid 4D-Var and Ensemble of Data Assimilations Lars Isaksen, Massimo Bonavita and Elias Holm Data Assimilation Section lars.isaksen@ecmwf.int Acknowledgements to: Mike Fisher and Marta Janiskova Assimilation at 1
The Hybrid 4D-Var and Ensemble of Data Assimilations 4D-Var Ensemble of Data Assimilations (EDA) Hybrid 4D-Var & EDA Assimilation at 2
will continue to use deterministic 4D-Var All observations within a 12-hour period (~17,000,000) are used simultaneously in one global (iterative) estimation problem Innovations are computed at the observation time using the high-resolution non-linear forecast model 4D-Var finds the 12-hour forecast that take account of the observations in a dynamically consistent way Based on a tangent linear and adjoint forecast models, that works in an 80,000,000 dimensional model subspace 9Z 12Z 15Z 18Z 21Z Assimilation at 3
Accuracy of Tangent-Linear and adjoint important: linearity issues Regularizations remove the most important threshold processes in physical parametrizations improving the validity of the tangent linear approximation Non-linear finite difference Thursday 15 March 2001 12UTC Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity 12 TL integration u-wind increments fc t+12, ~700 hpa Thursday 15 March 2001 12UTC Forecast t+12 VT: Friday 16 March 2001 00UTC Model Level 44 **u-velocity 12 8 4 2 8 4 2 1 1 0.5-0.5 0.5-0.5-1 -1-2 -2-4 -4-8 -8 Assimilation at -12 4-12
The Ensemble of Data Assimilations (EDA) Assimilation at 5
The Ensemble of Data Assimilations at (EDA) 10 (25 from November 2013) ensemble members using 4D-Var assimilations T399 (50km) outer loop, T95/T159 inner loops. (Deterministic 4D-Var: T1279 (16km) outer loop, T159/T255/T255 inner loops) Observations randomly perturbed according to their estimated errors SST perturbed with climatological perturbations Model error represented by stochastic methods (SPPT, Leutbecher, 2009) Assimilation at 6
The hybrid approach used at : EDA&4D-Var Hybrid approach: Use cycled, flow-dependent background error estimates from an Ensemble of Data Assimilations in a deterministic 4D-Var analysis. This hybrid formulation has many benefits: Introduces flow-dependent background errors into 4D-Var system Maintain the full rank representation of B and its implicit evolution inside the assimilation window More robust than pure EnKF for limited ensemble sizes Ensemble perturbations are used in 4D-Var control vector space, beneficial for e.g. assimilation of radiance observations Allows for flow-dependent Quality Control of observations Assimilation at 7
The EDA&4D-Var hybrid implementation at EDA Cycle x b +ε b i y+ε o i SST+ε SST i Analysis x a +ε i a Forecast x f +ε i f i=1,2,,10 (25) Variance post-process ε i f raw variances Variance Recalibration Variance Filtering EDA scaled variances 4DVar Cycle x b EDA scaled Var Analysis x a Forecast x b Assimilation at 8
Raw EDA variance estimates needs to be calibrated to become statistically consistent We performs an online calibration (Ensemble Variance Calibration; Kolczynsky et al., 2009, 2011; Bonavita et al., 2011) Calibration factors depend on latitude bands and parameter Calibration factors also depend on the size of the expected error Assimilation at 9
Sampling noise due to small ensemble size is a problem Noise filtering method used until June 2012: Use a spectral filter to disentangle noise from signal Truncation wavenumber is determined by maximizing signal-to-noise ratio of filtered variances (Raynaud et al., 2009; Bonavita et al., 2011) A more direct strategy applied now, based on two 50-member EDAs: 1. Sampling noise assumed a random process P Se n P Si S j 2. Time average sampling noise spectrum samples 1 3. Compute raw filters n and time average to smooth out P S 1 e noise P Sraw (based on Berre et al., 2010) 1 2 Assimilation at 10
Introducing flow-dependent background errors in 4D-Var In the 4D-Var, the B matrix is defined implicitly in terms of a transformation from the background departure (x-x b ) to a control variable χ: So that the implied B=LL T. (x-x b ) = Lχ In the current wavelet formulation (Fisher, 2003), the variable transform can be written as: x T is the balance operator x 1 1/ 2 1/ 2 b T Σb j C j, j Σ b is the gridpoint variance of background errors C j (λ,φ) is the vertical covariance matrix for wavelet index j ψ j are the set of radial basis function that define the wavelet transform. j Assimilation at 11
Introducing flow-dependent background errors in 4D-Var x x 1 1/ 2 1/ 2 b T Σb j C j, j C j (λ,φ) are full vertical covariance matrices, function of (λ,φ). They determine both the horizontal and vertical background error correlation structures; In standard 4D-Var T and C j are computed off-line using a climatology of EDA perturbations. Σ b is computed by random sampling of the static B matrix (randomization procedure, Fisher and Courtier, 1995) How do we make this error covariance model flow-dependent? We look for flow-dependent EDA estimates of Σ b and C j (λ,φ) j Assimilation at 12
Improved static background-error covariance statistics based on the latest EDA (implemented June 2012) Resolution upgrades and more observations since last update resulted in sharper structure functions: reduced correlation length scales both horizontally and vertically Assimilation at 13
EDA-based flow-dependent background errors for unbalanced control variables (T u,d u,lnsp u ) - June 2013 EDA Variances for the Unbalanced Control Vector (η u, (T,p s ) u ). Var Var M s u T, p N P T, u p s u Var Var T MVar M T Var T T, ps NVar N PVar u P u Var T, p s u Explained variance Ratio for divergence and temperature Derber, Bouttier, Fisher (1997) 14 Assimilation at Similar plot for the 2013 DA system
EDA-based flow-dependent background errors for unbalanced control variables (T u,d u,lnsp u ) - June 2013 Average unbalanced temperature (st.dev. in Kelvin) Previous bg error model for unbal. temp. EDA bg error for unbal. temp. Top of atmosphere Surface 90N 90S 90N 90S Assimilation at 15
Flow-dependent covariance estimation from an EDA Variance estimation needs an EDA sample size of ~10 Covariance estimation needs an EDA sample size of ~600 a) Background error covariances (JB) are computed in a postprocessing step of 25 member EDA b) EDA perturbations from the past 12 days are used for a weighted running mean (Sample size: 25*12*2=600) c) Continuously updated JB is used in deterministic 4D-Var Similar activities on-going at Météo-France (Varella et al. 2011) Assimilation at 16
Flow-dependent covariance estimation from an EDA EDA-based flow-dependent variances are computed for each analysis cycle - sufficient with 10 EDA members to estimate error-of-the day. St.dev of vorticity errors and Z at 500hPA Assimilation at 17
Flow-dependent covariance estimation from an EDA Covariance estimation requires a large sample size (order 600). This is computed with a lagged 12 days running average. Correlations are representative of prevailing weather patterns, not distinct weather features! St.dev of vorticity errors and Z at 500hPA Assimilation at 18
Why is flow-dependent JB better? Vertical correlations at (30N,140W) at 850hPa The 12-day averaging allows JB to cater for flow-regime changes The vertical correlations between 850hPa and the boundary layer change significantly from 10 th of January 2012 to 10 th of February 2012. 400 hpa 500 hpa 700 hpa 850 hpa 950 hpa Surface Assimilation at 19 Static JB 20120110 JB 20120210 JB
The next steps to improve the hybrid EDA & 4D-Var a) Reduce the time window used to compute the online JB (use more EDA forecast steps; hybrid with static JB; filtering of correlations) b) Introduce EDA errors and JB in the EDA members analysis (interactive EDA) c) Further extend the use of EDA errors for observation Quality Control d) Test EDA perturbations as EPS initial conditions towards a unified EDA and EPS Assimilation at 20
Reduce the time window used to compute the online JB Use more EDA forecast steps Hybrid with static JB; filtering of correlations Online JB with 12 day window, t+3h perturbations Online JB with 4 day window, t+0/3/6h perturbations Assimilation at 21
Hybrid EDA & 4DVAR improves forecast skill Impact of EDA based variances in hybrid 4D-Var Improved statistical noise filtering of EDA variances Static background errors updated, using latest EDA The introduction of flow-dependent background error variance and covariance estimated from the EDA has by far been the largest source of improvement in recent years in analysis and forecast skill at Assimilation at 22
Hybrid EDA & 4DVAR improves forecast skill Impact of EDA-based unbalanced control vector Reduction in Z RMSE (95% confidence, RAOBs) 200 hpa 500 hpa NH SH Impact of online JB Reduction in Z RMSE - 95% confidence NH 50 hpa SH 100 hpa 200 hpa 1000 hpa The introduction of flow-dependent background error variance and covariance estimated from the EDA has by far been the largest source of improvement in recent years in analysis and forecast skill at Assimilation at 500 hpa 1000 hpa 23
The background error model at has evolved over the past 30 years. The new flow-dependent wavelet-based B has many advantages: Compact model Good spectral resolution Conclusions and summary Non-separable (vertical and horizontal scales of background error covariance are non-separable: large horizontal scales tend to have deeper vertical correlations than small horizontal scales. It is important for a B model to retain this property) Non-homogeneous, based on wavelet formulation (Fisher,2003) Before: Isotropic Before: Static Now: Partially anisotropic (variances) Now: Flow-dependent Assimilation at 24
Conclusions and summary The EDA and hybrid EDA&4D-Var developments have been central to the recent analysis and forecast skill improvements at In general to estimate analysis uncertainty Improve the initial perturbations in the Ensemble Prediction June 2010 To estimate flow-dependent background error from the EDA for the balanced part of the control vector in 4D-Var May 2011 To improve observation QC decisions and usage in 4D-Var May 2011 To improve filtering of EDA sampling noise June 2012 To update the static background-error covariance statistics based on the latest EDA June 2012 To estimate flow-dependent background error estimates from the EDA for the unbalanced part of the control vector in 4D-Var June 2013 To estimate flow-dependent background correlations from the EDA variances in 4D-Var Nov. 2013 Assimilation at 25