An Ensemble Kalman Filter for NWP based on Variational Data Assimilation: VarEnKF

Size: px

Start display at page:

Download "An Ensemble Kalman Filter for NWP based on Variational Data Assimilation: VarEnKF"

Mervin Sanders
5 years ago
Views:

1 An Ensemble Kalman Filter for NWP based on Variational Data Assimilation: VarEnKF Blueprints for Next-Generation Data Assimilation Systems Workshop 8-10 March 2016 Mark Buehner Data Assimilation and Satellite Meteorology Research Section With contributions from Ron McTaggart-Cowan and Sylvain Heilliette

2 Context: EnVar Data Assimilation 4DEnVar replaced 4DVar for ECCC operational regional and global deterministic prediction systems Nov EnVar uses a variational assimilation approach in combination with the already available 4D ensemble covariances from the operational 256-member EnKF Future improvements to the ensembles will benefit both ensemble and deterministic prediction systems EnKF and EnVar estimate covariances from ensembles in a similar way, but important differences are unavoidable (e.g. localization, use of hybrid covariances) EnKF serially assimilates batches of perturbed obs to compute analysis increment for each member Page 2 March-9-16

3 Current Organization of the NWP Suites at Environment Canada Since 2014 implementation: Increasing role of global ensembles Global EnKF (256 mem) Global ensemble forecast (20 mem) Regional ensemble forecast (20 mem) Background error covariances Global EnVar Global deterministic forecast (GDPS) Page 3 March-9-16 Regional EnVar Regional deterministic forecast (RDPS) global system regional system

4 Motivation to Explore Alternative Ensemble Data Assimilation Approaches Only small fraction of Fortran code shared between EnKF and EnVar significant effort required to increase sharing Due to differences in EnKF and EnVar algorithms, changes to obs or covariances must be fully tested in both Due to computational cost, current EnKF algorithm limits the volume of observations (~40% of GDPS obs, no IR) EnVar uses theoretically better method for applying localization and has ability to use hybrid covariances, variational QC, and scale-dependent localization Motivation to examine variational approach for EnKF: experiments done to compare potential systems, not a clean comparison of algorithms Page 4 March-9-16

5 An EnKF based on Variational Approach Especially interesting for centers without an existing EnKF If a variational approach could be used for an EnKF with computational cost comparable to current EnKF algorithms, this would: 1. Reduce effort to maintain and improve systems (same unified code/algorithm used for all systems) 2. Reduce amount of required testing (same assimilation algorithm and obs used, therefore impact of changes more consistent for all systems) 3. Possibly improve quality of ensemble forecast (due to increased volume of assimilated obs and improved treatment of covariances) Page 5 March-9-16

6 Ensemble of Variational Analyses (EDA) Some centers now using this approach (ECMWF, M-F) The most direct approach is to use an ensemble of EnVar data assimilation cycles, each assimilating independently perturbed observation values: similar to EnKF, but very costly! Forecast Ensemble of Forecasts Analysis, x a Background, b x x b Analyses a( i) x, i 1: N ens Backgrounds b( i) x, i 1: N ens Observations, EnVar Analysis o y R Deterministic EnVar Ensemble B from EnKF Random System-error Perturbations + Perturbed observations o( i) y, i 1: N Page 6 March-9-16 ens Ensemble of EnVar Analyses Obs, y o Ensemble of EnVars R Ensemble B from EnKF or ens-envar

7 Ensemble of Full 4DEnVar Analyses The ensemble of full EnVar analyses results in a complete analysis for each member tested with 20 members The 20 EnVar members all use the 256-member EnKF ensemble to specify the B matrix Ensemble mean forecasts are improved by 6-12 h note: same model config used for forecasts with both approaches Root mean square error (solid) against global radiosondes and ensemble standard deviation (dashed) for 500hPa heights from EnKF and ensemble of 20 EnVars. EnKF (2hPa top) Ens-EnVar (0.1hPa top) Page 7 March-9-16

8 Separating Ensemble Mean and Perturbations The variational analysis for 20 members takes almost 2.5 hours on 1024 processors too expensive for operational use even for 20 members, but we want 256! To reduce computational cost, perform the analysis step separately for the ensemble mean and ensemble perturbations and simplify the problem for perturbations: x k a = x a + x k a, xk b = x b + x k b x k a = x a + x k a Mean update x b x a x a Page 8 March-9-16 Forecast from ensemble mean not actually required

9 Separating Ensemble Mean and Perturbations The variational analysis for 20 members takes almost 2.5 hours on 1024 processors too expensive for operational use even for 20 members, but we want 256! To reduce computational cost, perform the analysis step separately for the ensemble mean and ensemble perturbations and simplify the problem for perturbations: x k a = x a + x k a, xk b = x b + x k b x k a = x a + x k a x 2 b Perturbation update x 2 a Page 9 March-9-16 Forecast from ensemble mean not actually required

10 EnVar for the Mean Analysis Update For ensemble mean: full 4DEnVar analysis using ensemble mean background state and unperturbed observations to compute increment to ensemble mean: J x a = 1 2 xa T B 1 x a yo H(x b ) H x a T R 1 y o H(x b ) H x a Can use configuration almost identical to the well-tested deterministic system Variational approach suited for performing single analysis Page 10 March-9-16

11 EnVar for only the Mean Analysis Update A simple approach for incorporating more observations in the EnKF with little added cost would be to use EnVar to update the ensemble mean and the EnKF to update the perturbations: x k a = x k b + x a envar + x k a enkf Minimal additional development work, since EnVar can have nearly identical configuration as deterministic system Subset of GDPS obs Some centers get a similar benefit from recentering ensemble on deterministic analysis GEM GEM GEM GEM GEM EnKF Recentre ensemble GEM GEM GEM GEM GEM Compute mean EnVar Full Page set 11 March-9-16 of GDPS obs

12 Control member forecast results showing impact of using EnVar for only ensemble mean - Ensemble mean: EnVar with full set of GDPS obs* vs. Current EnKF* - Ensemble perturbations: Both use current EnKF - Experiments cover 3 January 15 January 2015 (26 forecasts) *Also includes non-zero inter-channel obs-error correlations, hybrid background-error covariances (10% B nmc + 90% B ens ), and variational QC *EnKF used older values of σ obs for AMSU-A/B and ATMS (mostly affects highest peaking AMSU-A channels that are too low) Page 12 March-9-16

13 Results: EnVar for mean, EnKF for perturbations Control member forecasts (deterministic forecast from mean analysis) 24h global forecasts 72h global forecasts U RH U RH Z T Z T Verification against ERA-interim Using EnVar with all GDPS obs to Page only 13 March-9-16 update the ensemble mean gives significant improvements for control member vs. Current EnKF

14 Using the Var System to Update Perturbations Var good for single analysis, not naturally suited to ensembles, so For perturbations: use a much cheaper, less precise approach to compute increment to all 256 ensemble perturbations x k a = xk a xk b Two approaches were examined: Stochastic-EVIL (Auligné et al., 2016, in review): Truncated eigen-decomposition approximation of analysis error covariance matrix P a (inverse of EnVar cost function Hessian) used to directly compute perturbation increment (K = P a H T R 1 ): x k a = P a H T R 1 r k Hx k b, where r k ~N 0, R Mean-Pert (Lorenc, 2015): Perform minimization possibly with fewer iterations or other strategies to efficiently solve equivalent variational problem for each perturbation analysis: J x k = 1 2 x k T B 1 x k r k Hx k b H x k T R 1 r k Hx k b H x k Page 14 March-9-16

15 Stochastic-EVIL for Updating Perturbations Tests with Stochastic-EVIL not promising: After initial tests with Lanczos minimizer, used a stand-alone iterative Eigen solver (ARPACK) to avoid limitations of obtaining Eigenvectors from variational minimization For full global system, requires prohibitively large number of Eigenvectors for reasonable update to ensemble perturbations (1600 still not enough! Equivalent to 3200 variational iterations) To be able to compute 1600 vectors requires significant simplifications to cost function (only climatological B) Calculation of Eigenvectors is inherently sequential, therefore difficult to parallelize over large number of processors Page 15 March-9-16

Analysis Increment to One Ensemble Perturbation For 300 hpa

iterations Direct update with 400 Hessian eigenvectors

Direct update with 1600 Hessian eigenvectors For similar

16 Analysis Increment to One Ensemble Perturbation For 300 hpa temperature: x k a = xk a xk b Ensemble of 20 EnVars, 70 iterations Direct update with 400 Hessian eigenvectors Perturbation minimization, 20 iterations Stochastic EVIL Direct update with 1600 Hessian eigenvectors For similar computational cost, minimization much more efficient than EVIL Page 16 March-9-16

17 Variational Minimization to Update Perturbations With Mean-Pert, need ways to speed up minimization: Reduce number of iterations Simplify B matrix used (3D instead of 4D, fewer ensemble members, or use only climatological B matrix) Reduce spatial resolution of analysis increment Reduce quantity of observations assimilated In the context of ensemble prediction, evidence that computing full perturbations with simple approaches works surprisingly well vs. sophisticated approaches: Magnusson et al. (2009): ECMWF system Raynaud and Bouttier (2015): Météo-France AROME Simplification to variational approach for perturbation increments should be less extreme than these Page 17 March-9-16

18 Variational Minimization to Update Perturbations With Mean-Pert, need ways to speed up minimization: Reduce number of iterations Simplify B matrix used (3D instead of 4D, fewer ensemble members, or use only climatological B matrix) Reduce spatial resolution of analysis increment Reduce quantity of observations assimilated What impact do these simplifications have on forecasts? Preliminary tests showed that reducing iterations and random system error perturbations changes distribution of spread Is there a combination of simplifications that sufficiently reduces the cost without significantly degrading forecasts? Page 18 March-9-16

19 Separate Minimization for Each Perturbation VarEnKF: Variational DA for mean and perturbations Initial test uses the following (extreme) simplifications: Keep same number of iterations as deterministic system (70) Only climatological B matrix with reduced resolution (3DVar) Reduced quantity of observations: no AIRS, IASI, CRIS, SSMIS, GeoRad (not used in current EnKF) The simplified B matrix and reduced volume of observations reduce the memory requirements while also decreasing the execution time Reduction in size of problem allows many jobs to be run in parallel: 1 Perturbation update has 2.5% the cost of full 4DEnVar! 256 members takes ~24min wall clock on 2048 processors Trivial to parallelize further (up to 256 jobs, each taking < 1min) Page 19 March-9-16

20 Impact of using 3DVar minimization vs. current EnKF for perturbation updates - Ensemble mean: Both use EnVar with full set of GDPS observations and same configuration - Ensemble perturbations: 3DVar vs. Current EnKF* each assimilating similar subset of observations - Experiments cover 3 January 15 January 2015 (26 forecasts) *EnKF used older values of σ obs for AMSU-A/B and ATMS (mostly affects highest peaking AMSU-A channels that are too low) Page 20 March-9-16

21 Perturbation increments computed with: 3DVar Current EnKF Results: 3DVar vs. current EnKF for perturbations Background ensemble Analysis ensemble (after adding system error perturbations) , 7 days after spin-up Page 21 March-9-16 Ensemble spread for Psfc (hpa) both experiments use EnVar for mean

22 Results: 3DVar vs. current EnKF for perturbations Control member forecasts (deterministic forecast from mean analysis) 24h global forecasts 72h global forecasts U RH U RH Z T Z T Verification against ERA-interim Using 3DVar with reduced set of obs Page 22 for March-9-16 perturbations is nearly equivalent to using Current EnKF for perturbations (both use EnVar for ens. mean)

23 Impact of VarEnKF* vs. Current EnKF* - Ensemble mean: EnVar with full set of GDPS obs* vs. Current EnKF* - Ensemble perturbations: 3DVar vs. Current EnKF* - Experiments cover 3 January 15 January 2015 (26 forecasts) *Also includes non-zero inter-channel obs-error correlations and hybrid background-error covariances (10% B nmc + 90% B ens ) *EnKF used older values of σ obs for AMSU-A/B and ATMS (mostly affects highest peaking AMSU-A channels that are too low) Page 23 March-9-16

24 Results: VarEnKF vs. current EnKF Control member forecasts (deterministic forecast from mean analysis) 24h global forecasts 72h global forecasts U RH U RH Z T Z T Verification against ERA-interim Using full 4DEnVar for mean and 3DVar Page 24 March-9-16 for perturbations (VarEnKF) gives significant improvement vs. using current EnKF

25 Results: CRPS for VarEnKF vs. current EnKF Page 25 March-9-16 VarEnKF gives similar or improved CRPS vs. Current EnKF

26 Conclusions Quality of ensemble forecasts improved by using 4DEnVar to update ensemble mean, nearly identical to deterministic system configuration: Much higher volume of assimilated observations than EnKF Non-zero inter-channel observation-error correlations Hybrid background-error covariances (10% / 90%) Variational Quality Control Severe simplifications to variational assimilation for updating ensemble perturbations: Have small impact relative to current EnKF (with comparable CPU time) Are there better simplifications? Should test using ensemble σ b Therefore, more efficient to dedicate development and computing resources to ensemble mean update than perturbation update: How general is this? Will it hold for a higher resolution regional system? What other data assimilation algorithms can be adapted to take advantage of this? e.g. EDA: 4DVar for mean, 3DVar for perturbations Page 26 March-9-16

27 Scale-dependent ensemble covariance localization NCAR MMM Seminar Series 11 March 2016 Mark Buehner Data Assimilation and Satellite Meteorology Research Section With contributions from Anna Shlyaeva and Jean-François Caron

28 Scale-dependent covariance localization Motivation Currently, EnVar uses single horizontal and vertical localization length scales, very similar to EnKF Comparing various studies, seems it is best to use different amount of localization depending on application: convective-scale assimilation: ~10km mesoscale assimilation: ~100km global-scale assimilation: ~1000km 3000km In the future, global systems may resolve convective scales Therefore, need a general approach for applying appropriate localization to wide range of scales in a single analysis procedure: Scale-dependent localization Possible in EnVar, since localization acts directly on modelspace covariances (not BH T and HBH T or R) Page 28 March-9-16

29 Scale-dependent covariance localization General Approach Ensemble perturbations decomposed with respect to a series of overlapping spectral wavebands (filter coefficients sum to 1 for each wavenumber) Apply scale-dependent spatial localization to the scaledecomposed ensemble covariances, both within-scale and between-scale covariances Keeping the between-scale covariances is necessary to maintain heterogeneity of ensemble covariances (Buehner and Charron, 2007; Buehner, 2012) Motivation different than spectral localization where the between-scale covariances are set to zero Page 29 March-9-16

Scale-dependent covariance localization 1D Idealized System Idealized system on 1D periodic domain Assume a simple true heterogeneous covariance function that is a spatially varying weighted average

30 Scale-dependent covariance localization 1D Idealized System Idealized system on 1D periodic domain Assume a simple true heterogeneous covariance function that is a spatially varying weighted average of 2 Gaussian functions with different length scales: total = small scale + large scale Length scales of Gaussian functions: large scale: 7 grid points small scale: 0.7 grid points Middle of domain dominated by small scale errors, both ends dominated by large scales Page 30 March-9-16

31 Scale-dependent covariance localization 1D Idealized System Ensemble perturbations decomposed with respect to 3 overlapping spectral wavebands: Page 31 March-9-16

32 Scale-dependent covariance localization 1D Idealized System Scale-dependent homogeneous spatial localization functions (Gaussian) are specified with length scales: 10, 3, and 1.5 grid points Localization of between-scale covariances constructed to ensure full matrix is positive-semidefinite: L i,j = (L i,i ) 1/2 (L j,j ) T/2 btwn scales i & j Within scale Between scale Page 32 March-9-16

33 Scale-dependent covariance localization 1D Idealized System Mostly large scale at both ends of the domain (top panel) and mostly small scale at the middle (bottom panel) Generate a random sample of 50 ensemble members and compute raw sample ensemble covariance Page 33 March-9-16

Stddev Error Mean Error Scale-dependent covariance localization 1D Idealized System From 5000 random realizations, compute mean and stddev of the error

34 Stddev Error Mean Error Scale-dependent covariance localization 1D Idealized System From 5000 random realizations, compute mean and stddev of the error of 50-member ensemble covariances with several types of localization: raw ens B localization 10 localization 1.5 scale-dependent loc. Page 34 March-9-16

35 Scale-dependent covariance localization 1D Idealized system: Conclusions When decomposed with respect to spatial scale: small-scale component of raw ensemble has spurious long-range covariances benefit from more severe localization of only smallscale component of ensemble covariances Variation in the amount of localization as a function of scale: reduces the between-scale covariances spectral localization this reduction corresponds with reducing the spatial heterogeneity not possible to keep all heterogeneity and severely increase localization of small scales Using scale-dependent spatial localization results in: similar mean error of covariance vs. only large-scale localization (both are much better than only small-scale localization) better stddev error of covariance vs. only large-scale localization, especially in areas where true covariances dominated by small-scales Page 35 March-9-16

36 Scale-dependent covariance localization Implementation in EnVar Current (standard) Approach Analysis increment computed from control vector (B 1/2 preconditioning) using: x x e k k j e L 1/ 2 k v k Scale-dependent Approach (Buehner and Shlyaeva, 2015, Tellus) Varying amounts of smoothing applied to same set of amplitudes for a given member k, j L 1/ 2 j where e k,j is scale j of normalized member k perturbation v k Page 36 9 mars 2016 k: member index k: member index j: scale index

37 Scale-dependent covariance localization 2D Sea Ice Ensemble Ensemble of sea ice concentration background fields (60 members, time-lagged ensemble) from the Canadian Regional Ice Prediction System ensemble of 3DVar analyses experiment Ensemble mean ice concentration Ensemble spread Page 37 March-9-16 Work of Anna Shlyaeva

38 Scale separation of ensemble perturbations with diffusion operator Apply diffusion with increasing length scales to the original ensemble perturbations Decompose into different scales by taking differences between perturbations before and after each level of diffusion Example: e original ensemble perturbation; D n diffusion with lengthscale n e 1 = D 10km (e) e 2 = D 30km (e 1 ) e 3 = D 100km (e 2 ) Scale 4 (smallest): e e 1 Scale 3: e 1 e 2 Scale 2: e 2 e 3 Scale 1 (largest): e 3 Scale-decomposed perturbations sum up to the original e Page 38 March-9-16 Work of Anna Shlyaeva

39 Scale separation with diffusion operator: Example of one ensemble perturbation Scale 4 (smallest) Scale 3 Original perturbation Scale 2 Scale 1 (largest) Page 39 March-9-16 Work of Anna Shlyaeva

40 Scale separation with diffusion operator: Ensemble spread for each scale Scale 4 (smallest) Scale 3 Full ensemble spread Scale 2 Scale 1 (largest) Page 40 March-9-16 Work of Anna Shlyaeva

41 Homogeneous correlation functions and chosen localization functions for each scales Page 41 March-9-16 Localization length scales: 500km, 150km, 50km, 30km (Gaussian-like functions) Work of Anna Shlyaeva

Assimilation of 2 observations One obs in

42 Assimilation of 2 observations One obs in area dominated by large-scale error, other in area of small-scale error Background field and obs 30km localization 500km localization No localization 150km localization Scale-dep. localization Page 42 March-9-16

43 Idealized data assimilation experiment setup True state: x t = x i (i th member) mean(x) = x t e i, e i ~ N(0,B) Background: x b = x t + e j e j = x j mean(x), e j ~ N(0,B) Observations: simulated by perturbing true state observe every 4th grid point, with random gaps to simulate "clouds e j (real background error) not used in ensemble B for assimilation Verification: compute analysis error for each scale (by decomposing x a x t by scale); averaged over 60 experiments Page 43 March-9-16 Work of Anna Shlyaeva

44 Results of the experiment Error Error (Ice Pack) Error (MIZ) Background Analysis 500km loc Analysis 30km loc Analysis S-D loc Largest scale Smallest scale Largest scale Smallest scale Largest scale Smallest scale Page 44 March-9-16 Work of Anna Shlyaeva

45 Scale-dependent covariance localization 2D Sea Ice Ensemble: Conclusions Scale separation can be performed using a diffusion operator (convenient for variational systems that use diffusion operator or recursive filter instead of spectral transform for modelling B) Strong spatial variation in partition of error wrt scale leads to strong spatial variation in strength of effective localization when using scale-dependent localization (similarity with adaptive localization approaches) Scale-dependent localization results in lowest analysis error for all scales in regions dominated by either smallscale or large-scale error in idealized DA experiments Page 45 March-9-16

46 Global NWP application: Horizontal Scale Decomposition Spectral filter response functions for decomposing with respect to 3 horizontal scale ranges km 2000 km 500 km Large scale Medium scale Small scale Page 46 9 mars 2016 Work of Jean-Francois Caron

47 Global NWP application: Horizontal Scale Decomposition Perturbations for ensemble member #001 Temperature at ~700hPa Full Large scale Small scale Medium scale +2 0 Page 47 9 mars Work of Jean-Francois Caron

Scale-dependent covariance localization Impact in single observation

the center of Hurricane Gonzalo (October 2014) hloc: 2800km +1 0-1 +1

(correlationlike) at 700 hpa resulting from various B matrices.

48 Scale-dependent covariance localization Impact in single observation DA experiments B ens Std hloc B ens No hloc 700 hpa T observation at the center of Hurricane Gonzalo (October 2014) hloc: 2800km B ens SD hloc B nmc Normalized temperature increments (correlationlike) at 700 hpa resulting from various B matrices hloc: 1500km Page 48 9 / mars 4000km 2016 / 10000km Work of Jean-Francois Caron

ens No hloc 700 hpa T observation at the center of

SD hloc B nmc +1 +1 Normalized temperature

49 Scale-dependent covariance localization Impact in single observation DA experiments B ens Std hloc B ens No hloc 700 hpa T observation at the center of a High Pressure hloc: 2800km B ens SD hloc B nmc Normalized temperature increments (correlationlike) at 700 hpa resulting from various B matrices hloc: 1500km Page 49 9 / mars 4000km 2016 / 10000km Work of Jean-Francois Caron

50 Scale-dependent covariance localization Forecast impact 1.5-month trialling (June-July 2014) in our global NWP system. 3DEnVar with 100% B ens used in both experiments 1) Control experiment with hloc = 2800 km, vloc = 2 units of ln(p) 2) Scale-Dependent experiment with a 3 horizontal-scale decomposition I. Small scale uses hloc = 1500 km II. Medium scale uses hloc = 2400 km III. Large scale with uses = 3300 km Ad hoc values Same vloc [2 units of ln(p)] for every horizontal-scale Page 50 9 mars 2016 Work of Jean-Francois Caron

51 Scale-dependent covariance localization Forecast impact Comparison against ERA-Interim Control Scale-Dependent U RH T+24h World Z T Page 51 9 mars 2016 Work of Jean-Francois Caron

52 Scale-dependent covariance localization Forecast impact Comparison against ERA-Interim Std Dev difference for U Control is better Scale-Dependent is better T+24h Zonal mean South Pole North Pole Page 52 9 mars 2016 Work of Jean-Francois Caron

53 Summary and Future Work Preliminary results using a horizontal-scale-dependent horizontal localization indicates small forecast improvements in our global NWP system. Expect larger improvements in a system with a larger range of scales and when assimilating dense high-resolution observations Up next Optimize the horizontal localization length scales used for each horizontal-scale band. An objective evaluation approach is needed. Tried several, so far without success! Examine the impact of horizontal-scale-dependent vertical localization Examine the impact of vertical-scale-dependent vertical localization. Page 53 9 mars 2016 Work of Jean-Francois Caron

54 Scale-dependent covariance localization General Conclusions Scale-dependent localization is feasible, but more expensive than single-scale localization: more spectral transforms or applications of diffusion operator per iteration To reduce computational cost, take advantage of lower resolution for large scales by degrading grid resolution (like with wavelets) Approach may provide net benefit by appropriately resolving background error over a wide range of scales with relatively small ensemble when assimilating all obs simultaneously Alternatives for future large-domain, high-resolution DA: Use broad localization (to avoid messing up large scales) with huge ensembles (to reduce sampling error for small scales) Other approaches seem more ad hoc: e.g. varying localization for different subsets of members, varying localization for different subsets of observations Page 54 March-9-16

55 Questions? Summary: VarEnKF: 4DEnVar for ensemble mean, simplified approach (3DVar) for ensemble perturbations Scale-dependent covariance localization to allow simultaneous estimation of a wide range of scales from assimilating all observations in EnVar Page 55 March-9-16

56 Extra Slides Follow Page 56 March-9-16

Scale-dependent covariance localization 1D Idealized System Within-scale and betweenscale localization matrices combined into a single multiscale localization matrix Note that

57 Scale-dependent covariance localization 1D Idealized System Within-scale and betweenscale localization matrices combined into a single multiscale localization matrix Note that between-scale blocks have diagonal values less than 1! This is a necessary consequence of requiring the matrix to be positivesemidefinite (we have no choice!) Page 57 March-9-16

58 Scale-dependent covariance localization 1D Idealized System The greater the change in localization length scale, the more reduced the between-scale localization function This spectral localization corresponds with a local spatial averaging of the covariance function, i.e. loss of heterogeneity Examples with 2 scales: reduction in between-scale localization (a) 10 and 8 (b) 10 and 3 (c) 10 and 0.2 Page 58 March-9-16

59 vertical level (scale) vertical level (scale) Scale-dependent covariance localization 1D Idealized System Positive-semidefiniteness required for physically realizable correlations, without it, localized matrix no longer guaranteed to be a covariance matrix Reduction in the between-scale localization function from changes in horizontal localization also occurs for between-vertical level localization, which is easier to interpret physically: Same severe horizontal localization for each level, Vertical correlations can be maintained: A B and C D, so A = C and B = D possible Very different horizontal localization at 2 levels, Impossible to maintain vertical correlations: A B and C = D, so A = C and B = D not possible A B A B C D C D horizontal position horizontal position Page 59 March-9-16

60 Scale-dependent covariance localization 1D Idealized System Apply various localization functions (right panel) and compare with true covariances: single, large-scale localization: 10 grid points single, small-scale localization: 1.5 grid points scale-dependent localization: 10, 3, 1.5 grid points For location where true covariances are dominated by large scale component: the small-scale localization does a very bad job, largescale localization is ok, scale-dependent similar, but less noisy Page 60 March-9-16

61 Scale-dependent covariance localization 1D Idealized System Apply various localization functions (right panel) and compare with true covariances: single, large-scale localization: 10 grid points single, small-scale localization: 1.5 grid points scale-dependent localization with 3 scales: 10, 3, 1.5 grid points Where true covariances are dominated by small scale component: small-scale loc. not quite as bad, but removes large-scale component, scale-dependent smoother and looks better than large-scale loc. Page 61 March-9-16

62 Scale-dependent covariance localization 1D Idealized System Look in more detail by dividing true covariance into 3 scales: Same for the ensemble covariance (between-scale cov. not symmetric!): Page 62 March-9-16

63 Scale-dependent covariance localization 1D Idealized System Look in more detail by dividing true covariance into 3 scales: Same for the ensemble covariance (between-scale cov. not symmetric!): Page 63 March-9-16

64 Scale-dependent covariance localization 1D Idealized System Same as previous slide, but shown separately for 2 locations only Note that unavoidable spectral localization with S-D spatial localization improves variance estimate no localization large-scale loc.: 10 grid points small-scale loc.: 1.5 grid points scale-dependent loc.: 10, 3, 1.5 grid points Page 64 March-9-16 Mean Error Stddev Error

65 Average standard deviation for different scales in the ice pack and MIZ Ice pack: large scales dominate MIZ: small scales dominate Largest scale Smallest scale Page 65 March-9-16

66 hpa Global NWP application: Horizontal Scale Decomposition Homogeneous horizontal correlation length scales Small scale Full Large scale 6-h temperature perturbation from 256-member EnKF Medium scale Page 66 9 mars 2016 km

The Big Leap: Replacing 4D-Var with 4D-EnVar and life ever since

The Big Leap: Replacing 4D-Var with 4D-EnVar and life ever since Symposium: 20 years of 4D-Var at ECMWF 26 January 2018 Mark Buehner 1, Jean-Francois Caron 1 and Ping Du 2 1 Data Assimilation and Satellite