1 How is balance of a forecast ensemble affected by adaptive and non-adaptive localization schemes? Ross Bannister Stefano Migliorini, Laura Baker, Ali Rudd NCEO, DIAMET, ESA University of Reading
2 Outline 1. Meteorological balance. (a) What and why? (b) How can assimilation lead to imbalance? (c) How to respect balance in data assimilation? (d) How much balance? 2. Ensemble data assimilation. (a) Sampling error. (b) Localization. (c) Balance (non-)preservation. (d) Possible mitigation strategies. 3. Balance diagnostics. 4. Adaptive and non-adaptive localization. (a) Spectral representation for static localization. (b) Static scheme 1 (S1). (c) Static scheme 2 (S2). (d) SENCORP adaptive localization. (e) ECO-RAP adaptive localization 1 (E1). (f) ECO-RAP adaptive localization 2 (E2). 5. Implied structure functions and balance diagnostics. 6. Conclusions. (a) Given an ensemble, what is the balance? (b) Localized diagnostics. (c) The meteorological case.
3 1 Meteorological balance 1(a) What is balance and why is it important to worry about? Initial conditions of meteorological models need to be appropriately balanced. Momentum equations Du Dt Dv Dt Dw Dt = fv 1 p ρ x D x, = fu 1 p ρ y D y, = 1 p ρ z g D x, f = 2Ω sin(y/a), Ω = 7.29 10 5 rads 1, a = 6.371 10 6 m, g = 9.806ms 1. Geostrophic balance fv 1 p ρ x = 0. Dimensionless variables u = Uũ, v = Uṽ, w = W w, p = P p, x = L x, z = H z, t = L/U t, Ro Dũ D t Ro Dṽ D t Ro W D w U D t = v P p fρul x D x, = ũ P p fρul ỹ D y, = P p fρuh z g fu D x, Ro = U fl = O(10 1 ), W U = O(10 2 ). Hydrostatic balance 1 p ρ z g = 0.
4 Geostrophic balance fv 1 p ρ x = 0. Geostrophic balance is characteristic of mid-latitude flow (small Ro wind follows the isobars). Hydrostatic balance 1 p ρ z g = 0. Some ageostrophic flow is needed in initial conditions to match the atmosphere. Too much ageostrophic flow can damage a forecast. Unbalanced motion relaxes to near balanced motion by geostrophic adjustment (gravity waves). All modern meteorological models are capable of supporting gravity waves so excessive gravity waves will cause a problem. Hydrostatic balance is characteristic of non-convective flow (small Ro and W/U). Meteorological models that permit convection explicitly must allow non-hydrostatic flow.
5 1(b) How can assimilation of observations lead to imbalance? Observations sample from the truth. The true manifold the model manifold. Observations are not perfect. Essentially discovered by L.F. Richardson in the 1920s.
6 1(c) How to respect balance in data assimilation? Initialization Post-posteriori filtering of imbalance according to a set of rules. Moves away from observations just assimilated. Forecast error covariance matrix x a = x f +P f H T ( R + HP f H T) 1 [ y Hx f ]. P f is used not just for regularization. P f contains the statistics of balance (e.g. for strong hydrostatic balance: in = x a x f (analysis increments), the correlation between [ p/ z] and [ρg] should be 1: p z + ρg = 0. P f in VAR is modelled using explicit balance conditions (called B). Example structure functions giving the output field (p, u or v down the side) associated with a point in the centre of the domain (either of p, u or v along the top). Red is positive, blue is negative.
7 1(d) How much balance should be in an analysis increment? Practically - we don t know. Have climatological ideas (e.g. winter average for mid-latitudes), but this could change from day-to-day (e.g. high-pressure vs. low pressure systems, fronts, convection, boundary layer characteristics, etc.) In the Kalman Filter this information will be wrapped-up in P f. In the Ensemble Kalman Filter this information is wrapped-up in the ensemble (see later). Want the analysis increments to be balanced in the way described by the Kalman update equation x a = x f + P f H T ( R + HP f H T) 1 [ y Hx f ].
8 2 Ensemble data assimilation 2(a) Sampling error Dynamical sample covariance P D (N) = 1 N 1 N l=1 P D (N) Rn n δx l δx T l = 1 N 1 XXT, Dynamical forecast ensemble X = {δx 1, δx 2,... δx N }, δx l R n X R n N Sample covariance error [E(δP D (N) )] ij σ iσ j N 1 (1 ( ) ) 2 [P]ij. σ i σ j v-p correlation function from ensemble with N = 24.
9 2(b) Localization P L (N,K) = P D (N) Ω (K) P D (N) Rn n Ω (K) R n n Notation: P D (N) forecast error covariance matrix from N ensemble members (rank N). Ω (K) localization / regularization matrix (rank K). Ω (K) is a correlation matrix.
10 2(c) Balance (non-)preservation Balance is defined by the null-space (or near null space) of P f. Localization changes the null-space. Localization assumes that structures become less important with distance from an observation. Some structures grow with distance. Example - can affect position of peaks in geostrophic lobes. Localization changes the values and gradients of fields 1. 1 Lorenc A.C., The potential of the ensemble Kalman Filter for NWP - a comparison with 4D-VAR, Quart. J. Roy. Meteor. Soc. 129, 3183-3203 (2003).
11 2(d) Possible mitigation strategies Avoid doing localization with highly anisotropic fields like u and v - apply to ψ and χ instead 2. OK for geostrophic balance, unclear what to do for other balances. Perform localization on control variables and introduce a balance operator. Like the Met Office s hybrid data assimilation system 3. OK when balances are known and appropriate. Adaptive localization schemes? SENCORP (Smoothed ENsemble COrrelations Raised to a Power) 4. ECO-RAP (Ensemble COrrelations Raised to A Power) 5 6. 2 Kepert J.S., Covariance localization and balance in an ensemble Kalman filter, Quart. J. Roy. Meteor. Soc. 135, 1157-1176, DOI:10.1002/qj.443 (2009). 3 Clayton A.M., Lorenc A.C. and Barker D.M., Operational implementation of a hybrid ensemble/4d-var global data assimilation system at the Met Office, Q.J.R. Meteorol. Soc., DOI:10.1002/qj.2054 (2012). 4 Bishop C.H. and Hodyss D., Flow adaptive moderation of spurious ensemble correlations and its used in ensemble-based data assimilation, Quart. J. Roy. Met. Soc. 133, 2029-2044 (2007), DOI:10.1002/qj.169. 5 Bishop C.H. and Hodyss D., Ensemble covariances adaptively localized with ECO-RAP, Part 1: Tests on simple error models, Tellus A 61, 84-96 (2009). 6 Bishop C.H. and Hodyss D., Ensemble covariances adaptively localized with ECO-RAP, Part 2: A strategy for the atmosphere, Tellus A 61, 97-111 (2009).
12 3 Balance diagnostics 3(a) Given an ensemble, what is the balance? General: equation of motion and two-term balance equation Q(x) t = A(x) + B(x) + C(x), 0 = A(x) + B(x). A(x) will be exactly anti-correlated with B(x) if this balance condition is obeyed exactly. Geostrophic balance (actually linear balance): the divergence equation Dδ Dt = M + W + horiz. Coriolis + metric + forcing + other, M = c p ( θv0 2 zπ + 2 zπ 0 θ v), W = k ( u + ( f) u). Hydrostatic balance: the vertical momentum equation Dw Dt = P + T + vert. Coriolis + metric + forcing + other, P = θ v0 Π z, T = Π 0 z θ v.
13 3(b) What is the balance correlation in a localized ensemble? Dynamical sample cov P D (N) = 1 N 1 N l=1 δx l δx T l = 1 N 1 XXT P D (N) Rn n Dynamical forecast ensemble X = {δx 1, δx 2,... δx N } X R n N Localized sample cov P L (N,K) = P D (N) Ω (K)
14 3(b) What is the balance correlation in a localized ensemble? Dynamical sample cov P D (N) = 1 N 1 N l=1 δx l δx T l = 1 N 1 XXT P D (N) Rn n Dynamical forecast ensemble X = {δx 1, δx 2,... δx N } X R n N Localized sample cov P L (N,K) = P D (N) Ω (K) Correlation Ω (K) = 1 K 1 K k=1 ω k ω T k = 1 K 1 KKT Ω (K) R n n Correlation ensemble K = {ω 1, ω 2,... ω K } K R n K
15 3(b) What is the balance correlation in a localized ensemble? Dynamical sample cov P D (N) = 1 N 1 N l=1 δx l δx T l = 1 N 1 XXT P D (N) Rn n Dynamical forecast ensemble X = {δx 1, δx 2,... δx N } X R n N Localized sample cov P L (N,K) = P D (N) Ω (K) Correlation Ω (K) = 1 K 1 K k=1 ω k ω T k = 1 K 1 KKT Ω (K) R n n Correlation ensemble K = {ω 1, ω 2,... ω K } K R n K Localized sample cov revisited P L (N,K) = 1 M 1 M δ x m δ x T m = m=1 1 M 1 X X T (M = NK) Localized ensemble X = {δ x1, δ x 2,... δ x M } X R n M δ x m = δx l ω k M 1 X = (N 1)(K 1) X K
16 3(c) The meteorological case 20/09/2011 MOGREPS: Met Office Global and Regional Ensemble Precipitation System
17 4 Adaptive and non-adaptive localization 4(a) Spectral representation (univariate non-adaptive localization for variable s) K s = F s Λ 1/2 s [K s ] rk = cos(k x r x + δs x ) cos(k y r y + δs)ν(r y z, k z ) }{{} λ H s (kx 2 + ky)λ 2 V s (k z ) }{{} F s Over-bar means normalize - make sum of squares of each row of matrix unity. Choose the # horiz. wns, K x\y, and # vert. modes, K z : K = K x\y K z. Λ 1/2 s In practice K n. 70 60 The first four vertical modes mode 1 mode 2 mode 3 mode 4 1 Static moderation function (either in horizontal or vertical) alpha = 1 alpha = 0 50 0.8 vertical level 40 30 function 0.6 0.4 20 10 0.2 0-10 -5 0 5 10 15 20 25 mode value 0 0 20 40 60 80 100 separation Figure 1: Left: First four vert. modes (ν(r z, k z )). Right: Moderation fns, λ H 2 s (rx\y ), λ V 2 s (rz ).
18 4(b) Static localization scheme 1 (S1) K S1 = F δu Λ 1/2 δu 0 0 0 0 0 F δv Λ 1/2 δv 0 0 0 0 0 F δw Λ 1/2 δw 0 0 0 0 0 F δπ Λ 1/2 δπ 0 0 0 0 0 F δθ Λ 1/2 δθ R n 5K 4(c) Static localization scheme 2 (S2) K S2 = F δu Λ 1/2 δu F δv Λ 1/2 δv F δw Λ 1/2 δw F δπ Λ 1/2 δπ F δθ Λ 1/2 δθ R n K
19 4(d) SENCORP (Smoothed ENsemble COrrelations Raised to a Power) localization Ω = C Q 1. From the ensemble members, δx l, create smoothed members, δw l. 2. Normalize. 3. Calculate correlation matrix C = 1 N 1 N δw l δwl T. l=1 4. C Q is the Schur power of C with itself Q times. K = {ω 1, ω 2,... ω K } = ω k = Q/2 Z and Q > 0 K 1 (N 1) QW W W K Rn K K 1 (N 1) Qδw l 1... δw lq K = N Q, but rank(c) < K
20 4(e) ECO-RAP (Ensemble COrrelations Raised to A Power) localization scheme 1 (E1) A combination of SENCORP and S1 F δu Λ 1/2 δu 0 0 0 0 0 F δv Λ 1/2 K E1 δv 0 0 0 = C Q 0 0 F δw Λ 1/2 δw 0 0 0 0 0 F δπ Λ 1/2 δπ 0 0 0 0 0 F δθ Λ 1/2 δθ R n 5K 4(f) ECO-RAP (Ensemble COrrelations Raised to A Power) localization scheme 2 (E2) A combination of SENCORP and S2 K E2 = C Q F δu Λ 1/2 δu F δv Λ 1/2 δv F δw Λ 1/2 δw F δπ Λ 1/2 δπ F δθ Λ 1/2 δθ R n K Practicality: expensive so have to restrict influence of C Q.
21 5 Implied structure functions and balance diagnostics (dynamic ensemble) (a) T-T (long/ht), (b) T-T (long/lat), (c) u-t (long/lat), (d) v-t long lat
22 5 Implied structure functions and balance diagnostics (static localization, S2) (a) T-T (long/ht), (b) T-T (long/lat), (c) u-t (long/lat), (d) v-t long lat
23 5 Implied structure functions and balance diagnostics (adaptive localization, SENCORP) (a) T-T (long/ht), (b) T-T (long/lat), (c) u-t (long/lat), (d) v-t long lat
24 5 Implied structure functions and balance diagnostics (adaptive localization, E2) (a) T-T (long/ht), (b) T-T (long/lat), (c) u-t (long/lat), (d) v-t long lat Fields not computed Fields not computed
25 5 Implied structure functions and balance diagnostics (dynamic ensemble) Geostrophic balance diagnostic Hydrostatic balance diagnostic 70 un-localized 70 un-localized 60 60 50 50 model level (0 is surface) 40 30 model level (0 is surface) 40 30 20 20 10 10 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced) 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced)
26 5 Implied structure functions and balance diagnostics (static localization, S2) Geostrophic balance diagnostic Hydrostatic balance diagnostic 70 un-localized S2 70 un-localized S2 60 60 50 50 model level (0 is surface) 40 30 model level (0 is surface) 40 30 20 20 10 10 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced) 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced)
27 5 Implied structure functions and balance diagnostics (adaptive localization, SENCORP) Geostrophic balance diagnostic Hydrostatic balance diagnostic 70 un-localized SENCORP 70 un-localized SENCORP 60 60 50 50 model level (0 is surface) 40 30 model level (0 is surface) 40 30 20 20 10 10 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced) 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced)
28 5 Implied structure functions and balance diagnostics (adaptive localization, E2) Geostrophic balance diagnostic Hydrostatic balance diagnostic 70 un-localized E2 70 un-localized E2 60 60 50 50 model level (0 is surface) 40 30 model level (0 is surface) 40 30 20 20 10 10 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced) 0-1 -0.5 0 0.5 1-1 (balanced), +1 (unbalanced)
29 6 Conclusions Balanced analyses for NWP. Crude DA does not respect balance. Not imposing balance when it should be,..., and imposing balance when it shouldn t be. Adding localization to EnKF disturbs balance. Balance diagnostics. Given an ensemble, find the correlation between leading terms. Apply to dynamical (raw) ensemble. Apply to localized ensemble (combining of dynamical and correlation ensemble). Correlation ensemble is squareroot of localization matrix. Localization schemes. Static (spectral) scheme (S2). SENCORP scheme (localization defined from smoothed ensemble). ECORAP (combination of S2 and SENCORP) scheme (S2). Findings SENCORP doesn t perform well. S2 performs well for geos. balance. E2 performs will for hydro. balance. Notes Many parameters (truncation, length-scales, order Q, other things). Look at other profiles and cases. Is it worth the computational effort? Other balances (anelastic, moisture,...).