Background Error Covariance! and GEN_BE!!! Tom Auligné! Gael Descombes!!!!!!

Transcription

1 2014 GSI Community Tutorial Background Error Covariance and GEN_BE Tom Auligné Gael Descombes Acknowledgments: R. Bannister, D. Barker, S. Rizvi, W-S. Wu Partial funding provided by the Air Force Weather Agency

2 Recap on Data Assimilation: Kalman Filter Algorithm Analysis step Hypotheses: observa*on and background errors are unbiased, normally distributed, uncorrelated, with known covariances The Best Linear Unbiased Es4mate (BLUE) is defined analy*cally as x a x f + K y o H(x f ) P a ( I KΗ)P f [ ] The solu*on is op*mal for minimum variance AND maximum likelihood It is equivalent to minimize the cost func4on ( ) = 1 2 x x b J x K = P f Η T ( ) T P f 1 ( x x b ) y o H x ( ΗP f Η T + R) 1 ( ( )) T R 1 y o H( x) ( ) Forecast step x f M(x a ) P f ΜP a Μ T + Q

3 Background Error: Practical Difficulties Defini*on of Background Error (BE) covariance: P f = ( x b x t ) x b x t Common prac*ce: the background x b is a (short- term) model forecast Assump*on: x b is the mean of a Gaussian distribu*on The covariance matrix P f defines the PDF of background errors Problem P f is a nxn matrix. Prohibi*ve cost to compute + store Need to use approxima*on B Prac*cal difficul*es Unfeasibly large size of B à covariance modeling Unknown true state x t à find surrogate Lack of sufficiently large popula*on of surrogates à reduced rank B ( ) T

4 Background Error: Mathematical Properties B is square and symmetric: eigenvalues of B are real and eigenvectors orthogonal B is posi*ve semi- definite: eigenvalues are posi*ve and the cost func*on convex in all direc*ons of the state space à guarantee of mimimum Eigenvectors can be associated with physical modes Large variance: Rossby- like slow structures Small variance: iner*o- gravity- like B is 4- dimensional u v p T Variance Auto- covariance Cross- covariance u v p T

5 Background Error: Role in the Assimilation BLUE analy*cal solu*on: x a x b = BΗ T ( ( )) ( ΗBΗ T + R) 1 y o H x b

6 Background Error: Role in the Assimilation BLUE analy*cal solu*on: x a x b = BΗ T ( ( )) ( ΗBΗ T + R) 1 y o H x b B weights the importance of the a- priori state B spreads out informa*on both horizontally and ver*cally in space to other (unobserved) variables B imposes balance (e.g. hydrosta*c and geostrophic) via sta*s*cal info B provides a means for observa*ons to act in synergy B is the last operator: the analysis increment lies in the subspace spanned by B

7 Modern Data Assimilation: Different Approaches Ensemble Kalman Filter N 1 e Monte Carlo es*ma*on B e = ( x N e 1 k x )( x k x ) T k =1 Limited- size ensemble à under- spread and sampling error Infla4on: with increased spread to avoid filter divergence Localiza4on (in space): for each model grid point, use only close observa*ons to compute the analysis increment: B = B e C Varia*onal DA (3D/4DVar): Sta*onary covariance model B = UU T B e Hybrid ensemble/varia*onal data assimila*on B = β e ( B e C) + β c ( UU T ) Cf. Whitaker tomorrow about GSI Hybrid

8 Modern Data Assimilation: Different Approaches Ensemble Kalman Filter Monte Carlo es*ma*on Limited- size ensemble à under- spread and sampling error Infla4on: with increased spread to avoid filter divergence Localiza4on (in space): for each model grid point, use only close observa*ons to compute the analysis increment: Varia*onal DA (3D/4DVar): Sta*onary covariance model B = UU T Hybrid ensemble/varia*onal data assimila*on

9 Background Error: Estimation Methods Analysis of innova*ons: separate background errors by assuming observa*on errors are uncorrelated spa*ally. Requires dense observing network. NMC method: differences b/w forecasts (e.g. 48h and 24h) valid at same *me. Assumes same model bias and covariances + uncorrelated errors. Canadian Quick method: uses forecast *me lags B 1 ( 2 x b ( t + 6) x b ( t) )( x b ( t + 6) x b ( t) ) T Ensemble method: requires a separate EnKF or a B matrix to perturb model appropriately B ( )( x 48 x 24 ) T B 1 2 x 48 x 24 1 N e 1 N e k =1 ( x k x )( x k x ) T

10 Background Error: Examples of auto-correlations From Descombes et al. 2014

11 Background Error: Examples of auto-correlations From Michel and Auligné 2010

12 Background Error: Covariance Modeling B = UU T B is symmetric posi*ve- definite A square root of B is modeled via the following sequence of operators U = U p U v U h S S Variance scaling factor (gridpoint space) U h Horizontal Transform (horizontal auto- correla*ons) U v Ver*cal Transform (ver*cal auto- correla*ons) Physical Transform (sta*s*cal balance) U p

13 Background Error: Covariance Modeling B U p U v U h.... S I

14 Covariance Model: Sequence of Operators U = U p U v U h S S rescale variance with inhomogeneous standard devia*ons (la*tudes) (alterna+vely Gridpoint variance fields, Binning) U h model horizontal auto- correla4ons through successive applica*ons of Recursive Filters = affordable approx to diffusion operator (alterna+vely Diffusion, Spectral, Wavelet diagonal) U v model ver4cal auto- correla4ons by applica*ons of Recursive Filters (alterna+vely homogeneous Empirical Orthogonal Func+ons (EOFs)) U p model cross- correla4ons between different analysis variables via sta*s*cal balance (linear) (alterna+vely LBE, NLBE)

15 Covariance Model: U h (horizontal transform) 4 passes of 1 st order 1 pass of 4 th order Gaussian From Wu et al Model fat tails via 3 successive applica*ons of Recursive Filters (with different length- scales)

16 Covariance Model: U h (horizontal) The globe is divided into 3 sub- domains with 2 blending zones (smooth transi*on): Two Cartesian polar patches (stereographic projec*on) A zonal band (account for scale factor) Paralleliza*on: from *les to slabs From Wu et al. 2002

17 Covariance Model: U p (physical transform) [u, v] à [ψ, χ] Sta*s*cal Balance: t = t u + t b = t u + Nψ where N is an empirical matrix that projects increments of stream func*on at one level to a ver*cal profile of the balanced part of temperature increments. $ ψ ' $ I ' $ ψ ' & χ ) & M I 0 0 0) & χ u ) & ) & )& ) & t ) = & N 0 I 0 0) & t u ) & Ps) & Q 0 0 I 0) & Ps u ) & % rh ) & )& ( % I * (% rh ) ( qoption=2 qoption=1 " " t % $ I 0 0 $ p ' $ # rh ' = $ 0 I 0 $ & rh b rh b $ σ rh b b # α t σ rh b σ rh b ( ) 1 N is la*tude dependent. M la*tude and height, Q height. ( ) 1 rh b p b ( ) 1 q b % '" t % ' $ p' ' $ ' # q& ' & Holm et al. (2002) χ b χ t b t 1 Clouds cw, q c, q i, etc.

18 Covariance Model: Parameters S Standard Devia4ons U h Length Scales U v U p Length Scales Regression Coefficients Calibra4on Step Forecast error surrogates GEN_BE {cor} {vz} {hwl} {vi} Data Assimila4on U = U p U v U h S

19 Preconditioning: Control Variable Transform The analysis corresponds to the minimum of the cost func*on J( x) = 1 ( 2 x x b) T P f 1 ( x x b ) y o H x Introducing The incremental formula*on becomes ( ) = 1 2 δxt B 1 δx d Ηδx J δx δx = x x b d = y o H x ( ) and assuming Precondi*oning with the Control Variable Transform (CVT) B = UU T δx = Uv ( ( )) T R 1 ( y o H( x) ) H( x + δx) H( x) + Ηδx ( )T R 1 d Ηδx J( v) = 1 2 vt v ( d ΗUv)T R 1 ( d ΗUv) ( )

20 Pseudo Single Observation Test: PSOT BLUE analy*cal solu*on: x a x b = BΗ T ( ( )) ( ΗBΗ T + R) 1 y o H x b Define a synthe*c observa*on such as [y o - H(x b )] = 1.0 ; H = I ; R = I Hence the analysis increment becomes x a - x b = B * delta vector Ac*va*on in the Namelist &SETUP oneobtest=.true. &SINGLEOB_TEST maginnov=1.,magoberr=1.,oneob_type= t, oblat=45.,oblon=270.,obpres=850., obdasme= ,obhourset=0.,

21 PSOT: Structure Functions Pseudo Single Obs Test can help trace a column of B Iden*fy its shorwalls Provide guidelines for tuning from Rizvi 2013

22 4DVar: Implicit Time Propagation of BE J( v) = 1 2 vt v ( d ΗMUv)T R 1 ( d ΗMUv) J(v) = v + Μ T Η T R 1 ( d ΗMUv) from Zhang et al. 2010

23 Background Error: Impact on Forecast 12 hr f/c bias/rmse for Sound T 24 hr f/c bias/rmse for Sound T Old BE New BE expa bias expa RMSE expb bias expb RMSE bias/rmse (K) Valid time Valid time h cycling experiments from Rizvi 2013

24 Covariance Model Parameters: Tuning Calibra4on Step Forecast error surrogates GEN_BE {cor} {vz} {hwl} {vi} Data Assimila4on U = U p U v U h S GSI namelist (apply to all variables) &BKGERR vs=1.0, hzscl=0.373, 0.746, 1.50 hswgt=0.45, 0.3, 0.25 GSI anavinfo (for each analysis variable) as/tsfc_sdv=1.0,1.0,0.5,0.7,0.7,0.5,1.0,1.0,,1.0,1.0

25 GEN_BE version 2.0

26 Why GEN_BE v2.0? New needs in DA, necessary to redesign the modeling of B for cloudy radiances, chemistry and aerosol applications A flexible tool that helps to investigate, to diagnose and to implement the modeling of B minimizing development efforts. A user friendly tool that allows to model B on different DA platforms (e.g. WRFDA, GSI, etc.) using different model input (WRF, UM,...) to unite the developments. Algorithms of stages are independent of the control variables (user choice). The control variables and their covariance errors are driven by a namelist file input. A broad set of control variables are available and easily extendable to new ones.

27 Structure of GEN_BE v2.0? WRF UM?

28 Test Case 6h WRF forecast at 15 km of resolution over the CONUS, valid at: 12:00z on June 3, members of the day (DART experiment, Romine et al. 2012)

29 Stage 0: Compute perturbations Convert model variables into analysis control variables. (meteorology & chemistry) Compute perturbations using either NMC or ENS method. (Parrish and Derber, 1992) Nomenclature of the Description control variables psi Stream function () chi Velocity potential (") vor Vorticity div Divergence u Horizontal wind component in x direction v Horizontal wind component in the y direction t Temperature ps Surface pressure (Pereira and Berre, 2006) rh qs qcloud qrain qice qsnow sst Relative humidity Specific humidity Cloud mixing ratio Rain mixing ratio Ice mixing ratio Snow mixing ratio Sea Surface Temperature

30 Stage 1: Binning Stage 1: remove mean from the perturbations computed in Stage 0 and define the binning option from the namelist. A type of averaging, called binning, is applied to increase the number of samples and to gather similar phenomena keeping heterogeneity. 7 binning options are currently available: static binning and geographical mask. New binning options can easily be implemented.

31 Stage 1: Binning options Bin_type Description 0 Binning by grid point. 1 Binning by vertical level along the x direction point of the model. 2 Binning by vertical heights and by latitude num_bins_lat. The parameters binwidth_lat and binwidth_hgt defined the width that splits the bins. 3 Binning by vertical level model and latitude dependent. The parameters lat_min, lat_max are computed from the model input data and the parameter binwidth_lat is defined in the namelist.input file. actually used in GSI 4 Binning by model vertical level and along the y direction. 5 Binning on vertical level model including all the horizontal point. 6 Average over all points. 7 Binning rain/no-rain by vertical levels and based on thresholds in the model background (Michel and al., 2011.). geographical mask, implemented in WRFDA

32 Stage 2: Balance Operator Stage 2 mimics correlated errors between the control variables performing linear regression. Diagnostics are useful to define them. First, the regression coefficient between 2 variables is computed. Then, an unbalanced variable is diagnosed by removing the balanced part (Bdiagonal blocks). Currently in GSI: {, ",T, ps,rh} regression #### $ {, " u,t u, ps u,rh}

33 Stage 2, diagnostics humidity RH/T Figure 14. Plot of pressure (hpa) against vertical model levels. Fig. 14. Plot of Pressure (hpa) against vertical model levels. 55 QS/T Bin rh [90%-100%] (Holm, ECMWF) Discussion Paper Discussion Paper Discussion Paper Discussion Paper GMDD 7, 1 62, 2014 QS/T Generalized Background Error covariance matrix model (GEN_BE v2.0) G. Descombes et al. Abstract Conclusions Tables J J Back Title Page Introduction References Figures I I Close Full Screen / Esc Printer-friendly Version Interactive Discussion 150 (hpa)

34 Control variables and covariances (NCEP, CV5) {, ",T, ps,rh} regression #### $ {, " u,t u, ps u,rh} &gen_be_cv Namelist options Description nb_cv 5, Number of control variables cv_list psi, chi, t, ps, rh, Variables used for the analysis fft_method 1,2 Conversion of u and v to psi and chi 1=Cosine, 2=Sine transform covar1 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, First variable do not have covariance covar2 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, Covariance of variable 1 (psi) and variable 2 (chi) covar3 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, Covariance of variable 1 (psi) with variable 3 (t) covar4 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, Covariance of variable 1 (psi) with variable 3 (ps) covar5 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Relative humidity univariate S T A G E 0 S T A G E 2

35 Example of multivariate approach (CV9) &gen_be_lenscale Namelist Options nb_cv 9, cv_list psi, chi, t, ps, rh, qcloud, qice, qrain, qsnow, covar1 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, covar2 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, Multivariate relative humidity (1) covar3 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, covar4 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, covar5 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, qcloud u (i, j,k) = qcloud(i, j,k) " $ # qcloud,rhu (b,k,l) rh u (i, j,l) N k l =1 Multivariate hydrometeors (2) covar6 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, Covar7 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, Covar8 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Univariate hydrometeors (3) Covar9 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, N k rh u (i, j,k) = rh(i, j,k) #" rh,tu (b,k,l)t u (i, j,l) " rh,psu (b,k)ps u (i, j) l=1 qcloud u (i, j,k) = qcloud(i, j,k) " $ # qcloud,rhu (b,k,l) rh u (i, j,l) N k l =1 (1) (2)

36 Example of multivariate approach By changing the namelist line covar6 to covar6=0,0,0,0,2,0,0,0 ; Eq. (2) becomes: By using 5 control variables, qs instead of rh and covar5=1,0,1,1,0,0 ; Eq. (3) becomes: l=1 q cloudu (i,j,k) = q cloud (i,j,k) q cloud,rh u (b,k)rh u (i,j,k) (2b) qs u (i,j,k) = qs(i,j,k) (Carron and Fillon, 2010) N k l=1 N N k l=1 qs, psi(b,k,l)psi(i,j,l) qs,t u (b,k,l)t u (i,j,l) qs,ps u (b,k)ps u (i,j) (1b)

37 Stage 3: Vertical transform Vertical length scale parameter defines the radius of influence for the correlated error of a variable and is an input for the recursive filters that spread out the information vertically. Diagnostic coming from the Daley (p110, 1991) formula: L v = 1 2 "(0) Ρ(0) the correlation taken at the origin L vp (k) = 1 2[1 "(k) "(k 1)] k vertical level

38 Stage 3: Vertical transform 150 (hpa)

39 Stage 4: Horizontal transform The horizontal length scale parameter defines the radius of influence for the correlated error of a variable and is an input for the recursive filters that spread out the information Horizontally. Diagnostic of field φ coming from the W. S. formula, 2002: L = 8 Variance(") Variance(# 2 ")

40 Stage 4: Horizontal transform 150 (hpa) Model Resolution = 15 km

41 Chemistry control variables (WRF-CHEM) (a) (b) (c) (d) Test performed from an experiment, 20 ensemble members, 36 km of resolution, 33 vertical levels, 12h forecast, BC Mozart Perturbations, Megan Emissions (Courtesy of J. Barré).

42 Chemistry control variables (WRF-CHEM) 1 Fig. A2. Horizontal length scale of O 3, NO 2, 2 Test Performed from an experiment, ensemble 20 members, 36km resolution, 33 vertical levels, 12h forecast, BC Mozart Perturbations, Megan Emissions (Courtesy of J. Barré). 3 Fig. A3. Vertical length scale of O 3, NO 2, NO

43 Conclusions The GEN_BE v2.0 is a flexible and user friendly community tool (multi model/da systems). The code allows diagnostics and to compute the parameters (variances, length scales and regression coefficients) that model B for a large set of control of variables and options. The code can provide the statistics (input parameters) that model B for GSI (and WRFDA). Development may be needed in the GSI code to have more flexibility to handle new control variables and balance operators in the modeled B part.

44 Background Error Covariance: References Bannister, 2008a: A review of forecast error covariance sta*s*cs in atmospheric varia*onal data assimila*on. I: Characteris*cs and measurements of forecast error covariances. QJRMS. Bannister, 2008: A review of forecast error covariance sta*s*cs in atmospheric varia*onal data assimila*on. II Modelling the forecast error covariance sta*s*cs. QJRMS. Michel and Auligné, 2010: Inhomogeneous Background Error Modeling and Es*ma*on over Antarc*ca. MWR. Purser et al., 2003a: Numerical aspects of the applica*on of recursive filters to varia*onal sta*s*cal analysis. Part I: Spa*ally homogeneous and isotropic Gaussian covariances. MWR. Purser et al., 2003b: Numerical aspects of the applica*on of recursive filters to varia*onal sta*s*cal analysis. Part II: Spa*ally inhomogeneous and anisotropic general covariances. MWR. Holm et al., 2002: Assimila*on and Modelling of the Hydrological Cycle: ECMWF s Status and Plans. ECMWF technical memorandum. Rizvi, 2013: WRFDA Background Error Es*ma*on. WRFDA Tutorial. Wu, 2013: Background and Observa*on Error Es*ma*on and Tuning. GSI Tutorial. Descombes et al., 2014: Generalized Background Error Covariance Matrix Model (GEN_BE v2.0). GMDD [submi{ed].

45 Background Error Covariance: References (cont d) Caron, J. F. and Fillion L.: An Examina*on of Background Error Correla*ons between Mass and Rota*onal Wind over Precipita*on Regions. Mon. Weather Rev., 138 (2), , doi: h{p://dx.doi.org/ /2009mwr2998.1, Daley, R.: Atmospheric Data Analysis. Cambridge Univeristy Press, Michel Y., Auligné T. and Montmerle T.: Heterogeneous convec*ve- scale Background Error Covariances with the inclusion of hydrometeor variables. Mon. Weather Rev., 139, 9, , doi: h{p://dx.doi.org/ /2011mwr3632.1, Parrish, D. F., and J. C. Derber: The Na*onal Meteorological Center s Spectral Sta*s*cal- interpola*on Analysis System. Mon. Weather Rev., 120, , Pereira, M. B., and Berre, L.: The Use of an Ensemble Approach to Study the Background Error Covariances in a Global NWP Model, Mon. Weather Rev., 134, , doi: h{p://dx.doi.org/ /mwr3189.1, Romine G., Weisman M., Manning K., Wang W., Schwartz C., Anderson J. and Snyder C.: The Use of WRF- DART Analyses for 3 km Explicit Convec*ve Forecasts in Support of the 2012 DC3 Field Program. 13th WRF Users Workshop, Boulder, CO, June , WS2012/5.2, Wu, W., S., Purser, R., J., and Parrish, D., F.: Three- Dimensional Varia*onal Analysis with Spa*ally Inhomogeneous Covariances, Mon. Weather Rev., 130, , doi: h{p://dx.doi.org/ / (2002)130<2905:tdvaws>2.0.co;2, 2002.

46 2014 GSI Community Tutorial The End