Background and observation error covariances Data Assimilation & Inverse Problems from Weather Forecasting to Neuroscience

Transcription

1 Background and observation error covariances Data Assimilation & Inverse Problems from Weather Forecasting to Neuroscience Sarah Dance School of Mathematical and Physical Sciences, University of Reading July of 30

2 Outline Introduction Background errors Observation error covariances References 2 of 30

3 3D-Var J(x) = ( x x b) T B 1 ( x x b) + (y H(x)) T R 1 (y H(x)). 3 of 30

4 Introduction Background errors Observation error covariances References 4 of 30

5 Statistical viewpoint of background errors x t = x b + ε b ; x t, x b, ε b R n 5 of 30

6 Statistical viewpoint of background errors x t = x b + ε b ; x t, x b, ε b R n (1) The background errors are assumed unbiased, E(ε b ) = 0 5 of 30

7 Statistical viewpoint of background errors x t = x b + ε b ; x t, x b, ε b R n (1) The background errors are assumed unbiased, E(ε b ) = 0 with covariance E [ ε b (ε b ) T ] = B 5 of 30

8 Statistical viewpoint of background errors x t = x b + ε b ; x t, x b, ε b R n (1) The background errors are assumed unbiased, E(ε b ) = 0 with covariance E [ ε b (ε b ) T ] = B and Gaussian distributed ε b N(0, B) 5 of 30

9 Tikhonov regularization viewpoint The minimization problem, min J(x) = (y H(x)) T R 1 (y H(x)). is ill-posed, since there are not enough observations to define x uniquely. 6 of 30

10 Tikhonov regularization viewpoint The minimization problem, min J(x) = (y H(x)) T R 1 (y H(x)). is ill-posed, since there are not enough observations to define x uniquely. The addition of the background term can be viewed as regularization. B is then chosen to give a balance between fitting the observations closely and making the problem easier to solve. 6 of 30

11 Role of B in the analysis It is instructive to consider the solution for the analysis in the following form, x a = x b + BH T (HBH T + R) T (y Hx b ), (2) (compare with Kalman filter). 7 of 30

12 Role of B in the analysis It is instructive to consider the solution for the analysis in the following form, x a = x b + BH T (HBH T + R) T (y Hx b ), (2) (compare with Kalman filter). Thus, it is clear that the analysis increments are linear combinations of the columns of B. 7 of 30

13 Spreading and smoothing 8 of 30 Polly Smith, 2010

14 Multivariate effects Figure from Ross Bannister 9 of 30

15 Pre-conditioning with B B has a large effect on the number of iterations required to solve the variational assimilation problem (e.g., Haben et al. (2011)). 10 of 30

16 Pre-conditioning with B B has a large effect on the number of iterations required to solve the variational assimilation problem (e.g., Haben et al. (2011)). In NWP, a control variable transform is used to precondition with B. By setting x = Uχ; B = UU T 10 of 30

17 Pre-conditioning with B B has a large effect on the number of iterations required to solve the variational assimilation problem (e.g., Haben et al. (2011)). In NWP, a control variable transform is used to precondition with B. By setting x = Uχ; B = UU T (3) We can transform the background term in the cost function from ( x x b) T ( B 1 x x b) (χ χ b) T ( χ χ b). 10 of 30

18 Estimating and modelling B Since we do not know the truth, we can only guess what B really is! 11 of 30

19 Estimating and modelling B Since we do not know the truth, we can only guess what B really is! Typically we use forecast differences (difference in lead time) or an ensemble (of forecasts or analyses) to provide a proxy for background errors 11 of 30

20 Estimating and modelling B Since we do not know the truth, we can only guess what B really is! Typically we use forecast differences (difference in lead time) or an ensemble (of forecasts or analyses) to provide a proxy for background errors These are typically transformed from model variables to a reduced set of variables - this reduces the cost and also imposes dynamical balances. Bannister (2008a,b) provides a comprehensive review of these methods. 11 of 30

21 Background errors summary Background errors play a large role in Spreading and smoothing the increments Setting the multivariate balance properties of the analysis Pre-conditioning by B is often used in variational DA - the control variable transform B cannot be estimated directly so it is often estimated using forecast differences, or ensembles (of forecasts or analyses) as a proxy. 12 of 30

22 Introduction Background errors Observation error covariances References 13 of 30

23 What are observation errors? In data assimilation, we consider the observation equation y = H(x) + ε o. We assume ε o is unbiased and has covariance R such that R ij = E(ε o i εo j ). 14 of 30

24 Where do observation errors come from? The error vector, ε o, contains errors from three main sources: Instrument noise regular calibrations should ensure noise is uncorrelated between channels usually provided by instrument manufacturer 15 of 30

25 Where do observation errors come from? The error vector, ε o, contains errors from three main sources: Instrument noise regular calibrations should ensure noise is uncorrelated between channels usually provided by instrument manufacturer Observation processing and forward model error e.g., mis-representation of gaseous contributors undetected cloud 15 of 30

26 Where do observation errors come from? The error vector, ε o, contains errors from three main sources: Instrument noise regular calibrations should ensure noise is uncorrelated between channels usually provided by instrument manufacturer Observation processing and forward model error e.g., mis-representation of gaseous contributors undetected cloud Representativity error contrasting model and observation resolutions observations resolve spatial scales or features that the model cannot contributes to cross channel observation error correlations 15 of 30

27 Problems dealing with observation error correlations Magnitude and character of observation error correlations largely unknown - can only be estimated in a statistical sense, not observed directly 16 of 30

28 Problems dealing with observation error correlations Magnitude and character of observation error correlations largely unknown - can only be estimated in a statistical sense, not observed directly In the early days of data assimilation, there were fewer observations from remote sensing, so historically the most effort has been put into modelling and preconditioning B. 16 of 30

29 Problems dealing with observation error correlations Magnitude and character of observation error correlations largely unknown - can only be estimated in a statistical sense, not observed directly In the early days of data assimilation, there were fewer observations from remote sensing, so historically the most effort has been put into modelling and preconditioning B. Observations thinned spatially to reduce the correlations, and the R matrix treated as diagonal. 16 of 30

30 Problems dealing with observation error correlations Magnitude and character of observation error correlations largely unknown - can only be estimated in a statistical sense, not observed directly In the early days of data assimilation, there were fewer observations from remote sensing, so historically the most effort has been put into modelling and preconditioning B. Observations thinned spatially to reduce the correlations, and the R matrix treated as diagonal. However with higher resolution forecasting we need to retain the observed information on finer scales. 16 of 30

31 What do we lose? Dealing properly with correlations would allow More information to be extracted from observations (Stewart et al., 2008) Shannon Info. Content 1 2 ln B (H T R 1 t H+B 1 ) 1 +K (R t R f )K T B = I, R t structure for AMV observations (Bormann et al., 2003) 17 of 30

32 Structure of representativity error Use a method of Daley (1993); Liu and Rabier (2003) to calculate approximate representativity error covariance for Met Office model. Idea: ε H = H(x t ) H(x). 18 of 30

33 Structure of representativity error Use a method of Daley (1993); Liu and Rabier (2003) to calculate approximate representativity error covariance for Met Office model. Idea: ε H = H(x t ) H(x). Assume: Model state is a truncation of a high resolution true state. 18 of 30

34 Experiment, Waller et al. (2013) True states are temperature and humidity from the Met Office UKV (1.5km) model. Truncation is x8 (12km grid). Observed at all model grid points. Two cases Case 1: cloudy/convection Case 2: slow tracking deep depression 19 of 30

35 Case 1 Data Temperature (K) Log(Specific humidity) (kg/kg) 20 of 30

36 Conclusions from Experiment Errors of representativity: are correlated, state and time dependent vary with height throughout the atmosphere are more significant for humidity than for temperature are reduced by using a finer model discretization mesh depend only on the distance between the observations 21 of 30

37 How can we estimate observation errors? Desroziers (2005) Assume linear observation operator and DA statistics are the true statistics. The background state, x b, and observation vector, y, are approximations to the true state of the atmosphere, x t, y = Hx t + ε o x t = x b + ε b where ε o and ε b are the observation and background errors respectively. The best linear unbiased estimate of the true state, x a, is given by 22 of 30 x a = x b + K(y Hx b ) def = x b + Kd o b ; K = BHT (HBH T + R) 1

38 Desrozier s statistical method Background innovation d o b = y Hx b = Hx t + ε o Hx b 23 of 30

39 Desrozier s statistical method Background innovation d o b = y Hx b = Hx t + ε o Hx b = ε o + Hε b 23 of 30

40 Desrozier s statistical method Background innovation d o b = y Hx b = Hx t + ε o Hx b = ε o + Hε b Analysis innovation d o a = y Hx a = y H(x b + Kd o b ) 23 of 30

41 Desrozier s statistical method Background innovation d o b = y Hx b = Hx t + ε o Hx b = ε o + Hε b Analysis innovation d o a = y Hx a = y H(x b + Kd o b ) = (I HK)d o b 23 of 30

42 Desrozier s statistical method Background innovation d o b = y Hx b = Hx t + ε o Hx b = ε o + Hε b Analysis innovation d o a = y Hx a = y H(x b + Kd o b ) = (I HK)d o b = R(HBH T + R) 1 d o b 23 of 30

43 Desroziers et al. (2005) statistical method R estimate ( ) E d o a (d o b )T = E (R(HBH ) T + R) 1 d o b (do b )T 24 of 30

44 Desroziers et al. (2005) statistical method R estimate ( ) E d o a (d o b )T = E (R(HBH ) T + R) 1 d o b (do b )T = R(HBH T + R) 1 E ((ε ) o + Hε b )(ε o + Hε b ) T 24 of 30

45 Desroziers et al. (2005) statistical method R estimate ( ) E d o a (d o b )T = E (R(HBH ) T + R) 1 d o b (do b )T = R(HBH T + R) 1 E ((ε ) o + Hε b )(ε o + Hε b ) T = R(HBH T + R) 1 (HBH T + R) 24 of 30

46 Desroziers et al. (2005) statistical method R estimate ( ) E d o a (d o b )T = E (R(HBH ) T + R) 1 d o b (do b )T = R(HBH T + R) 1 E ((ε ) o + Hε b )(ε o + Hε b ) T = R(HBH T + R) 1 (HBH T + R) = R 24 of 30

47 Example - application to IASI data - Stewart et al. (2013) UK Met Office global 4D-Var system and 139 channels of IASI data (clear sky, sea surface observations only), 2073 observations (after observation processing and QC) from 17 July Note: Non-symmetric structure Index are window channels sensitive to surface temperature and emissivity , , are sensitive to water vapour 25 of 30

48 Does it really work? To derive the diagnostic, we assumed that H was linear, and that the statistics used in the assimilation were the true statistics. One problem is that the diagnostic does not in general give in a symmetric matrix could symmetrize as 1 2 (R + RT ). How do we know how good our estimate is? 26 of 30

49 Observation Errors Summary A proper accounting for observation error correlations is needed, especially for high resolution forecasting Assimilation output diagnostic techniques could be used to diagnose the correlations The Desroziers et al. (2005) technique relies on unrealistic assumptions, but some simple experiments show that the diagnostic may nevertheless give reasonable results The resulting statistics are noisy - and finding R 1 might be problematic 27 of 30

50 Workshop on Correlated Observation Errors Workshop on correlated errors planned for April/May If interested in attending, please let myself or Nancy Nichols know. 28 of 30

51 R. N. Bannister. A review of forecast error covariance statistics in atmospheric variational data assimilation. i: Characteristics and measurements of forecast error covariances. Quarterly Journal of the Royal Meteorological Society, 134(637): , 2008a. R. N. Bannister. A review of forecast error covariance statistics in atmospheric variational data assimilation. ii: Modelling the forecast error covariance statistics. Quarterly Journal of the Royal Meteorological Society, 134(637): , 2008b. N. Bormann, S. Saarinen, G. Kelly, and J.-N. Thépaut. The spatial structure of observation errors in Atmospheric Motion Vectors for geostationary satellite data. Monthly Weather Review, 131: , R. Daley. Estimating observation error statistics for atmospheric data assimilation. Ann. Geophys., 11: , G. Desroziers, L. Berre, B. Chapnik, and P. Poli. Diagnosis of observation, background and analysis-error statistics in observation space. Quarterly Journal of the Royal Meteorological Society, 131(613): , S. A. Haben, A. S. Lawless, and N. K. Nichols. Conditioning of incremental variational data assimilation, with application to the met office system. Tellus A, 63(4): , Z.-Q. Liu and F. Rabier. The potential of high-density observations for numerical weather prediction: A study with simulated observations. Q.J.R.Meteorol.Soc., 129: , L. M. Stewart, S. L. Dance, N. K. Nichols, J. R. Eyre, and J. Cameron. Estimating interchannel observation error correlations for iasi radiance data in the met office system. Quarterly Journal of the Royal Meteorological Society, doi: /qj L.M. Stewart, S.L. Dance, and N.K. Nichols. Correlated observation errors in data assimilation. Int.J.Numer.Meth.Fluids, 56: , J. A. Waller, S. L. Dance, A. S. Lawless, N. K. Nichols, and J. R. Eyre. Representativity error for temperature and humidity using the met office high-resolution model. Quarterly Journal of the Royal Meteorological Society, doi: /qj of 30