A data-driven method for improving the correlation estimation in serial ensemble Kalman filter

Transcription

1 A data-driven method for improving the correlation estimation in serial ensemble Kalman filter Michèle De La Chevrotière, 1 John Harlim 2 1 Department of Mathematics, Penn State University, 2 Department of Mathematics and Department of Meteorology, Penn State University The 7th EnKF Data Assimilation Workshop May 16

2 Outline 1. Correlation statistics in EnKF 2. Learning algorithm for improving the correlation estimation 3. Serial EnKF with transformed correlations 4. Numerical experiments on the Lorenz-96 model. Conclusion Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 2 / 18

3 Correlation statistics in EnKF Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 3 / 18

4 EnKF in linear and Gaussian settings In the linear and Gaussian, and under certain assumptions (controllability, observability, uncorrelated noises), the Kalman posterior error covariance matrix P a will converge to the fixed point Ĉ of Ĉ = (I KH)(F ĈF + Q), where K = ĈH (HĈH + R) 1 is the Kalman gain, F is the deterministic linear forecast model with noise covariance matrix Q, H is the linear observation operator, and R is the observation error covariance matrix. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 4 / 18

5 EnKF in linear and Gaussian settings In the linear and Gaussian, and under certain assumptions (controllability, observability, uncorrelated noises), the Kalman posterior error covariance matrix P a will converge to the fixed point Ĉ of Ĉ = (I KH)(F ĈF + Q), where K = ĈH (HĈH + R) 1 is the Kalman gain, F is the deterministic linear forecast model with noise covariance matrix Q, H is the linear observation operator, and R is the observation error covariance matrix. EnKF construct an ensemble of analysis solutions {x a,k } K k=1 so that the ensemble mean and covariance match the Kalman solution. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 4 / 18

6 EnKF in linear and Gaussian settings In the linear and Gaussian, and under certain assumptions (controllability, observability, uncorrelated noises), the Kalman posterior error covariance matrix P a will converge to the fixed point Ĉ of Ĉ = (I KH)(F ĈF + Q), where K = ĈH (HĈH + R) 1 is the Kalman gain, F is the deterministic linear forecast model with noise covariance matrix Q, H is the linear observation operator, and R is the observation error covariance matrix. EnKF construct an ensemble of analysis solutions {x a,k } K k=1 so that the ensemble mean and covariance match the Kalman solution. Here x a R N, and K is the ensemble size. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 4 / 18

7 EnKF in linear and Gaussian settings In the linear and Gaussian, and under certain assumptions (controllability, observability, uncorrelated noises), the Kalman posterior error covariance matrix P a will converge to the fixed point Ĉ of Ĉ = (I KH)(F ĈF + Q), where K = ĈH (HĈH + R) 1 is the Kalman gain, F is the deterministic linear forecast model with noise covariance matrix Q, H is the linear observation operator, and R is the observation error covariance matrix. EnKF construct an ensemble of analysis solutions {x a,k } K k=1 so that the ensemble mean and covariance match the Kalman solution. Here x a R N, and K is the ensemble size. The scheme approximates both prior and posterior covariances by the ensemble covariance matrix, e.g., P a 1 K 1 XXT, where X is the matrix R N K of analysis ensemble perturbations from the ensemble mean (the k th column of X is X a,k = x a,k x a,k ). Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 4 / 18

8 EnKF in linear and Gaussian settings In the linear and Gaussian, and under certain assumptions (controllability, observability, uncorrelated noises), the Kalman posterior error covariance matrix P a will converge to the fixed point Ĉ of Ĉ = (I KH)(F ĈF + Q), where K = ĈH (HĈH + R) 1 is the Kalman gain, F is the deterministic linear forecast model with noise covariance matrix Q, H is the linear observation operator, and R is the observation error covariance matrix. EnKF construct an ensemble of analysis solutions {x a,k } K k=1 so that the ensemble mean and covariance match the Kalman solution. Here x a R N, and K is the ensemble size. The scheme approximates both prior and posterior covariances by the ensemble covariance matrix, e.g., P a 1 K 1 XXT, where X is the matrix R N K of analysis ensemble perturbations from the ensemble mean (the k th column of X is X a,k = x a,k x a,k ). In the linear and Gaussian setting, the EnKF solution is optimal with finite ensemble if x a,k t x a t N (0, Ĉ) in the limit as t. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 4 / 18

9 Motivation Nonlinear and non-gaussian case: estimating the correlation ρ = D 1/2 CD 1/2, D = diag(c), is much more difficult because ρ is time-dependent and does not equilibrate to an invariant ρeq. Furthermore, the EnKF sample correlation, rk, is such that r (t) lim rk (t) ρ(t), t. K Miche le De La Chevrotie re Data-driven method for correlation estimation in serial EnKF / 18

10 Motivation Nonlinear and non-gaussian case: estimating the correlation ρ = D 1/2 CD 1/2, D = diag(c), is much more difficult because ρ is time-dependent and does not equilibrate to an invariant ρeq. Furthermore, the EnKF sample correlation, rk, is such that r (t) lim rk (t) ρ(t), t. K Since estimating the true correlation ρ is very difficult, then instead, we try to estimate r. Miche le De La Chevrotie re Data-driven method for correlation estimation in serial EnKF / 18

11 Motivation Nonlinear and non-gaussian case: estimating the correlation ρ = D 1/2 CD 1/2, D = diag(c), is much more difficult because ρ is time-dependent and does not equilibrate to an invariant ρeq. Furthermore, the EnKF sample correlation, rk, is such that r (t) lim rk (t) ρ(t), t. K Since estimating the true correlation ρ is very difficult, then instead, we try to estimate r. How can we transform rk into r? Miche le De La Chevrotie re Data-driven method for correlation estimation in serial EnKF / 18

12 Motivation Nonlinear and non-gaussian case: estimating the correlation ρ = D 1/2 CD 1/2, D = diag(c), is much more difficult because ρ is time-dependent and does not equilibrate to an invariant ρeq. Furthermore, the EnKF sample correlation, rk, is such that r (t) lim rk (t) ρ(t), t. K Since estimating the true correlation ρ is very difficult, then instead, we try to estimate r. How can we transform rk into r? The idea: I. Find a linear map L that transforms the undersampled correlation matrix rk into a sample of improved correlation matrix: LT rk r, Miche le De La Chevrotie re L RN N M. Data-driven method for correlation estimation in serial EnKF / 18

13 Motivation Nonlinear and non-gaussian case: estimating the correlation ρ = D 1/2 CD 1/2, D = diag(c), is much more difficult because ρ is time-dependent and does not equilibrate to an invariant ρeq. Furthermore, the EnKF sample correlation, rk, is such that r (t) lim rk (t) ρ(t), t. K Since estimating the true correlation ρ is very difficult, then instead, we try to estimate r. How can we transform rk into r? The idea: I. Find a linear map L that transforms the undersampled correlation matrix rk into a sample of improved correlation matrix: L RN N M State Variables State Variables State Variables Miche le De La Chevrotie re State Variables State Variables State Variables r r ^ rl ^ r rl rl^ LTrKLTrK LTrK rk rk 1 Observation Variables 1 Observation Variables Observation Variables rk r L r EnKF L= LT rk r, State Variables State Variables State Variables Data-driven method for correlation estimation in serial EnKF / 18

14 Motivation Nonlinear and non-gaussian case: estimating the correlation ρ = D 1/2 CD 1/2, D = diag(c), is much more difficult because ρ is time-dependent and does not equilibrate to an invariant ρeq. Furthermore, the EnKF sample correlation, rk, is such that r (t) lim rk (t) ρ(t), t. K Since estimating the true correlation ρ is very difficult, then instead, we try to estimate r. How can we transform rk into r? The idea: I. Find a linear map L that transforms the undersampled correlation matrix rk into a sample of improved correlation matrix: L RN N M State Variables State Variables State Variables State Variables State Variables State Variables r r ^ rl ^ r rl rl^ LTrKLTrK LTrK rk rk 1 Observation Variables 1 Observation Variables Observation Variables rk r L r EnKF L= LT rk r, State Variables State Variables State Variables II. Implement the map L into the serial LLS EnKF of Anderson (03). Miche le De La Chevrotie re Data-driven method for correlation estimation in serial EnKF / 18

15 Learning algorithm for improving the correlation estimation Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 6 / 18

16 A minimization problem For nonlinear filtering problems, where h(x) R N R M, we consider the sample cross correlation: r K xy = D 1/2 X ( 1 K 1 XY )D 1/2 Y R N M. [The K th column of Y is Y k = h(x a,k ) y a, where y a = K k=1 h(x a,k )/K] Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 7 / 18

17 A minimization problem For nonlinear filtering problems, where h(x) R N R M, we consider the sample cross correlation: r K xy = D 1/2 X ( 1 K 1 XY )D 1/2 Y R N M. [The K th column of Y is Y k = h(x a,k ) y a, where y a = K k=1 h(x a,k )/K] Onward: suppress the subscript xy (write r K xy as r K ; ˆr = lim K r K ). Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 7 / 18

18 A minimization problem For nonlinear filtering problems, where h(x) R N R M, we consider the sample cross correlation: r K xy = D 1/2 X ( 1 K 1 XY )D 1/2 Y R N M. [The K th column of Y is Y k = h(x a,k ) y a, where y a = K k=1 h(x a,k )/K] Onward: suppress the subscript xy (write r K xy as r K ; ˆr = lim K r K ). We seek to find a linear map L such that for every pair (i, j), L(, i, j) takes the sample correlations r K (, y j) to the limiting correlation ˆr(x i, y j). Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 7 / 18

19 A minimization problem For nonlinear filtering problems, where h(x) R N R M, we consider the sample cross correlation: r K xy = D 1/2 X ( 1 K 1 XY )D 1/2 Y R N M. [The K th column of Y is Y k = h(x a,k ) y a, where y a = K k=1 h(x a,k )/K] Onward: suppress the subscript xy (write r K xy as r K ; ˆr = lim K r K ). We seek to find a linear map L such that for every pair (i, j), L(, i, j) takes the sample correlations r K (, y j) to the limiting correlation ˆr(x i, y j). Inspired by Anderson (12), we formulate: min (r K (, y j) L(, i, j) ˆr(x i, y j)) 2 p(r K ˆr)p(ˆr) dr K dˆr. L(,i,j) [ 1,1] [ 1,1] Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 7 / 18

20 A minimization problem For nonlinear filtering problems, where h(x) R N R M, we consider the sample cross correlation: r K xy = D 1/2 X ( 1 K 1 XY )D 1/2 Y R N M. [The K th column of Y is Y k = h(x a,k ) y a, where y a = K k=1 h(x a,k )/K] Onward: suppress the subscript xy (write r K xy as r K ; ˆr = lim K r K ). We seek to find a linear map L such that for every pair (i, j), L(, i, j) takes the sample correlations r K (, y j) to the limiting correlation ˆr(x i, y j). Inspired by Anderson (12), we formulate: min (r K (, y j) L(, i, j) ˆr(x i, y j)) 2 p(r K ˆr)p(ˆr) dr K dˆr. L(,i,j) [ 1,1] [ 1,1] In practice, can use any reliable correlation statistics r L to approximate the limiting correlation ˆr, so that r L p(ˆr). (e.g. EnKF with large ensemble L). Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 7 / 18

21 Learning the map L 1. Train offline: Generate an OSSE assimilation analysis ensemble {x a,k t with L 1 and T assimilation cycles. } T,L t,k=1 Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 8 / 18

22 Learning the map L 1. Train offline: Generate an OSSE assimilation analysis ensemble {x a,k t with L 1 and T assimilation cycles. 2. Compute {r L t } T t=1, so that r L t p(ˆr). } T,L t,k=1 Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 8 / 18

23 Learning the map L 1. Train offline: Generate an OSSE assimilation analysis ensemble {x a,k t with L 1 and T assimilation cycles. 2. Compute {r L t } T t=1, so that r L t p(ˆr). 3. Compute {r K t } T t=1, subsampling K out of L members, so that r K t p(r K ˆr = r L t ). } T,L t,k=1 Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 8 / 18

24 Learning the map L 1. Train offline: Generate an OSSE assimilation analysis ensemble {x a,k t with L 1 and T assimilation cycles. 2. Compute {r L t } T t=1, so that r L t p(ˆr). 3. Compute {r K t } T t=1, subsampling K out of L members, so that r K t p(r K ˆr = r L t ). 4. Monte Carlo approximation to the minimization problem: } T,L t,k=1 or 1 min L(,i,j) T T t=1 (r K t (, y j) L(, i, j) r L t (x i, y j) ) 2. row of A u entry of b LSP min u Au b 2 when u = (A A) 1 A b Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 8 / 18

25 Learning the map L 1. Train offline: Generate an OSSE assimilation analysis ensemble {x a,k t with L 1 and T assimilation cycles. 2. Compute {r L t } T t=1, so that r L t p(ˆr). 3. Compute {r K t } T t=1, subsampling K out of L members, so that r K t p(r K ˆr = r L t ). 4. Monte Carlo approximation to the minimization problem: } T,L t,k=1 or 1 min L(,i,j) T T t=1 (r K t (, y j) L(, i, j) r L t (x i, y j) ) 2. row of A u entry of b LSP min u Au b 2 when u = (A A) 1 A b Offline training, little computational overhead, almost no tuning (K, T, L). Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 8 / 18

26 Learning the map L 1. Train offline: Generate an OSSE assimilation analysis ensemble {x a,k t with L 1 and T assimilation cycles. 2. Compute {r L t } T t=1, so that r L t p(ˆr). 3. Compute {r K t } T t=1, subsampling K out of L members, so that r K t p(r K ˆr = r L t ). 4. Monte Carlo approximation to the minimization problem: } T,L t,k=1 or 1 min L(,i,j) T T t=1 (r K t (, y j) L(, i, j) r L t (x i, y j) ) 2. row of A u entry of b LSP min u Au b 2 when u = (A A) 1 A b Offline training, little computational overhead, almost no tuning (K, T, L). L is a scalar: the map found can be viewed as a type of data-driven localization function (L d ). Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 8 / 18

27 Serial EnKF with transformed correlations Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 9 / 18

28 Adapted serial LLS EnKF (Anderson, 13) LLS EnKF assimilates observations j = 1,..., M one at a time: 1 for j = 1 to M do /* Loop over observations */ 2 y a j = y b j + (Py b j y j + R jj ) 1 Py b j y j (yj o yb j) ; 3 y a,k j = y a Rjj j + R jj (y b,k +Py b j y j y b j j); 4 y k j = ya,k j y b,k j ; /* compute the obs update */ x a,k = x b,k + P xy b j y P k b y j y j j 6 end ; /* regresses the update onto x */ The straightforward modification is to replace in line : P b xy j = D 1/2 X r(, y j)d 1/2 Y D 1/2 X L(,, j) r(, y j )D 1/2 Y, where D X, D Y and r are estimated form the EnKF with ensemble size K. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF / 18

29 Numerical experiments on the Lorenz-96 model Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 11 / 18

30 Linear direct observations Maps L and L d found by regressing onto ETKF L00 N = 40 state variables, (top) and 40 (bottom) observations observation time steps 1 (left) and (right) 40 obs obs L., L., L., L., obs obs L., L., L., L., n = 1 n = (a) ensemble members. n = 1 n = (b) 40 ensemble members. Localized structure of the mappings; L d (dashed black) is an envelope for the mappings L. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 12 / 18

31 Linear direct observations En En nn = = 1 (c) L En n=en =1 nen n= En En En En= n En n =1 nen = En En En En obs obs En En En 1. EnEn En obs obs En En obs0. 1 En En En obs obs En En obs obs Filtering results with adapted LLS EnKF + covariance inflation nen =En n= (d) Ld. Time mean RMSE normalized by the RMSE of the ETKF using 00 ensemble members shown for the serial EnKF using L and Ld, plotted against inflation factor and K Miche le De La Chevrotie re Data-driven method for correlation estimation in serial EnKF 13 / 18

32 Linear direct observations Filtering results Best tuned inflation values observations, observation time step n = 1, comparison of 3 localization methods Time Mean of RMSE Time Mean of Spread ETKF L Ld GC Obs error Time mean of RMSE (left) and spread (right) as a function of ensemble size GC is more performant for K > but is extensively tuned! RMSE.0 2. Spread Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 14 /

33 Linear direct observations Regression using different products observations, observation time step n = 1, all use best tuned inflation values Time Mean of RMSE Time Mean of Spread ETKF L (against ETKF) Ld (against ETKF) L (against GC) Ld (against GC) GC 1 Ensemble Size 1 Ensemble Size Two choices of regression products: (1) ETKF with 00 ensemble members and (2) serial EnKF with GC localization with optimal half-width. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 1 / 18

34 Linear indirect observations Maps L and L d found by regressing onto ETKF L00 observations, ensemble members obs L., L., L., n = 1 n = n = Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 16 / 18

35 Time M Linear indirect observations Filtering results with localization and best tuned inflation values observations, observation time step n = Time M Time Mean of RMSE Time Mean of Spread ETKF L Ld GC Obs error Time Mean of Spread Ensemble Size Ensemble Size e Mean of Spread Time mean of RMSE (left) and spread (right) Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 17 / 18

36 Summary A data-driven method for improving the correlation estimation in serial ensemble Kalman filter is introduced. The method finds a linear map that transforms, at each assimilation cycle, the poorly estimated sample correlation into a sample of improved correlation. This map is obtained from an offline training procedure with almost no tuning as the solution of a linear regression problem that uses appropriate sample correlation statistics obtained from historical data assimilation product. In numerical tests with the Lorenz-96 model for ranges of cases of linear and nonlinear observation models, the proposed scheme improves the filter estimates, especially when ensemble size is small relative to the dimension of the state space. Reference: M. De La Chevrotière, J. Harlim, A data-driven method for improving the correlation estimation in serial ensemble Kalman filter (16), Mon. Wea. Rev., submitted. Michèle De La Chevrotière Data-driven method for correlation estimation in serial EnKF 18 / 18