Performance of ensemble Kalman filters with small ensembles

Transcription

1 Performance of ensemble Kalman filters with small ensembles Xin T Tong Joint work with Andrew J. Majda National University of Singapore Sunday 28 th May, 2017 X.Tong EnKF performance 1 / 31

2 Content Ensemble Kalman filter (EnKF) High dimensional challenges for EnKF. Low rank scenario: A low effective dimension p, EnKF reaches its proclaimed performance, ensemble size K > Cp for a constant C. Sparse/Localized scenario: EnKF with domain localization, with a stable localized structure reaches its proclaimed performance, if the ensemble size K > C L log d for a constant C L. X.Tong EnKF performance 2 / 31

3 Filtering Signal-observation system with random coefficients Signal: X n+1 = A n X n + B n + ξ n+1, ξ n+1 N (0, Σ n ) Observation: Y n+1 = HX n+1 + ζ n+1, ζ n+1 N (0, σ 2 oi q ) Goal: estimate X n based on Y 1,..., Y n A n, B n, Σ m, σ o, H sequence of given matrices and vectors. A n can be unstable sometimes. X.Tong EnKF performance 3 / 31

4 Weather forecast Signal: X n+1 = A n X n + B n + ξ n+1, Observation: Y n+1 = HX n+1 + ζ n+1. Weather forecast: Signal: atmosphere and ocean, follows a PDE. Obs: weather station, satellite, sensors... Main challenge: high dimension, d X.Tong EnKF performance 4 / 31

5 Kalman filter Use Gaussian: X n Y1...n N (m n, R n ) Forecast step: ˆm n+1 = A n m n + B n, R n+1 = A n R n A T n + Σ n. Assimilation step: apply the Kalman update rule m n+1 = ˆm n+1 + G( R n+1 )(Y n+1 H ˆm n+1 ), R n+1 = K( R n+1 ) G(C) = CH T (σ 2 oi q + HCH T ) 1, Complexity: O(d 3 ). K(C) = C G(C)HC Posterior at t = n Prior+Obs at t = n + 1 forecast assimilate Posterior at t = n + 1 X.Tong EnKF performance 5 / 31

6 Sampling+Gaussian Monte Carlo: use samples to represent a distribution: X (1),..., X (K) p, 1 K K δ X (k) p. k=1 Ensemble {X (k) n } K k=1 to represent N (X n, C n ) X n = X (k) n K, C n = 1 K 1 k (X (k) n X n ) (X (k) n X n ). X.Tong EnKF performance 6 / 31

7 Ensemble Kalman filter Forecast step X (k) n+1 = A nx n (k) + B n + ξ (k) n+1, Ĉ n+1 = Ŝn+1ŜT n+1 K 1. Assimilation step X (k) (k) n+1 = X n+1 + G(Ĉn+1)(Y (k) n+1 H X n+1 + ζ(k) n+1 ). Gain matrix: G(C) = CH T (σ 2 oi q + HCH T ) 1. Complexity O(K 2 d). Posterior at t = n Prior+Obs at t = n + 1 Posterior at t = n + 1 X (k) n+1 X (k) forecast: M.C. assimilate n+1 X (k) n X.Tong EnKF performance 7 / 31

8 Application and theory of EnKF Pros and Cons K = 100 ensembles can forecast d = 10 6 dimensional systems. Fast weather forecast and oil reservoir management K 2 d d 3. No systematic theoretical understanding: why does it work? Literature Focused on showing ensemble version (X n, C n ) (m n, R n ) Require K (Mandel, Cobb, Beezley 11) Fixed d sufficiently large K, A < 1 (Del Moral, Tugaut 16) Fixed K, well definedness E X (k) n 2 < (Law, Kelly, Stuart, 14) Fixed K, boundedness sup n E X (k) n 2 < (Tong, Majda, Kelly 15) Missing: performance analysis with K d. X.Tong EnKF performance 8 / 31

9 Application and theory of EnKF Pros and Cons K = 100 ensembles can forecast d = 10 6 dimensional systems. Fast weather forecast and oil reservoir management K 2 d d 3. No systematic theoretical understanding: why does it work? Literature Focused on showing ensemble version (X n, C n ) (m n, R n ) Require K (Mandel, Cobb, Beezley 11) Fixed d sufficiently large K, A < 1 (Del Moral, Tugaut 16) Fixed K, well definedness E X (k) n 2 < (Law, Kelly, Stuart, 14) Fixed K, boundedness sup n E X (k) n 2 < (Tong, Majda, Kelly 15) Missing: performance analysis with K d. X.Tong EnKF performance 8 / 31

10 Application and theory of EnKF Pros and Cons K = 100 ensembles can forecast d = 10 6 dimensional systems. Fast weather forecast and oil reservoir management K 2 d d 3. No systematic theoretical understanding: why does it work? Literature Focused on showing ensemble version (X n, C n ) (m n, R n ) Require K (Mandel, Cobb, Beezley 11) Fixed d sufficiently large K, A < 1 (Del Moral, Tugaut 16) Fixed K, well definedness E X (k) n 2 < (Law, Kelly, Stuart, 14) Fixed K, boundedness sup n E X (k) n 2 < (Tong, Majda, Kelly 15) Missing: performance analysis with K d. X.Tong EnKF performance 8 / 31

11 Challenges from high dimension Ensemble size K to represent uncertainty of dimension d: Spurious correlation in high dimension. Suppose X n (k) N (0, I d ) i.i.d, by Bai-Yin s law C n I d d/k with large probability K k=1 (X(k) n Rank deficiency: C n = [ Cn 0 Has rank(c n ) K 1, see as 0 0 Xn)(X(k) n Xn)T K 1 ] } K-1 } d-k+1 Instability of the dynamics: Ĉ n+1 = A n C n A T n + Σ n What if span(c n ) does not cover expanding directions? X.Tong EnKF performance 9 / 31

12 Heuristic answers For high dimensional numerical problems, Low rank structure Sparse structure can be exploited for efficient computation. In EnKF, efficient performance with Low effective dimension. Localized covariance structure. X.Tong EnKF performance 10 / 31

13 Low Effective Dimension X.Tong EnKF performance 11 / 31

14 Low effective dimension Assume there is a effective dimension p < K d Most uncertainty lies in p directions, others below a threshold ρ, Cn 0 Cn O(ρ) } K O(ρ) O(ρ) } d-k+1 How are the filter ensemble attracted to this subspace. Lorenz system propagation. Video: MIT Aero Astro X.Tong EnKF performance 12 / 31

15 EnKF variant for enhanced fidelity Techniques to enhance fidelity Spurious correlation in high dimension. Projecting to p principal directions of K(Ĉn+1) The leftover direction: assume error strength is ρ Rank deficiency: additive inflation ρi d C ρ n = ρi d + K k=1 (X(k) n X n )(X n (k) K 1 X n ) T = [ ] Cn + ρi K ρi d K+1 The under represented direction: assume error strength is ρ. Instability of the dynamics. Increase noise strength X n+1 k = A n+1 X n (k) + ξ (k) n+1, ξ(k) n+1 Σ+ n Σ n Σ + n = [ρa n A T n + Σ n ρ/ri d ], Σ + n indicates the system instability. Also multiplicate covariance with a ratio: Ĉ n rĉn. X.Tong EnKF performance 13 / 31

16 EnKF variant for enhanced fidelity Techniques to enhance fidelity Spurious correlation in high dimension. Projecting to p principal directions of K(Ĉn+1) The leftover direction: assume error strength is ρ Rank deficiency: additive inflation ρi d C ρ n = ρi d + K k=1 (X(k) n X n )(X n (k) K 1 X n ) T = [ ] Cn + ρi K ρi d K+1 The under represented direction: assume error strength is ρ. Instability of the dynamics. Increase noise strength X n+1 k = A n+1 X n (k) + ξ (k) n+1, ξ(k) n+1 Σ+ n Σ n Σ + n = [ρa n A T n + Σ n ρ/ri d ], Σ + n indicates the system instability. Also multiplicate covariance with a ratio: Ĉ n rĉn. X.Tong EnKF performance 13 / 31

17 EnKF variant for enhanced fidelity Techniques to enhance fidelity Spurious correlation in high dimension. Projecting to p principal directions of K(Ĉn+1) The leftover direction: assume error strength is ρ Rank deficiency: additive inflation ρi d C ρ n = ρi d + K k=1 (X(k) n X n )(X n (k) K 1 X n ) T = [ ] Cn + ρi K ρi d K+1 The under represented direction: assume error strength is ρ. Instability of the dynamics. Increase noise strength X n+1 k = A n+1 X n (k) + ξ (k) n+1, ξ(k) n+1 Σ+ n Σ n Σ + n = [ρa n A T n + Σ n ρ/ri d ], Σ + n indicates the system instability. Also multiplicate covariance with a ratio: Ĉ n rĉn. X.Tong EnKF performance 13 / 31

18 Main result: low effective dimension Theorem (Majda, T. 16) Suppose the system has a low effective filter dimension p, there is a variant of EnKF with a constant C, such that the EnKF reaches its proclaimed performance, if the sample size K > cp. Remained questions: What is the effective dimension: Intuitively, it is the number C n eigenvalues above ρ. C n depends on random sampling realization. Practically, can be checked while EnKF is running. Theoretical a-priori conditions for small effective dimension. How to describe EnKF performance. X.Tong EnKF performance 14 / 31

19 Main result: low effective dimension Theorem (Majda, T. 16) Suppose the system has a low effective filter dimension p, there is a variant of EnKF with a constant C, such that the EnKF reaches its proclaimed performance, if the sample size K > cp. Remained questions: What is the effective dimension: Intuitively, it is the number C n eigenvalues above ρ. C n depends on random sampling realization. Practically, can be checked while EnKF is running. Theoretical a-priori conditions for small effective dimension. How to describe EnKF performance. X.Tong EnKF performance 14 / 31

20 Main result: low effective dimension Theorem (Majda, T. 16) Suppose the system has a low effective filter dimension p, there is a variant of EnKF with a constant C, such that the EnKF reaches its proclaimed performance, if the sample size K > cp. Remained questions: What is the effective dimension: Intuitively, it is the number C n eigenvalues above ρ. C n depends on random sampling realization. Practically, can be checked while EnKF is running. Theoretical a-priori conditions for small effective dimension. How to describe EnKF performance. X.Tong EnKF performance 14 / 31

21 Main result: low effective dimension Theorem (Majda, T. 16) Suppose the system has a low effective filter dimension p, there is a variant of EnKF with a constant C, such that the EnKF reaches its proclaimed performance, if the sample size K > cp. Remained questions: What is the effective dimension: Intuitively, it is the number C n eigenvalues above ρ. C n depends on random sampling realization. Practically, can be checked while EnKF is running. Theoretical a-priori conditions for small effective dimension. How to describe EnKF performance. X.Tong EnKF performance 14 / 31

22 Kalman filter reference Consider another signal-observation system: Signal : X n+1 = ra n X n + B n + ξ n+1, ξ n+1 N (0, Σ n) Observation : Y n+1 = HX n+1 + ζ n+1, ζ n+1 N (0, I q ) An inflated version with Σ n = r 2 Σ + n + r 2 ρi d Signal: X n+1 = A n X n + B n + ξ n+1, ξ n+1 N (0, Σ n ) Observation: Y n+1 = HX n+1 + ξ n+1, ζ n+1 N (0, I q ) (X n, Y n) has its own Kalman filter (m n, R n), with covariance update R n+1 = K(r 2 A n R na T n + Σ n) X.Tong EnKF performance 15 / 31

23 Kalman filter reference Consider another signal-observation system: Signal : X n+1 = ra n X n + B n + ξ n+1, ξ n+1 N (0, Σ n) Observation : Y n+1 = HX n+1 + ζ n+1, ζ n+1 N (0, I q ) An inflated version with Σ n = r 2 Σ + n + r 2 ρi d Signal: X n+1 = A n X n + B n + ξ n+1, ξ n+1 N (0, Σ n ) Observation: Y n+1 = HX n+1 + ξ n+1, ζ n+1 N (0, I q ) (X n, Y n) has its own Kalman filter (m n, R n), with covariance update R n+1 = K(r 2 A n R na T n + Σ n) X.Tong EnKF performance 15 / 31

24 Kalman filter reference Consider another signal-observation system: Signal : X n+1 = ra n X n + B n + ξ n+1, ξ n+1 N (0, Σ n) Observation : Y n+1 = HX n+1 + ζ n+1, ζ n+1 N (0, I q ) An inflated version with Σ n = r 2 Σ + n + r 2 ρi d Signal: X n+1 = A n X n + B n + ξ n+1, ξ n+1 N (0, Σ n ) Observation: Y n+1 = HX n+1 + ξ n+1, ζ n+1 N (0, I q ) (X n, Y n) has its own Kalman filter (m n, R n), with covariance update R n+1 = K(r 2 A n R na T n + Σ n) X.Tong EnKF performance 15 / 31

25 Low effective dimension (X n, Y n) has its own Kalman filter (m n, R n), with covariance update R n+1 = K(r 2 A n R na T n + Σ n) Such update often has a stationary solution R n (Bougerol 93) C n R n with large probability Assumption (Low effective dimension) The original system has an effective dimension p if A covariance sequence R n has at most p eigenvalues above ρ Rank(Σ + n ) p A n A T n + Σ n /ρ has at p eigenvalues above 1/r Assumption (Uniform observability) A 1 n, A n, Σ n, R 1 n, R n are all bounded in operator norm. The m step observability Gramian m k=1 AT k,1 HT k H ka k,1 c m I d X.Tong EnKF performance 16 / 31

26 Proclaimed performance Proclaimed/estimated performance EnKF estimates X n by X n = 1 (k) K X n. Error e n = X n X n. Covariance : Ee n e T n = Ee n e n. EnKF estimates its performance by ensemble covariance C ρ n. Can it captures the error covariance? EC ρ n Ee n e n Quantification via Mahalanobis distance v 2 C = vt [C] 1 v: e n C ρ n = e T n [C ρ n] 1 e n. Idealistic: e n N (0, C ρ n), then e n C ρ n d. X.Tong EnKF performance 17 / 31

27 Proclaimed performance Proclaimed/estimated performance EnKF estimates X n by X n = 1 (k) K X n. Error e n = X n X n. Covariance : Ee n e T n = Ee n e n. EnKF estimates its performance by ensemble covariance C ρ n. Can it captures the error covariance? EC ρ n Ee n e n Quantification via Mahalanobis distance v 2 C = vt [C] 1 v: e n C ρ n = e T n [C ρ n] 1 e n. Idealistic: e n N (0, C ρ n), then e n C ρ n d. X.Tong EnKF performance 17 / 31

28 Main theorem Theorem (Majda, T. 16) Suppose the signal observation system is uniformly observable with m steps, and has a effective dimension p. Then for any c, there are C, F, D F, M n, so that if K > Cp E e n C ρ n r n 6 EF (C0 ) e 0 2 C 0 + 2m + M n d With the constants bounded by F (C) D F exp(d F log 3 C ), lim sup M n n 1 + c 1 r m 6. X.Tong EnKF performance 18 / 31

29 Main theorem Theorem (Majda, T. 16) Suppose the signal observation system is uniformly observable with m steps, and has a effective dimension p. Then for any c, there are C, F, D F, M n, so that if K > Cp E e n C ρ n r n 6 EF (C0 ) e 0 2 C 0 + 2m + M n d With the constants bounded by F (C) D F exp(d F log 3 C ), lim sup M n n 1 + c 1 r m 6. X.Tong EnKF performance 18 / 31

30 Corollary: accuracy System with frequent accurate observations: Signal: X n+1 = A n X n + B n + ɛξ n+1, ξ n+1 N (0, Σ n ) Observation: Y n+1 = HX n+1 + ɛζ n+1, ζ n+1 N (0, σoi 2 q ) Corollary: ensemble covariance C n ɛ, also E e n ɛr n 6 DR + ρef (ɛ 1 C 0 ) e m+ ɛ(d C ρ R + ρ)dm n. 0 X.Tong EnKF performance 19 / 31

31 Localized covariance X.Tong EnKF performance 20 / 31

32 Local interaction High dimension often comes from dense grids. Interaction often is local: PDE discritization. Example: Lorenz 96 model ẋ i (t) = (x i+1 x i 2 )x i 1 x i dt + F, i = 1,, d Information travels along interaction, and is dissipated. In linear systems: the matrices have short bandwidth. X n+1 = A n X n. X.Tong EnKF performance 21 / 31

33 Sparsity: local covariance Correlation of Lorenz 96 X.Tong EnKF performance 22 / 31 Correlation depends on information propagation. Correlation decays quickly with the distance. Covariance is localized with a structure Φ, e.g. Φ(x) = ρ x [Ĉn] i,j Φ( i j ) Φ(x) [0, 1] is decreasing. Distance can be general

34 Covariance Localization Spurious correlation may exist for far away terms. Localization: simply ignore far away correlations. Implementation: Schur product with a mask [Ĉn D L ] i,j = [Ĉn] i,j [D L ] i,j Use Ĉn D L to describe uncertainty [D L ] i,j = φ( i j ), with a radius L. Gaspari-Cohn matrix: φ(x) = exp( 4x 2 /L 2 )1 i j L. Cutoff/Branding matrix: φ(x) = 1 i j L. Also resolves rank deficiency, e.g = X.Tong EnKF performance 23 / 31

35 Domain localization Two types LEnKF: Domain localization and covariance tempering. Domain localization with radius l: Assume H is a partial observation matrix Use information in I i = {j : i j l} to update component i C i = P Ii CP Ii, X (k) n = G i (C) = C i H T (σ 2 oi q + HC i H T ) 1 G L (C) = e i e T i G i (C) X (k) n + G L (Ĉn)(Y n H X (k) n + ζ (k) n ). X.Tong EnKF performance 24 / 31

36 Advantage with localization Intuitively, ignoring the long distance covariance terms, reduces the sampling difficulty, and necessary sampling size. Theorem (Bickel, Levina 08) If X (1),..., X (K) N (0, Σ), denote C = 1 K K k=1 X(k) X (k). D L 1 = max i j D L i,j.there is a constant c, and for any t > 0 P( C D L Σ D L > D L 1 t) 8 exp(2 log d ck min{t, t 2 }) This indicates that K D L 2 1 log d is the necessary sample size. D L is independent of d, e.g, the cut-off/branding matrix, [D L cut] i,j = 1 i j L, D L cut 1 2L. L = 4l in our application. X.Tong EnKF performance 25 / 31

37 Advantage with localization Intuitively, ignoring the long distance covariance terms, reduces the sampling difficulty, and necessary sampling size. Theorem (Bickel, Levina 08) If X (1),..., X (K) N (0, Σ), denote C = 1 K K k=1 X(k) X (k). D L 1 = max i j D L i,j.there is a constant c, and for any t > 0 P( C D L Σ D L > D L 1 t) 8 exp(2 log d ck min{t, t 2 }) This indicates that K D L 2 1 log d is the necessary sample size. D L is independent of d, e.g, the cut-off/branding matrix, [D L cut] i,j = 1 i j L, D L cut 1 2L. L = 4l in our application. X.Tong EnKF performance 25 / 31

38 Advantage with localization Intuitively, ignoring the long distance covariance terms, reduces the sampling difficulty, and necessary sampling size. Theorem (Bickel, Levina 08) If X (1),..., X (K) N (0, Σ), denote C = 1 K K k=1 X(k) X (k). D L 1 = max i j D L i,j.there is a constant c, and for any t > 0 P( C D L Σ D L > D L 1 t) 8 exp(2 log d ck min{t, t 2 }) This indicates that K D L 2 1 log d is the necessary sample size. D L is independent of d, e.g, the cut-off/branding matrix, [D L cut] i,j = 1 i j L, D L cut 1 2L. L = 4l in our application. X.Tong EnKF performance 25 / 31

39 Main result: localized EnKF Theorem (T. 17) Suppose the system coefficients have bandwidth l, and the LEnKF ensemble covariance admits a stable localized structure, then for any δ > 0, LEnKf reaches its proclaimed performance with high probability 1 O(δ): 1 1 T T t=1 P(E S ê n ê n (1 + δ)(ĉn D 4l cut + ρi d )) 1 T D 0 + D 1 δ, if the sample size K > D l,δ log d. E S conditioned on the information of the sampling noise realization. Stable localized structure: with local structure function Φ, e.g. Φ(x) = λ x, [Ĉn] i,j M n Φ( i j ), T EM n T M. n=1 X.Tong EnKF performance 26 / 31

40 Main result: localized EnKF Theorem (T. 17) Suppose the system coefficients have bandwidth l, and the LEnKF ensemble covariance admits a stable localized structure, then for any δ > 0, LEnKf reaches its proclaimed performance with high probability 1 O(δ): 1 1 T T t=1 P(E S ê n ê n (1 + δ)(ĉn D 4l cut + ρi d )) 1 T D 0 + D 1 δ, if the sample size K > D l,δ log d. E S conditioned on the information of the sampling noise realization. Stable localized structure: with local structure function Φ, e.g. Φ(x) = λ x, [Ĉn] i,j M n Φ( i j ), T EM n T M. n=1 X.Tong EnKF performance 26 / 31

41 Main result: localized EnKF Theorem (T. 17) Suppose the system coefficients have bandwidth l, and the LEnKF ensemble covariance admits a stable localized structure, then for any δ > 0, LEnKf reaches its proclaimed performance with high probability 1 O(δ): 1 1 T T t=1 P(E S ê n ê n (1 + δ)(ĉn D 4l cut + ρi d )) 1 T D 0 + D 1 δ, if the sample size K > D l,δ log d. E S conditioned on the information of the sampling noise realization. Stable localized structure: with local structure function Φ, e.g. Φ(x) = λ x, [Ĉn] i,j M n Φ( i j ), T EM n T M. n=1 X.Tong EnKF performance 26 / 31

42 LEnKF inconsistency An intrinsic inconsistency in LEnKF. Use Ĉn D L as covariance. Target covariance by Bayes formula (I G L (Ĉn)H)[Ĉn D L ](I G L (Ĉn)H) T + σog 2 L (G L ) T. LEnKF implementation X (k) n = X (k) n + G L (Ĉn)(Y n H X (k) n + ζ (k) n ) Average ensemble covariance C n D L = [(I G L (Ĉn)H)Ĉn(I G L (Ĉn)H) T +σ 2 og L (G L ) T ] D L. Difference: commuting the localization and Kalman update. Previously investigated numerically by Nerger 2015, the inconsistency can lead to error growth. X.Tong EnKF performance 27 / 31

43 When is the inconsistency small? localization is applied, covariance is assumed localized. Given localized structure Φ, find M n so that [Ĉn] i,j M n Φ( i j ). Interestingly, when D L is D cut 4l, the Localization inconsistency CM n Φ(2l). If 2l is large, Φ(x) = λ x, this difference can be controlled. Localized covariance leads to small localization inconsistency. Therefore, we need M n to be a stable sequence, T EM n T M. n=1 X.Tong EnKF performance 28 / 31

44 When is the inconsistency small? localization is applied, covariance is assumed localized. Given localized structure Φ, find M n so that [Ĉn] i,j M n Φ( i j ). Interestingly, when D L is D cut 4l, the Localization inconsistency CM n Φ(2l). If 2l is large, Φ(x) = λ x, this difference can be controlled. Localized covariance leads to small localization inconsistency. Therefore, we need M n to be a stable sequence, T EM n T M. n=1 X.Tong EnKF performance 28 / 31

45 When is covariance localized? Practical perspective Simply assumed. Numerically checked. Theoretical perspective: does covariance localize for any stochastic system? Linear system: covariance can be computed. Nonlinear: difficult, e.g. Lorenz 96. LEnKF: difficult since assimilation is nonlinear. Under strong conditions: Weak local interaction, strong dissipation. Sparse observation for simplicity. Also scales with the noise strength. t=0 t=1 t=2 t= X.Tong EnKF performance 29 / 31

46 When is covariance localized? Practical perspective Simply assumed. Numerically checked. Theoretical perspective: does covariance localize for any stochastic system? Linear system: covariance can be computed. Nonlinear: difficult, e.g. Lorenz 96. LEnKF: difficult since assimilation is nonlinear. Under strong conditions: Weak local interaction, strong dissipation. Sparse observation for simplicity. Also scales with the noise strength. t=0 t=1 t=2 t= X.Tong EnKF performance 29 / 31

47 Summary EnKF perform well, with a small sample size, if There is a low effective dimension. There is a localized a covariance structure. X.Tong EnKF performance 30 / 31

48 Reference Performance of Ensemble Kalman filters in large dimensions. to appear on CPAM arxiv: Performance analysis of local ensemble Kalman filter. to appear on arxiv Rigorous accuracy and robustness analysis for two-scale reduced random Kalman filters in high dimensions. arxiv: Links and slides can be found at mattxin. Thank you! X.Tong EnKF performance 31 / 31