Ensemble Data Assimila.on for Climate System Component Models

Transcription

1 Ensemble Data Assimila.on for Climate System Component Models Jeffrey Anderson Na.onal Center for Atmospheric Research In collabora.on with: Alicia Karspeck, Kevin Raeder, Tim Hoar, Nancy Collins IMA 11 March 213 1

2 What is Data Assimila.on? Observa.ons combined with a Model forecast + to produce an analysis (best possible es.mate). IMA 11 March 213 2

3 Example: Estimating the Temperature Outside An observation has a value ( *), IMA 11 March 213 3

4 Example: Estimating the Temperature Outside An observation has a value ( *), and an error distribution (red curve) that is associated with the instrument. IMA 11 March 213 4

5 Example: Estimating the Temperature Outside Thermometer outside measures 1C. Instrument builder says thermometer is unbiased with +/-.8C gaussian error. IMA 11 March 213 5

6 Example: Estimating the Temperature Outside Thermometer outside measures 1C. The red plot is observed. ( ) P T T o, probability of temperature given that T o was IMA 11 March 213 6

7 Example: Estimating the Temperature Outside We also have a prior estimate of temperature. The green curve is prior information. C ( ) P T C ; probability of temperature given all available IMA 11 March 213 7

8 Example: Estimating the Temperature Outside Prior information C can include: 1. Observations of things besides T; 2. Model forecast made using observations at earlier times; 3. A priori physical constraints ( T > C ); 4. Climatological constraints ( -3C < T < 4C ). IMA 11 March 213 8

9 Bayes Theorem: Combining the Prior Estimate and Observation ( ) = P T o T,C P T T o,c Posterior: Probability of T given observations and Prior. Also called update or analysis. ( )P T C Normalization ( ) Prior Likelihood: Probability that T o is observed if T is true value and given prior information C. IMA 11 March 213 9

10 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization IMA 11 March 213 1

11 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization IMA 11 March

12 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization IMA 11 March

13 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization IMA 11 March

14 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization IMA 11 March

15 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization Generally no analytic solution for Posterior. IMA 11 March

16 Combining the Prior Estimate and Observation P ( T T o, C ) = P ( T o T, C ) P ( T C ) normalization Gaussian Prior and Likelihood -> Gaussian Posterior IMA 11 March

17 For Gaussian prior and likelihood Prior Combining the Prior Estimate and Observation Likelihood Then, Posterior P( T C) = Normal( T p,σ p ) P( T o T,C) = Normal( T o,σ o ) P( T T o,c) = Normal( T u,σ u ) With σ u = ( σ 2 2 p + σ ) 1 o T u = σ 2 [ u σ 2 p T p + σ 2 o T ] o IMA 11 March

18 The One-Dimensional Kalman Filter 1. Suppose we have a linear forecast model L A. If temperature at time t 1 = T 1, then temperature at t 2 = t 1 + Δt is T 2 = L(T 1 ) B. Example: T 2 = T 1 + ΔtT 1 IMA 11 March

19 The One-Dimensional Kalman Filter 1. Suppose we have a linear forecast model L. A. If temperature at time t 1 = T 1, then temperature at t 2 = t 1 + Δt is T 2 = L(T 1 ). B. Example: T 2 = T 1 + ΔtT If posterior estimate at time t 1 is Normal(T u,1, σ u,1 ) then prior at t 2 is Normal(T p,2, σ p,2 ). T p,2 = T u,1,+ ΔtT u,1 σ p,2 = (Δt + 1) σ u,1 IMA 11 March

20 The One-Dimensional Kalman Filter 1. Suppose we have a linear forecast model L. A. If temperature at time t 1 = T 1, then temperature at t 2 = t 1 + Δt is T 2 = L(T 1 ). B. Example: T 2 = T 1 + ΔtT If posterior estimate at time t 1 is Normal(T u,1, σ u,1 ) then prior at t 2 is Normal(T p,2, σ p,2 ). 3. Given an observation at t 2 with distribution Normal(t o, σ o ) the likelihood is also Normal(t o, σ o ). IMA 11 March 213 2

21 The One-Dimensional Kalman Filter 1. Suppose we have a linear forecast model L. A. If temperature at time t 1 = T 1, then temperature at t 2 = t 1 + Δt is T 2 = L(T 1 ). B. Example: T 2 = T 1 + ΔtT If posterior estimate at time t 1 is Normal(T u,1, σ u,1 ) then prior at t 2 is Normal(T p,2, σ p,2 ). 3. Given an observation at t 2 with distribution Normal(t o, σ o ) the likelihood is also Normal(t o, σ o ). 4. The posterior at t 2 is Normal(T u,2, σ u,2 ) where T u,2 and σ u,2 come from page 17. IMA 11 March

22 A One-Dimensional Ensemble Kalman Filter Represent a prior pdf by a sample (ensemble) of N values: IMA 11 March

23 A One-Dimensional Ensemble Kalman Filter Represent a prior pdf by a sample (ensemble) of N values: Use sample mean T = T n N n =1 and sample standard deviation N σ T = ( T n T ) 2 N 1 to determine a corresponding continuous distribution N n =1 ( ) ( ) Normal T,σ T IMA 11 March

24 A One-Dimensional Ensemble Kalman Filter: Model Advance If posterior ensemble at time t 1 is T 1,n, n = 1,, N IMA 11 March

25 A One-Dimensional Ensemble Kalman Filter: Model Advance If posterior ensemble at time t 1 is T 1,n, n = 1,, N, advance each member to time t 2 with model, T 2,n = L(T 1, n ) n = 1,,N. IMA 11 March

26 A One-Dimensional Ensemble Kalman Filter: Model Advance Same as advancing continuous pdf at time t 1 IMA 11 March

27 A One-Dimensional Ensemble Kalman Filter: Model Advance Same as advancing continuous pdf at time t 1 to time t 2 with model L. IMA 11 March

28 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation IMA 11 March

29 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation Fit a Gaussian to the sample. IMA 11 March

30 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation Get the observation likelihood. IMA 11 March 213 3

31 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation Compute the continuous posterior PDF. IMA 11 March

32 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation Use a deterministic algorithm to adjust the ensemble. IMA 11 March

33 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation First, shift the ensemble to have the exact mean of the posterior. IMA 11 March

34 A One-Dimensional Ensemble Kalman Filter: Assimilating an Observation First, shift the ensemble to have the exact mean of the posterior. Second, linearly contract to have the exact variance of the posterior. Sample statistics are identical to Kalman filter. IMA 11 March

35 Initial Comments on (Ensemble) Kalman Filter Ø KF optimal for linear model, gaussian likelihood. Ø In KF, only mean and variance have meaning. Ø The deterministic Ensemble KF gives identical mean, variance. Ø The original Ensemble KF uses a Monte Carlo algorithm for the observation impact; has sampling error. Ø Ensemble allows computation of many other statistics. Ø What do they mean? Not entirely clear. Ø Example: Kurtosis. Completely constrained by initial ensemble. It is problem specific whether this is even defined! IMA 11 March

36 Multivariate Kalman Filter Product of d-dimensional normals is normal N ( µ 1,Σ 1 )N ( µ 2, Σ 2 ) = cn ( µ,σ) Covariance: Mean: Σ = Σ 1 1 ( 1 + Σ 2 ) 1 µ = Σ( Σ 1 1 µ 1 + Σ 1 2 µ 2 ) Weight: c = ( 2π ) d Σ 1 + Σ 2 { ( ) T ( Σ 1 + Σ 2 ) 1 ( µ 2 µ 1 ) } 1 exp 1 2 µ2 µ 1 Weight normalizes away. IMA 11 March

37 Multivariate Ensemble Kalman Filter Ø So far, we have an observation likelihood for single variable. Ø Suppose the model prior has additional variables. Ø KF equivalent to linear regression to update additional variables. Ø Need ensemble size > d to represent d-dimensional normal. IMA 11 March

38 Ensemble filters: Updating additional prior state variables Assume that all we know is prior joint distribution. One variable is observed. What should happen to the unobserved variable? IMA 11 March

39 Ensemble filters: Updating additional prior state variables Assume that all we know is prior joint distribution. One variable is observed. Compute increments for prior ensemble members of observed variable. IMA 11 March

40 Ensemble filters: Updating additional prior state variables Assume that all we know is prior joint distribution. One variable is observed. Using only increments guarantees that if observation had no impact on observed variable, unobserved variable is unchanged (highly desirable). IMA 11 March 213 4

41 Ensemble filters: Updating additional prior state variables Assume that all we know is prior joint distribution. How should the unobserved variable be impacted? First choice: least squares. Equivalent to linear regression. Same as assuming binormal prior. IMA 11 March

42 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. How should the unobserved variable be impacted? First choice: least squares. Begin by finding least squares fit. IMA 11 March

43 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Next, regress the observed variable increments onto increments for the unobserved variable. Equivalent to first finding image of increment in joint space. IMA 11 March

44 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Next, regress the observed variable increments onto increments for the unobserved variable. Equivalent to first finding image of increment in joint space. IMA 11 March

45 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Next, regress the observed variable increments onto increments for the unobserved variable. Equivalent to first finding image of increment in joint space. IMA 11 March

46 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Next, regress the observed variable increments onto increments for the unobserved variable. Equivalent to first finding image of increment in joint space. IMA 11 March

47 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Next, regress the observed variable increments onto increments for the unobserved variable. Equivalent to first finding image of increment in joint space. IMA 11 March

48 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Regression: Equivalent to first finding image of increment in joint space. Then projecting from joint space onto unobserved priors. IMA 11 March

49 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Regression: Equivalent to first finding image of increment in joint space. Then projecting from joint space onto unobserved priors. IMA 11 March

50 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Regression: Equivalent to first finding image of increment in joint space. Then projecting from joint space onto unobserved priors. IMA 11 March 213 5

51 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Regression: Equivalent to first finding image of increment in joint space. Then projecting from joint space onto unobserved priors. IMA 11 March

52 Ensemble filters: Updating additional prior state variables Have joint prior distribution of two variables. Regression: Equivalent to first finding image of increment in joint space. Then projecting from joint space onto unobserved priors. IMA 11 March

53 Comments on multidimensional (Ensemble) Kalman Filter Ø KF optimal for linear model, gaussian likelihood. Ø The deterministic Ensemble KF gives identical mean, covariance with sufficiently large ensemble size. Ø Basic ensemble filter fails for ensemble too small. Ø For nonlinear model, non-gaussian likelihood, all bets are off. Ø Both deterministic and stochastic Ensemble KFs become Monte Carlo algorithms. IMA 11 March

54 Ensemble Filter for Large Geophysical Models 1. Use model to advance ensemble (3 members here) to.me at which next observa.on becomes available. Ensemble state es.mate axer using previous observa.on (analysis) Ensemble state at.me of next observa.on (prior) IMA 11 March

55 Ensemble Filter for Large Geophysical Models 2. Get prior ensemble sample of observa.on, y = h(x), by applying forward operator h to each ensemble member. Theory: observa.ons from instruments with uncorrelated errors can be done sequen.ally. IMA 11 March

56 Ensemble Filter for Large Geophysical Models 3. Get observed value and observa.onal error distribu.on from observing system. IMA 11 March

57 Ensemble Filter for Large Geophysical Models 4. Find the increments for the prior observa.on ensemble (this is a scalar problem for uncorrelated observa.on errors). Note: Difference between various ensemble filters is primarily in observa.on increment calcula.on. IMA 11 March

58 Ensemble Filter for Large Geophysical Models 5. Use ensemble samples of y and each state variable to linearly regress observa.on increments onto state variable increments. Theory: impact of observa.on increments on each state variable can be handled independently! IMA 11 March

59 Ensemble Filter for Large Geophysical Models 6. When all ensemble members for each state variable are updated, there is a new analysis. Integrate to.me of next observa.on IMA 11 March

60 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March 213 6

61 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

62 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

63 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

64 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

65 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

66 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

67 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

68 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

69 Ensemble Filter for Lorenz Variable Model 4 state variables: X1, X2,..., X4. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. Acts something like synop.c weather around a la.tude band time State Variable IMA 11 March

70 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 12 truth ensemble State Variable IMA 11 March 213 7

71 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 14 truth ensemble State Variable IMA 11 March

72 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 16 truth ensemble State Variable IMA 11 March

73 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 18 truth ensemble State Variable IMA 11 March

74 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 11 truth ensemble State Variable IMA 11 March

75 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 112 truth ensemble State Variable IMA 11 March

76 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 114 truth ensemble State Variable IMA 11 March

77 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 116 truth ensemble State Variable IMA 11 March

78 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 118 truth ensemble State Variable IMA 11 March

79 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 12 truth ensemble State Variable IMA 11 March

80 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 122 truth ensemble State Variable IMA 11 March 213 8

81 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 124 truth ensemble State Variable IMA 11 March

82 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 126 truth ensemble State Variable IMA 11 March

83 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 128 truth ensemble State Variable IMA 11 March

84 Lorenz- 96 is sensi.ve to small perturba.ons Introduce 2 ensemble state es.mates. Each is slightly perturbed for each of the 4- variables at.me 1. Refer to unperturbed control integra.on as truth time 13 truth ensemble State Variable IMA 11 March

85 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 21 truth ensemble obs State Variable IMA 11 March

86 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 23 truth ensemble obs State Variable IMA 11 March

87 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 25 truth ensemble obs State Variable IMA 11 March

88 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 27 truth ensemble obs State Variable IMA 11 March

89 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 29 truth ensemble obs State Variable IMA 11 March

90 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 211 truth ensemble obs State Variable IMA 11 March 213 9

91 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 213 truth ensemble obs State Variable IMA 11 March

92 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 215 truth ensemble obs State Variable IMA 11 March

93 Assimilate observa.ons from 4 random loca.ons each step. Observa.ons generated by interpola.ng truth to sta.on loca.on. Simulate observa.onal error: Add random draw from N(, 16) to each. Start from climatological 2- member ensemble time 217 truth ensemble obs State Variable IMA 11 March

94 Assimilate observa.ons from 4 random loca.ons each step. This isn t working very well. Ensemble spread is reduced, but..., Ensemble is inconsistent with truth most places time 219 truth ensemble obs State Variable Confident and wrong. Confident and right. IMA 11 March

95 Some Error Sources in Ensemble Filters 2. Obs. operator error; Representa.veness 3. Observa.on error 4. Sampling Error; Gaussian Assump.on 5. Sampling Error; Assuming Linear Sta.s.cal Rela.on 1. Model error IMA 11 March

96 Observa.ons impact unrelated state variables through sampling error. Expected Sample Correlation Members.1 2 Members 4 Members 8 Members True Correlation Plot shows expected absolute value of sample correla.on vs. true correla.on. Unrelated obs. reduce spread, increase error. Anack with localiza.on. Reduce impact of observa.on on weakly correlated state variables. Let weight go to zero for many unrelated variables to save on compu.ng. IMA 11 March

97 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 21 truth ensemble obs No Localization 1 1 time 21 truth ensemble obs State Variable IMA 11 March

98 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 23 truth ensemble obs No Localization 1 1 time 23 truth ensemble obs State Variable IMA 11 March

99 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 25 truth ensemble obs No Localization 1 1 time 25 truth ensemble obs State Variable IMA 11 March

100 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 27 truth ensemble obs No Localization 1 1 time 27 truth ensemble obs State Variable IMA 11 March 213 1

101 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 29 truth ensemble obs No Localization 1 1 time 29 truth ensemble obs State Variable IMA 11 March

102 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 211 truth ensemble obs No Localization 1 1 time 211 truth ensemble obs State Variable IMA 11 March

103 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 213 truth ensemble obs No Localization 1 1 time 213 truth ensemble obs State Variable IMA 11 March

104 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 215 truth ensemble obs No Localization 1 1 time 215 truth ensemble obs State Variable IMA 11 March

105 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. 1 Localization from Hierarchical Filter 1 time 217 truth ensemble obs No Localization 1 1 time 217 truth ensemble obs State Variable IMA 11 March

106 Lorenz- 96 Assimila.on with localiza.on of observa.on impact. Localization from Hierarchical Filter This ensemble is more consistent with truth time 219 truth ensemble obs No Localization 1 time 219 truth ensemble obs State Variable IMA 11 March

107 Localiza.on computed by empirical offline computa.on. 1 Localization from Hierarchical Filter 1 1 time 219 truth ensemble obs Sample Observation Localizations State Variable IMA 11 March

108 Localiza.on in dry dynamical core / -4,. + 23!*&%&'() -! ,-./- 1-!"#$%&'() ,-./ !"#$%&'() Localization for V ob. on U state variables. Has statistically significant quadrupole structure in horizontal. Localization can have lots of structure in realistic models. IMA 11 March

109 Some Error Sources in Ensemble Filters 2. Obs. operator error; Representa.veness 3. Observa.on error 4. Sampling Error; Gaussian Assump.on 5. Sampling Error; Assuming Linear Sta.s.cal Rela.on 1. Model error IMA 11 March

110 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = 6. Note Axis! Time evolu.on for first state variable shown. Assimila.ng model quickly diverges from true model. IMA 11 March

111 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 31 truth ensemble obs State Variable IMA 11 March

112 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 36 truth ensemble obs State Variable IMA 11 March

113 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 311 truth ensemble obs State Variable IMA 11 March

114 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 316 truth ensemble obs State Variable IMA 11 March

115 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 321 truth ensemble obs State Variable IMA 11 March

116 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 326 truth ensemble obs State Variable IMA 11 March

117 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 331 truth ensemble obs State Variable IMA 11 March

118 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 336 truth ensemble obs State Variable IMA 11 March

119 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 341 truth ensemble obs State Variable IMA 11 March

120 Assimila.ng in the presence of simulated model error. dxi / dt = (Xi+1 - Xi- 2)Xi- 1 - Xi + F. For truth, use F = 8. In assimila.ng model, use F = time 346 truth ensemble obs State Variable This isn t working again! It will just keep geqng worse. IMA 11 March

121 How to deal with model error in assimila.on? Ø Just fix the model! (May not be such an easy thing to do) Ø Model the model error and include this in prior: If we knew the error accurately could just fix the model. Modeling or parameterizing model uncertainty in prior can help. This looks like a job for stochas.c modeling. IMA 11 March

122 Assimila.on fix for model error. Ø Use prior variance infla.on. Ø Simply increase prior ensemble variance of each state variable before compu.ng observa.on increments. Ø Adap.ve algorithms use observa.ons to guide this. Probability Prior PDF Inflated S.D. Inflate SD by 1.5 Variance by Obs IMA 11 March

123 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 31 truth ensemble obs No Inflation time 31 truth ensemble obs State Variable IMA 11 March

124 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 36 truth ensemble obs No Inflation time 36 truth ensemble obs State Variable IMA 11 March

125 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 311 truth ensemble obs No Inflation time 311 truth ensemble obs State Variable IMA 11 March

126 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 316 truth ensemble obs No Inflation time 316 truth ensemble obs State Variable IMA 11 March

127 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 321 truth ensemble obs No Inflation time 321 truth ensemble obs State Variable IMA 11 March

128 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 326 truth ensemble obs No Inflation time 326 truth ensemble obs State Variable IMA 11 March

129 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 331 truth ensemble obs No Inflation time 331 truth ensemble obs State Variable IMA 11 March

130 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 336 truth ensemble obs No Inflation time 336 truth ensemble obs State Variable IMA 11 March

131 Assimila.ng with Infla.on in presence of model error. Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm inflation Adaptive State Space Inflation time 341 truth ensemble obs No Inflation time 341 truth ensemble obs State Variable IMA 11 March

132 Assimila.ng with Infla.on in presence of model error Infla.on is a func.on of state variable and.me. Automa.cally selected by adap.ve infla.on algorithm. It can work, even in presence of severe model error. 1 1 inflation Adaptive State Space Inflation time 346 truth ensemble obs No Inflation time 346 truth ensemble obs State Variable IMA 11 March

133 Atmospheric Ensemble Reanalysis, O(1 million) atmospheric obs are assimilated every day. Assimilation uses 8 members of 2 o FV CAM forced by a single ocean (Hadley+ NCEP-OI2) and produces a very competitive reanalysis. 5 hpa GPH Feb IMA 11 March

134 266 hpa U wind infla.on Infla.on is large where model bias is detected. Mostly where there are dense observa.ons. U at 266 hpa, t = 8/1 1 5 New South Wales Colorado 8/4 8/9 8/14 8/19 8/24 8/ inflation factor IMA 11 March

135 Some Error Sources in Ensemble Filters 2. Obs. operator error; Representa.veness 3. Observa.on error 4. Sampling Error; Gaussian Assump.on 5. Sampling Error; Assuming Linear Sta.s.cal Rela.on 1. Model error IMA 11 March

136 Obs DART Current Ocean (POP) Assimila.on DATM 3D state 2D forcing from CAM assimila.on POP Coupler 3D restart 2D forcing IMA 11 March

137 World Ocean Database T,S observa.on counts These counts are for 1998 & 1999 and are representa.ve. FLOAT_SALINITY 682 FLOAT_TEMPERATURE DRIFTER_TEMPERATURE MOORING_SALINITY MOORING_TEMPERATURE BOTTLE_SALINITY BOTTLE_TEMPERATURE CTD_SALINITY CTD_TEMPERATURE STD_SALINITY 674 STD_TEMPERATURE 677 XCTD_SALINITY 3328 XCTD_TEMPERATURE 579 MBT_TEMPERATURE 5826 XBT_TEMPERATURE APB_TEMPERATURE temperature observa.on error standard devia.on ==.5 K. salinity observa.on error standard devia.on ==.5 msu. IMA 11 March

138 Physical Space: 1998/1999 SST Anomaly from HadOI- SST Coupled Free Run POP forced by observed atmosphere (hindcast) Ensemble Assimila.on 48 POP oceans Forced by 48 CAM reanalyses IMA 11 March

139 Challenges where ocean model is unable, or unwilling, to simulate reality. Example: cross sec.on along Kuroshio; model separates too far north. Hurrell SST o N 4 o N 35 o N 3 o N Regarded to be accurate. 25 o N 12 o E 132 o E 144 o E 156 o E 168 o E 18 o W POP SST o N 4 o N 35 o N 3 o N Free run of POP, the warm water is too far North. 25 o N 12 o E 132 o E 144 o E 156 o E 168 o E 18 o W 5 IMA 11 March

140 Challenges in correc.ng posi.on of Kuroshio. 6- day assimila.on star.ng from model climatology on 1 January 24. Temperature Temperature Temperature.me = 1days.me = 2days.me = 3days La.tude La.tude La.tude Ini.ally warm water goes too far north. Many observa.ons are rejected (red), but others (blue) move temperature gradient south. Adap.ve infla.on increases ensemble spread as assimila.on struggles to push model towards obs..me = 4days.me = 5days.me = 6days IMA 11 March

141 Challenges in correc.ng posi.on of Kuroshio. 6- day assimila.on star.ng from model climatology on 1 January 24. Green dashed line is posterior at previous.me, Blue dashed line is prior at current.me, Ensembles are thin lines. Model forecasts finally fail due Observa.ons keep pulling the warm water to the south; to numerical issues. Black Model forecasts con.nue to quickly move warm water dashes show original model further.me north. = 1days Infla.on con.nues to.me increase = 2days spread..me = 3days state from 1 January. Temperature Temperature Temperature.me = 4days.me = 5days.me = 6days La.tude La.tude La.tude IMA 11 March

142 Challenges in correc.ng posi.on of Kuroshio. 6- day assimila.on star.ng from model climatology on 1 January 24. Assimila.on cannot force model to fit observa.ons. Model cannot represent the observa.ons. Use of adap.ve infla.on leads to eventual model failure. Reduced adap.ve infla.on can lead to compromise between observa.ons and model. Increasing the error associated with forward operator can ameliorate, but.me = 1days.me = 2days.me = 3days what do the answers mean? Temperature Temperature Temperature.me = 4days.me = 5days.me = 6days La.tude La.tude La.tude IMA 11 March

143 All Obs DART CAM Fully coupled assimila.on will need data from all components at the same.me POP Coupler CICE CLM Each component corrected by all kinds of observa.ons IMA 11 March

144 Ensemble Data Assimila.on for Large Geophysical Models Ø Very certain that model predic.ons are different from observa.ons. Ø Very certain that small correla.ons have large errors. Ø Moderately confident that large correla.ons are realis.c. Ø Very uncertain about state es.mates in sparse/unobserved regions. Ø Must Calibrate and Validate results (adap.ve infla.on/localiza.on). Ø There may not be enough observa.ons to do this many places. Ø First order of business: Improving models. DA can help with this. Ø Stochas.c models seem like a logical way forward. IMA 11 March

145 Code to implement all of the algorithms discussed are freely available from: hnp:// IMA 11 March