Adaptive Inflation for Ensemble Assimilation

Adaptive Inflation for Ensemble Assimilation Jeffrey Anderson IMAGe Data Assimilation Research Section (DAReS) Thanks to Ryan Torn, Nancy Collins, Tim Hoar, Hui Liu, Kevin Raeder, Xuguang Wang Anderson: Balcones Springs 1 4/1/8

. h errors; Representativeness Some Error Sources in Ensemble Filters 3. Gross Obs. Errors 4. Sampling Error; Gaussian Assumption y y h h h t k+ t k * * 1. Model Error 5. Sampling Error; Assuming Linear Statistical Relation Anderson: Balcones Springs 4/1/8

All Error Sources are Expected to Lead to Too Little Variance 3. Gross Obs. Errors. h errors; Representativeness 4. Sampling Error; Gaussian Assumption y y h h h t k+ t k * * 1. Model Error 5. Sampling Error; Assuming Linear Statistical Relation Increase variance by linearly inflating ensemble around mean. Can do this to PRIOR before computing obs. operator or, In OBS. SPACE after computing obs. operator or, To POSTERIOR after doing regression. Anderson: Balcones Springs 3 4/1/8

Pretty Much the Simplest Low-Order Model 1. Single state variable, linear growth around. x t + 1 = α x t α= in first examples.. For perfect model, truth is perpetually. 3. Variable observed directly with synthetic noise. 4. Model error in assimilating model is additive. x t + 1 = αx t + γ γ=.1 in first examples. Case 1: obserr variance 1., 5 member EnKF. Case : same as 1). but obserr variance 1. with 1 obs per time. Case 3: same as 1). but observed every 1th step. Case 4: same as 1). but member EAKF, γ=.1. Anderson: Balcones Springs 4 4/1/8

Results for BEST time constant inflation; Prior and Posterior Lowest Prior RMS does NOT mean spread/rms is unity. Anderson: Balcones Springs 5 4/1/8

Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 y y o. Actual 3.698 SDs.4 Expected Separation Inflated S.D.. S.D. Probability 4 4 Distance, D, from prior mean y to obs. is λ Prob. y o is observed given λ: N, λσ prior + σ obs = N(, θ) py ( o λ) = ( Πθ ) 1 exp( D θ ) Anderson: Balcones Springs 6 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. 1 1 3 4 Observation: y Prior λ PDF 1 Prior PDF Obs. Likelihood We've assumed a gaussian for prior p( λ Y prev ) Recall that can be evaluated from normal PDF.. py ( o λ) 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 7 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4 Obs. Likelihood Repeat for a range of values of λ.. 1 1 3 4 Observation: y Prior λ PDF Now must get posterior in same form as prior (gaussian). 1 Posterior Likelihood y observed given λ 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 8 4/1/8

State Space Adaptive Inflation Computations so far adapt inflation for observation space. What is relation between observation and state space inflation? Have to use prior ensemble observation/state joint distribution. y y Regress changes in inflation onto state variable inflation. h h h t k+ t k * * Anderson: Balcones Springs 9 4/1/8

Spatially varying adaptive inflation algorithm: Have a distribution for λ at each time for each state variable, λ s,i. Use prior correlation from ensemble to determine impact of λ s,i on prior variance for given observation. If γ is correlation between state variable i and observation then θ = [ 1 + γ( λ si, 1) ] σ prior + σ obs Equation for finding mode of posterior is now full 1th order: Analytic solution appears unlikely. Can do Taylor expansion of θ around λ s,i. Retaining linear term is normally quite accurate. There is an analytic solution to find mode of product in this case!. Anderson: Balcones Springs 1 4/1/8

Prior Adaptive Inflation in Simplest Model with Model Error There is analytic solution for the inflation that minimizes prior RMS. Adaptive is trying to make spread and RMS agree. Anderson: Balcones Springs 11 4/1/8

Prior Adaptive Inflation in Simplest Model with Model Error Anderson: Balcones Springs 1 4/1/8

Prior Adaptive Inflation in Simplest Model with Model Error Adaptive Prior Works Marginally in Simplest Model. Anderson: Balcones Springs 13 4/1/8

Why do we need spatially-varying inflation? 1. Observation density can be spatially-varying. S.H. vs. N.H. for global. Near hurricane vortex vs. oceanic environment. In squall line vs. non-convecting environment. Large enough inflation to do well in one may blow-up in the other.. Sampling error will be larger in densely observed regions. 3. Model error may also be spatially-varying. Anderson: Balcones Springs 14 4/1/8

Model: CAM 3.1 T85L6. Adaptive Inflation in Global NWP Initialized from a climatological distribution (huge spread). Observations: Radiosondes, ACARS, Satellite Winds. Subset of observations used in NCAR/NCEP reanalysis. Anderson: Balcones Springs 15 4/1/8

Adaptive Inflation in CAM; 5 hpa T Obs. Space Prior RMS, Spread Anderson: Balcones Springs 16 4/1/8

Adaptive Inflation in CAM after 1-Month; 66 hpa U Anderson: Balcones Springs 17 4/1/8

Adaptive Inflation in CAM 1. Largest inflation caused by model error in densely observed regions.. RMS reduced, spread increased. 3. Fewer observations rejected. Note: really need to verify on set of trusted obs. 4. Inflation distribution has no time tendency. Can still cause blow-ups if inflation gets orphaned. Example: Inflation is larger near Antarctica in summer. When winter comes, obs. disappear. Inflation will just stay at previous values. Anderson: Balcones Springs 18 4/1/8

Really Need a Model of Time Evolution of Inflation Estimate Simple and Naive: 1. Mean inflation value is damped towards 1 as function of time.. Inflation variance is damped towards some climatological value. 3. Example in CAM: Mean is damped 1% every 6 hours. Variance is held fixed at.6. Anderson: Balcones Springs 19 4/1/8

Damped Adaptive Inflation in CAM after 1-Month; 66 hpa U Anderson: Balcones Springs 4/1/8

Damped Adaptive Inflation in CAM: Obs. Space 5 hpa T Anderson: Balcones Springs 1 4/1/8

Damped Adaptive Inflation 1. Works well in CAM until... Reanalysis BUFR files had most obs. missing on 8 Jan., 7... Then, it blew up.. Can this work with a hurricane or thunderstorm? Observations may be quite variable in time/space. Model error may also be quite variable. People have been trying this with some success. Anderson: Balcones Springs 4/1/8

General Thoughts 1. Don t expect ensembles to have consistent spread and RMS, Unless you want increased RMS error. Adaptive aims for consistency, so not optimal. Ensembles that look like random samples won t come from EnKF.. When model error dominates lack of spread, Need to change the general model of inflation. Work with model error and sampling error separately. For model error, model as additive in time? 3. Need trusted observation sets for comparing assimilations. Everything shown here is available for play in DART: www.image.ucar.edu/dares/dart Anderson: Balcones Springs 3 4/1/8

Model/Filter Error; Filter Divergence and Variance Inflation 1. History of observations and physical system => true distribution.. Sampling error, some model errors lead to insufficient prior variance. Probability 1.5 Variance Deficient PDF "TRUE" Prior PDF 4 3 1 3. Naive solution is Variance inflation: just increase spread of prior 4. For ensemble member i, inflate( x i ) = λ( x i x) + x. Anderson: Balcones Springs 4 4/1/8

Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 Probability.4. S.D. Actual 4.714 SDs Expected Separation S.D. 4 4 σ prior. Expected(prior mean - observation) = + σ obs. Assumes that prior and observation are supposed to be unbiased. Is it model error or random chance? Anderson: Balcones Springs 5 4/1/8

Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 Probability.4. Inflated S.D. Actual 3.698 SDs Expected Separation S.D. 4 4 σ prior. Expected(prior mean - observation) = + σ obs. 3. Inflating increases expected separation. Increases apparent consistency between prior and observation. Anderson: Balcones Springs 6 4/1/8

Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 y y o. Actual 3.698 SDs.4 Expected Separation Inflated S.D.. S.D. Probability 4 4 Distance, D, from prior mean y to obs. is λ Prob. y o is observed given λ: N, λσ prior + σ obs = N(, θ) py ( o λ) = ( Πθ ) 1 exp( D θ ) Anderson: Balcones Springs 7 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. Prior PDF Obs. Likelihood 1 1 3 4 Observation: y Prior λ PDF 1 1 3 4 5 6 Obs. Space Inflation Factor: λ Assume prior is gaussian; p( λ Y prev ) = N( λ p, σ λ, p ). Anderson: Balcones Springs 8 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. 1 1 3 4 Observation: y Prior λ PDF 1 Prior PDF Obs. Likelihood We've assumed a gaussian for prior p( λ Y prev ) Recall that can be evaluated from normal PDF.. py ( o λ) 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 9 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. 1 Obs. Likelihood Inflated Prior λ =.75 Get py ( o λ=.75) from normal PDF. 1 1 3 4 Observation: y Prior λ PDF Multiply by p( λ =.75 Y prev ) to get p( λ =.75 Y prev, y o ) 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 3 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. 1 Obs. Likelihood Inflated Prior λ = 1.5 Get py ( λ = 1.5 ) o 1 1 3 4 Observation: y Prior λ PDF from normal PDF. Multiply by to get p( λ = 1.5 Y prev ) p( λ = 1.5 Y prev, y o ) 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 31 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. 1 Obs. Likelihood Inflated Prior λ =.5 Get py ( λ =. ) o 1 1 3 4 Observation: y Prior λ PDF from normal PDF. Multiply by to get p( λ =. Y prev ) p( λ =. Y prev, y o ) 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 3 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4 Obs. Likelihood Repeat for a range of values of λ.. 1 1 3 4 Observation: y Prior λ PDF Now must get posterior in same form as prior (gaussian). 1 Posterior Likelihood y observed given λ 1 3 4 5 6 Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 33 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. Obs. Likelihood Very little information about λ in a single observation. 1 1 3 4 Observation: y Prior λ PDF 1 Posterior 1 Obs. Space Inflation Factor: λ Posterior and prior are very similar. Normalized posterior indistinguishable from prior. p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 34 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. Obs. Likelihood Very little information about λ in a single observation..1.1 1 1 3 4 Observation: y λ: Posterior Prior Max density shifted to right 1 Obs. Space Inflation Factor: λ Posterior and prior are very similar. Difference shows slight shift to larger values of λ. p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 35 4/1/8

Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4. Obs. Likelihood One option is to use Gaussian prior for λ. 1 1 3 4 Observation: y Find Max by search Prior λ PDF 1 Max is new λ mean 1 Obs. Space Inflation Factor: λ Select max (mode) of posterior as mean of updated Gaussian. Do a fit for updated standard deviation. p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 36 4/1/8

Adaptive Inflation for Ensemble Filtering A. Computing updated inflation mean,. Mode of py ( o λ)p( λ Y prev ) can be found analytically! Solving py ( o λ)p( λ Y prev ) λ= leads to 6th order poly in θ This can be reduced to a cubic equation and solved to give mode. λ u New is set to the mode. This is relatively cheap compared to computing regressions. λ u Anderson: Balcones Springs 37 4/1/8

Adaptive Inflation for Ensemble Filtering A. Computing updated inflation variance, σ λ, u 1. Evaluate numerator at mean λ u and second point, e.g. λ u + σ λ, p.. Find σ λ, u so N( λ u, σ λ, u ) goes through p( λ u ) and p( λ u + σ λ, p ). 3. Compute as σ λ, u = σ λ, p lnr where r = p( λ u + σ λ, p ) p( λ u ). Anderson: Balcones Springs 38 4/1/8

State Space Adaptive Inflation Computations so far adapt inflation for observation space. What is relation between observation and state space inflation? Have to use prior ensemble observation/state joint distribution. y y Regress changes in inflation onto state variable inflation. h h h t k+ t k * * Anderson: Balcones Springs 39 4/1/8

Spatially varying adaptive inflation algorithm: Have a distribution for λ at each time for each state variable, λ s,i. Use prior correlation from ensemble to determine impact of λ s,i on prior variance for given observation. If γ is correlation between state variable i and observation then θ = [ 1 + γ( λ si, 1) ] σ prior + σ obs Equation for finding mode of posterior is now full 1th order: Analytic solution appears unlikely. Can do Taylor expansion of θ around λ s,i. Retaining linear term is normally quite accurate. There is an analytic solution to find mode of product in this case!. Anderson: Balcones Springs 4 4/1/8

Hierarchical Bayesian Methods for Adaptive Filters: Summary 1. Localization: Run an ensemble of ensembles. Use regression coefficient signal-to-noise ratio to minimize error.. Inflation: Use each observation twice. Once to adjust parameter (inflation) of filter system. Second time to adjust mean and variance of estimate. Anderson: Balcones Springs 41 4/1/8