Adaptive Inflation for Ensemble Assimilation
|
|
- Philomena George
- 5 years ago
- Views:
Transcription
1 Adaptive Inflation for Ensemble Assimilation Jeffrey Anderson IMAGe Data Assimilation Research Section (DAReS) Thanks to Ryan Torn, Nancy Collins, Tim Hoar, Hui Liu, Kevin Raeder, Xuguang Wang Anderson: Balcones Springs 1 4/1/8
2 . h errors; Representativeness Some Error Sources in Ensemble Filters 3. Gross Obs. Errors 4. Sampling Error; Gaussian Assumption y y h h h t k+ t k * * 1. Model Error 5. Sampling Error; Assuming Linear Statistical Relation Anderson: Balcones Springs 4/1/8
3 All Error Sources are Expected to Lead to Too Little Variance 3. Gross Obs. Errors. h errors; Representativeness 4. Sampling Error; Gaussian Assumption y y h h h t k+ t k * * 1. Model Error 5. Sampling Error; Assuming Linear Statistical Relation Increase variance by linearly inflating ensemble around mean. Can do this to PRIOR before computing obs. operator or, In OBS. SPACE after computing obs. operator or, To POSTERIOR after doing regression. Anderson: Balcones Springs 3 4/1/8
4 Pretty Much the Simplest Low-Order Model 1. Single state variable, linear growth around. x t + 1 = α x t α= in first examples.. For perfect model, truth is perpetually. 3. Variable observed directly with synthetic noise. 4. Model error in assimilating model is additive. x t + 1 = αx t + γ γ=.1 in first examples. Case 1: obserr variance 1., 5 member EnKF. Case : same as 1). but obserr variance 1. with 1 obs per time. Case 3: same as 1). but observed every 1th step. Case 4: same as 1). but member EAKF, γ=.1. Anderson: Balcones Springs 4 4/1/8
5 Results for BEST time constant inflation; Prior and Posterior Lowest Prior RMS does NOT mean spread/rms is unity. Anderson: Balcones Springs 5 4/1/8
6 Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 y y o. Actual SDs.4 Expected Separation Inflated S.D.. S.D. Probability 4 4 Distance, D, from prior mean y to obs. is λ Prob. y o is observed given λ: N, λσ prior + σ obs = N(, θ) py ( o λ) = ( Πθ ) 1 exp( D θ ) Anderson: Balcones Springs 6 4/1/8
7 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Observation: y Prior λ PDF 1 Prior PDF Obs. Likelihood We've assumed a gaussian for prior p( λ Y prev ) Recall that can be evaluated from normal PDF.. py ( o λ) Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 7 4/1/8
8 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4 Obs. Likelihood Repeat for a range of values of λ Observation: y Prior λ PDF Now must get posterior in same form as prior (gaussian). 1 Posterior Likelihood y observed given λ Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 8 4/1/8
9 State Space Adaptive Inflation Computations so far adapt inflation for observation space. What is relation between observation and state space inflation? Have to use prior ensemble observation/state joint distribution. y y Regress changes in inflation onto state variable inflation. h h h t k+ t k * * Anderson: Balcones Springs 9 4/1/8
10 Spatially varying adaptive inflation algorithm: Have a distribution for λ at each time for each state variable, λ s,i. Use prior correlation from ensemble to determine impact of λ s,i on prior variance for given observation. If γ is correlation between state variable i and observation then θ = [ 1 + γ( λ si, 1) ] σ prior + σ obs Equation for finding mode of posterior is now full 1th order: Analytic solution appears unlikely. Can do Taylor expansion of θ around λ s,i. Retaining linear term is normally quite accurate. There is an analytic solution to find mode of product in this case!. Anderson: Balcones Springs 1 4/1/8
11 Prior Adaptive Inflation in Simplest Model with Model Error There is analytic solution for the inflation that minimizes prior RMS. Adaptive is trying to make spread and RMS agree. Anderson: Balcones Springs 11 4/1/8
12 Prior Adaptive Inflation in Simplest Model with Model Error Anderson: Balcones Springs 1 4/1/8
13 Prior Adaptive Inflation in Simplest Model with Model Error Adaptive Prior Works Marginally in Simplest Model. Anderson: Balcones Springs 13 4/1/8
14 Why do we need spatially-varying inflation? 1. Observation density can be spatially-varying. S.H. vs. N.H. for global. Near hurricane vortex vs. oceanic environment. In squall line vs. non-convecting environment. Large enough inflation to do well in one may blow-up in the other.. Sampling error will be larger in densely observed regions. 3. Model error may also be spatially-varying. Anderson: Balcones Springs 14 4/1/8
15 Model: CAM 3.1 T85L6. Adaptive Inflation in Global NWP Initialized from a climatological distribution (huge spread). Observations: Radiosondes, ACARS, Satellite Winds. Subset of observations used in NCAR/NCEP reanalysis. Anderson: Balcones Springs 15 4/1/8
16 Adaptive Inflation in CAM; 5 hpa T Obs. Space Prior RMS, Spread Anderson: Balcones Springs 16 4/1/8
17 Adaptive Inflation in CAM after 1-Month; 66 hpa U Anderson: Balcones Springs 17 4/1/8
18 Adaptive Inflation in CAM 1. Largest inflation caused by model error in densely observed regions.. RMS reduced, spread increased. 3. Fewer observations rejected. Note: really need to verify on set of trusted obs. 4. Inflation distribution has no time tendency. Can still cause blow-ups if inflation gets orphaned. Example: Inflation is larger near Antarctica in summer. When winter comes, obs. disappear. Inflation will just stay at previous values. Anderson: Balcones Springs 18 4/1/8
19 Really Need a Model of Time Evolution of Inflation Estimate Simple and Naive: 1. Mean inflation value is damped towards 1 as function of time.. Inflation variance is damped towards some climatological value. 3. Example in CAM: Mean is damped 1% every 6 hours. Variance is held fixed at.6. Anderson: Balcones Springs 19 4/1/8
20 Damped Adaptive Inflation in CAM after 1-Month; 66 hpa U Anderson: Balcones Springs 4/1/8
21 Damped Adaptive Inflation in CAM: Obs. Space 5 hpa T Anderson: Balcones Springs 1 4/1/8
22 Damped Adaptive Inflation 1. Works well in CAM until... Reanalysis BUFR files had most obs. missing on 8 Jan., 7... Then, it blew up.. Can this work with a hurricane or thunderstorm? Observations may be quite variable in time/space. Model error may also be quite variable. People have been trying this with some success. Anderson: Balcones Springs 4/1/8
23 General Thoughts 1. Don t expect ensembles to have consistent spread and RMS, Unless you want increased RMS error. Adaptive aims for consistency, so not optimal. Ensembles that look like random samples won t come from EnKF.. When model error dominates lack of spread, Need to change the general model of inflation. Work with model error and sampling error separately. For model error, model as additive in time? 3. Need trusted observation sets for comparing assimilations. Everything shown here is available for play in DART: Anderson: Balcones Springs 3 4/1/8
24 Model/Filter Error; Filter Divergence and Variance Inflation 1. History of observations and physical system => true distribution.. Sampling error, some model errors lead to insufficient prior variance. Probability 1.5 Variance Deficient PDF "TRUE" Prior PDF Naive solution is Variance inflation: just increase spread of prior 4. For ensemble member i, inflate( x i ) = λ( x i x) + x. Anderson: Balcones Springs 4 4/1/8
25 Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 Probability.4. S.D. Actual SDs Expected Separation S.D. 4 4 σ prior. Expected(prior mean - observation) = + σ obs. Assumes that prior and observation are supposed to be unbiased. Is it model error or random chance? Anderson: Balcones Springs 5 4/1/8
26 Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 Probability.4. Inflated S.D. Actual SDs Expected Separation S.D. 4 4 σ prior. Expected(prior mean - observation) = + σ obs. 3. Inflating increases expected separation. Increases apparent consistency between prior and observation. Anderson: Balcones Springs 6 4/1/8
27 Adaptive Inflation for Ensemble Filtering 1. For observed variable, have estimate of prior-observed inconsistency..8 Prior PDF Obs. Likelihood.6 y y o. Actual SDs.4 Expected Separation Inflated S.D.. S.D. Probability 4 4 Distance, D, from prior mean y to obs. is λ Prob. y o is observed given λ: N, λσ prior + σ obs = N(, θ) py ( o λ) = ( Πθ ) 1 exp( D θ ) Anderson: Balcones Springs 7 4/1/8
28 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Prior PDF Obs. Likelihood Observation: y Prior λ PDF Obs. Space Inflation Factor: λ Assume prior is gaussian; p( λ Y prev ) = N( λ p, σ λ, p ). Anderson: Balcones Springs 8 4/1/8
29 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Observation: y Prior λ PDF 1 Prior PDF Obs. Likelihood We've assumed a gaussian for prior p( λ Y prev ) Recall that can be evaluated from normal PDF.. py ( o λ) Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 9 4/1/8
30 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Obs. Likelihood Inflated Prior λ =.75 Get py ( o λ=.75) from normal PDF Observation: y Prior λ PDF Multiply by p( λ =.75 Y prev ) to get p( λ =.75 Y prev, y o ) Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 3 4/1/8
31 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Obs. Likelihood Inflated Prior λ = 1.5 Get py ( λ = 1.5 ) o Observation: y Prior λ PDF from normal PDF. Multiply by to get p( λ = 1.5 Y prev ) p( λ = 1.5 Y prev, y o ) Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 31 4/1/8
32 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Obs. Likelihood Inflated Prior λ =.5 Get py ( λ =. ) o Observation: y Prior λ PDF from normal PDF. Multiply by to get p( λ =. Y prev ) p( λ =. Y prev, y o ) Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 3 4/1/8
33 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ..6.4 Obs. Likelihood Repeat for a range of values of λ Observation: y Prior λ PDF Now must get posterior in same form as prior (gaussian). 1 Posterior Likelihood y observed given λ Obs. Space Inflation Factor: λ p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 33 4/1/8
34 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Obs. Likelihood Very little information about λ in a single observation Observation: y Prior λ PDF 1 Posterior 1 Obs. Space Inflation Factor: λ Posterior and prior are very similar. Normalized posterior indistinguishable from prior. p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 34 4/1/8
35 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Obs. Likelihood Very little information about λ in a single observation Observation: y λ: Posterior Prior Max density shifted to right 1 Obs. Space Inflation Factor: λ Posterior and prior are very similar. Difference shows slight shift to larger values of λ. p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 35 4/1/8
36 Adaptive Inflation for Ensemble Filtering Use Bayesian statistics to get estimate of inflation factor, λ Obs. Likelihood One option is to use Gaussian prior for λ Observation: y Find Max by search Prior λ PDF 1 Max is new λ mean 1 Obs. Space Inflation Factor: λ Select max (mode) of posterior as mean of updated Gaussian. Do a fit for updated standard deviation. p( λ Y prev, y o ) = py ( o λ)p( λ Y prev ) normalization. Anderson: Balcones Springs 36 4/1/8
37 Adaptive Inflation for Ensemble Filtering A. Computing updated inflation mean,. Mode of py ( o λ)p( λ Y prev ) can be found analytically! Solving py ( o λ)p( λ Y prev ) λ= leads to 6th order poly in θ This can be reduced to a cubic equation and solved to give mode. λ u New is set to the mode. This is relatively cheap compared to computing regressions. λ u Anderson: Balcones Springs 37 4/1/8
38 Adaptive Inflation for Ensemble Filtering A. Computing updated inflation variance, σ λ, u 1. Evaluate numerator at mean λ u and second point, e.g. λ u + σ λ, p.. Find σ λ, u so N( λ u, σ λ, u ) goes through p( λ u ) and p( λ u + σ λ, p ). 3. Compute as σ λ, u = σ λ, p lnr where r = p( λ u + σ λ, p ) p( λ u ). Anderson: Balcones Springs 38 4/1/8
39 State Space Adaptive Inflation Computations so far adapt inflation for observation space. What is relation between observation and state space inflation? Have to use prior ensemble observation/state joint distribution. y y Regress changes in inflation onto state variable inflation. h h h t k+ t k * * Anderson: Balcones Springs 39 4/1/8
40 Spatially varying adaptive inflation algorithm: Have a distribution for λ at each time for each state variable, λ s,i. Use prior correlation from ensemble to determine impact of λ s,i on prior variance for given observation. If γ is correlation between state variable i and observation then θ = [ 1 + γ( λ si, 1) ] σ prior + σ obs Equation for finding mode of posterior is now full 1th order: Analytic solution appears unlikely. Can do Taylor expansion of θ around λ s,i. Retaining linear term is normally quite accurate. There is an analytic solution to find mode of product in this case!. Anderson: Balcones Springs 4 4/1/8
41 Hierarchical Bayesian Methods for Adaptive Filters: Summary 1. Localization: Run an ensemble of ensembles. Use regression coefficient signal-to-noise ratio to minimize error.. Inflation: Use each observation twice. Once to adjust parameter (inflation) of filter system. Second time to adjust mean and variance of estimate. Anderson: Balcones Springs 41 4/1/8
DART_LAB Tutorial Section 5: Adaptive Inflation
DART_LAB Tutorial Section 5: Adaptive Inflation UCAR 14 The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings and conclusions or recommendations
More informationIntroduction to Ensemble Kalman Filters and the Data Assimilation Research Testbed
Introduction to Ensemble Kalman Filters and the Data Assimilation Research Testbed Jeffrey Anderson, Tim Hoar, Nancy Collins NCAR Institute for Math Applied to Geophysics pg 1 What is Data Assimilation?
More informationReducing the Impact of Sampling Errors in Ensemble Filters
Reducing the Impact of Sampling Errors in Ensemble Filters Jeffrey Anderson NCAR Data Assimilation Research Section The National Center for Atmospheric Research is sponsored by the National Science Foundation.
More informationEnsemble Data Assimila.on and Uncertainty Quan.fica.on
Ensemble Data Assimila.on and Uncertainty Quan.fica.on Jeffrey Anderson, Alicia Karspeck, Tim Hoar, Nancy Collins, Kevin Raeder, Steve Yeager Na.onal Center for Atmospheric Research Ocean Sciences Mee.ng
More informationEnsemble Data Assimilation and Uncertainty Quantification
Ensemble Data Assimilation and Uncertainty Quantification Jeff Anderson National Center for Atmospheric Research pg 1 What is Data Assimilation? Observations combined with a Model forecast + to produce
More informationSea Ice Data Assimilation in the Arctic via DART/CICE5 in the CESM1
Sea Ice Data Assimilation in the Arctic via DART/CICE5 in the CESM1 Yongfei Zhang and Cecilia Bitz 1 Jeffrey Anderson, Nancy Collins, Jonathan Hendricks, Tim Hoar, and Kevin Raeder 2 1 University of Washington,
More informationDART Tutorial Sec'on 9: More on Dealing with Error: Infla'on
DART Tutorial Sec'on 9: More on Dealing with Error: Infla'on UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on. Any opinions, findings and conclusions or recommenda'ons
More informationData Assimilation Research Testbed Tutorial
Data Assimilation Research Testbed Tutorial Section 2: How should observations of a state variable impact an unobserved state variable? Multivariate assimilation. Single observed variable, single unobserved
More informationDART Tutorial Part IV: Other Updates for an Observed Variable
DART Tutorial Part IV: Other Updates for an Observed Variable UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on. Any opinions, findings and conclusions or recommenda'ons
More informationData Assimilation Research Testbed Tutorial
Data Assimilation Research Testbed Tutorial Section 3: Hierarchical Group Filters and Localization Version 2.: September, 26 Anderson: Ensemble Tutorial 9//6 Ways to deal with regression sampling error:
More informationLocalization and Correlation in Ensemble Kalman Filters
Localization and Correlation in Ensemble Kalman Filters Jeff Anderson NCAR Data Assimilation Research Section The National Center for Atmospheric Research is sponsored by the National Science Foundation.
More informationEnsemble Data Assimila.on for Climate System Component Models
Ensemble Data Assimila.on for Climate System Component Models Jeffrey Anderson Na.onal Center for Atmospheric Research In collabora.on with: Alicia Karspeck, Kevin Raeder, Tim Hoar, Nancy Collins IMA 11
More informationToward improved initial conditions for NCAR s real-time convection-allowing ensemble. Ryan Sobash, Glen Romine, Craig Schwartz, and Kate Fossell
Toward improved initial conditions for NCAR s real-time convection-allowing ensemble Ryan Sobash, Glen Romine, Craig Schwartz, and Kate Fossell Storm-scale ensemble design Can an EnKF be used to initialize
More informationDART_LAB Tutorial Section 2: How should observations impact an unobserved state variable? Multivariate assimilation.
DART_LAB Tutorial Section 2: How should observations impact an unobserved state variable? Multivariate assimilation. UCAR 2014 The National Center for Atmospheric Research is sponsored by the National
More informationUsing DART Tools for CAM Development
Using DART Tools for CAM Development Kevin Raeder, For DART: Jeff Anderson, Tim Hoar, Nancy Collins, Johnny Hendricks CSEG: Alice Bertini, Mariana Vertenstein, Steve Goldhaber, Jim Edwards And: Nick Pedatella
More informationGaussian Filtering Strategies for Nonlinear Systems
Gaussian Filtering Strategies for Nonlinear Systems Canonical Nonlinear Filtering Problem ~u m+1 = ~ f (~u m )+~ m+1 ~v m+1 = ~g(~u m+1 )+~ o m+1 I ~ f and ~g are nonlinear & deterministic I Noise/Errors
More informationRelative Merits of 4D-Var and Ensemble Kalman Filter
Relative Merits of 4D-Var and Ensemble Kalman Filter Andrew Lorenc Met Office, Exeter International summer school on Atmospheric and Oceanic Sciences (ISSAOS) "Atmospheric Data Assimilation". August 29
More informationNumerical Weather Prediction: Data assimilation. Steven Cavallo
Numerical Weather Prediction: Data assimilation Steven Cavallo Data assimilation (DA) is the process estimating the true state of a system given observations of the system and a background estimate. Observations
More informationAccepted in Tellus A 2 October, *Correspondence
1 An Adaptive Covariance Inflation Error Correction Algorithm for Ensemble Filters Jeffrey L. Anderson * NCAR Data Assimilation Research Section P.O. Box 3000 Boulder, CO 80307-3000 USA Accepted in Tellus
More informationDART Tutorial Section 18: Lost in Phase Space: The Challenge of Not Knowing the Truth.
DART Tutorial Section 18: Lost in Phase Space: The Challenge of Not Knowing the Truth. UCAR The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings
More informationModal view of atmospheric predictability
Modal view of atmospheric predictability Nedjeljka Žagar University of Ljubljana, Ljubljana, Slovenia Based on Žagar, N., R. Buizza and J. Tribbia, J. Atmos. Sci., 0, and Žagar, N., J. Anderson, N. Collins,
More informationMulti-instance CESM for Fully Coupled Data Assimilation Capability using DART
Multi-instance CESM for Fully Coupled Data Assimilation Capability using DART Kevin Raeder, Tim Hoar, Nancy Collins, Jeffrey Anderson: DAReS Mariana Vertenstein, Tony Craig, Jim Edwards: SEWG Alicia Karspeck:
More informationStatus of CESM data assimilation using the DART system
Status of CESM data assimilation using the DART system Alicia R. Karspeck (aliciak@ucar.edu) DARES personnel: Nancy Collins, Jeff Anderson, John Hendricks, Kevin Raeder, Tim Hoar CISL/ASAP: Brian Dobbins
More informationMulti-instance CESM for Fully Coupled Data Assimilation Capability using DART
Multi-instance CESM for Fully Coupled Data Assimilation Capability using DART Kevin Raeder, Tim Hoar, Nancy Collins, Jeffrey Anderson: DAReS Mariana Vertenstein, Tony Craig, Jim Edwards: SEWG Jen Kay:
More informationFundamentals of Data Assimila1on
014 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO July 14-16, 014 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University
More informationHow well do we know the climatological characteristics of the North Atlantic jet stream? Isla Simpson, CAS, CDG, NCAR
How well do we know the climatological characteristics of the North Atlantic jet stream? Isla Simpson, CAS, CDG, NCAR A common bias among GCMs is that the Atlantic jet is too zonal One particular contour
More informationThe Canadian approach to ensemble prediction
The Canadian approach to ensemble prediction ECMWF 2017 Annual seminar: Ensemble prediction : past, present and future. Pieter Houtekamer Montreal, Canada Overview. The Canadian approach. What are the
More informationImproved analyses and forecasts with AIRS retrievals using the Local Ensemble Transform Kalman Filter
Improved analyses and forecasts with AIRS retrievals using the Local Ensemble Transform Kalman Filter Hong Li, Junjie Liu, and Elana Fertig E. Kalnay I. Szunyogh, E. J. Kostelich Weather and Chaos Group
More informationDART Ini)al Condi)ons for a Refined Grid CAM- SE Forecast of Hurricane Katrina. Kevin Raeder (IMAGe) Colin Zarzycki (ASP)
DART Ini)al Condi)ons for a Refined Grid CAM- SE Forecast of Hurricane Katrina Kevin Raeder (IMAGe) Colin Zarzycki (ASP) 1 Mo)va)on Thousands of processors on current supercomputers. - > new CAM dynamical
More information4. DATA ASSIMILATION FUNDAMENTALS
4. DATA ASSIMILATION FUNDAMENTALS... [the atmosphere] "is a chaotic system in which errors introduced into the system can grow with time... As a consequence, data assimilation is a struggle between chaotic
More informationA Non-Gaussian Ensemble Filter Update for Data Assimilation
4186 M O N T H L Y W E A T H E R R E V I E W VOLUME 138 A Non-Gaussian Ensemble Filter Update for Data Assimilation JEFFREY L. ANDERSON NCAR Data Assimilation Research Section,* Boulder, Colorado (Manuscript
More informationData assimilation; comparison of 4D-Var and LETKF smoothers
Data assimilation; comparison of 4D-Var and LETKF smoothers Eugenia Kalnay and many friends University of Maryland CSCAMM DAS13 June 2013 Contents First part: Forecasting the weather - we are really getting
More informationEnsemble Assimilation of Global Large-Scale Precipitation
Ensemble Assimilation of Global Large-Scale Precipitation Guo-Yuan Lien 1,2 in collaboration with Eugenia Kalnay 2, Takemasa Miyoshi 1,2 1 RIKEN Advanced Institute for Computational Science 2 University
More informationSTA 414/2104, Spring 2014, Practice Problem Set #1
STA 44/4, Spring 4, Practice Problem Set # Note: these problems are not for credit, and not to be handed in Question : Consider a classification problem in which there are two real-valued inputs, and,
More informationData AssimilaJon with CLM & DART
Data AssimilaJon with CLM & DART Tim Hoar: Na#onal Center for Atmospheric Research with a whole lot of help from: Bill Sacks, Mariana Vertenstein, Tony Craig, Jim Edwards: NCAR Andrew Fox: Na#onal Ecological
More informationData Assimilation for Tropospheric CO. Avelino F. Arellano, Jr. Atmospheric Chemistry Division National Center for Atmospheric Research
Data Assimilation for Tropospheric CO Avelino F. Arellano, Jr. Atmospheric Chemistry Division National Center for Atmospheric Research Caveat: Illustrative rather than quantitative, applied rather than
More informationLand Surface Data AssimilaEon: DART and CLM
Land Surface Data AssimilaEon: DART and CLM Yongfei Zhang: University of Texas Aus2n Tim Hoar, Bill Sacks, Tony Craig, Jeff Anderson: Na2onal Center for Atmospheric Research Andrew Fox: Na2onal Ecological
More informationGeneralized Linear Models for Non-Normal Data
Generalized Linear Models for Non-Normal Data Today s Class: 3 parts of a generalized model Models for binary outcomes Complications for generalized multivariate or multilevel models SPLH 861: Lecture
More informationDART and Land Data Assimila7on
DART and Land Data Assimila7on Tim Hoar: Na#onal Center for Atmospheric Research with a whole lot of help from: Jeff Anderson, Nancy Collins, Kevin Raeder, Bill Sacks: NCAR Yongfei Zhang: University of
More informationJuli I. Rubin. NRC Postdoctoral Research Associate Naval Research Laboratory, Monterey, CA
Development of the Ensemble Navy Aerosol Analysis Prediction System and its application of the Data Assimilation Research Testbed in Support of Aerosol Forecasting Juli I. Rubin NRC Postdoctoral Research
More informationConvective-scale data assimilation in the Weather Research and Forecasting model using a nonlinear ensemble filter
Convective-scale data assimilation in the Weather Research and Forecasting model using a nonlinear ensemble filter Jon Poterjoy, Ryan Sobash, and Jeffrey Anderson National Center for Atmospheric Research
More informationAspects of the practical application of ensemble-based Kalman filters
Aspects of the practical application of ensemble-based Kalman filters Lars Nerger Alfred Wegener Institute for Polar and Marine Research Bremerhaven, Germany and Bremen Supercomputing Competence Center
More informationEnsemble Kalman Filters for WRF-ARW. Chris Snyder MMM and IMAGe National Center for Atmospheric Research
Ensemble Kalman Filters for WRF-ARW Chris Snyder MMM and IMAGe National Center for Atmospheric Research Preliminaries Notation: x = modelʼs state w.r.t. some discrete basis, e.g. grid-pt values y = Hx
More informationFundamentals of Data Assimilation
National Center for Atmospheric Research, Boulder, CO USA GSI Data Assimilation Tutorial - June 28-30, 2010 Acknowledgments and References WRFDA Overview (WRF Tutorial Lectures, H. Huang and D. Barker)
More information1 Using standard errors when comparing estimated values
MLPR Assignment Part : General comments Below are comments on some recurring issues I came across when marking the second part of the assignment, which I thought it would help to explain in more detail
More informationImproving GFS 4DEnVar Hybrid Data Assimilation System Using Time-lagged Ensembles
Improving GFS 4DEnVar Hybrid Data Assimilation System Using Time-lagged Ensembles Bo Huang and Xuguang Wang School of Meteorology University of Oklahoma, Norman, OK, USA Acknowledgement: Junkyung Kay (OU);
More informationSome Applications of WRF/DART
Some Applications of WRF/DART Chris Snyder, National Center for Atmospheric Research Mesoscale and Microscale Meteorology Division (MMM), and Institue for Mathematics Applied to Geoscience (IMAGe) WRF/DART
More informationCOMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017
COMS 4721: Machine Learning for Data Science Lecture 10, 2/21/2017 Prof. John Paisley Department of Electrical Engineering & Data Science Institute Columbia University FEATURE EXPANSIONS FEATURE EXPANSIONS
More informationRadar data assimilation using a modular programming approach with the Ensemble Kalman Filter: preliminary results
Radar data assimilation using a modular programming approach with the Ensemble Kalman Filter: preliminary results I. Maiello 1, L. Delle Monache 2, G. Romine 2, E. Picciotti 3, F.S. Marzano 4, R. Ferretti
More informationErgodicity in data assimilation methods
Ergodicity in data assimilation methods David Kelly Andy Majda Xin Tong Courant Institute New York University New York NY www.dtbkelly.com April 15, 2016 ETH Zurich David Kelly (CIMS) Data assimilation
More informationA Global Atmospheric Model. Joe Tribbia NCAR Turbulence Summer School July 2008
A Global Atmospheric Model Joe Tribbia NCAR Turbulence Summer School July 2008 Outline Broad overview of what is in a global climate/weather model of the atmosphere Spectral dynamical core Some results-climate
More informationBayesian Regression Linear and Logistic Regression
When we want more than point estimates Bayesian Regression Linear and Logistic Regression Nicole Beckage Ordinary Least Squares Regression and Lasso Regression return only point estimates But what if we
More informationA Note on the Particle Filter with Posterior Gaussian Resampling
Tellus (6), 8A, 46 46 Copyright C Blackwell Munksgaard, 6 Printed in Singapore. All rights reserved TELLUS A Note on the Particle Filter with Posterior Gaussian Resampling By X. XIONG 1,I.M.NAVON 1,2 and
More informationHypothesis Testing. Econ 690. Purdue University. Justin L. Tobias (Purdue) Testing 1 / 33
Hypothesis Testing Econ 690 Purdue University Justin L. Tobias (Purdue) Testing 1 / 33 Outline 1 Basic Testing Framework 2 Testing with HPD intervals 3 Example 4 Savage Dickey Density Ratio 5 Bartlett
More informationMultimodel Ensemble forecasts
Multimodel Ensemble forecasts Calibrated methods Michael K. Tippett International Research Institute for Climate and Society The Earth Institute, Columbia University ERFS Climate Predictability Tool Training
More informationHybrid variational-ensemble data assimilation. Daryl T. Kleist. Kayo Ide, Dave Parrish, John Derber, Jeff Whitaker
Hybrid variational-ensemble data assimilation Daryl T. Kleist Kayo Ide, Dave Parrish, John Derber, Jeff Whitaker Weather and Chaos Group Meeting 07 March 20 Variational Data Assimilation J Var J 2 2 T
More informationGAUSSIAN PROCESS REGRESSION
GAUSSIAN PROCESS REGRESSION CSE 515T Spring 2015 1. BACKGROUND The kernel trick again... The Kernel Trick Consider again the linear regression model: y(x) = φ(x) w + ε, with prior p(w) = N (w; 0, Σ). The
More informationStatistical Methods in Particle Physics
Statistical Methods in Particle Physics Lecture 11 January 7, 2013 Silvia Masciocchi, GSI Darmstadt s.masciocchi@gsi.de Winter Semester 2012 / 13 Outline How to communicate the statistical uncertainty
More informationNew filtering options in CAM
AMWG, February, 2010 New filtering options in CAM Peter Hjort Lauritzen (NCAR) Art Mirin (LLNL) John Truesdale (NCAR) Kevin Raeder (NCAR) Jeff Anderson (NCAR) Why new filtering options in CAM4? 1. Excessive
More informationModeling Data with Linear Combinations of Basis Functions. Read Chapter 3 in the text by Bishop
Modeling Data with Linear Combinations of Basis Functions Read Chapter 3 in the text by Bishop A Type of Supervised Learning Problem We want to model data (x 1, t 1 ),..., (x N, t N ), where x i is a vector
More informationStatistics 203: Introduction to Regression and Analysis of Variance Penalized models
Statistics 203: Introduction to Regression and Analysis of Variance Penalized models Jonathan Taylor - p. 1/15 Today s class Bias-Variance tradeoff. Penalized regression. Cross-validation. - p. 2/15 Bias-variance
More informationImpact of COSMIC observations in a whole atmosphere-ionosphere data assimilation model
Impact of COSMIC observations in a whole atmosphere-ionosphere data assimilation model Nick Pedatella 1,2, Hanli Liu 1, Jing Liu 1, Jeffrey Anderson 3, and Kevin Raeder 3 1 High Altitude Observatory, NCAR
More informationAHW Ensemble Data Assimilation and Forecasting System
AHW Ensemble Data Assimilation and Forecasting System Ryan D. Torn, Univ. Albany, SUNY Chris Davis, Wei Wang, Jimy Dudhia, Tom Galarneau, Chris Snyder, James Done, NCAR/MMM Overview Since participation
More informationEvaluating Methods to Account for System Errors in Ensemble Data Assimilation
3078 M O N T H L Y W E A T H E R R E V I E W VOLUME 140 Evaluating Methods to Account for System Errors in Ensemble Data Assimilation JEFFREY S. WHITAKER AND THOMAS M. HAMILL NOAA/Earth System Research
More informationStatistical Models with Uncertain Error Parameters (G. Cowan, arxiv: )
Statistical Models with Uncertain Error Parameters (G. Cowan, arxiv:1809.05778) Workshop on Advanced Statistics for Physics Discovery aspd.stat.unipd.it Department of Statistical Sciences, University of
More informationAn Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation
An Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation A. Shah 1,2, M. E. Gharamti 1, L. Bertino 1 1 Nansen Environmental and Remote Sensing Center 2 University of Bergen
More informationReport on the Joint SRNWP workshop on DA-EPS Bologna, March. Nils Gustafsson Alex Deckmyn.
Report on the Joint SRNWP workshop on DA-EPS Bologna, 22-24 March Nils Gustafsson Alex Deckmyn http://www.smr.arpa.emr.it/srnwp/ Purpose of the workshop On the one hand, data assimilation techniques require
More informationJi-Sun Kang and Eugenia Kalnay
Review of Probability Wilks, Chapter Ji-Sun Kang and Eugenia Kalnay AOSC630 Spring, 008, updated in 07 Definition Event: set, class, or group of possible uncertain outcomes A compound event can be decomposed
More informationModel comparison. Christopher A. Sims Princeton University October 18, 2016
ECO 513 Fall 2008 Model comparison Christopher A. Sims Princeton University sims@princeton.edu October 18, 2016 c 2016 by Christopher A. Sims. This document may be reproduced for educational and research
More informationJi-Sun Kang. Pr. Eugenia Kalnay (Chair/Advisor) Pr. Ning Zeng (Co-Chair) Pr. Brian Hunt (Dean s representative) Pr. Kayo Ide Pr.
Carbon Cycle Data Assimilation Using a Coupled Atmosphere-Vegetation Model and the LETKF Ji-Sun Kang Committee in charge: Pr. Eugenia Kalnay (Chair/Advisor) Pr. Ning Zeng (Co-Chair) Pr. Brian Hunt (Dean
More informationApplication of Radio Occultation Data in Analyses and Forecasts of Tropical Cyclones Using an Ensemble Assimilation System
Application of Radio Occultation Data in Analyses and Forecasts of Tropical Cyclones Using an Assimilation System Hui Liu, Jeff Anderson, and Bill Kuo NCAR Acknowledgment: C. Snyder, Y. Chen, T. Hoar,
More informationScalable Compu-ng Challenges in Ensemble Data Assimila-on. FRCRC Symposium Nancy Collins NCAR - IMAGe/DAReS 14 Aug 2013
Scalable Compu-ng Challenges in Ensemble Data Assimila-on FRCRC Symposium Nancy Collins NCAR - IMAGe/DAReS 14 Aug 2013 Overview What is Data Assimila-on? What is DART? Current Work on Highly Scalable Systems
More informationDART Tutorial Sec'on 5: Comprehensive Filtering Theory: Non-Iden'ty Observa'ons and the Joint Phase Space
DART Tutorial Sec'on 5: Comprehensive Filtering Theory: Non-Iden'ty Observa'ons and the Joint Phase Space UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on.
More informationOverview. Probabilistic Interpretation of Linear Regression Maximum Likelihood Estimation Bayesian Estimation MAP Estimation
Overview Probabilistic Interpretation of Linear Regression Maximum Likelihood Estimation Bayesian Estimation MAP Estimation Probabilistic Interpretation: Linear Regression Assume output y is generated
More informationRelevance Vector Machines
LUT February 21, 2011 Support Vector Machines Model / Regression Marginal Likelihood Regression Relevance vector machines Exercise Support Vector Machines The relevance vector machine (RVM) is a bayesian
More informationTesting Restrictions and Comparing Models
Econ. 513, Time Series Econometrics Fall 00 Chris Sims Testing Restrictions and Comparing Models 1. THE PROBLEM We consider here the problem of comparing two parametric models for the data X, defined by
More informationECE295, Data Assimila0on and Inverse Problems, Spring 2015
ECE295, Data Assimila0on and Inverse Problems, Spring 2015 1 April, Intro; Linear discrete Inverse problems (Aster Ch 1 and 2) Slides 8 April, SVD (Aster ch 2 and 3) Slides 15 April, RegularizaFon (ch
More informationExploring and extending the limits of weather predictability? Antje Weisheimer
Exploring and extending the limits of weather predictability? Antje Weisheimer Arnt Eliassen s legacy for NWP ECMWF is an independent intergovernmental organisation supported by 34 states. ECMWF produces
More informationHave a better understanding of the Tropical Cyclone Products generated at ECMWF
Objectives Have a better understanding of the Tropical Cyclone Products generated at ECMWF Learn about the recent developments in the forecast system and its impact on the Tropical Cyclone forecast Learn
More informationHierarchical Bayes Ensemble Kalman Filter
Hierarchical Bayes Ensemble Kalman Filter M Tsyrulnikov and A Rakitko HydroMetCenter of Russia Wrocław, 7 Sep 2015 M Tsyrulnikov and A Rakitko (HMC) Hierarchical Bayes Ensemble Kalman Filter Wrocław, 7
More informationDART Tutorial Sec'on 1: Filtering For a One Variable System
DART Tutorial Sec'on 1: Filtering For a One Variable System UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on. Any opinions, findings and conclusions or recommenda'ons
More informationCan hybrid-4denvar match hybrid-4dvar?
Comparing ensemble-variational assimilation methods for NWP: Can hybrid-4denvar match hybrid-4dvar? WWOSC, Montreal, August 2014. Andrew Lorenc, Neill Bowler, Adam Clayton, David Fairbairn and Stephen
More informationChoosing among models
Eco 515 Fall 2014 Chris Sims Choosing among models September 18, 2014 c 2014 by Christopher A. Sims. This document is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported
More informationFernando Prates. Evaluation Section. Slide 1
Fernando Prates Evaluation Section Slide 1 Objectives Ø Have a better understanding of the Tropical Cyclone Products generated at ECMWF Ø Learn the recent developments in the forecast system and its impact
More informationPATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Chapter 1. Introduction Shuai Huang April 21, 2014 Outline 1 What is Machine Learning? 2 Curve Fitting 3 Probability Theory 4 Model Selection 5 The curse of dimensionality
More informationThe University of Texas at Austin, Jackson School of Geosciences, Austin, Texas 2. The National Center for Atmospheric Research, Boulder, Colorado 3
Assimilation of MODIS Snow Cover and GRACE Terrestrial Water Storage Data through DART/CLM4 Yong-Fei Zhang 1, Zong-Liang Yang 1, Tim J. Hoar 2, Hua Su 1, Jeffrey L. Anderson 2, Ally M. Toure 3,4, and Matthew
More informationA data-driven method for improving the correlation estimation in serial ensemble Kalman filter
A data-driven method for improving the correlation estimation in serial ensemble Kalman filter Michèle De La Chevrotière, 1 John Harlim 2 1 Department of Mathematics, Penn State University, 2 Department
More informationEvaluation of a non-local observation operator in assimilation of. CHAMP radio occultation refractivity with WRF
Evaluation of a non-local observation operator in assimilation of CHAMP radio occultation refractivity with WRF Hui Liu, Jeffrey Anderson, Ying-Hwa Kuo, Chris Snyder, and Alain Caya National Center for
More informationIntroduction to data assimilation and least squares methods
Introduction to data assimilation and least squares methods Eugenia Kalnay and many friends University of Maryland October 008 (part 1 Contents (1 Forecasting the weather - we are really getting better!
More informationEnhancing information transfer from observations to unobserved state variables for mesoscale radar data assimilation
Enhancing information transfer from observations to unobserved state variables for mesoscale radar data assimilation Weiguang Chang and Isztar Zawadzki Department of Atmospheric and Oceanic Sciences Faculty
More informationChanging predictability characteristics of Arctic sea ice in a warming climate
Changing predictability characteristics of Arctic sea ice in a warming climate Marika Holland 1 Laura Landrum 1, John Mioduszewski 2, Steve Vavrus 2, Muyin Wang 3 1. NCAR, 2. U. Wisconsin-Madison, 3. NOAA
More informationCS Homework 3. October 15, 2009
CS 294 - Homework 3 October 15, 2009 If you have questions, contact Alexandre Bouchard (bouchard@cs.berkeley.edu) for part 1 and Alex Simma (asimma@eecs.berkeley.edu) for part 2. Also check the class website
More informationNonparametric Bayesian Methods (Gaussian Processes)
[70240413 Statistical Machine Learning, Spring, 2015] Nonparametric Bayesian Methods (Gaussian Processes) Jun Zhu dcszj@mail.tsinghua.edu.cn http://bigml.cs.tsinghua.edu.cn/~jun State Key Lab of Intelligent
More informationDART Tutorial Part II: How should observa'ons impact an unobserved state variable? Mul'variate assimila'on.
DART Tutorial Part II: How should observa'ons impact an unobserved state variable? Mul'variate assimila'on. UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on.
More informationP 1.86 A COMPARISON OF THE HYBRID ENSEMBLE TRANSFORM KALMAN FILTER (ETKF)- 3DVAR AND THE PURE ENSEMBLE SQUARE ROOT FILTER (EnSRF) ANALYSIS SCHEMES
P 1.86 A COMPARISON OF THE HYBRID ENSEMBLE TRANSFORM KALMAN FILTER (ETKF)- 3DVAR AND THE PURE ENSEMBLE SQUARE ROOT FILTER (EnSRF) ANALYSIS SCHEMES Xuguang Wang*, Thomas M. Hamill, Jeffrey S. Whitaker NOAA/CIRES
More informationWhat do we know about EnKF?
What do we know about EnKF? David Kelly Kody Law Andrew Stuart Andrew Majda Xin Tong Courant Institute New York University New York, NY April 10, 2015 CAOS seminar, Courant. David Kelly (NYU) EnKF April
More informationSTRONGLY COUPLED ENKF DATA ASSIMILATION
STRONGLY COUPLED ENKF DATA ASSIMILATION WITH THE CFSV2 Travis Sluka Acknowledgements: Eugenia Kalnay, Steve Penny, Takemasa Miyoshi CDAW Toulouse Oct 19, 2016 Outline 1. Overview of strongly coupled DA
More informationIntroduction to Machine Learning Midterm Exam
10-701 Introduction to Machine Learning Midterm Exam Instructors: Eric Xing, Ziv Bar-Joseph 17 November, 2015 There are 11 questions, for a total of 100 points. This exam is open book, open notes, but
More informationInconsistency of Bayesian inference when the model is wrong, and how to repair it
Inconsistency of Bayesian inference when the model is wrong, and how to repair it Peter Grünwald Thijs van Ommen Centrum Wiskunde & Informatica, Amsterdam Universiteit Leiden June 3, 2015 Outline 1 Introduction
More informationStatistical Distribution Assumptions of General Linear Models
Statistical Distribution Assumptions of General Linear Models Applied Multilevel Models for Cross Sectional Data Lecture 4 ICPSR Summer Workshop University of Colorado Boulder Lecture 4: Statistical Distributions
More information