Fundamentals of Data Assimila1on
|
|
- Gordon Francis
- 6 years ago
- Views:
Transcription
1 2015 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO August 11-14, 2015 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University Fort Collins, Colorado
2 Outline! Motivation! Basic concepts! General challenges of data assimilation! Data assimilation methodologies! Relevance of forecast error covariance! Future of DA 2
3 Mo1va1on for Data Assimila1on! Prediction Models o o o models are described by a set of equations used to simulate the real- world processes and predict their future behavior. In geosciences, these equations typically refer to a system of partial differential equations (PDE) Various parameters can impact the performance of a PDE model: Initial conditions (IC), model errors (ME), empirical parameters (EP), Our knowledge of these parameters is never perfect, implying uncertainty of the prediction! Observations o o o Various measurements are available: satellite, radar, ships, aircraft, surface, Measuring instruments have errors, other errors (e.g., representativeness) Measurements have numerous errors, implying uncertainty of observations 3
4 Forecast! Uncertain forecast can be represented by a dynamic- stochastic process dx t = m(x t,t)dt + g(x t,t)dβ t m = Dynamics/model time evolution g = Stochastic forcing x β = State vector = Random vector! State vector (x) o o A (smallest) subset of variables defining a dynamical/physical system Typically it refers to the initial conditions only, but it may also include model errors and empirical parameters x = ( p T q u,v q cloud q snow O 3 T soil q soil ) T p = ( p 1 p N ) T T = ( T 1 T N ) T... Prediction models are fundamentally probabilistic! 4
5 Observa1ons! In general, observed variables are nonlinearly related to model variables! Observations include instrument and representativeness errors h y ε = Nonlinear mapping from model to observations = Observation vector = Observation error y = h(x t ) + ε! Given the probabilistic character of the model state (x t ) and the existence of observation errors, the observation transformation equation implies the probabilistic character of observations 5
6 Uncertainty Defines how reliable is a state vector estimate Measure of missing knowledge Important for decision- making Classical information theory (Shannon): - uncertainty is related to probability and entropy - probability and entropy are measures of incomplete knowledge Hurricane Ike (2008) wind speed probability Entropy and order 6
7 Data assimila1on process Observations Analysis Guess forecast Mathematical method used to blend the information from models and observations is called data assimilation. Data assimilation has a goal of producing optimal estimates of the state and its uncertainty.
8 Data assimila1on is probabilis1c! Insufficient knowledge of input information implies insufficient knowledge of the output: o o model equations and input parameters are imperfect, thus model forecast will be imperfect observations are imperfect! Uncertainties and imperfect knowledge are best measured by probability. o o make an assumption regarding adequate probability, or, (if possible) form a histogram and deduce best probability distribution Input to data assimilation could be represented by a joint probability density function (pdf) p(x,y ) X = State variable, Y = Observations 8
9 Bayesian principle in data assimila1on! Bayes theorem: create independent probabilities p(x,y ) = p(y )p(x Y ) = p(x)p(y X) p(x Y ) = p(y X)p(X) p(y ) o o p(x) Prior PDF p(x Y) Conditional PDF! It is implicitly assumed that it is easier to calculate the prior and the conditional PDFs than the joint PDF! A learning algorithm: probability estimate is updated as additional evidence is acquired 9
10 Prior and condi1onal probability density func1ons! Prior : Defines the knowledge about dynamical state before new observations (Y N ) are assimilated p(x ) = p(x Y N 1 Y 1 ) o Y N-1 Y 1 = Old observations o Y N = New observation not yet used! Conditional probability of new observations with respect to the prior state p(y N X ) o Y N = New observations 10
11 Gaussian assump1on! Probability Density Function (PDF) can be highly nonlinear, or with numerous unknown parameters that need to be estimated! One of the simplest and most widely applicable PDFs is the Gaussian PDF o Errors of physical processes tend to accumulate near zero (e.g., small errors dominate) o Gaussian PDF has the smallest number of unknown parameters (e.g., mean and covariance first two moments) 1 " (z µ) p(z) = exp$ σ 2π # 2σ 2 2 % ' & 11
12 Other relevant non- Gaussian PDFs Double exponential (Laplacian) PDF p L (x µ,b) = 1 2b exp x µ b! Wind speed errors can be described using Laplacian PDF! Sharp gradient fields (e.g., atmospheric fronts) exhibit Laplacian PDF There are other non- Gaussian (skewed) PDFs that are relevant: - Lognormal PDF (humidity, cloud variables) - Gamma function PDFs (precipitation) 12
13 Mul1variate Gaussian prior PDF! Multivariate Gaussian PDF p(x) = 1 exp " ( 2π ) N /2 1/2 $ 1 P 2 x x f # f ( ) T P 1 ( f x x f ) o x = state variable, P f = covariance, f = index denoting forecast o = determinant, N = dimension of state vector x % ' & } } } Covariance is independent of x Error of the forecast is defined as a difference between states Only mean and covariance are required # p(x) exp% 1 2 x x f $ ( ) T P 1 ( f x x f ) & ( ' 13
14 Mul1variate Gaussian condi1onal PDF! Probability of observations given the prior state # p( y x) exp% 1 $ 2 y h(x) ( ) T R 1 ( y h(x) ) o y = observations, R = observation covariance o h = nonlinear observation operator (mapping from model to observations) & ( ' } } } Zero mean is implicitly assumed in the above equation Error of the observation is defined as a difference between the observation and the model guess value at observation location Some observations, such as satellite, have a non- zero mean error 14
15 Data assimila1on with Gaussian PDFs! Maximum a- posteriori estimate: Find optimal state X opt that maximizes the posterior probability density function p(x Y) X opt = argmax x p(x Y )! Minimum variance estimate: Find optimal state X opt with the smallest error (variance) [L= loss function of conditional mean] X opt = argmin x E( L[E(x y)] ) Both estimates are identical for Gaussian PDF, otherwise they differ 15
16 Gaussian posterior PDF! Given the prior and conditional PDF, the posterior is # p(x y) exp% 1 $ 2 y h(x) ( ) T R 1 ( y h(x) ) 1 2 x x f ( ) T P 1 ( f x x f ) & ( '! Log- likelihood function (also referred to as the cost function) f (x) = log p(x y) = 1 " 2 # x x f $ % T Pf 1 " # x x f $ % + 1 " # 2 y h(x) $ %T R 1 " # y h(x) $ % Consequence: Minimize cost function = Maximize posterior PDF X opt = argmax x p(x y) = argmin f (x) x 16
17 Op1mality condi1ons for the minimum Taylor expansion for multivariate functional: f (x + ε p) = f (x) + ε p T [ f ]+ 1 2 ε 2 p T [ 2 f ]p + O(ε 3 ) ε p - constant parameter - perturbation vector 2 f > 0 f = 0 x - control vector f = g(x) - gradient 2 f = G(x) - Hessian f (x + ε p) = f (x) + ε p T g ε 2 p T Gp + O(ε 3 ) g(x*) = 0 G(x*) > 0 17
18 Minimize cost function: Op1mal solu1on f (x) = 1 2 x x f T Pf 1 x x f + 1 [ 2 y h(x) ] T R 1 [ y h(x) ] (1) g(x) = 0 P f 1 x x f H T R 1 [ y h(x) ] = 0 H = h x (2) G(x) > 0 z T Gz > 0 for any z G = P f 1 + H T R 1 H > 0 Since P 1 f + H T R 1 H minimum exists. is positive definite and symmetric, the 18
19 Role of Hessian (second deriva1ve) Optimal solution requires inverse Hessian g(x) = P f 1 x x f H T R 1 [ y h(x) ] f (x) G = P f 1 + H T R 1 H Gd = g d = G 1 ( g) Best condition for the system of linear equations: x x f = E T w G = EE T f (x) G w = E 1 (EE T )E T = I 19
20 One- point DA algorithm deriva1on Minimize quadratic cost function: f (x) = 1 2 x x f T Pf 1 x x f (1) f (x) = 0 P f [ 2 y Hx ] T R 1 [ y Hx] x x f H T R 1 [ y Hx] = 0 x = (I + P f H T R 1 H ) 1 (x f + P f H T R 1 y) One- point DA with single observation at a grid- point 2 2 H I P f σ f R σ o x a = (I + σ f 2 σ o 2 ) 1 (x f + σ f 2 σ o 2 y) 20
21 One- point DA (1) x a = (I + σ f 2 σ o 2 ) 1 (x f + σ f 2 σ o 2 y) x a = 2 σ o σ 2 2 f + σ o 2 x σ f + f σ 2 2 f + σ o y x a = α x f + βy α + β =? Analysis is a linear combination of the first guess and observation vectors, or analysis is an interpolation from observation and first guess uncertainty defines the interpolation weights 21
22 One- point DA (2) Note that interpolation weights are normalized: 2 σ o σ 2 2 f + σ o + 2 σ f σ 2 2 f + σ o = σ 2 2 o + σ f σ 2 f + σ = 1 2 α + β = 1 o x a = α x f + (1 α )y α = σ o 2 σ f 2 + σ o 2 Normalization of weights assures that the analysis will be between the guess and the observation α = 2 σ o σ 2 f + σ = o σ f σ o 2 +1 = 1 1+ σ f σ o 1 2 Only the ratio between uncertainties is important! 22
23 One- point DA (3) x a = 2 σ o σ 2 2 f + σ o x 2 σ f + 1 o σ 2 2 f + σ o y Observation error Forecast error a = σ o b = σ f (1) Large confidence in observations: a < b (2) Equal confidence in observations and first guess: a = b First guess x b b Optimal analysis First guess x b b Optimal analysis x a a x a a y Observation y Observation Interpretation of data assimilation is simple, the complexity comes from high dimensional state and nonlinear operators 23
24 Example 1 σ f (1) Given =, what is the analysis? σ o x a = 1 1+ ω 2 x f + ω 2 1+ ω 2 y ω = σ f σ o x a = lim 1+ ω 2 x f + lim ω 1 1+ ω 2 y ω 2 ω lim ω 1 1+ ω 2 = 0 ω 2 lim ω 1+ ω 2 = lim 1 ω 1 ω +1 2 = 1 x a = y No confidence in forecast implies the analysis at observation 24
25 Example 2 σ f (1) Given = 0, what is the analysis? σ o x a = 1 1+ ω 2 x f + ω 2 1+ ω 2 y ω = σ f σ o x a 1 ω 2 = lim ω 0 1+ ω 2 x f + lim ω 0 1+ ω 2 y lim ω ω 2 = 1 ω 2 lim ω 0 1+ ω 2 = 0 x a = x f No confidence in observations implies no impact of data assimilation 25
26 Challenges of realis1c data assimila1on Challenges are:! High dimensionality of state and observations o impacts degrees of freedom of forecast error covariance, acceptable choices of DA methodology! Nonlinearity of simulated physical processes and observation operators o Need capability to handle nonlinearities! Computation o Costly integration of realistic forecast models, matrix inversion! Observation errors o Bias correction, correlated observation errors! Multivariate character of the DA problem o Dynamical stability of the analysis 26
27 Impact of high dimensionality on matrices in data assimila1on Several possible formula8ons of data assimila8on (linear) analysis equa8ons: (1) (2) x = x f + P f H T ( HP f H T + R) 1 [y h(x)] x = x f + (P f 1 + H T R 1 H ) 1 H T R 1 [y h(x)] KF, observa8on- space VAR model- space VAR (3) x = x f + P f 1/2 (I + P f T /2 H T R 1 HP f 1/2 ) 1 P f T /2 H T R 1 [y h(x)] sqrt ensemble- space Q: Which version is best for high- dimensional applica8ons? It all depends on the limitation of matrix inversion due to dimensionality 27
28 Possible approxima1ons used in prac1ce (1) Neglect forecast covariance impact in matrix inverse: HP f H T + R R x = x f + P f H T R 1 [y h(x)] (2) Neglect observation impact in matrix inverse: H T R 1 H ~ 0 x = x f + P f H T R 1 [y h(x)] (3) Use all, but in low- dimensional subspace: dim(i + P f T /2 H T R 1 HP f 1/2 ) ~ O(100) x = x f + P f 1/2 (I + P f T /2 H T R 1 HP f 1/2 ) 1 P f T /2 H T R 1 [y h(x)] 28
29 Prac1cal data assimila1on algorithms: Basic methods! Variational data assimilation (3D- Var, 4D- Var) o Maximum a- posteriori estimate o Iterative minimization has advantage for nonlinear operators o Forecast uncertainty is pre- defined, time- independent (e.g., static) o Forecast uncertainty has all degrees of freedom o Employs an adjoint (e.g., transpose) operator! Ensemble Kalman filter data assimilation (EnKF, EnSRF) o Minimum variance estimate o Assumed linear KF analysis solution o Statistical sampling of forecast error covariance o Forecast uncertainty is flow- dependent (e.g., ensemble forecasts) o Reduced number of degrees of freedom o No need for an adjoint, use difference of nonlinear functions
30 Varia1onal data assimila1on 3D- Var cost function (one observation time): f (x) = 1 2 x x f T Pf 1 x x f + 1 [ 2 y h(x) ] T R 1 [ y h(x) ] 4D- Var cost function (sum over observation times): f (x) = 1 2 x x f T Pf 1 x x f T k=1 [ y h(m(x)) ] T k R 1 k y h(m(x)) [ ] k o 4D- Var allows smooth transition to forecast after data assimilation o 4D- Var analysis is more costly to calculate than 3D- Var o 3D- Var can considerably improve with better definition of the background error covariance (P f )
31 Prac1cal varia1onal DA f (x) = 1 2 x x f T Pf 1 x x f + 1 [ 2 y h(x) ] T R 1 [ y h(x) ] Forecast error covariance is high- dimensional (O(10 14 )) Matrix inversion in high- dimensions is costly/impossible to compute and likely ill- condi8oned Always use a change of variable: x x f = P f 1/2 w OR x x f = P f w f (w) = 1 2 wt w y h(x f + P 1/2 f w) This is practical if we assume a simple, or diagonal observation error covariance Otherwise, need to think of another change of variable T R 1 y h(x f + P 1/2 f w)
32 What is ensemble data assimila1on? (1) Forecast uncertainty is calculated from multiple model forecasts (ensembles) FCST ERROR M(x) } Forecast uncertainty Initial uncertainty { t t+1 TIME (2) Analysis employs the ensemble information and produces uncertainty Dynamical model (phase) space Observation space K(x) X X X 32
33 Ensemble Kalman filters Monte-Carlo (Evensen 1994; Houtekamer and Mitchell 1998) (1) Perturb observations using an assumed observation error PDF N(0,R) (2) Calculate ensemble of analyses, for each realization of observations x i a = x i f + [P f H T ens ] i ([HP f H T ] ens i + R) 1 y i h(x f i ) x a = 1 N Deterministic, square-root filters: (Bishop et al. 2001; Anderson 2003; Whitaker and Hamill 2002; Hunt et al. 2007) i x i a (1) No perturbed observations, use actual measurements (2) Only a single analysis is calculated x a ( f = x i + [P f H T ens ] i ([HP f H T ] ens i + R) 1 y i h(x f i ) ) y i = y + ε i x a = x f + [P f H T ens ] i ([HP f H T ] ens i + R) 1 y h(x f ) h(x f i ) h(x f )
34 Flow- dependent forecast error covariance grid- point x time obs 2 obs 1 Geographically distant observations can bring more information than close- by observations, if in a dynamically significant region
35 Impact of sta1c error covariance grid- point x Correlation length scale time obs 2 obs 1 Low- valued information (obs 2 ) will be assimilated instead of a high- valued information (obs 1 )
36 Forecast error covariance: Degrees of Freedom Singular Value Decomposition [Golub and van Loan (1989)] P 1/2 f = VΣW T T = σ i v i w i i P f = P 1/2 1/2 f ( P f ) T = VΣW T ( WΣV T ) = VΣ 2 V T 2 σ i v i v i T i v i, w i singular vectors σ i singular value V = ( v 1 v 2 v ) = N! # # # # # " 1 v 1 1 v 2 2 v 1 2 v 2 1 v N 2 v N N v S 1 N v S 2 N v S N $ & & & & & % " $ $ Σ = $ $ # σ σ σ N % ' ' ' ' & N S state vector dimension N N S σ 1 σ 2 σ N 0 " $ 1 i = j v T i v j = δ ij = # %$ 0 i j DOF = number of singular values greater than zero = rank of the matrix
37 Reduced rank aspect of (ensemble) data assimila1on Only a limited number of ensembles can be calculated: - high cost of forecast model integra1on - high- dimensional state vector Full model (phase) space Ensemble (phase) space Observations outside ensemble space cannot be assimilated Hybrid varia1onal- ensemble methods improve DOF problem by crea1ng uncertainty in all parts of the model space (e.g., combine flow- dependent and sta1c error covariance)
38 Why is forecast error covariance important? x a = x f + P f H T z obs = H T ( HP f H T + R) 1 [y h(x f )] = P f z obs ( HP f H T + R) 1 [y h(x f )] The analysis update is x a x f = i σ i 2 v i v i T z obs = i µ i v i µ i = σ i 2 v i T z obs! Analysis update is a linear combina1on of forecast error covariance singular vectors! Analysis increments are defined in the subspace spanned by forecast error covariance singular vectors
39 Prac1cal data assimila1on algorithms Hybrid methods! Hybrid variational- ensemble data assimilation J(x,α ) = P f 1/2 = β f o Combined static and flow- dependent error covariance o Iterative minimization o Sequential method ( δ x f ) T 1 P VAR ( δ x f ) + β e (α ) T ( P ENS L) 1 (α ) y Hδ xtot T R 1 y Hδ x tot δx tot = δx f + k α k P ENS 1/2 " # $ % k 1 β f + 1 β e =1! 4D- EN- VAR o 4- D control variable: simultaneous adjustment in time and space o Sequential method o Increased dimension of the control vector o Reduced rank, but could be used as a hybrid
40 What are future possibili1es for DA methodology? (1) Particle filters - solution to curse of dimensionality may exist - computational improvements may help with respect to large ensemble size requirement (2) Other hybrid options - improve existing hybrid DA methods by other means (3) Maximum entropy filters, information filters - more general than using a typical log- likelihood function - relate to Shannon information theory (4) Development of reduced- rank modeling - if reduced- rank modeling becomes a reality, it would imply reduced number of DOF - this would directly benefit EnsDA by alleviating the issue of small ensemble size (5) Improved computing efficiency - parallel computing on large number of CPUs
41 Future of data assimila1on! Practice: o Great need for combining information from observations and models with increased dimensionality and complexity o High temporal frequency observations (e.g., geostationary satellite) o Coupled data assimilation! Theory: o Increase generality of the mathematical formalism (fewer auxiliary parameters) o Reduce number of assumptions - Gaussian pdf, - uncorellated observation errors, 41
Fundamentals of Data Assimila1on
014 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO July 14-16, 014 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University
More informationRelative Merits of 4D-Var and Ensemble Kalman Filter
Relative Merits of 4D-Var and Ensemble Kalman Filter Andrew Lorenc Met Office, Exeter International summer school on Atmospheric and Oceanic Sciences (ISSAOS) "Atmospheric Data Assimilation". August 29
More informationA new Hierarchical Bayes approach to ensemble-variational data assimilation
A new Hierarchical Bayes approach to ensemble-variational data assimilation Michael Tsyrulnikov and Alexander Rakitko HydroMetCenter of Russia College Park, 20 Oct 2014 Michael Tsyrulnikov and Alexander
More information(Extended) Kalman Filter
(Extended) Kalman Filter Brian Hunt 7 June 2013 Goals of Data Assimilation (DA) Estimate the state of a system based on both current and all past observations of the system, using a model for the system
More informationPar$cle Filters Part I: Theory. Peter Jan van Leeuwen Data- Assimila$on Research Centre DARC University of Reading
Par$cle Filters Part I: Theory Peter Jan van Leeuwen Data- Assimila$on Research Centre DARC University of Reading Reading July 2013 Why Data Assimila$on Predic$on Model improvement: - Parameter es$ma$on
More informationKalman Filter and Ensemble Kalman Filter
Kalman Filter and Ensemble Kalman Filter 1 Motivation Ensemble forecasting : Provides flow-dependent estimate of uncertainty of the forecast. Data assimilation : requires information about uncertainty
More informationMaximum Likelihood Ensemble Filter Applied to Multisensor Systems
Maximum Likelihood Ensemble Filter Applied to Multisensor Systems Arif R. Albayrak a, Milija Zupanski b and Dusanka Zupanski c abc Colorado State University (CIRA), 137 Campus Delivery Fort Collins, CO
More information4. DATA ASSIMILATION FUNDAMENTALS
4. DATA ASSIMILATION FUNDAMENTALS... [the atmosphere] "is a chaotic system in which errors introduced into the system can grow with time... As a consequence, data assimilation is a struggle between chaotic
More informationEnsemble square-root filters
Ensemble square-root filters MICHAEL K. TIPPETT International Research Institute for climate prediction, Palisades, New Yor JEFFREY L. ANDERSON GFDL, Princeton, New Jersy CRAIG H. BISHOP Naval Research
More informationLocal Ensemble Transform Kalman Filter
Local Ensemble Transform Kalman Filter Brian Hunt 11 June 2013 Review of Notation Forecast model: a known function M on a vector space of model states. Truth: an unknown sequence {x n } of model states
More informationFundamentals of Data Assimilation
National Center for Atmospheric Research, Boulder, CO USA GSI Data Assimilation Tutorial - June 28-30, 2010 Acknowledgments and References WRFDA Overview (WRF Tutorial Lectures, H. Huang and D. Barker)
More informationarxiv: v1 [physics.ao-ph] 23 Jan 2009
A Brief Tutorial on the Ensemble Kalman Filter Jan Mandel arxiv:0901.3725v1 [physics.ao-ph] 23 Jan 2009 February 2007, updated January 2009 Abstract The ensemble Kalman filter EnKF) is a recursive filter
More informationA Note on the Particle Filter with Posterior Gaussian Resampling
Tellus (6), 8A, 46 46 Copyright C Blackwell Munksgaard, 6 Printed in Singapore. All rights reserved TELLUS A Note on the Particle Filter with Posterior Gaussian Resampling By X. XIONG 1,I.M.NAVON 1,2 and
More informationDynamic System Identification using HDMR-Bayesian Technique
Dynamic System Identification using HDMR-Bayesian Technique *Shereena O A 1) and Dr. B N Rao 2) 1), 2) Department of Civil Engineering, IIT Madras, Chennai 600036, Tamil Nadu, India 1) ce14d020@smail.iitm.ac.in
More informationIntroduction to Particle Filters for Data Assimilation
Introduction to Particle Filters for Data Assimilation Mike Dowd Dept of Mathematics & Statistics (and Dept of Oceanography Dalhousie University, Halifax, Canada STATMOS Summer School in Data Assimila5on,
More informationLagrangian Data Assimilation and Its Application to Geophysical Fluid Flows
Lagrangian Data Assimilation and Its Application to Geophysical Fluid Flows Laura Slivinski June, 3 Laura Slivinski (Brown University) Lagrangian Data Assimilation June, 3 / 3 Data Assimilation Setup:
More informationImproving GFS 4DEnVar Hybrid Data Assimilation System Using Time-lagged Ensembles
Improving GFS 4DEnVar Hybrid Data Assimilation System Using Time-lagged Ensembles Bo Huang and Xuguang Wang School of Meteorology University of Oklahoma, Norman, OK, USA Acknowledgement: Junkyung Kay (OU);
More informationHow 4DVAR can benefit from or contribute to EnKF (a 4DVAR perspective)
How 4DVAR can benefit from or contribute to EnKF (a 4DVAR perspective) Dale Barker WWRP/THORPEX Workshop on 4D-Var and Ensemble Kalman Filter Intercomparisons Sociedad Cientifica Argentina, Buenos Aires,
More informationEnsemble 4DVAR and observa3on impact study with the GSIbased hybrid ensemble varia3onal data assimila3on system. for the GFS
Ensemble 4DVAR and observa3on impact study with the GSIbased hybrid ensemble varia3onal data assimila3on system for the GFS Xuguang Wang University of Oklahoma, Norman, OK xuguang.wang@ou.edu Ting Lei,
More informationWeak Constraints 4D-Var
Weak Constraints 4D-Var Yannick Trémolet ECMWF Training Course - Data Assimilation May 1, 2012 Yannick Trémolet Weak Constraints 4D-Var May 1, 2012 1 / 30 Outline 1 Introduction 2 The Maximum Likelihood
More informationComparing Variational, Ensemble-based and Hybrid Data Assimilations at Regional Scales
Comparing Variational, Ensemble-based and Hybrid Data Assimilations at Regional Scales Meng Zhang and Fuqing Zhang Penn State University Xiang-Yu Huang and Xin Zhang NCAR 4 th EnDA Workshop, Albany, NY
More informationShort tutorial on data assimilation
Mitglied der Helmholtz-Gemeinschaft Short tutorial on data assimilation 23 June 2015 Wolfgang Kurtz & Harrie-Jan Hendricks Franssen Institute of Bio- and Geosciences IBG-3 (Agrosphere), Forschungszentrum
More informationConvective-scale data assimilation in the Weather Research and Forecasting model using a nonlinear ensemble filter
Convective-scale data assimilation in the Weather Research and Forecasting model using a nonlinear ensemble filter Jon Poterjoy, Ryan Sobash, and Jeffrey Anderson National Center for Atmospheric Research
More informationConvergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit
Convergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit Evan Kwiatkowski, Jan Mandel University of Colorado Denver December 11, 2014 OUTLINE 2 Data Assimilation Bayesian Estimation
More informationLocalization in the ensemble Kalman Filter
Department of Meteorology Localization in the ensemble Kalman Filter Ruth Elizabeth Petrie A dissertation submitted in partial fulfilment of the requirement for the degree of MSc. Atmosphere, Ocean and
More informationUNDERSTANDING DATA ASSIMILATION APPLICATIONS TO HIGH-LATITUDE IONOSPHERIC ELECTRODYNAMICS
Monday, June 27, 2011 CEDAR-GEM joint workshop: DA tutorial 1 UNDERSTANDING DATA ASSIMILATION APPLICATIONS TO HIGH-LATITUDE IONOSPHERIC ELECTRODYNAMICS Tomoko Matsuo University of Colorado, Boulder Space
More information4DEnVar. Four-Dimensional Ensemble-Variational Data Assimilation. Colloque National sur l'assimilation de données
Four-Dimensional Ensemble-Variational Data Assimilation 4DEnVar Colloque National sur l'assimilation de données Andrew Lorenc, Toulouse France. 1-3 décembre 2014 Crown copyright Met Office 4DEnVar: Topics
More informationCan hybrid-4denvar match hybrid-4dvar?
Comparing ensemble-variational assimilation methods for NWP: Can hybrid-4denvar match hybrid-4dvar? WWOSC, Montreal, August 2014. Andrew Lorenc, Neill Bowler, Adam Clayton, David Fairbairn and Stephen
More informationApplication of the Ensemble Kalman Filter to History Matching
Application of the Ensemble Kalman Filter to History Matching Presented at Texas A&M, November 16,2010 Outline Philosophy EnKF for Data Assimilation Field History Match Using EnKF with Covariance Localization
More informationThe Canadian approach to ensemble prediction
The Canadian approach to ensemble prediction ECMWF 2017 Annual seminar: Ensemble prediction : past, present and future. Pieter Houtekamer Montreal, Canada Overview. The Canadian approach. What are the
More informationGaussian Process Approximations of Stochastic Differential Equations
Gaussian Process Approximations of Stochastic Differential Equations Cédric Archambeau Dan Cawford Manfred Opper John Shawe-Taylor May, 2006 1 Introduction Some of the most complex models routinely run
More informationVariational data assimilation
Background and methods NCEO, Dept. of Meteorology, Univ. of Reading 710 March 2018, Univ. of Reading Bayes' Theorem Bayes' Theorem p(x y) = posterior distribution = p(x) p(y x) p(y) prior distribution
More informationPractical Aspects of Ensemble-based Kalman Filters
Practical Aspects of Ensemble-based Kalman Filters Lars Nerger Alfred Wegener Institute for Polar and Marine Research Bremerhaven, Germany and Bremen Supercomputing Competence Center BremHLR Bremen, Germany
More informationApplications of information theory in ensemble data assimilation
QUARTERLY JOURNAL OF THE ROYAL METEOROLOGICAL SOCIETY Q. J. R. Meteorol. Soc. 33: 533 545 (007) Published online in Wiley InterScience (www.interscience.wiley.com).3 Applications of information theory
More informationData assimilation in the geosciences An overview
Data assimilation in the geosciences An overview Alberto Carrassi 1, Olivier Talagrand 2, Marc Bocquet 3 (1) NERSC, Bergen, Norway (2) LMD, École Normale Supérieure, IPSL, France (3) CEREA, joint lab École
More informationAdvancing Data AssimilaJon Science for OperaJonal Hydrology: Methodology, ComputaJon, and Algorithms
2014 CAHMDA & HEPEX- DAFOH Workshop September 8-12, 2014 The University of Texas at AusJn Advancing Data AssimilaJon Science for OperaJonal Hydrology: Methodology, ComputaJon, and Algorithms Milija Zupanski
More informationA Comparison between the 4DVAR and the Ensemble Kalman Filter Techniques for Radar Data Assimilation
NOVEMBER 2005 C A Y A E T A L. 3081 A Comparison between the 4DVAR and the Ensemble Kalman Filter Techniques for Radar Data Assimilation A. CAYA, J.SUN, AND C. SNYDER National Center for Atmospheric Research,*
More informationCross-validation methods for quality control, cloud screening, etc.
Cross-validation methods for quality control, cloud screening, etc. Olaf Stiller, Deutscher Wetterdienst Are observations consistent Sensitivity functions with the other observations? given the background
More informationA Spectral Approach to Linear Bayesian Updating
A Spectral Approach to Linear Bayesian Updating Oliver Pajonk 1,2, Bojana V. Rosic 1, Alexander Litvinenko 1, and Hermann G. Matthies 1 1 Institute of Scientific Computing, TU Braunschweig, Germany 2 SPT
More informationAddressing the nonlinear problem of low order clustering in deterministic filters by using mean-preserving non-symmetric solutions of the ETKF
Addressing the nonlinear problem of low order clustering in deterministic filters by using mean-preserving non-symmetric solutions of the ETKF Javier Amezcua, Dr. Kayo Ide, Dr. Eugenia Kalnay 1 Outline
More informationNumerical Weather Prediction: Data assimilation. Steven Cavallo
Numerical Weather Prediction: Data assimilation Steven Cavallo Data assimilation (DA) is the process estimating the true state of a system given observations of the system and a background estimate. Observations
More information1. Current atmospheric DA systems 2. Coupling surface/atmospheric DA 3. Trends & ideas
1 Current issues in atmospheric data assimilation and its relationship with surfaces François Bouttier GAME/CNRM Météo-France 2nd workshop on remote sensing and modeling of surface properties, Toulouse,
More informationForecasting and data assimilation
Supported by the National Science Foundation DMS Forecasting and data assimilation Outline Numerical models Kalman Filter Ensembles Douglas Nychka, Thomas Bengtsson, Chris Snyder Geophysical Statistics
More informationIntroduction to Ensemble Kalman Filters and the Data Assimilation Research Testbed
Introduction to Ensemble Kalman Filters and the Data Assimilation Research Testbed Jeffrey Anderson, Tim Hoar, Nancy Collins NCAR Institute for Math Applied to Geophysics pg 1 What is Data Assimilation?
More informationHierarchical Bayes Ensemble Kalman Filter
Hierarchical Bayes Ensemble Kalman Filter M Tsyrulnikov and A Rakitko HydroMetCenter of Russia Wrocław, 7 Sep 2015 M Tsyrulnikov and A Rakitko (HMC) Hierarchical Bayes Ensemble Kalman Filter Wrocław, 7
More informationAn Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation
An Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation A. Shah 1,2, M. E. Gharamti 1, L. Bertino 1 1 Nansen Environmental and Remote Sensing Center 2 University of Bergen
More informationThe Ensemble Kalman Filter:
p.1 The Ensemble Kalman Filter: Theoretical formulation and practical implementation Geir Evensen Norsk Hydro Research Centre, Bergen, Norway Based on Evensen 23, Ocean Dynamics, Vol 53, No 4 p.2 The Ensemble
More informationNonlinear ensemble data assimila/on in high- dimensional spaces. Peter Jan van Leeuwen Javier Amezcua, Mengbin Zhu, Melanie Ades
Nonlinear ensemble data assimila/on in high- dimensional spaces Peter Jan van Leeuwen Javier Amezcua, Mengbin Zhu, Melanie Ades Data assimila/on: general formula/on Bayes theorem: The solu/on is a pdf!
More informationEnKF Review. P.L. Houtekamer 7th EnKF workshop Introduction to the EnKF. Challenges. The ultimate global EnKF algorithm
Overview 1 2 3 Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation 6th EnKF Purpose EnKF equations localization After the 6th EnKF (2014), I decided with Prof. Zhang to summarize progress
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing
More informationEnsemble Data Assimilation and Uncertainty Quantification
Ensemble Data Assimilation and Uncertainty Quantification Jeff Anderson National Center for Atmospheric Research pg 1 What is Data Assimilation? Observations combined with a Model forecast + to produce
More informationP 1.86 A COMPARISON OF THE HYBRID ENSEMBLE TRANSFORM KALMAN FILTER (ETKF)- 3DVAR AND THE PURE ENSEMBLE SQUARE ROOT FILTER (EnSRF) ANALYSIS SCHEMES
P 1.86 A COMPARISON OF THE HYBRID ENSEMBLE TRANSFORM KALMAN FILTER (ETKF)- 3DVAR AND THE PURE ENSEMBLE SQUARE ROOT FILTER (EnSRF) ANALYSIS SCHEMES Xuguang Wang*, Thomas M. Hamill, Jeffrey S. Whitaker NOAA/CIRES
More informationDATA ASSIMILATION FOR FLOOD FORECASTING
DATA ASSIMILATION FOR FLOOD FORECASTING Arnold Heemin Delft University of Technology 09/16/14 1 Data assimilation is the incorporation of measurement into a numerical model to improve the model results
More informationAdaptive Data Assimilation and Multi-Model Fusion
Adaptive Data Assimilation and Multi-Model Fusion Pierre F.J. Lermusiaux, Oleg G. Logoutov and Patrick J. Haley Jr. Mechanical Engineering and Ocean Science and Engineering, MIT We thank: Allan R. Robinson
More informationEnsemble 4DVAR for the NCEP hybrid GSI EnKF data assimilation system and observation impact study with the hybrid system
Ensemble 4DVAR for the NCEP hybrid GSI EnKF data assimilation system and observation impact study with the hybrid system Xuguang Wang School of Meteorology University of Oklahoma, Norman, OK OU: Ting Lei,
More informationVariational Data Assimilation Current Status
Variational Data Assimilation Current Status Eĺıas Valur Hólm with contributions from Mike Fisher and Yannick Trémolet ECMWF NORDITA Solar and stellar dynamos and cycles October 2009 Eĺıas Valur Hólm (ECMWF)
More informationA Stochastic Collocation based. for Data Assimilation
A Stochastic Collocation based Kalman Filter (SCKF) for Data Assimilation Lingzao Zeng and Dongxiao Zhang University of Southern California August 11, 2009 Los Angeles Outline Introduction SCKF Algorithm
More informationConsider the joint probability, P(x,y), shown as the contours in the figure above. P(x) is given by the integral of P(x,y) over all values of y.
ATMO/OPTI 656b Spring 009 Bayesian Retrievals Note: This follows the discussion in Chapter of Rogers (000) As we have seen, the problem with the nadir viewing emission measurements is they do not contain
More informationGaussian processes. Chuong B. Do (updated by Honglak Lee) November 22, 2008
Gaussian processes Chuong B Do (updated by Honglak Lee) November 22, 2008 Many of the classical machine learning algorithms that we talked about during the first half of this course fit the following pattern:
More informationProbabilistic numerics for deep learning
Presenter: Shijia Wang Department of Engineering Science, University of Oxford rning (RLSS) Summer School, Montreal 2017 Outline 1 Introduction Probabilistic Numerics 2 Components Probabilistic modeling
More informationQuarterly Journal of the Royal Meteorological Society
Quarterly Journal of the Royal Meteorological Society Effects of sequential or simultaneous assimilation of observations and localization methods on the performance of the ensemble Kalman filter Journal:
More informationComputer Vision Group Prof. Daniel Cremers. 9. Gaussian Processes - Regression
Group Prof. Daniel Cremers 9. Gaussian Processes - Regression Repetition: Regularized Regression Before, we solved for w using the pseudoinverse. But: we can kernelize this problem as well! First step:
More informationEnsemble Data Assimila.on and Uncertainty Quan.fica.on
Ensemble Data Assimila.on and Uncertainty Quan.fica.on Jeffrey Anderson, Alicia Karspeck, Tim Hoar, Nancy Collins, Kevin Raeder, Steve Yeager Na.onal Center for Atmospheric Research Ocean Sciences Mee.ng
More informationReview of Covariance Localization in Ensemble Filters
NOAA Earth System Research Laboratory Review of Covariance Localization in Ensemble Filters Tom Hamill NOAA Earth System Research Lab, Boulder, CO tom.hamill@noaa.gov Canonical ensemble Kalman filter update
More informationGaussian Filtering Strategies for Nonlinear Systems
Gaussian Filtering Strategies for Nonlinear Systems Canonical Nonlinear Filtering Problem ~u m+1 = ~ f (~u m )+~ m+1 ~v m+1 = ~g(~u m+1 )+~ o m+1 I ~ f and ~g are nonlinear & deterministic I Noise/Errors
More informationEnhancing information transfer from observations to unobserved state variables for mesoscale radar data assimilation
Enhancing information transfer from observations to unobserved state variables for mesoscale radar data assimilation Weiguang Chang and Isztar Zawadzki Department of Atmospheric and Oceanic Sciences Faculty
More informationCS 6140: Machine Learning Spring What We Learned Last Week. Survey 2/26/16. VS. Model
Logis@cs CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Assignment
More informationHybrid Variational Ensemble Data Assimilation for Tropical Cyclone
Hybrid Variational Ensemble Data Assimilation for Tropical Cyclone Forecasts Xuguang Wang School of Meteorology University of Oklahoma, Norman, OK Acknowledgement: OU: Ting Lei, Yongzuo Li, Kefeng Zhu,
More informationCS 6140: Machine Learning Spring 2016
CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa?on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Logis?cs Assignment
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationRobust Ensemble Filtering With Improved Storm Surge Forecasting
Robust Ensemble Filtering With Improved Storm Surge Forecasting U. Altaf, T. Buttler, X. Luo, C. Dawson, T. Mao, I.Hoteit Meteo France, Toulouse, Nov 13, 2012 Project Ensemble data assimilation for storm
More informationIn the derivation of Optimal Interpolation, we found the optimal weight matrix W that minimizes the total analysis error variance.
hree-dimensional variational assimilation (3D-Var) In the derivation of Optimal Interpolation, we found the optimal weight matrix W that minimizes the total analysis error variance. Lorenc (1986) showed
More informationLecture 3: Pattern Classification
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 1 2 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mixtures
More informationAutonomous Navigation for Flying Robots
Computer Vision Group Prof. Daniel Cremers Autonomous Navigation for Flying Robots Lecture 6.2: Kalman Filter Jürgen Sturm Technische Universität München Motivation Bayes filter is a useful tool for state
More informationEnsemble Kalman Filter
Ensemble Kalman Filter Geir Evensen and Laurent Bertino Hydro Research Centre, Bergen, Norway, Nansen Environmental and Remote Sensing Center, Bergen, Norway The Ensemble Kalman Filter (EnKF) Represents
More informationDART_LAB Tutorial Section 2: How should observations impact an unobserved state variable? Multivariate assimilation.
DART_LAB Tutorial Section 2: How should observations impact an unobserved state variable? Multivariate assimilation. UCAR 2014 The National Center for Atmospheric Research is sponsored by the National
More informationStochastic Collocation Methods for Polynomial Chaos: Analysis and Applications
Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications Dongbin Xiu Department of Mathematics, Purdue University Support: AFOSR FA955-8-1-353 (Computational Math) SF CAREER DMS-64535
More informationMAXIMUM LIKELIHOOD ENSEMBLE FILTER: THEORETICAL ASPECTS. Milija Zupanski
MAXIMUM LIKELIHOOD ENSEMBLE FILTER: THEORETICAL ASPECTS Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University Foothills Campus Fort Collins, CO 8053-1375 ZupanskiM@cira.colostate.edu
More information2D Image Processing. Bayes filter implementation: Kalman filter
2D Image Processing Bayes filter implementation: Kalman filter Prof. Didier Stricker Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de
More informationLocal Ensemble Transform Kalman Filter: An Efficient Scheme for Assimilating Atmospheric Data
Local Ensemble Transform Kalman Filter: An Efficient Scheme for Assimilating Atmospheric Data John Harlim and Brian R. Hunt Department of Mathematics and Institute for Physical Science and Technology University
More informationThe Ensemble Kalman Filter:
p.1 The Ensemble Kalman Filter: Theoretical formulation and practical implementation Geir Evensen Norsk Hydro Research Centre, Bergen, Norway Based on Evensen, Ocean Dynamics, Vol 5, No p. The Ensemble
More informationComputer Vision Group Prof. Daniel Cremers. 4. Gaussian Processes - Regression
Group Prof. Daniel Cremers 4. Gaussian Processes - Regression Definition (Rep.) Definition: A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution.
More informationUsing Observations at Different Spatial. Scales in Data Assimilation for. Environmental Prediction. Joanne A. Waller
UNIVERSITY OF READING DEPARTMENT OF MATHEMATICS AND STATISTICS Using Observations at Different Spatial Scales in Data Assimilation for Environmental Prediction Joanne A. Waller Thesis submitted for the
More informationStructural Reliability
Structural Reliability Thuong Van DANG May 28, 2018 1 / 41 2 / 41 Introduction to Structural Reliability Concept of Limit State and Reliability Review of Probability Theory First Order Second Moment Method
More informationAn introduction to data assimilation. Eric Blayo University of Grenoble and INRIA
An introduction to data assimilation Eric Blayo University of Grenoble and INRIA Data assimilation, the science of compromises Context characterizing a (complex) system and/or forecasting its evolution,
More informationThe Big Leap: Replacing 4D-Var with 4D-EnVar and life ever since
The Big Leap: Replacing 4D-Var with 4D-EnVar and life ever since Symposium: 20 years of 4D-Var at ECMWF 26 January 2018 Mark Buehner 1, Jean-Francois Caron 1 and Ping Du 2 1 Data Assimilation and Satellite
More informationAccepted in Tellus A 2 October, *Correspondence
1 An Adaptive Covariance Inflation Error Correction Algorithm for Ensemble Filters Jeffrey L. Anderson * NCAR Data Assimilation Research Section P.O. Box 3000 Boulder, CO 80307-3000 USA Accepted in Tellus
More informationSmoothers: Types and Benchmarks
Smoothers: Types and Benchmarks Patrick N. Raanes Oxford University, NERSC 8th International EnKF Workshop May 27, 2013 Chris Farmer, Irene Moroz Laurent Bertino NERSC Geir Evensen Abstract Talk builds
More informationInverse problems and uncertainty quantification in remote sensing
1 / 38 Inverse problems and uncertainty quantification in remote sensing Johanna Tamminen Finnish Meterological Institute johanna.tamminen@fmi.fi ESA Earth Observation Summer School on Earth System Monitoring
More informationA Comparison of Error Subspace Kalman Filters
Tellus 000, 000 000 (0000) Printed 4 February 2005 (Tellus LATEX style file v2.2) A Comparison of Error Subspace Kalman Filters By LARS NERGER, WOLFGANG HILLER and JENS SCHRÖTER Alfred Wegener Institute
More informationFour-Dimensional Ensemble Kalman Filtering
Four-Dimensional Ensemble Kalman Filtering B.R. Hunt, E. Kalnay, E.J. Kostelich, E. Ott, D.J. Patil, T. Sauer, I. Szunyogh, J.A. Yorke, A.V. Zimin University of Maryland, College Park, MD 20742, USA Ensemble
More informationMethods of Data Assimilation and Comparisons for Lagrangian Data
Methods of Data Assimilation and Comparisons for Lagrangian Data Chris Jones, Warwick and UNC-CH Kayo Ide, UCLA Andrew Stuart, Jochen Voss, Warwick Guillaume Vernieres, UNC-CH Amarjit Budiraja, UNC-CH
More informationEnsemble forecasting and flow-dependent estimates of initial uncertainty. Martin Leutbecher
Ensemble forecasting and flow-dependent estimates of initial uncertainty Martin Leutbecher acknowledgements: Roberto Buizza, Lars Isaksen Flow-dependent aspects of data assimilation, ECMWF 11 13 June 2007
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond Department of Biomedical Engineering and Computational Science Aalto University January 26, 2012 Contents 1 Batch and Recursive Estimation
More informationEnsemble Data Assimila.on for Climate System Component Models
Ensemble Data Assimila.on for Climate System Component Models Jeffrey Anderson Na.onal Center for Atmospheric Research In collabora.on with: Alicia Karspeck, Kevin Raeder, Tim Hoar, Nancy Collins IMA 11
More information2D Image Processing. Bayes filter implementation: Kalman filter
2D Image Processing Bayes filter implementation: Kalman filter Prof. Didier Stricker Dr. Gabriele Bleser Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche
More informationToward improved initial conditions for NCAR s real-time convection-allowing ensemble. Ryan Sobash, Glen Romine, Craig Schwartz, and Kate Fossell
Toward improved initial conditions for NCAR s real-time convection-allowing ensemble Ryan Sobash, Glen Romine, Craig Schwartz, and Kate Fossell Storm-scale ensemble design Can an EnKF be used to initialize
More informationDART Tutorial Part IV: Other Updates for an Observed Variable
DART Tutorial Part IV: Other Updates for an Observed Variable UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on. Any opinions, findings and conclusions or recommenda'ons
More informationMobile Robot Localization
Mobile Robot Localization 1 The Problem of Robot Localization Given a map of the environment, how can a robot determine its pose (planar coordinates + orientation)? Two sources of uncertainty: - observations
More informationOrganization. I MCMC discussion. I project talks. I Lecture.
Organization I MCMC discussion I project talks. I Lecture. Content I Uncertainty Propagation Overview I Forward-Backward with an Ensemble I Model Reduction (Intro) Uncertainty Propagation in Causal Systems
More information