Fundamentals of Data Assimila1on
|
|
- Phebe Jones
- 5 years ago
- Views:
Transcription
1 014 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO July 14-16, 014 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University Fort Collins, Colorado
2 Outline Motivation for data assimilation Basic data assimilation General challenges of data assimilation Data assimilation methodologies Relevance of forecast error covariance Future of DA
3 Mo1va1on for Data Assimila1on Models and observations o models are described by a set of equations used to simulate the real- world processes and predict their future behavior o In geosciences, these equations typically refer to a system of partial differential equations (PDE) o Various parameters can impact the performance of a PDE model: Initial conditions (IC), model errors (ME), empirical parameters (EP), o Our knowledge of these parameters is never perfect, implying uncertainty! State vector (x) o o o A (smallest) subset of variables defining a dynamical/physical system Typically it refers to the initial conditions only In general, it may include initial conditions, model errors, and empirical parameters x = ( p T q u,v q cloud q snow O 3 T soil q soil ) T p = ( p 1 p N ) T T = ( T 1 T N ) T... 3
4 Uncertainty Defines how reliable is the state vector estimate Measure of missing knowledge Important for decision- making Classical information theory: - uncertainty is related to probability and entropy - probability and entropy are measures of incomplete knowledge Hurricane Ike (008) wind speed probability Entropy and order 4
5 What is data assimila1on? How can we improve model prediction? o o o By improving general parameters of a modeling system (IC, ME, EP) Also, could improve model equations: include missing processes, coupling, spatiotemporal resolution (if discrete), How to improve IC, ME, EP? Use observations (measurements) as a source of information about the real- world. If we believe that a model has a skill, could also use past model performances as an additional information: (1) If model has no skill, use observations as the only source of information about the real world, employ statistics. () If a model have a meaningful role, use both the model and observations as sources of information. Employ a combination of physics/dynamics, probability, and statistics Mathematical method used to blend the information from models and observations is called data assimilation. Data assimilation has a goal of producing optimal estimates of the state and its uncertainty. 5
6 Data assimila1on process Observations Guess forecast Analysis Various observation types and model forecast are combined to obtain an improved model state (analysis)
7 Forecast Assume that forecast is a dynamic- stochastic process dx t = m(x t,t)dt + g(x t,t)dβ t m = Dynamics/model time evolution g = Stochastic forcing x β = State vector = Random vector The probability density (p) evolution of the process is described by the Fokker- Planck equation p(x,t) t + [ p(x,t)m(x,t) ] x = 1 p(x,t)g (x,t) x - - Diffusion equation for pdfs Huge dimensions and numerous unknowns, not used in practice Information from the model is fundamentally probabilistic 7
8 Observa1ons h y ε Assume that observed variables are nonlinearly related to model variables Observations include instrument and representativeness errors = Nonlinear mapping from model to observations = Observation vector = Observation error y = h(x t ) + ε Given the probabilistic character of the model state (x t ) and the existence of observation errors, the observation transformation equation implies the probabilistic character of observations 8
9 Data assimila1on is probabilis1c Model equations are imperfect. Also, IC, ME, EP are not perfectly known, implying model forecast will have an error Observations are imperfect: instrument errors, representativeness error (unexplained by the model Since observations and model forecast are input to data assimilation, DA has also an error (e.g., insufficient knowledge of the input implies insufficient knowledge of the output) Uncertainties and imperfect knowledge are best measured by probability o o make an assumption regarding adequate probability, or, (if possible) form a histogram and deduce best probability distribution 9
10 Bayesian principle in data assimila1on Bayes theorem: create independent probabilities p(x Y ) = p(y X)p(X) p(y ) o o o X = State variable, Y = Observations p(x) Prior PDF p(x Y) Conditional PDF It is implicitly assumed that it is easier to calculate the prior and the conditional PDFs than the joint PDF [p(x,y)] A learning algorithm: probability estimate is updated as additional evidence is acquired 10
11 Prior and condi1onal probability density func1ons Prior : Defines the knowledge about dynamical state before new observations (Y N ) are assimilated p(x ) = p(x Y N 1 Y 1 ) o Y N-1 Y 1 = Old observations o Y N = New observation not yet used Conditional probability of new observations with respect to the prior state p(y N X ) o Y N = New observations 11
12 Gaussian assump1on Probability Density Function (PDF) can be highly nonlinear, or with numerous unknown parameters that need to be estimated One of the simplest and most widely applicable PDFs is the Gaussian PDF o Errors of physical processes tend to accumulate near zero (e.g., small errors dominate) o Gaussian PDF has the smallest number of unknown parameters (e.g., mean and covariance first two moments) 1 " (z µ) p(z) = exp$ σ π # σ % ' & 1
13 Other relevant non- Gaussian PDFs Double exponential (Laplacian) PDF p L (x µ,b) = 1 b exp x µ b Wind speed errors can be described using Laplacian PDF Sharp gradient fields (e.g., atmospheric fronts) exhibit Laplacian PDF There are other non- Gaussian (skewed) PDFs that are relevant: - Lognormal PDF (humidity, cloud variables) - Gamma function PDFs (precipitation) 13
14 Gaussian prior and condi1onal PDFs Multivariate Gaussian PDFs prior conditional # p(x) exp% 1 x x f $ # p( y x) exp% 1 $ y h(x) ( ) T P 1 f ( x x f ) & ( ' ( ) T R 1 ( y h(x) ) & ( ' o x = state variable, P f = covariance o y = observations, R = observation covariance o h = nonlinear observation operator (mapping from model to observations) } } } Covariance is independent of x Only mean and covariance are required Zero mean is implicitly assumed in the above equation 14
15 Data assimila1on with Gaussian PDFs Maximum a- posteriori estimate: Find optimal state X opt that maximizes the posterior probability density function p(x Y) X opt = argmax x p(x Y ) Minimum variance estimate: Find optimal state X opt with the smallest error (variance) [L= loss function of conditional mean] X opt = argmin x E( L[E(x y)] ) Both estimates are identical for Gaussian PDF, otherwise they differ 15
16 Gaussian posterior PDF and cost func1on Given the prior and conditional PDF, the posterior is # p(x y) exp% 1 $ y h(x) ( ) T R 1 ( y h(x) ) 1 x x f ( ) T P 1 ( f x x f ) & ( ' Log- likelihood function (also referred to as the cost function) f (x) = log p(x y) = 1 " # x x f $ % T Pf 1 " # x x f $ % + 1 " # y h(x) $ %T R 1 " # y h(x) $ % Consequence: Minimize cost function = Maximize posterior PDF X opt = argmax x p(x y) = argmin f (x) x 16
17 Op1mality condi1ons for the minimum (1) First variation equal to zero δ f (x) = 0 () Second variation greater than zero δ f (x) > 0 δ f (x) > 0 δ f (x) = 0 Taylor expansion for function with higher- order derivatives: f (x + δ x) f (x) = δ f (x) + 1! δ f (x) + 1 3! δ 3 f (x) + Data assimilation typically utilizes only first two variations Keeping more terms improves nonlinear capability of DA 17
18 First and second varia1on of a func1on First variation δ f (x) = f (x) f (x),δ x = x x T δ x Second variation δ f (x) = δ[δ f (x)] Neglect second variation of x: δ x 0 δ f (x) δ x, f (x) x δ x = δ x ( ) T f (x) x δ x First and second variation of a function imply the use of inner products 18
19 Minimize cost function: Op1mal solu1on f (x) = 1 x x f T Pf 1 x x f + 1 [ y h(x) ] T R 1 [ y h(x) ] (1) δ f (x) = 0 δ f (x) = δ x P f 1 [ ] T P f 1 { x x f + H T R 1 [ y h(x) ]} x x f + H T R 1 [ y h(x) ] = 0 () δ f (x) > 0 δ f (x) = δ x [ ] T δ P f 1 { x x f + H T R 1 [ y h(x) ]} = δ x { }δ x [ ] T P f 1 + H T R 1 H Since P 1 f + H T R 1 H minimum exists. is positive definite and symmetric, the 19
20 One- point DA algorithm deriva1on Minimize quadratic cost function: f (x) = 1 x x f T Pf 1 x x f (1) δ f (x) = 0 P f [ y Hx ] T R 1 [ y Hx] x x f H T R 1 [ y Hx] = 0 x = (I + P f H T R 1 H ) 1 (x f + P f H T R 1 y) One- point DA with observations at a grid- point H I P f σ f R σ R x a = (I + σ f σ R ) 1 (x f + σ f σ R y) 0
21 One- point DA (1) x a = (I + σ f σ R ) 1 (x f + σ f σ R y) x a = σ R σ f + σ R x σ f + f σ f + σ R y x a = α x f + βy α + β =? Analysis is a linear combination of the first guess and observation vectors, or analysis is an interpolation from observation and first guess uncertainty defines the interpolation weights 1
22 One- point DA () Note that interpolation weights are normalized: σ R σ f + σ R + σ f σ f + σ R = σ R + σ f σ f + σ = 1 α + β = 1 R x a = α x f + (1 α )y α = σ R σ f + σ R Normalization of weights assures that the analysis will be between the guess and the observation α = σ R σ f + σ = 1 R σ f σ R +1 = 1 1+ σ f σ R 1 Only the ratio between uncertainties is important!
23 One- point DA (3) x a = σ R σ f + σ R x f + 1 σ R σ f + σ R y Observation error Forecast error a = σ R b = σ f (1) Large confidence in observations: a > b () Equal confidence in observations and first guess: a = b First guess x b b Optimal analysis First guess x b b Optimal analysis x a a x a a y Observation y Observation Interpretation of data assimilation is simple, the complexity comes from high dimensional state and nonlinear operators 3
24 Example 1 1 x a = 1+ σ f σ R σ f σ x f + R 1+ σ f σ R y x x f = (0,0) x a = (5,0) x x y = (10,0) σ f (1) Given = 1, what are the coordinates of x σ a? R x a 1 1 = 1+1 (0,0) (10,0) = 1 (0,0) + 1 (10,0) = , = (5,0) Equal confidence implies the analysis is in the middle 4
25 Example 1 x a = 1+ σ f σ R σ f σ x f + R 1+ σ f σ R y x x f = (0,0) x a = (9,0) x x y = (10,0) σ () Given f = 3, what are the coordinates of x a? σ R x a 1 9 = 1+ 9 (0,0) (10,0) = 1 10 (0,0) (10,0) = , = (9,0) More confidence in observation implies the analysis is close to observation 5
26 Example 3 1 x a = 1+ σ f σ R σ f σ x f + R 1+ σ f σ R y x x a = (1,0) x x f = (0,0) x y = (10,0) σ f (3) Given = 1, what are the coordinates of x σ R 3 a? 1 1 x a = 1+ 1 (0,0) (10,0) = 9 10 (0,0) (10,0) = , = (1,0) More confidence in first guess implies the analysis is close to the guess 6
27 Challenges of realis1c data assimila1on High dimensionality of state and observations o impacts degrees of freedom of forecast error covariance, acceptable choices of DA methodology Nonlinearity of simulated physical processes and observation operators o Need capability to handle nonlinearities Computation o Costly integration of realistic forecast models, matrix inversion Observation errors o Bias correction, correlated observation errors Multivariate character of the DA problem o Dynamical stability of the analysis 7
28 Prac1cal data assimila1on algorithms: Basic methods Variational data assimilation (3D- Var, 4D- Var) o Maximum a- posteriori estimate o Iterative minimization has advantage for nonlinear operators o Forecast uncertainty pre- defined (e.g., static) o Forecast uncertainty has all degrees of freedom o Employs an adjoint (e.g., transpose) operator Ensemble Kalman filter data assimilation (EnKF, EnSRF) o Minimum variance o Assumed linear KF solution o Statistical sampling of forecast error covariance o Forecast uncertainty is flow- dependent (e.g., ensemble forecasts) o Reduced number of degrees of freedom o No need for an adjoint, use difference of nonlinear functions
29 Varia1onal cost func1on 3D- Var cost function (one observation time): f (x) = 1 x x f T Pf 1 x x f + 1 [ y h(x) ] T R 1 [ y h(x) ] 4D- Var cost function (sum over observation times): f (x) = 1 x x f T Pf 1 x x f + 1 T k=1 [ y h(m(x)) ] T k R 1 k y h(m(x)) [ ] k o 4D- Var allows smooth transition to forecast after data assimilation o 4D- Var analysis is more costly to calculate than 3D- Var o 3D- Var can considerably improve with better definition of the background error covariance (P f )
30 What is ensemble data assimila1on? (1) Forecast uncertainty is calculated from multiple model forecasts (ensembles) FCST ERROR M(x) } Forecast uncertainty Initial uncertainty { t t+1 TIME () Analysis employs the ensemble information and produces uncertainty Dynamical model (phase) space Observation space K(x) X X X 30
31 Flow- dependent forecast error covariance grid- point x time obs obs 1 Geographically distant observations can bring more information than close- by observations, if in a dynamically significant region
32 Impact of sta1c error covariance grid- point x Correlation length scale time obs obs 1 Low- valued information (obs ) will be assimilated instead of a high- valued information (obs 1 )
33 Reduced rank aspect of (ensemble) data assimila1on Only a limited number of ensembles can be calculated: - high cost of forecast model integra1on - high- dimensional state vector Full model (phase) space Ensemble (phase) space Observations outside ensemble space cannot be assimilated Hybrid varia1onal- ensemble methods improve DOF problem by crea1ng uncertainty in all parts of the model space (e.g., combine flow- dependent and sta1c error covariance)
34 Why is forecast error covariance important? Singular Value Decomposition [Golub and van Loan (1989)] P 1/ f = VΣW T T = σ i v i w i i P f = P 1/ 1/ f ( P f ) T = VΣW T ( W ΣV T ) = VΣ V T = σ i i v i v i T x a = x f + P f H T z obs = H T ( HP f H T + R) 1 [y h(x f )] = P f z obs ( HP f H T + R) 1 [y h(x f )] The analysis update is x a x f = i σ i v i v i T z obs = i µ i v i µ i = σ i v i T z obs Analysis update is a linear combina1on of forecast error covariance singular vectors Analysis increments are defined in the subspace spanned by forecast error covariance singular vectors
35 Prac1cal data assimila1on algorithms Hybrid methods Hybrid variational- ensemble data assimilation J(x,α ) = P f 1/ = β f o Combined static and flow- dependent error covariance o Iterative minimization o Sequential method ( δ x f ) T 1 P VAR ( δ x f ) + β e (α ) T ( P ENS L) 1 (α ) + 1 y Hδ xtot T R 1 y Hδ x tot δx tot = δx f + k α k P ENS 1/ " # $ % k 1 β f + 1 β e =1 4D- EN- VAR o 4- D control variable: simultaneous adjustment in time and space o Sequential method o Increased dimension of the control vector o Reduced rank, but could be used as a hybrid
36 Future of data assimila1on Practice: o great need for blending information from observations and models o Coupled data assimilation (DA for coupled modeling systems) o high temporal frequency observations (e.g., geostationary satellite) Theory: o more general mathematical formalism (fewer auxiliary parameters) o reduce number of assumptions (e.g., Gaussian pdf), 36
Fundamentals of Data Assimila1on
2015 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO August 11-14, 2015 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University
More informationRelative Merits of 4D-Var and Ensemble Kalman Filter
Relative Merits of 4D-Var and Ensemble Kalman Filter Andrew Lorenc Met Office, Exeter International summer school on Atmospheric and Oceanic Sciences (ISSAOS) "Atmospheric Data Assimilation". August 29
More informationPar$cle Filters Part I: Theory. Peter Jan van Leeuwen Data- Assimila$on Research Centre DARC University of Reading
Par$cle Filters Part I: Theory Peter Jan van Leeuwen Data- Assimila$on Research Centre DARC University of Reading Reading July 2013 Why Data Assimila$on Predic$on Model improvement: - Parameter es$ma$on
More information4. DATA ASSIMILATION FUNDAMENTALS
4. DATA ASSIMILATION FUNDAMENTALS... [the atmosphere] "is a chaotic system in which errors introduced into the system can grow with time... As a consequence, data assimilation is a struggle between chaotic
More information(Extended) Kalman Filter
(Extended) Kalman Filter Brian Hunt 7 June 2013 Goals of Data Assimilation (DA) Estimate the state of a system based on both current and all past observations of the system, using a model for the system
More informationA new Hierarchical Bayes approach to ensemble-variational data assimilation
A new Hierarchical Bayes approach to ensemble-variational data assimilation Michael Tsyrulnikov and Alexander Rakitko HydroMetCenter of Russia College Park, 20 Oct 2014 Michael Tsyrulnikov and Alexander
More informationFundamentals of Data Assimilation
National Center for Atmospheric Research, Boulder, CO USA GSI Data Assimilation Tutorial - June 28-30, 2010 Acknowledgments and References WRFDA Overview (WRF Tutorial Lectures, H. Huang and D. Barker)
More informationVariational data assimilation
Background and methods NCEO, Dept. of Meteorology, Univ. of Reading 710 March 2018, Univ. of Reading Bayes' Theorem Bayes' Theorem p(x y) = posterior distribution = p(x) p(y x) p(y) prior distribution
More informationUNDERSTANDING DATA ASSIMILATION APPLICATIONS TO HIGH-LATITUDE IONOSPHERIC ELECTRODYNAMICS
Monday, June 27, 2011 CEDAR-GEM joint workshop: DA tutorial 1 UNDERSTANDING DATA ASSIMILATION APPLICATIONS TO HIGH-LATITUDE IONOSPHERIC ELECTRODYNAMICS Tomoko Matsuo University of Colorado, Boulder Space
More informationIntroduction to Particle Filters for Data Assimilation
Introduction to Particle Filters for Data Assimilation Mike Dowd Dept of Mathematics & Statistics (and Dept of Oceanography Dalhousie University, Halifax, Canada STATMOS Summer School in Data Assimila5on,
More informationIn the derivation of Optimal Interpolation, we found the optimal weight matrix W that minimizes the total analysis error variance.
hree-dimensional variational assimilation (3D-Var) In the derivation of Optimal Interpolation, we found the optimal weight matrix W that minimizes the total analysis error variance. Lorenc (1986) showed
More information1. Current atmospheric DA systems 2. Coupling surface/atmospheric DA 3. Trends & ideas
1 Current issues in atmospheric data assimilation and its relationship with surfaces François Bouttier GAME/CNRM Météo-France 2nd workshop on remote sensing and modeling of surface properties, Toulouse,
More informationDynamic System Identification using HDMR-Bayesian Technique
Dynamic System Identification using HDMR-Bayesian Technique *Shereena O A 1) and Dr. B N Rao 2) 1), 2) Department of Civil Engineering, IIT Madras, Chennai 600036, Tamil Nadu, India 1) ce14d020@smail.iitm.ac.in
More informationLagrangian Data Assimilation and Its Application to Geophysical Fluid Flows
Lagrangian Data Assimilation and Its Application to Geophysical Fluid Flows Laura Slivinski June, 3 Laura Slivinski (Brown University) Lagrangian Data Assimilation June, 3 / 3 Data Assimilation Setup:
More informationConvective-scale data assimilation in the Weather Research and Forecasting model using a nonlinear ensemble filter
Convective-scale data assimilation in the Weather Research and Forecasting model using a nonlinear ensemble filter Jon Poterjoy, Ryan Sobash, and Jeffrey Anderson National Center for Atmospheric Research
More informationApplication of the Ensemble Kalman Filter to History Matching
Application of the Ensemble Kalman Filter to History Matching Presented at Texas A&M, November 16,2010 Outline Philosophy EnKF for Data Assimilation Field History Match Using EnKF with Covariance Localization
More informationAn Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation
An Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation A. Shah 1,2, M. E. Gharamti 1, L. Bertino 1 1 Nansen Environmental and Remote Sensing Center 2 University of Bergen
More informationWeak Constraints 4D-Var
Weak Constraints 4D-Var Yannick Trémolet ECMWF Training Course - Data Assimilation May 1, 2012 Yannick Trémolet Weak Constraints 4D-Var May 1, 2012 1 / 30 Outline 1 Introduction 2 The Maximum Likelihood
More informationConsider the joint probability, P(x,y), shown as the contours in the figure above. P(x) is given by the integral of P(x,y) over all values of y.
ATMO/OPTI 656b Spring 009 Bayesian Retrievals Note: This follows the discussion in Chapter of Rogers (000) As we have seen, the problem with the nadir viewing emission measurements is they do not contain
More informationShort tutorial on data assimilation
Mitglied der Helmholtz-Gemeinschaft Short tutorial on data assimilation 23 June 2015 Wolfgang Kurtz & Harrie-Jan Hendricks Franssen Institute of Bio- and Geosciences IBG-3 (Agrosphere), Forschungszentrum
More informationEnsemble Data Assimilation and Uncertainty Quantification
Ensemble Data Assimilation and Uncertainty Quantification Jeff Anderson National Center for Atmospheric Research pg 1 What is Data Assimilation? Observations combined with a Model forecast + to produce
More informationCross-validation methods for quality control, cloud screening, etc.
Cross-validation methods for quality control, cloud screening, etc. Olaf Stiller, Deutscher Wetterdienst Are observations consistent Sensitivity functions with the other observations? given the background
More informationAdvancing Data AssimilaJon Science for OperaJonal Hydrology: Methodology, ComputaJon, and Algorithms
2014 CAHMDA & HEPEX- DAFOH Workshop September 8-12, 2014 The University of Texas at AusJn Advancing Data AssimilaJon Science for OperaJonal Hydrology: Methodology, ComputaJon, and Algorithms Milija Zupanski
More informationGaussian Process Approximations of Stochastic Differential Equations
Gaussian Process Approximations of Stochastic Differential Equations Cédric Archambeau Dan Cawford Manfred Opper John Shawe-Taylor May, 2006 1 Introduction Some of the most complex models routinely run
More informationProbabilistic numerics for deep learning
Presenter: Shijia Wang Department of Engineering Science, University of Oxford rning (RLSS) Summer School, Montreal 2017 Outline 1 Introduction Probabilistic Numerics 2 Components Probabilistic modeling
More informationData assimilation; comparison of 4D-Var and LETKF smoothers
Data assimilation; comparison of 4D-Var and LETKF smoothers Eugenia Kalnay and many friends University of Maryland CSCAMM DAS13 June 2013 Contents First part: Forecasting the weather - we are really getting
More informationCan hybrid-4denvar match hybrid-4dvar?
Comparing ensemble-variational assimilation methods for NWP: Can hybrid-4denvar match hybrid-4dvar? WWOSC, Montreal, August 2014. Andrew Lorenc, Neill Bowler, Adam Clayton, David Fairbairn and Stephen
More informationAutonomous Navigation for Flying Robots
Computer Vision Group Prof. Daniel Cremers Autonomous Navigation for Flying Robots Lecture 6.2: Kalman Filter Jürgen Sturm Technische Universität München Motivation Bayes filter is a useful tool for state
More information2D Image Processing. Bayes filter implementation: Kalman filter
2D Image Processing Bayes filter implementation: Kalman filter Prof. Didier Stricker Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche Intelligenz http://av.dfki.de
More informationA Note on the Particle Filter with Posterior Gaussian Resampling
Tellus (6), 8A, 46 46 Copyright C Blackwell Munksgaard, 6 Printed in Singapore. All rights reserved TELLUS A Note on the Particle Filter with Posterior Gaussian Resampling By X. XIONG 1,I.M.NAVON 1,2 and
More informationKalman Filter and Ensemble Kalman Filter
Kalman Filter and Ensemble Kalman Filter 1 Motivation Ensemble forecasting : Provides flow-dependent estimate of uncertainty of the forecast. Data assimilation : requires information about uncertainty
More informationNumerical Weather Prediction: Data assimilation. Steven Cavallo
Numerical Weather Prediction: Data assimilation Steven Cavallo Data assimilation (DA) is the process estimating the true state of a system given observations of the system and a background estimate. Observations
More informationConvergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit
Convergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit Evan Kwiatkowski, Jan Mandel University of Colorado Denver December 11, 2014 OUTLINE 2 Data Assimilation Bayesian Estimation
More informationEnhancing information transfer from observations to unobserved state variables for mesoscale radar data assimilation
Enhancing information transfer from observations to unobserved state variables for mesoscale radar data assimilation Weiguang Chang and Isztar Zawadzki Department of Atmospheric and Oceanic Sciences Faculty
More informationIntroduction to Ensemble Kalman Filters and the Data Assimilation Research Testbed
Introduction to Ensemble Kalman Filters and the Data Assimilation Research Testbed Jeffrey Anderson, Tim Hoar, Nancy Collins NCAR Institute for Math Applied to Geophysics pg 1 What is Data Assimilation?
More informationHow 4DVAR can benefit from or contribute to EnKF (a 4DVAR perspective)
How 4DVAR can benefit from or contribute to EnKF (a 4DVAR perspective) Dale Barker WWRP/THORPEX Workshop on 4D-Var and Ensemble Kalman Filter Intercomparisons Sociedad Cientifica Argentina, Buenos Aires,
More informationEnsemble Data Assimila.on and Uncertainty Quan.fica.on
Ensemble Data Assimila.on and Uncertainty Quan.fica.on Jeffrey Anderson, Alicia Karspeck, Tim Hoar, Nancy Collins, Kevin Raeder, Steve Yeager Na.onal Center for Atmospheric Research Ocean Sciences Mee.ng
More informationStability of Ensemble Kalman Filters
Stability of Ensemble Kalman Filters Idrissa S. Amour, Zubeda Mussa, Alexander Bibov, Antti Solonen, John Bardsley, Heikki Haario and Tuomo Kauranne Lappeenranta University of Technology University of
More informationCS491/691: Introduction to Aerial Robotics
CS491/691: Introduction to Aerial Robotics Topic: State Estimation Dr. Kostas Alexis (CSE) World state (or system state) Belief state: Our belief/estimate of the world state World state: Real state of
More informationMethods of Data Assimilation and Comparisons for Lagrangian Data
Methods of Data Assimilation and Comparisons for Lagrangian Data Chris Jones, Warwick and UNC-CH Kayo Ide, UCLA Andrew Stuart, Jochen Voss, Warwick Guillaume Vernieres, UNC-CH Amarjit Budiraja, UNC-CH
More informationGaussian Process Approximations of Stochastic Differential Equations
Gaussian Process Approximations of Stochastic Differential Equations Cédric Archambeau Centre for Computational Statistics and Machine Learning University College London c.archambeau@cs.ucl.ac.uk CSML
More informationForecasting and data assimilation
Supported by the National Science Foundation DMS Forecasting and data assimilation Outline Numerical models Kalman Filter Ensembles Douglas Nychka, Thomas Bengtsson, Chris Snyder Geophysical Statistics
More informationVariational data assimilation of lightning with WRFDA system using nonlinear observation operators
Variational data assimilation of lightning with WRFDA system using nonlinear observation operators Virginia Tech, Blacksburg, Virginia Florida State University, Tallahassee, Florida rstefane@vt.edu, inavon@fsu.edu
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing
More informationAn introduction to data assimilation. Eric Blayo University of Grenoble and INRIA
An introduction to data assimilation Eric Blayo University of Grenoble and INRIA Data assimilation, the science of compromises Context characterizing a (complex) system and/or forecasting its evolution,
More informationImproving GFS 4DEnVar Hybrid Data Assimilation System Using Time-lagged Ensembles
Improving GFS 4DEnVar Hybrid Data Assimilation System Using Time-lagged Ensembles Bo Huang and Xuguang Wang School of Meteorology University of Oklahoma, Norman, OK, USA Acknowledgement: Junkyung Kay (OU);
More information4DEnVar. Four-Dimensional Ensemble-Variational Data Assimilation. Colloque National sur l'assimilation de données
Four-Dimensional Ensemble-Variational Data Assimilation 4DEnVar Colloque National sur l'assimilation de données Andrew Lorenc, Toulouse France. 1-3 décembre 2014 Crown copyright Met Office 4DEnVar: Topics
More informationA Spectral Approach to Linear Bayesian Updating
A Spectral Approach to Linear Bayesian Updating Oliver Pajonk 1,2, Bojana V. Rosic 1, Alexander Litvinenko 1, and Hermann G. Matthies 1 1 Institute of Scientific Computing, TU Braunschweig, Germany 2 SPT
More informationDART Ini)al Condi)ons for a Refined Grid CAM- SE Forecast of Hurricane Katrina. Kevin Raeder (IMAGe) Colin Zarzycki (ASP)
DART Ini)al Condi)ons for a Refined Grid CAM- SE Forecast of Hurricane Katrina Kevin Raeder (IMAGe) Colin Zarzycki (ASP) 1 Mo)va)on Thousands of processors on current supercomputers. - > new CAM dynamical
More informationInverse problems and uncertainty quantification in remote sensing
1 / 38 Inverse problems and uncertainty quantification in remote sensing Johanna Tamminen Finnish Meterological Institute johanna.tamminen@fmi.fi ESA Earth Observation Summer School on Earth System Monitoring
More informationLecture 3: Pattern Classification
EE E6820: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 1 2 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mixtures
More informationEnKF Review. P.L. Houtekamer 7th EnKF workshop Introduction to the EnKF. Challenges. The ultimate global EnKF algorithm
Overview 1 2 3 Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation 6th EnKF Purpose EnKF equations localization After the 6th EnKF (2014), I decided with Prof. Zhang to summarize progress
More informationThe priority program SPP1167 Quantitative Precipitation Forecast PQP and the stochastic view of weather forecasting
The priority program SPP1167 Quantitative Precipitation Forecast PQP and the stochastic view of weather forecasting Andreas Hense 9. November 2007 Overview The priority program SPP1167: mission and structure
More informationMaximum Likelihood Ensemble Filter Applied to Multisensor Systems
Maximum Likelihood Ensemble Filter Applied to Multisensor Systems Arif R. Albayrak a, Milija Zupanski b and Dusanka Zupanski c abc Colorado State University (CIRA), 137 Campus Delivery Fort Collins, CO
More information2D Image Processing. Bayes filter implementation: Kalman filter
2D Image Processing Bayes filter implementation: Kalman filter Prof. Didier Stricker Dr. Gabriele Bleser Kaiserlautern University http://ags.cs.uni-kl.de/ DFKI Deutsches Forschungszentrum für Künstliche
More informationBasic Verification Concepts
Basic Verification Concepts Barbara Brown National Center for Atmospheric Research Boulder Colorado USA bgb@ucar.edu Basic concepts - outline What is verification? Why verify? Identifying verification
More informationData Modeling & Analysis Techniques. Probability & Statistics. Manfred Huber
Data Modeling & Analysis Techniques Probability & Statistics Manfred Huber 2017 1 Probability and Statistics Probability and statistics are often used interchangeably but are different, related fields
More informationECE521 week 3: 23/26 January 2017
ECE521 week 3: 23/26 January 2017 Outline Probabilistic interpretation of linear regression - Maximum likelihood estimation (MLE) - Maximum a posteriori (MAP) estimation Bias-variance trade-off Linear
More informationKalman filtering and friends: Inference in time series models. Herke van Hoof slides mostly by Michael Rubinstein
Kalman filtering and friends: Inference in time series models Herke van Hoof slides mostly by Michael Rubinstein Problem overview Goal Estimate most probable state at time k using measurement up to time
More informationEnsemble forecasting and flow-dependent estimates of initial uncertainty. Martin Leutbecher
Ensemble forecasting and flow-dependent estimates of initial uncertainty Martin Leutbecher acknowledgements: Roberto Buizza, Lars Isaksen Flow-dependent aspects of data assimilation, ECMWF 11 13 June 2007
More informationParametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationRobust Ensemble Filtering With Improved Storm Surge Forecasting
Robust Ensemble Filtering With Improved Storm Surge Forecasting U. Altaf, T. Buttler, X. Luo, C. Dawson, T. Mao, I.Hoteit Meteo France, Toulouse, Nov 13, 2012 Project Ensemble data assimilation for storm
More informationLocal Ensemble Transform Kalman Filter
Local Ensemble Transform Kalman Filter Brian Hunt 11 June 2013 Review of Notation Forecast model: a known function M on a vector space of model states. Truth: an unknown sequence {x n } of model states
More informationGraphical Models for Collaborative Filtering
Graphical Models for Collaborative Filtering Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Sequence modeling HMM, Kalman Filter, etc.: Similarity: the same graphical model topology,
More informationVariational Data Assimilation Current Status
Variational Data Assimilation Current Status Eĺıas Valur Hólm with contributions from Mike Fisher and Yannick Trémolet ECMWF NORDITA Solar and stellar dynamos and cycles October 2009 Eĺıas Valur Hólm (ECMWF)
More informationCS 6140: Machine Learning Spring What We Learned Last Week. Survey 2/26/16. VS. Model
Logis@cs CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Assignment
More informationDART Tutorial Sec'on 5: Comprehensive Filtering Theory: Non-Iden'ty Observa'ons and the Joint Phase Space
DART Tutorial Sec'on 5: Comprehensive Filtering Theory: Non-Iden'ty Observa'ons and the Joint Phase Space UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on.
More informationCS 6140: Machine Learning Spring 2016
CS 6140: Machine Learning Spring 2016 Instructor: Lu Wang College of Computer and Informa?on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: luwang@ccs.neu.edu Logis?cs Assignment
More informationAdaptive Data Assimilation and Multi-Model Fusion
Adaptive Data Assimilation and Multi-Model Fusion Pierre F.J. Lermusiaux, Oleg G. Logoutov and Patrick J. Haley Jr. Mechanical Engineering and Ocean Science and Engineering, MIT We thank: Allan R. Robinson
More informationEnsemble square-root filters
Ensemble square-root filters MICHAEL K. TIPPETT International Research Institute for climate prediction, Palisades, New Yor JEFFREY L. ANDERSON GFDL, Princeton, New Jersy CRAIG H. BISHOP Naval Research
More informationLagrangian Data Assimilation and Manifold Detection for a Point-Vortex Model. David Darmon, AMSC Kayo Ide, AOSC, IPST, CSCAMM, ESSIC
Lagrangian Data Assimilation and Manifold Detection for a Point-Vortex Model David Darmon, AMSC Kayo Ide, AOSC, IPST, CSCAMM, ESSIC Background Data Assimilation Iterative process Forecast Analysis Background
More informationMobile Robot Localization
Mobile Robot Localization 1 The Problem of Robot Localization Given a map of the environment, how can a robot determine its pose (planar coordinates + orientation)? Two sources of uncertainty: - observations
More informationDART Tutorial Part IV: Other Updates for an Observed Variable
DART Tutorial Part IV: Other Updates for an Observed Variable UCAR The Na'onal Center for Atmospheric Research is sponsored by the Na'onal Science Founda'on. Any opinions, findings and conclusions or recommenda'ons
More informationMachine Learning for Signal Processing Bayes Classification and Regression
Machine Learning for Signal Processing Bayes Classification and Regression Instructor: Bhiksha Raj 11755/18797 1 Recap: KNN A very effective and simple way of performing classification Simple model: For
More informationDART_LAB Tutorial Section 5: Adaptive Inflation
DART_LAB Tutorial Section 5: Adaptive Inflation UCAR 14 The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings and conclusions or recommendations
More informationEnsemble 4DVAR and observa3on impact study with the GSIbased hybrid ensemble varia3onal data assimila3on system. for the GFS
Ensemble 4DVAR and observa3on impact study with the GSIbased hybrid ensemble varia3onal data assimila3on system for the GFS Xuguang Wang University of Oklahoma, Norman, OK xuguang.wang@ou.edu Ting Lei,
More informationM.Sc. in Meteorology. Numerical Weather Prediction
M.Sc. in Meteorology UCD Numerical Weather Prediction Prof Peter Lynch Meteorology & Climate Cehtre School of Mathematical Sciences University College Dublin Second Semester, 2005 2006. Text for the Course
More informationTangent-linear and adjoint models in data assimilation
Tangent-linear and adjoint models in data assimilation Marta Janisková and Philippe Lopez ECMWF Thanks to: F. Váňa, M.Fielding 2018 Annual Seminar: Earth system assimilation 10-13 September 2018 Tangent-linear
More informationDATA ASSIMILATION FOR FLOOD FORECASTING
DATA ASSIMILATION FOR FLOOD FORECASTING Arnold Heemin Delft University of Technology 09/16/14 1 Data assimilation is the incorporation of measurement into a numerical model to improve the model results
More informationSmoothers: Types and Benchmarks
Smoothers: Types and Benchmarks Patrick N. Raanes Oxford University, NERSC 8th International EnKF Workshop May 27, 2013 Chris Farmer, Irene Moroz Laurent Bertino NERSC Geir Evensen Abstract Talk builds
More informationSome Applications of WRF/DART
Some Applications of WRF/DART Chris Snyder, National Center for Atmospheric Research Mesoscale and Microscale Meteorology Division (MMM), and Institue for Mathematics Applied to Geoscience (IMAGe) WRF/DART
More informationNew Fast Kalman filter method
New Fast Kalman filter method Hojat Ghorbanidehno, Hee Sun Lee 1. Introduction Data assimilation methods combine dynamical models of a system with typically noisy observations to obtain estimates of the
More informationHierarchical Bayes Ensemble Kalman Filter
Hierarchical Bayes Ensemble Kalman Filter M Tsyrulnikov and A Rakitko HydroMetCenter of Russia Wrocław, 7 Sep 2015 M Tsyrulnikov and A Rakitko (HMC) Hierarchical Bayes Ensemble Kalman Filter Wrocław, 7
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationThe Local Ensemble Transform Kalman Filter (LETKF) Eric Kostelich. Main topics
The Local Ensemble Transform Kalman Filter (LETKF) Eric Kostelich Arizona State University Co-workers: Istvan Szunyogh, Brian Hunt, Ed Ott, Eugenia Kalnay, Jim Yorke, and many others http://www.weatherchaos.umd.edu
More informationSignal Processing - Lecture 7
1 Introduction Signal Processing - Lecture 7 Fitting a function to a set of data gathered in time sequence can be viewed as signal processing or learning, and is an important topic in information theory.
More informationEfficient Data Assimilation for Spatiotemporal Chaos: a Local Ensemble Transform Kalman Filter
Efficient Data Assimilation for Spatiotemporal Chaos: a Local Ensemble Transform Kalman Filter arxiv:physics/0511236 v1 28 Nov 2005 Brian R. Hunt Institute for Physical Science and Technology and Department
More informationSLAM Techniques and Algorithms. Jack Collier. Canada. Recherche et développement pour la défense Canada. Defence Research and Development Canada
SLAM Techniques and Algorithms Jack Collier Defence Research and Development Canada Recherche et développement pour la défense Canada Canada Goals What will we learn Gain an appreciation for what SLAM
More informationEnKF-based particle filters
EnKF-based particle filters Jana de Wiljes, Sebastian Reich, Wilhelm Stannat, Walter Acevedo June 20, 2017 Filtering Problem Signal dx t = f (X t )dt + 2CdW t Observations dy t = h(x t )dt + R 1/2 dv t.
More informationMachine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 11 Maximum Likelihood as Bayesian Inference Maximum A Posteriori Bayesian Gaussian Estimation Why Maximum Likelihood? So far, assumed max (log) likelihood
More informationData Assimilation: Finding the Initial Conditions in Large Dynamical Systems. Eric Kostelich Data Mining Seminar, Feb. 6, 2006
Data Assimilation: Finding the Initial Conditions in Large Dynamical Systems Eric Kostelich Data Mining Seminar, Feb. 6, 2006 kostelich@asu.edu Co-Workers Istvan Szunyogh, Gyorgyi Gyarmati, Ed Ott, Brian
More informationAdaptive Inflation for Ensemble Assimilation
Adaptive Inflation for Ensemble Assimilation Jeffrey Anderson IMAGe Data Assimilation Research Section (DAReS) Thanks to Ryan Torn, Nancy Collins, Tim Hoar, Hui Liu, Kevin Raeder, Xuguang Wang Anderson:
More informationThe ECMWF Hybrid 4D-Var and Ensemble of Data Assimilations
The Hybrid 4D-Var and Ensemble of Data Assimilations Lars Isaksen, Massimo Bonavita and Elias Holm Data Assimilation Section lars.isaksen@ecmwf.int Acknowledgements to: Mike Fisher and Marta Janiskova
More informationAssessing Potential Impact of Air Pollutant Observations from the Geostationary Satellite on Air Quality Prediction through OSSEs
Assessing Potential Impact of Air Pollutant Observations from the Geostationary Satellite on Air Quality Prediction through OSSEs Ebony Lee 1,2, Seon Ki Park 1,2,3, and Milija Županski4 1 Department of
More informationLecture 6: Multiple Model Filtering, Particle Filtering and Other Approximations
Lecture 6: Multiple Model Filtering, Particle Filtering and Other Approximations Department of Biomedical Engineering and Computational Science Aalto University April 28, 2010 Contents 1 Multiple Model
More informationUniversität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters
More informationGaussian processes. Chuong B. Do (updated by Honglak Lee) November 22, 2008
Gaussian processes Chuong B Do (updated by Honglak Lee) November 22, 2008 Many of the classical machine learning algorithms that we talked about during the first half of this course fit the following pattern:
More informationEnsemble Kalman Filter based snow data assimilation
Ensemble Kalman Filter based snow data assimilation (just some ideas) FMI, Sodankylä, 4 August 2011 Jelena Bojarova Sequential update problem Non-linear state space problem Tangent-linear state space problem
More informationDevelopment of the NCAR 4D-REKF System and Comparison with RTFDDA, DART-EaKF and WRFVAR
Development of the NCAR 4D-REKF System and Comparison with RTFDDA, DART-EaKF and WRFVAR Yubao Liu, Yonghui Wu, Linlin Pan, Al Bourgeois Thanks: Jason Knievel, Scott Swerdlin, Xin Zhang, and Xiang-Yu Huang
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond Department of Biomedical Engineering and Computational Science Aalto University January 26, 2012 Contents 1 Batch and Recursive Estimation
More information