Bayesian Statistics and Data Assimilation. Jonathan Stroud. Department of Statistics The George Washington University
|
|
- Patrick Briggs
- 5 years ago
- Views:
Transcription
1 Bayesian Statistics and Data Assimilation Jonathan Stroud Department of Statistics The George Washington University 1
2 Outline Motivation Bayesian Statistics Parameter Estimation in Data Assimilation Combined State and Parameter Estimation within EnKF Physical Parameters Observation Error Variance Observation Error Covariance 2
3 Motivation Physical models and data assimilation systems often involve unknown parameters: physical model parameters error covariance parameters covariance inflation factors localization radius Use data to estimate parameters, either off-line or sequentially. Two common approaches to parameter estimation Maximum likelihood estimation Bayesian estimation 3
4 Statistical Methods for Parameter Estimation Maximum Likelihood approach Specify a likelihood function for the data. Choose parameters to maximize this function. Bayesian approach Parameters are assigned a prior probability distribution. Update the prior distribution using Bayes Theorem. Summarize the parameters using the posterior distribution. 4
5 Bayesian Parameter Estimation A Bayesian model includes the following components (where d = data; α = unknown parameters): Likelihood function: p(d α) Prior distribution: p(α) Posterior distribution (via Bayes Theorem): p(α d) = p(α)p(d α) p(d) The parameters can be summarized using the posterior mean, standard deviation, or 95% posterior intervals. 5
6 Example 1: Normal Data with Unknown Mean Let d 1,...,d n be iid samples from a normal distribution with unknown mean θ and variance v. The likelihood function is p(d θ) = (2πv) n/2 exp { ( d θ) 2 }. 2v/n The standard prior distribution is a normal: θ N(θ b,v b ) p(θ) = (2πv b ) 1/2 exp { (θ θ b) 2 } 2v b Posterior distribution (likelihood prior): p(θ d) exp { ( d θ) 2 2v/n (θ θ b) 2 2v b } 6
7 Example 1: Normal Data with Unknown Mean The posterior distribution is normal θ d N(θ a,v a ) with mean and information θ a = (v/n)θ b +v b d v b +(v/n) v 1 a = v 1 b +(v/n) 1 The posterior mean is a weighted average of the prior mean and the sample mean. The posterior information is the sum of the prior and data information. 7
8 Example 1: Normal Data with Unknown Mean Prior N(0,1) Likelihood N(2,4) Posterior N(0.4,0.8)
9 Example 2: Normal with Unknown Variance Let d 1,...,d n be iid samples from a normal distribution with mean zero and unknown variance v. The likelihood function is p(d v) = (2πv) n/2 exp ( 1 2v The prior distribution is inverse gamma, v IG(m/2,s/2) n i=1 d 2 i p(v) = (s/2)m/2 ( Γ(m/2) v m/2 1 exp s ). 2v ). The posterior distribution is p(v d) v (m+n)/2 1 exp ( s + d 2 i 2v ). 9
10 Example 2: Normal with Unknown Variance The posterior is also inverse gamma v d IG ( m+n 2, s + di 2 2 ). The parameters are updated by adding the sample size and the data sum of squares, respectively. 10
11 Example 2: Normal with Unknown Variance 0.5 Prior IG(5,10) mean= Likelihood IG(10,50) mean= Posterior IG(15,60) mean=
12 Conjugate Priors The normal and inverse gamma priors are called conjugate priors. These occur when the prior and posterior distribution belong to the same family. These are convenient because the distribution can be updated by updating a set of sufficient statistics. Other examples of conjugate priors: Inverse-Wishart, for the covariance matrix of a normal. Normal-Inverse Gamma, for the mean and variance of normal. Dirichlet, for the probabilities of a discrete distribution. 12
13 Sequential Bayesian Estimation If the data are assimilated sequentially, we want to update the parameters α after each new observation d 1,d 2,...,d n. Under the Bayesian approach, this requires calculating the sequence of posterior distributions p(α d 1 ), p(α d 1,d 2 ),..., p(α d 1,d 2,...,d n ) This is done by applying Bayes theorem recursively after each new observation: p(α d 1,...,d k ) p(d k α)p(α d 1,...,d k 1 ), for each k. 13
14 Sequence of Posterior Distributions Posterior t=1 Posterior t= MLE=2.9 Mode=1 Mean= MLE=1 Mode=1.6 Mean= Posterior t=2 MLE=0.2 Mode=0.9 Mean= Posterior t=4 MLE=11 Mode=2.2 Mean= Posterior t=100 MLE=9 Mode=2 Mean= Posterior t=200 MLE=8.6 Mode=2 Mean=
15 Sequential Bayesian Estimates MLEs of alpha Sequential Estimates Posterior Mean Posterior Mode 90% Posterior CI
16 Convergence of the Posterior Distribution 1. The posterior distribution converges to the true parameter value. 2. If the model is correct, and certain regularity conditions hold, the posterior distribution converges to a normal distribution with mean equal to the true value and covariance equal to the asymptotic covariance matrix. 16
17 Parameter Estimation in Data Assimilation Many parameter estimation methods have been proposed for data assimilation systems. Maximum likelihood estimation Dee (1995) and Dee & Da Silva (1999): Error covariances Mitchell & Houtekamer (2000): Error covariances (EnKF) Li, Kalnay & Miyoshi (2007): Variance/Covariance inflation Bayesian estimation Anderson & Anderson (1999): State augmentation Stroud & Bengtsson (2007): Observation error variance Anderson (2007, 2009): Covariance inflation factors Miyoshi (2011): Covariance inflation factors 17
18 Estimation of Physical Parameters State augmentation is used to estimate unknown parameters θ in the physical model M(x t,θ). Define the augmented state vector z t = (x t,θ t ), and the augmented model as x t θ t = M(x t 1,θ t 1 ) θ t 1 + w t. 0 Specify an initial prior distribution, θ 0 p(θ 0 ). Then, standard data assimilation methods are applied to z t to estimate the posterior distribution, p(θ d 1,...,d t ), at each time t. 18
19 Example: Lorenz 63 Model Model equations: dx/dt = σ(y x) dy/dt = ρx y xz dz/dt = xy βz z y x The state vector is x = (x,y,z), and parameter is θ = (σ,ρ,β) The parameters σ = 10,ρ = 28,β = 8/3 give the famous butterfly. Generate data with time step dt =.01, and observation noise = 1. ETKF/state augmentation on z t = (x t,θ t ) with ensemble size 100 and variance inflation factor 4. 19
20 Sequential Parameter Estimates: Lorenz σ Posterior Mean 90% Posterior CI ρ Posterior Mean 90% Posterior CI β Posterior Mean 90% Posterior CI
21 Estimation of Covariance Parameters State augmentation does not work well for parameters in the background or error covariance matrices, P, Q, and R. Dee (1995), D&D (1999) and Mitchell & Houtekamer (2000) proposed Maximum Likelihood estimation for these parameters. Assuming the innovations d are normal with mean zero and covariance D(α), the likelihood function is p(d α) D(α) 1/2 exp { 1 } 2 d D(α) 1 d and the maximum likelihood estimator is ˆα ML = argmax α p(d α). 21
22 Estimation of Covariance Parameters Maximum Likelihood (ML) works well for large samples, but has problems for recursive estimation. D95 and MH00 proposed the recursive ML estimator: ) ˆα t = (1 γ t )ˆα t 1 +γ t (argmax α p(d t α). Setting γ t = 1/t defines ˆα t as the mean of the ML estimates. They also defined ˆα t as the median of the ML estimates. 22
23 Simple Scalar Example Mitchell & Houtekamer (2000) proposed the following example: Generate data d N(0,2+α), with true value α =.3. Since α 0, the single sample ML estimator is ˆα ML = max ( 0,d 2 2 ) 23
24 Mitchell & Houtekamer Example Recursive estimates for α. 0.5 True Mean Median Bayes The recursive ML estimators do not converge to the true value. The Bayes estimator does converge. 24
25 Bayesian Parameter Estimation in the EnKF We propose the following generic EnKF algorithm for combined estimation of states z and covariance parameters α. 1. Assume a prior distribution for the parameters α p(α). 2. Generate a forecast ensemble of parameters and states: α f i p(α) z f i p(z α f i ) 3. Update the prior distribution via Bayes Theorem: p(α d) p(α)p(d α) 4. Generate an analysis ensemble of parameters and states: α i p(α d) z i p(z α i,d) 25
26 Model 1: Unknown Observation Variance Stroud & Bengtsson (2007) considered the case where R = αr, Q = αq and D = αd. 1. Assume an inverse gamma prior distribution: α IG(n/2,s/2). 2. Generate the forecast ensemble: α f i IG(n/2,s/2), z f t,i M(z t 1,i )+N(0,α f i Q ) 3. Update the parameters of the inverse gamma distribution: n = n+p, s = s +d (D ) 1 d 4. Generate the analysis ensemble: α i IG(n /2,s /2) z t,i z f t,i +K(d+N(0,α i R )) 26
27 Example: Lorenz 96 Model The Lorenz 96 model mimics advection on a latitude circle. The model is highly nonlinear (chaotic), containing quadratic terms. ẋ t,j = (x t,j+1 x t,j 2 )x t,j 1 x t,j +F. The state vector has 40 variables x = (x 1,...,x 40 ). The model parameter is F, the forcing variable. 27
28 Model & Assimilation Settings Time step dt =.05 or.25. Forcing F = 8 (known or unknown). Observations at every location. Error covariances: Q = 0, R = αi; true α = 4. EnKF with ensemble size m = 100. Covariance localization with cutoff radius c = 10. Covariance inflation factor 1. 28
29 Sequential Bayesian Estimates of α (dt =.05) 5 α Y 0 ~IG(1.5,6) 4.5 α t 500 α Y ~IG(15,240) α α Y ~IG(15,15) t α t 29
30 Sequential Bayesian Estimates of α (dt =.25) 5 α Y 0 ~IG(1.5,6) 4.5 α t α Y ~IG(15,240) α t α Y ~IG(15,15) α t 30
31 Sequential Bayesian Estimates of (α, F) 5 α Y 0 ~IG(15,15) 10 F Y 0 ~N(8,1) α 4 F t t 5 α Y 0 ~IG(15,15) 10 F Y 0 ~N(8,1) α 4 F t t 31
32 Sequential Estimates of (α, F): Sparse Network 5 α Y 0 ~IG(15,15) 10 F Y 0 ~N(8,1) α 4 F t t 5 α Y 0 ~IG(15,15) 10 F Y 0 ~N(8,1) α 4 F t t 32
33 Spatially- and Temporally-Varying Scale Factors 5 α 1 Y 0 ~IG(1.5, 6) 2.5 α 2 Y 0 ~IG(1.5, 3) 4.5 α 1 4 α t t 5 α 1 Y 0 ~IG(1.5, 6) 11 α 2 Y 0 ~IG(1.5, 13.5) α 1 4 α t t 33
34 Estimation of Spatial Correlation Parameters: Discrete Representation Assume R is defined by a covariance model K(r;α). 1. Assume a discrete prior on a grid of parameter values α : α Mult(α,π) 2. Generate the forecast ensemble. 3. Estimate the innovation mean d and covariance, D(α). 4. Update the posterior distribution: p(α d) Mult(α α,π)p(d α) = Mult(α α,π ) 5. Generate the analysis ensemble: α i Mult(α,π ) z t,i z f t,i +K(α i )(d+n(0,r(α i ))) 34
35 Estimation of Spatial Correlation Parameters: Gaussian Approximation Assume R is defined by a covariance model K(r;α). 1. Assume a normal prior on the parameters: α N(m,C) 2. Generate the forecast ensemble. 3. Estimate the innovation mean d and covariance, D(α). 4. Update the posterior distribution: p(α d) N(α m,c)p(d α) N(α m,c ) 5. Generate the analysis ensemble: α i N(m,C ) z t,i z f t,i +K(α i )(d+n(0,r(α i ))) 35
36 Grid vs Normal Posteriors: Linear Model True: t= Grid: t= Normal: t= True: t= Grid: t= Normal: t= True: t= Grid: t= Normal: t=
37 Sequential Posterior Estimates: Linear Model 0.5 γ σ γ θ γ α
38 Lorenz 96 Model & Assimilation Settings Time step dt =.01. Perfect model, F = 8 known. Observations at 40 locations. R defined by the Matérn correlation model: α ( r ν ( r ) K(r) = Kν Γ(ν)2 λ) ν 1 ; α,λ,ν > 0. λ EnKF with ensemble size m = 100 Covariance localization with radius r = 12. No covariance inflation. 38
39 Sequential Posterior Distributions: Discrete t=0 t= t=5 t= t=50 t=
40 Sequential Bayesian Estimates: Discrete α Posterior Mean 95% Interval λ Posterior Mean 95% Interval ν Posterior Mean 95% Interval
41 Conclusions Bayesian methods are useful for parameter estimation in DA. Two new algorithms for combined state and parameter estimation within EnKF. Easily combined with state augmentation. Good convergence properties (unlike recursive ML). Conjugate priors allow for easy updating. Would love to collaborate with you on this topic! 41
42 Computational Methods Bayesian and ML methods rely heavily on calculation of the likelihood. Several approximate methods have been proposed for computing the likelihood for large spatial data sets Spectral approximations (Whittle, 1953) Approximate likelihood (Vecchia, 1988) Covariance localization (Kaufman et al., 2008) These methods can be applied in data assimilation systems. 42
Parametric Models. Dr. Shuang LIANG. School of Software Engineering TongJi University Fall, 2012
Parametric Models Dr. Shuang LIANG School of Software Engineering TongJi University Fall, 2012 Today s Topics Maximum Likelihood Estimation Bayesian Density Estimation Today s Topics Maximum Likelihood
More informationAdaptive ensemble Kalman filtering of nonlinear systems
Adaptive ensemble Kalman filtering of nonlinear systems Tyrus Berry George Mason University June 12, 213 : Problem Setup We consider a system of the form: x k+1 = f (x k ) + ω k+1 ω N (, Q) y k+1 = h(x
More informationA new Hierarchical Bayes approach to ensemble-variational data assimilation
A new Hierarchical Bayes approach to ensemble-variational data assimilation Michael Tsyrulnikov and Alexander Rakitko HydroMetCenter of Russia College Park, 20 Oct 2014 Michael Tsyrulnikov and Alexander
More informationState and Parameter Estimation in Stochastic Dynamical Models
State and Parameter Estimation in Stochastic Dynamical Models Timothy DelSole George Mason University, Fairfax, Va and Center for Ocean-Land-Atmosphere Studies, Calverton, MD June 21, 2011 1 1 collaboration
More informationSmoothers: Types and Benchmarks
Smoothers: Types and Benchmarks Patrick N. Raanes Oxford University, NERSC 8th International EnKF Workshop May 27, 2013 Chris Farmer, Irene Moroz Laurent Bertino NERSC Geir Evensen Abstract Talk builds
More informationGraphical Models for Collaborative Filtering
Graphical Models for Collaborative Filtering Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Sequence modeling HMM, Kalman Filter, etc.: Similarity: the same graphical model topology,
More informationForecasting and data assimilation
Supported by the National Science Foundation DMS Forecasting and data assimilation Outline Numerical models Kalman Filter Ensembles Douglas Nychka, Thomas Bengtsson, Chris Snyder Geophysical Statistics
More informationPMR Learning as Inference
Outline PMR Learning as Inference Probabilistic Modelling and Reasoning Amos Storkey Modelling 2 The Exponential Family 3 Bayesian Sets School of Informatics, University of Edinburgh Amos Storkey PMR Learning
More informationWhat do we know about EnKF?
What do we know about EnKF? David Kelly Kody Law Andrew Stuart Andrew Majda Xin Tong Courant Institute New York University New York, NY April 10, 2015 CAOS seminar, Courant. David Kelly (NYU) EnKF April
More informationErgodicity in data assimilation methods
Ergodicity in data assimilation methods David Kelly Andy Majda Xin Tong Courant Institute New York University New York NY www.dtbkelly.com April 15, 2016 ETH Zurich David Kelly (CIMS) Data assimilation
More informationFundamentals of Data Assimilation
National Center for Atmospheric Research, Boulder, CO USA GSI Data Assimilation Tutorial - June 28-30, 2010 Acknowledgments and References WRFDA Overview (WRF Tutorial Lectures, H. Huang and D. Barker)
More informationAdaptive Data Assimilation and Multi-Model Fusion
Adaptive Data Assimilation and Multi-Model Fusion Pierre F.J. Lermusiaux, Oleg G. Logoutov and Patrick J. Haley Jr. Mechanical Engineering and Ocean Science and Engineering, MIT We thank: Allan R. Robinson
More information(Extended) Kalman Filter
(Extended) Kalman Filter Brian Hunt 7 June 2013 Goals of Data Assimilation (DA) Estimate the state of a system based on both current and all past observations of the system, using a model for the system
More informationVariational Mixture of Gaussians. Sargur Srihari
Variational Mixture of Gaussians Sargur srihari@cedar.buffalo.edu 1 Objective Apply variational inference machinery to Gaussian Mixture Models Demonstrates how Bayesian treatment elegantly resolves difficulties
More informationBayesian Regression Linear and Logistic Regression
When we want more than point estimates Bayesian Regression Linear and Logistic Regression Nicole Beckage Ordinary Least Squares Regression and Lasso Regression return only point estimates But what if we
More informationEnKF and filter divergence
EnKF and filter divergence David Kelly Andrew Stuart Kody Law Courant Institute New York University New York, NY dtbkelly.com December 12, 2014 Applied and computational mathematics seminar, NIST. David
More informationMethods of Data Assimilation and Comparisons for Lagrangian Data
Methods of Data Assimilation and Comparisons for Lagrangian Data Chris Jones, Warwick and UNC-CH Kayo Ide, UCLA Andrew Stuart, Jochen Voss, Warwick Guillaume Vernieres, UNC-CH Amarjit Budiraja, UNC-CH
More informationApplication of the Ensemble Kalman Filter to History Matching
Application of the Ensemble Kalman Filter to History Matching Presented at Texas A&M, November 16,2010 Outline Philosophy EnKF for Data Assimilation Field History Match Using EnKF with Covariance Localization
More informationThe Ensemble Kalman Filter:
p.1 The Ensemble Kalman Filter: Theoretical formulation and practical implementation Geir Evensen Norsk Hydro Research Centre, Bergen, Norway Based on Evensen 23, Ocean Dynamics, Vol 53, No 4 p.2 The Ensemble
More informationShort tutorial on data assimilation
Mitglied der Helmholtz-Gemeinschaft Short tutorial on data assimilation 23 June 2015 Wolfgang Kurtz & Harrie-Jan Hendricks Franssen Institute of Bio- and Geosciences IBG-3 (Agrosphere), Forschungszentrum
More informationStochastic Collocation Methods for Polynomial Chaos: Analysis and Applications
Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications Dongbin Xiu Department of Mathematics, Purdue University Support: AFOSR FA955-8-1-353 (Computational Math) SF CAREER DMS-64535
More informationA Note on the Particle Filter with Posterior Gaussian Resampling
Tellus (6), 8A, 46 46 Copyright C Blackwell Munksgaard, 6 Printed in Singapore. All rights reserved TELLUS A Note on the Particle Filter with Posterior Gaussian Resampling By X. XIONG 1,I.M.NAVON 1,2 and
More informationarxiv: v1 [physics.ao-ph] 23 Jan 2009
A Brief Tutorial on the Ensemble Kalman Filter Jan Mandel arxiv:0901.3725v1 [physics.ao-ph] 23 Jan 2009 February 2007, updated January 2009 Abstract The ensemble Kalman filter EnKF) is a recursive filter
More informationFundamentals of Data Assimila1on
014 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO July 14-16, 014 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University
More informationLecture 4: Dynamic models
linear s Lecture 4: s Hedibert Freitas Lopes The University of Chicago Booth School of Business 5807 South Woodlawn Avenue, Chicago, IL 60637 http://faculty.chicagobooth.edu/hedibert.lopes hlopes@chicagobooth.edu
More informationBayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 2: Bayesian Basics https://people.orie.cornell.edu/andrew/orie6741 Cornell University August 25, 2016 1 / 17 Canonical Machine Learning
More informationEnsemble Data Assimilation and Uncertainty Quantification
Ensemble Data Assimilation and Uncertainty Quantification Jeff Anderson National Center for Atmospheric Research pg 1 What is Data Assimilation? Observations combined with a Model forecast + to produce
More informationEnsemble Kalman Filter
Ensemble Kalman Filter Geir Evensen and Laurent Bertino Hydro Research Centre, Bergen, Norway, Nansen Environmental and Remote Sensing Center, Bergen, Norway The Ensemble Kalman Filter (EnKF) Represents
More informationNon-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
More informationUNDERSTANDING DATA ASSIMILATION APPLICATIONS TO HIGH-LATITUDE IONOSPHERIC ELECTRODYNAMICS
Monday, June 27, 2011 CEDAR-GEM joint workshop: DA tutorial 1 UNDERSTANDING DATA ASSIMILATION APPLICATIONS TO HIGH-LATITUDE IONOSPHERIC ELECTRODYNAMICS Tomoko Matsuo University of Colorado, Boulder Space
More informationData assimilation with and without a model
Data assimilation with and without a model Tim Sauer George Mason University Parameter estimation and UQ U. Pittsburgh Mar. 5, 2017 Partially supported by NSF Most of this work is due to: Tyrus Berry,
More informationMultivariate Normal & Wishart
Multivariate Normal & Wishart Hoff Chapter 7 October 21, 2010 Reading Comprehesion Example Twenty-two children are given a reading comprehsion test before and after receiving a particular instruction method.
More informationEnKF Localization Techniques and Balance
EnKF Localization Techniques and Balance Steven Greybush Eugenia Kalnay, Kayo Ide, Takemasa Miyoshi, and Brian Hunt Weather Chaos Meeting September 21, 2009 Data Assimilation Equation Scalar form: x a
More informationIntroduction to Probabilistic Machine Learning
Introduction to Probabilistic Machine Learning Piyush Rai Dept. of CSE, IIT Kanpur (Mini-course 1) Nov 03, 2015 Piyush Rai (IIT Kanpur) Introduction to Probabilistic Machine Learning 1 Machine Learning
More informationDART_LAB Tutorial Section 5: Adaptive Inflation
DART_LAB Tutorial Section 5: Adaptive Inflation UCAR 14 The National Center for Atmospheric Research is sponsored by the National Science Foundation. Any opinions, findings and conclusions or recommendations
More informationSYDE 372 Introduction to Pattern Recognition. Probability Measures for Classification: Part I
SYDE 372 Introduction to Pattern Recognition Probability Measures for Classification: Part I Alexander Wong Department of Systems Design Engineering University of Waterloo Outline 1 2 3 4 Why use probability
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationChapter 3: Maximum-Likelihood & Bayesian Parameter Estimation (part 1)
HW 1 due today Parameter Estimation Biometrics CSE 190 Lecture 7 Today s lecture was on the blackboard. These slides are an alternative presentation of the material. CSE190, Winter10 CSE190, Winter10 Chapter
More informationMachine Learning. Lecture 4: Regularization and Bayesian Statistics. Feng Li. https://funglee.github.io
Machine Learning Lecture 4: Regularization and Bayesian Statistics Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 207 Overfitting Problem
More informationLecture 2: Priors and Conjugacy
Lecture 2: Priors and Conjugacy Melih Kandemir melih.kandemir@iwr.uni-heidelberg.de May 6, 2014 Some nice courses Fred A. Hamprecht (Heidelberg U.) https://www.youtube.com/watch?v=j66rrnzzkow Michael I.
More informationDensity Estimation: ML, MAP, Bayesian estimation
Density Estimation: ML, MAP, Bayesian estimation CE-725: Statistical Pattern Recognition Sharif University of Technology Spring 2013 Soleymani Outline Introduction Maximum-Likelihood Estimation Maximum
More informationData assimilation in high dimensions
Data assimilation in high dimensions David Kelly Courant Institute New York University New York NY www.dtbkelly.com February 12, 2015 Graduate seminar, CIMS David Kelly (CIMS) Data assimilation February
More informationData assimilation with and without a model
Data assimilation with and without a model Tyrus Berry George Mason University NJIT Feb. 28, 2017 Postdoc supported by NSF This work is in collaboration with: Tim Sauer, GMU Franz Hamilton, Postdoc, NCSU
More informationBayesian data analysis in practice: Three simple examples
Bayesian data analysis in practice: Three simple examples Martin P. Tingley Introduction These notes cover three examples I presented at Climatea on 5 October 0. Matlab code is available by request to
More informationRelevance Vector Machines
LUT February 21, 2011 Support Vector Machines Model / Regression Marginal Likelihood Regression Relevance vector machines Exercise Support Vector Machines The relevance vector machine (RVM) is a bayesian
More informationFour-Dimensional Ensemble Kalman Filtering
Four-Dimensional Ensemble Kalman Filtering B.R. Hunt, E. Kalnay, E.J. Kostelich, E. Ott, D.J. Patil, T. Sauer, I. Szunyogh, J.A. Yorke, A.V. Zimin University of Maryland, College Park, MD 20742, USA Ensemble
More informationAn Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation
An Efficient Ensemble Data Assimilation Approach To Deal With Range Limited Observation A. Shah 1,2, M. E. Gharamti 1, L. Bertino 1 1 Nansen Environmental and Remote Sensing Center 2 University of Bergen
More informationA Bayesian Perspective on Residential Demand Response Using Smart Meter Data
A Bayesian Perspective on Residential Demand Response Using Smart Meter Data Datong-Paul Zhou, Maximilian Balandat, and Claire Tomlin University of California, Berkeley [datong.zhou, balandat, tomlin]@eecs.berkeley.edu
More informationLagrangian Data Assimilation and Its Application to Geophysical Fluid Flows
Lagrangian Data Assimilation and Its Application to Geophysical Fluid Flows Laura Slivinski June, 3 Laura Slivinski (Brown University) Lagrangian Data Assimilation June, 3 / 3 Data Assimilation Setup:
More informationLocal Ensemble Transform Kalman Filter: An Efficient Scheme for Assimilating Atmospheric Data
Local Ensemble Transform Kalman Filter: An Efficient Scheme for Assimilating Atmospheric Data John Harlim and Brian R. Hunt Department of Mathematics and Institute for Physical Science and Technology University
More informationSTA414/2104 Statistical Methods for Machine Learning II
STA414/2104 Statistical Methods for Machine Learning II Murat A. Erdogdu & David Duvenaud Department of Computer Science Department of Statistical Sciences Lecture 3 Slide credits: Russ Salakhutdinov Announcements
More informationBayesian Model Comparison:
Bayesian Model Comparison: Modeling Petrobrás log-returns Hedibert Freitas Lopes February 2014 Log price: y t = log p t Time span: 12/29/2000-12/31/2013 (n = 3268 days) LOG PRICE 1 2 3 4 0 500 1000 1500
More informationLecture 3. Linear Regression II Bastian Leibe RWTH Aachen
Advanced Machine Learning Lecture 3 Linear Regression II 02.11.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de This Lecture: Advanced Machine Learning Regression
More informationKernel methods, kernel SVM and ridge regression
Kernel methods, kernel SVM and ridge regression Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Collaborative Filtering 2 Collaborative Filtering R: rating matrix; U: user factor;
More informationIntroduction to Machine Learning
Introduction to Machine Learning Generative Models Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574 1
More informationFundamentals of Data Assimila1on
2015 GSI Community Tutorial NCAR Foothills Campus, Boulder, CO August 11-14, 2015 Fundamentals of Data Assimila1on Milija Zupanski Cooperative Institute for Research in the Atmosphere Colorado State University
More informationConvergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit
Convergence of Square Root Ensemble Kalman Filters in the Large Ensemble Limit Evan Kwiatkowski, Jan Mandel University of Colorado Denver December 11, 2014 OUTLINE 2 Data Assimilation Bayesian Estimation
More informationIntroduction to Bayesian Inference
University of Pennsylvania EABCN Training School May 10, 2016 Bayesian Inference Ingredients of Bayesian Analysis: Likelihood function p(y φ) Prior density p(φ) Marginal data density p(y ) = p(y φ)p(φ)dφ
More informationPattern Recognition and Machine Learning. Bishop Chapter 2: Probability Distributions
Pattern Recognition and Machine Learning Chapter 2: Probability Distributions Cécile Amblard Alex Kläser Jakob Verbeek October 11, 27 Probability Distributions: General Density Estimation: given a finite
More informationA Spectral Approach to Linear Bayesian Updating
A Spectral Approach to Linear Bayesian Updating Oliver Pajonk 1,2, Bojana V. Rosic 1, Alexander Litvinenko 1, and Hermann G. Matthies 1 1 Institute of Scientific Computing, TU Braunschweig, Germany 2 SPT
More informationCOS513 LECTURE 8 STATISTICAL CONCEPTS
COS513 LECTURE 8 STATISTICAL CONCEPTS NIKOLAI SLAVOV AND ANKUR PARIKH 1. MAKING MEANINGFUL STATEMENTS FROM JOINT PROBABILITY DISTRIBUTIONS. A graphical model (GM) represents a family of probability distributions
More informationDART_LAB Tutorial Section 2: How should observations impact an unobserved state variable? Multivariate assimilation.
DART_LAB Tutorial Section 2: How should observations impact an unobserved state variable? Multivariate assimilation. UCAR 2014 The National Center for Atmospheric Research is sponsored by the National
More informationBayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 3 Stochastic Gradients, Bayesian Inference, and Occam s Razor https://people.orie.cornell.edu/andrew/orie6741 Cornell University August
More informationBayesian Estimation of DSGE Models 1 Chapter 3: A Crash Course in Bayesian Inference
1 The views expressed in this paper are those of the authors and do not necessarily reflect the views of the Federal Reserve Board of Governors or the Federal Reserve System. Bayesian Estimation of DSGE
More informationPart 6: Multivariate Normal and Linear Models
Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of
More informationLocal Ensemble Transform Kalman Filter
Local Ensemble Transform Kalman Filter Brian Hunt 11 June 2013 Review of Notation Forecast model: a known function M on a vector space of model states. Truth: an unknown sequence {x n } of model states
More informationEstimation Theory. as Θ = (Θ 1,Θ 2,...,Θ m ) T. An estimator
Estimation Theory Estimation theory deals with finding numerical values of interesting parameters from given set of data. We start with formulating a family of models that could describe how the data were
More information. Also, in this case, p i = N1 ) T, (2) where. I γ C N(N 2 2 F + N1 2 Q)
Supplementary information S7 Testing for association at imputed SPs puted SPs Score tests A Score Test needs calculations of the observed data score and information matrix only under the null hypothesis,
More informationECONOMETRIC METHODS II: TIME SERIES LECTURE NOTES ON THE KALMAN FILTER. The Kalman Filter. We will be concerned with state space systems of the form
ECONOMETRIC METHODS II: TIME SERIES LECTURE NOTES ON THE KALMAN FILTER KRISTOFFER P. NIMARK The Kalman Filter We will be concerned with state space systems of the form X t = A t X t 1 + C t u t 0.1 Z t
More informationData assimilation : Basics and meteorology
Data assimilation : Basics and meteorology Olivier Talagrand Laboratoire de Météorologie Dynamique, École Normale Supérieure, Paris, France Workshop on Coupled Climate-Economics Modelling and Data Analysis
More informationCovariance function estimation in Gaussian process regression
Covariance function estimation in Gaussian process regression François Bachoc Department of Statistics and Operations Research, University of Vienna WU Research Seminar - May 2015 François Bachoc Gaussian
More informationLinear Models A linear model is defined by the expression
Linear Models A linear model is defined by the expression x = F β + ɛ. where x = (x 1, x 2,..., x n ) is vector of size n usually known as the response vector. β = (β 1, β 2,..., β p ) is the transpose
More informationParametric Techniques Lecture 3
Parametric Techniques Lecture 3 Jason Corso SUNY at Buffalo 22 January 2009 J. Corso (SUNY at Buffalo) Parametric Techniques Lecture 3 22 January 2009 1 / 39 Introduction In Lecture 2, we learned how to
More informationNonlinear Filtering. With Polynomial Chaos. Raktim Bhattacharya. Aerospace Engineering, Texas A&M University uq.tamu.edu
Nonlinear Filtering With Polynomial Chaos Raktim Bhattacharya Aerospace Engineering, Texas A&M University uq.tamu.edu Nonlinear Filtering with PC Problem Setup. Dynamics: ẋ = f(x, ) Sensor Model: ỹ = h(x)
More informationThe University of Reading
The University of Reading A hybrid sequential data assimilation scheme for model state and parameter estimation P.J. Smith, S.L. Dance and N.K. Nichols MATHEMATICS REPORT 2/2010 Department of Mathematics
More informationBayesian Regularization
Bayesian Regularization Aad van der Vaart Vrije Universiteit Amsterdam International Congress of Mathematicians Hyderabad, August 2010 Contents Introduction Abstract result Gaussian process priors Co-authors
More informationObservation Impact Assessment for Dynamic. Data-Driven Coupled Chaotic System
Applied Mathematical Sciences, Vol. 10, 2016, no. 45, 2239-2248 HIKARI Ltd, www.m-hikari.com http://dx.doi.org/10.12988/ams.2016.65170 Observation Impact Assessment for Dynamic Data-Driven Coupled Chaotic
More informationComputer Vision Group Prof. Daniel Cremers. 2. Regression (cont.)
Prof. Daniel Cremers 2. Regression (cont.) Regression with MLE (Rep.) Assume that y is affected by Gaussian noise : t = f(x, w)+ where Thus, we have p(t x, w, )=N (t; f(x, w), 2 ) 2 Maximum A-Posteriori
More informationKalman Filter and Ensemble Kalman Filter
Kalman Filter and Ensemble Kalman Filter 1 Motivation Ensemble forecasting : Provides flow-dependent estimate of uncertainty of the forecast. Data assimilation : requires information about uncertainty
More informationMachine Learning 4771
Machine Learning 4771 Instructor: Tony Jebara Topic 11 Maximum Likelihood as Bayesian Inference Maximum A Posteriori Bayesian Gaussian Estimation Why Maximum Likelihood? So far, assumed max (log) likelihood
More informationParametric Techniques
Parametric Techniques Jason J. Corso SUNY at Buffalo J. Corso (SUNY at Buffalo) Parametric Techniques 1 / 39 Introduction When covering Bayesian Decision Theory, we assumed the full probabilistic structure
More informationGaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
More informationClick Prediction and Preference Ranking of RSS Feeds
Click Prediction and Preference Ranking of RSS Feeds 1 Introduction December 11, 2009 Steven Wu RSS (Really Simple Syndication) is a family of data formats used to publish frequently updated works. RSS
More information1 Bayesian Linear Regression (BLR)
Statistical Techniques in Robotics (STR, S15) Lecture#10 (Wednesday, February 11) Lecturer: Byron Boots Gaussian Properties, Bayesian Linear Regression 1 Bayesian Linear Regression (BLR) In linear regression,
More informationEnKF and Catastrophic filter divergence
EnKF and Catastrophic filter divergence David Kelly Kody Law Andrew Stuart Mathematics Department University of North Carolina Chapel Hill NC bkelly.com June 4, 2014 SIAM UQ 2014 Savannah. David Kelly
More informationBayesian Ingredients. Hedibert Freitas Lopes
Normal Prior s Ingredients Hedibert Freitas Lopes The University of Chicago Booth School of Business 5807 South Woodlawn Avenue, Chicago, IL 60637 http://faculty.chicagobooth.edu/hedibert.lopes hlopes@chicagobooth.edu
More informationThe Metropolis-Hastings Algorithm. June 8, 2012
The Metropolis-Hastings Algorithm June 8, 22 The Plan. Understand what a simulated distribution is 2. Understand why the Metropolis-Hastings algorithm works 3. Learn how to apply the Metropolis-Hastings
More information4. DATA ASSIMILATION FUNDAMENTALS
4. DATA ASSIMILATION FUNDAMENTALS... [the atmosphere] "is a chaotic system in which errors introduced into the system can grow with time... As a consequence, data assimilation is a struggle between chaotic
More informationAn Introduction to Bayesian Machine Learning
1 An Introduction to Bayesian Machine Learning José Miguel Hernández-Lobato Department of Engineering, Cambridge University April 8, 2013 2 What is Machine Learning? The design of computational systems
More informationGaussian Models
Gaussian Models ddebarr@uw.edu 2016-04-28 Agenda Introduction Gaussian Discriminant Analysis Inference Linear Gaussian Systems The Wishart Distribution Inferring Parameters Introduction Gaussian Density
More informationRelative Merits of 4D-Var and Ensemble Kalman Filter
Relative Merits of 4D-Var and Ensemble Kalman Filter Andrew Lorenc Met Office, Exeter International summer school on Atmospheric and Oceanic Sciences (ISSAOS) "Atmospheric Data Assimilation". August 29
More informationA data-driven method for improving the correlation estimation in serial ensemble Kalman filter
A data-driven method for improving the correlation estimation in serial ensemble Kalman filter Michèle De La Chevrotière, 1 John Harlim 2 1 Department of Mathematics, Penn State University, 2 Department
More informationHierarchical Modeling for Univariate Spatial Data
Hierarchical Modeling for Univariate Spatial Data Geography 890, Hierarchical Bayesian Models for Environmental Spatial Data Analysis February 15, 2011 1 Spatial Domain 2 Geography 890 Spatial Domain This
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond Department of Biomedical Engineering and Computational Science Aalto University January 26, 2012 Contents 1 Batch and Recursive Estimation
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationHOW TO IMPROVE FORECASTING WITH BRED VECTORS BY USING THE GEOMETRIC NORM
HOW TO IMPROVE FORECASTING WITH BRED VECTORS BY USING THE GEOMETRIC NORM Diego Pazó Instituto de Física de Cantabria (IFCA), CSIC-UC, Santander (Spain) Santander OUTLINE 1) Lorenz '96 model. 2) Definition
More informationThe Ensemble Kalman Filter:
p.1 The Ensemble Kalman Filter: Theoretical formulation and practical implementation Geir Evensen Norsk Hydro Research Centre, Bergen, Norway Based on Evensen, Ocean Dynamics, Vol 5, No p. The Ensemble
More informationLecture 16: State Space Model and Kalman Filter Bus 41910, Time Series Analysis, Mr. R. Tsay
Lecture 6: State Space Model and Kalman Filter Bus 490, Time Series Analysis, Mr R Tsay A state space model consists of two equations: S t+ F S t + Ge t+, () Z t HS t + ɛ t (2) where S t is a state vector
More informationAccepted in Tellus A 2 October, *Correspondence
1 An Adaptive Covariance Inflation Error Correction Algorithm for Ensemble Filters Jeffrey L. Anderson * NCAR Data Assimilation Research Section P.O. Box 3000 Boulder, CO 80307-3000 USA Accepted in Tellus
More informationGibbs Sampling in Endogenous Variables Models
Gibbs Sampling in Endogenous Variables Models Econ 690 Purdue University Outline 1 Motivation 2 Identification Issues 3 Posterior Simulation #1 4 Posterior Simulation #2 Motivation In this lecture we take
More information