Data-Driven Bayesian Model Selection: Parameter Space Dimension Reduction using Automatic Relevance Determination Priors
|
|
- Bruce Sparks
- 5 years ago
- Views:
Transcription
1 Data-Driven : Parameter Space Dimension Reduction using Priors Mohammad Khalil mkhalil@sandia.gov, Livermore, CA Workshop on Uncertainty Quantification and Data-Driven Modeling Austin, Texas March 23-24, 217 is a multi-mission laboratory managed and operated by Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energys Nuclear Security Administration under contract DE-AC4-94AL85.
2 Overview Predictive Optimal Predictive Optimal M. Khalil Data-Driven using 2 / 26
3 Why Model? Why Model? Predictive Optimal Model selection is the task of selecting a physical/statistical model from a set of candidate models, given data. When dealing with nontrivial physics under limited a priori understanding of the system, multiple plausible models can be envisioned to represent the system with a reasonable accuracy. A complex model may overfit the data but results in a higher model prediction uncertainty. A simpler model may misfit the data but results in a lower model prediction uncertainty. An optimal model provides a balance between data-fit and prediction uncertainty. Common approaches: Cross-validation Akaike information criterion (AIC) Bayesian information criterion (BIC) (Bayesian) Model evidence M. Khalil Data-Driven using 3 / 26
4 Inverse Problems Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work forward model noisy observations model parameters Forward Problem: Given model parameters, predict clean observations Inverse Problem: Given noisy observations, infer model parameters observations are inherently noisy with unknown (or weakly known) noise model sparse in space and time (insufficient resolution) problem typically ill-posed, i.e. no guarantee of solution existence nor uniqueness Predictive Optimal M. Khalil Data-Driven using 4 / 26
5 Bayes Theorem Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work Predictive Optimal The parameters φ are treated as a random vector. Using Bayes rule, one can write posterior likelihood prior evidence pdf prior likelihood posterior p(φ,m) is the prior pdf of φ: induces regularization p(d φ, M) is the likelihood pdf: describes data misfit p(φ d,m) is the posterior pdf of φ the full Bayesian solution: Not a single point estimate but a probability density Completely characterizes the uncertainty in φ Used in simulations for prediction under uncertainty For parameter inference alone, it is sufficient to consider 1 1 u p(φ d,m) p(d φ,m)p(φ M) M. Khalil Data-Driven using 5 / 26
6 Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work Stages of Bayesian Inference Bayesian inverse modeling from real data is often an iterative process: Select a model (parameters + priors) Using available data, perform model calibration: Parameter inference Using posterior parameter pdf, compute model evidence: Model selection Refine model or propose new model and repeat Stage 1 Stage 2 Stage 3 I have a model and parameter priors I have more than one plausible model None of the models is clearly the best Predictive Optimal Parameter inference: assume I have an accurate model Model selection: compute relative plausibility of models given data Model averaging: obtain posterior predictive density of QoI averaged over plausible models M. Khalil Data-Driven using 6 / 26
7 Model Evidence and Bayes Factor Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work Predictive Optimal When there are competing models, Bayesian model selection allows us to obtain their relative probabilities in light of the data and prior information The best model is then the one which strikes an optimum balance between quality of fit and predictivity Model evidence: An integral of the likelihood over the prior, or marginalized (averaged) likelihood p(d M) = p(d φ,m)p(φ,m)dφ Model posterior/plausibility: Obtained using Bayes Theorem p(m d) p(d M)p(M) Relative model posterior probabilities: Obtained using Bayes factor Posterior odds = Bayes' factor x prior odds Bayes' factor = relative model evidence M. Khalil Data-Driven using 7 / 26
8 Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work Predictive Optimal Model Evidence and Occam s Razor Bayes model evidence balances quality of fit vs unwarranted model complexity It does that by penalizing wasted parameter space and thereby rewarding highly predictive models Prior Likelihood Penalizes complex models Prior automatic Occam s razor effect Likelihood The parameter prior plays a decisive role as it reflects the available parameter space under the model M prior to assimilating data. M. Khalil Data-Driven using 8 / 26
9 Model Evidence: Nested Models Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work Nested models are investigated often in practice: a more complex model, M 1, with prior p(φ,m), which reduces to a simpler nested model, M, for a certain value of the parameter, φ = φ = Question: Is the extra complexity of M 1 warranted by the data? Define: We have: Prior Likelihood Predictive Optimal Wasted parameter space Favors simpler model mismatch between prediction and likelihood Favors more complex model M. Khalil Data-Driven using 9 / 26
10 e M t P 3 r n n Inverse Problems Bayes Theorem Stages of Bayesian Inference Bayes Factor Occam s Razor Model Evidence: Nested Models : Occam s Razor at Work : Occam s Razor at Work Generate 6 noisy data points from the true model given by y i = 1 + x 2 i + ǫ i ǫ i N (,1) Question: Not knowing the true model, what is the best model? We propose polynomials of increasing order: M : y = a + ǫ. M 5 : y = a + a 1 x + a 2 x 2 + a 3 x 3 + a 4 x 4 + a 5 x 5 + ǫ ior pr t e Predictive Optimal true 2 true rm M. Khalil Data-Driven using 1 / 26
11 Challenges with Challenges with : ARD Priors Predictive Optimal Model evidence is extremely sensitive to prior parameter pdfs Missing out on better candidate models: The number of possible models grows rapidly with the number of possible terms in the physical/statistical model For the previous example, the number of possible models of order up to and including 6 is N M = number of k combinations up to and including 5 6 ( ) 6 = k k=1 = 6! 1! 5! + 6! 2! 4! + 6! 3! 3! + 6! 4! 2! + 6! 5! 1! + 6! 6!! = 63 For polynomials of maximum order of 1, 123 possible models! Solution: (ARD) M. Khalil Data-Driven using 11 / 26
12 Challenges with : ARD Priors A parametrized prior distribution known as ARD prior is assigned to the unknown model parameters ARD prior pdf is a Gaussian with zero mean and unknown variance (could also use Laplace priors, etc...) The hyper-parameters, α, are estimated using the data by performing evidence maximization or type-ii maximum likelihood estimation Prior : p(φ α,m) Posterior : p(φ d,α,m) Type II likelihood : p(d α,m) = p(d φ,m)p(φ α,m)dφ Predictive Optimal M. Khalil Data-Driven using 12 / 26
13 : ARD Priors Challenges with : ARD Priors Revisiting the previous example with the true model given by y i = 1+x 2 i +ǫ i ǫ i N (,1) Question: What is the best model nested under the model: y = a +a 1 x+a 2 x 2 +a 3 x 3 +a 4 x 4 +a 5 x 5 +ǫ Predictive Optimal Optimizer Iteration Optimizer Iteration Optimizer Iteration Convergence could be improved with better optimizer Optimizer Iteration Optimizer Iteration Optimizer Iteration Optimizer Iteration Type-II likelihood (model evidence) M. Khalil Data-Driven using 13 / 26
14 Nonlinear Modeling in Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Limit cycle oscillation (LCO) is observed in wind tunnel experiments for 2-D rigid airfoil in transitional Re regime Pure pitch LCO due to nonlinear aerodynamic loads Objective: Inverse modeling of nonlinear oscillations with an aim to understand and quantify the contribution of unsteady and nonlinear aerodynamics. Predictive Optimal Nor alized T me M. Khalil Data-Driven using 14 / 26
15 Research Group/Resources Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Philippe Bisaillon, Ph.D. candidate, Carleton University Rimple Sandhu, Ph.D. candidate, Carleton University Dominique Poirel, Royal Military College (RMC) of Canada Abhijit Sarkar, Carleton University Chris Pettit, United States Naval Academy Predictive Optimal HPC lab at Carleton University Wind tunnel at RMC M. Khalil Data-Driven using 15 / 26
16 Previous Work Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Predictive Optimal Start with a candidate model set: I EA θ +D θ +Kθ +K θ 3 = D sign θ ρu2 c 2 sc M (θ, M 1 : C M = e 1 θ +e 2 θ +e3 θ 3 +e 4 θ 2 θ +σξ(τ) M 6 :. ) θ, θ C M + (B 1 +B 2 ) C M +C M = e 1 θ +e 2 θ +e3 θ 3 +e 4 θ 2 θ +e5 θ 5 B 1 B 2 B 1 B 2 + (2c 6c 7 +.5) θ + c... 6θ +σξ(τ) B 1 B 2 B 1 B 2 We observe the pitch degree-of-freedom (DOF): d k = θ(t k )+ǫ k We perform Bayesian model selection in discrete model space Sandhu et al., JCP, 216 Sandhu et al., CMAME, 214 Khalil et al., JSV, 213 M. Khalil Data-Driven using 16 / 26
17 Use of ARD Priors Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Start with an encompassing model: I EA θ +D θ +Kθ +K θ 3 = D sign θ ρu2 c 2 sc M C M B +C M = a 1 θ +a 2 θ +a3 θ 3 +a 4 θ 2 θ +a5 θ 5 +a 6 θ 4 θ + c 6 B θ +σξ(τ) We would like to find the optimal model nested under the overly-prescribed encompassing model Predictive Optimal M. Khalil Data-Driven using 17 / 26
18 Hybrid approach: ARD Priors vs Fixed Priors Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors We assign prior distributions by categorizing parameters based on prior knowledge about the aerodynamics as Required ( φ α ) or Contentious (φα ) C M B +C M = a 1 θ +a 2 θ +a3 θ 3 +a 4 θ 2 θ +a5 θ 5 +a 6 θ 4 θ + c 6 B θ +σξ(τ) Predictive Optimal M. Khalil Data-Driven using 18 / 26
19 Hierarchical Bayes Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Predictive Optimal Using hierarchical Bayes approach Posterior pdf p(α d) of hyper-parameter vector α: p(α d) p(d α)p(α) For a fixed hyper-prior p(α), Task: Stochastic optimization: α MAP = arg max α p(α d) Model evidence as a function of hyper-parameter, Task: Evidence computation: p(d α) = p(d φ)p(φ α)dφ Parameter likelihood computation, Task: State estimation: p(d φ) = n d k=1 p ( d k u j(k),φ ) p ( u j(k) d 1:k 1,φ ) du j(k) M. Khalil Data-Driven using 19 / 26
20 Numerical techniques Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Evidence computation: Chib-Jeliazkov method, Power posteriors, Nested sampling, Annealed importance sampling, Harmonic mean estimator, adaptive Gauss-Hermite quadrature; and many others MCMC sampler for Chib-Jeliazkov method: Metropolis-Hastings, Gibbs, tmcmc, adaptive Metropolis, Delayed Rejection Adaptive Metropolis (DRAM); and many others State estimation: Kalman filter, extended Kalman filter, unscented Kalman filter, ensemble Kalman filter, particle filter; and many others. Results are in: R. Sandhu, C. Pettit, M. Khalil, D. Poirel, A. Sarkar, Bayesian model selection using automatic relevance determination for nonlinear dynamical systems, Computer Methods in Applied Mechanics and Engineering, in press. Predictive Optimal M. Khalil Data-Driven using 2 / 26
21 Numerical Results: ARD Priors Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors Predictive Optimal M. Khalil Data-Driven using 21 / 26
22 Numerical Results: ARD vs Flat Priors Nonlinear Modeling in Previous Work Use of ARD Priors Hybrid approach: ARD Priors vs Fixed Priors Hierarchical Bayes Numerical techniques ARD Priors ARD vs Flat Priors We compare selected marginal and joint pdfs for (a) ARD priors with optimal hyper-parameters, and (b) flat priors ARD priors able to remove superfluous parameters while having insignificant effect on the posterior pdfs of important parameters Predictive Optimal M. Khalil Data-Driven using 22 / 26
23 Predictive Predictive Predictive Optimal Collaborators: Jina Lee, Maher Salloum () Objective: Replace computationally expensive simulations of physical systems with response predictions constructed at the wavelet coefficient level Procedure: Perform compressed sensing of high-dimensional system response from full-order model simulations Model resulting low-dimensional wavelet coefficients using autoregressive-moving-average (ARMA) model x t = p ϕ i x t i + i=1 q θ j ǫ t j ǫ t N (,1) j=1 y t = x t +ζ t ζ t N (,γ 2) Parameters likelihood for ϕ i, θ j and γ involves a state estimation using the Kalman filter Model selection, i.e. determining model orders p and q, is performed using Akaike information criterion (AIC) M. Khalil Data-Driven using 23 / 26
24 Wavelet Coefficient Predictions For illustration we consider the transient response of the 2D heat equation on a square domain with randomly chosen holes (for added heterogeneity) Compressed sensing is performed and 7 dominant wavelet coefficients are modeled Predictive Predictive Optimal M. Khalil Data-Driven using 24 / 26
25 Optimal Predictive Optimal Optimal Collaborators: Layal Hakim, Guilhem Lacaze, Khachik Sargsyan, Habib Najm, Joe Oefelein () Objective: Calibrate a simple chemical model against computations from a detailed kinetic model Simple model with an embedded parameterization of model error using polynomial chaos expansions Optimal placement of model error achieved via Bayesian model selection (Bayes factor) Bayes' factors M. Khalil Data-Driven using 25 / 26
26 Predictive Optimal Presented a framework for data-driven model selection using ARD prior pdfs ARD priors enable the transformation of the model selection problem from the discrete model space into the continuous hyper-parameter space Allow for parameter space dimension reduction informed by noisy observations of the system Applications: Nonlinear dynamical systems modeled using stochastic ordinary differential equations (ARD priors) Predictive (AIC) Optimal (Bayes factor) ARD priors able to remove superfluous parameters while having insignificant effect on the posterior pdfs of important parameters M. Khalil Data-Driven using 26 / 26
Bayesian Model Selection and Parameter Estimation for Strongly Nonlinear Dynamical Systems
Bayesian Model Selection and Parameter Estimation for Strongly Nonlinear Dynamical Systems A thesis submitted to the Faculty of Graduate and Postdoctoral Affairs in partial fulfillment of the requirements
More informationBayesian Inference in Astronomy & Astrophysics A Short Course
Bayesian Inference in Astronomy & Astrophysics A Short Course Tom Loredo Dept. of Astronomy, Cornell University p.1/37 Five Lectures Overview of Bayesian Inference From Gaussians to Periodograms Learning
More informationUncertainty Quantification for Machine Learning and Statistical Models
Uncertainty Quantification for Machine Learning and Statistical Models David J. Stracuzzi Joint work with: Max Chen, Michael Darling, Stephen Dauphin, Matt Peterson, and Chris Young Sandia National Laboratories
More informationBayesian room-acoustic modal analysis
Bayesian room-acoustic modal analysis Wesley Henderson a) Jonathan Botts b) Ning Xiang c) Graduate Program in Architectural Acoustics, School of Architecture, Rensselaer Polytechnic Institute, Troy, New
More informationDynamic System Identification using HDMR-Bayesian Technique
Dynamic System Identification using HDMR-Bayesian Technique *Shereena O A 1) and Dr. B N Rao 2) 1), 2) Department of Civil Engineering, IIT Madras, Chennai 600036, Tamil Nadu, India 1) ce14d020@smail.iitm.ac.in
More informationHierarchical sparse Bayesian learning for structural health monitoring. with incomplete modal data
Hierarchical sparse Bayesian learning for structural health monitoring with incomplete modal data Yong Huang and James L. Beck* Division of Engineering and Applied Science, California Institute of Technology,
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2016
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2016 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationUncertainty quantification for inverse problems with a weak wave-equation constraint
Uncertainty quantification for inverse problems with a weak wave-equation constraint Zhilong Fang*, Curt Da Silva*, Rachel Kuske** and Felix J. Herrmann* *Seismic Laboratory for Imaging and Modeling (SLIM),
More informationIntro BCS/Low Rank Model Inference/Comparison Summary References. UQTk. A Flexible Python/C++ Toolkit for Uncertainty Quantification
A Flexible Python/C++ Toolkit for Uncertainty Quantification Bert Debusschere, Khachik Sargsyan, Cosmin Safta, Prashant Rai, Kenny Chowdhary bjdebus@sandia.gov Sandia National Laboratories, Livermore,
More informationBayesian System Identification based on Hierarchical Sparse Bayesian Learning and Gibbs Sampling with Application to Structural Damage Assessment
Bayesian System Identification based on Hierarchical Sparse Bayesian Learning and Gibbs Sampling with Application to Structural Damage Assessment Yong Huang a,b, James L. Beck b,* and Hui Li a a Key Lab
More informationA note on Reversible Jump Markov Chain Monte Carlo
A note on Reversible Jump Markov Chain Monte Carlo Hedibert Freitas Lopes Graduate School of Business The University of Chicago 5807 South Woodlawn Avenue Chicago, Illinois 60637 February, 1st 2006 1 Introduction
More informationFrequentist-Bayesian Model Comparisons: A Simple Example
Frequentist-Bayesian Model Comparisons: A Simple Example Consider data that consist of a signal y with additive noise: Data vector (N elements): D = y + n The additive noise n has zero mean and diagonal
More informationIntroduction to Statistical modeling: handout for Math 489/583
Introduction to Statistical modeling: handout for Math 489/583 Statistical modeling occurs when we are trying to model some data using statistical tools. From the start, we recognize that no model is perfect
More informationLecture : Probabilistic Machine Learning
Lecture : Probabilistic Machine Learning Riashat Islam Reasoning and Learning Lab McGill University September 11, 2018 ML : Many Methods with Many Links Modelling Views of Machine Learning Machine Learning
More informationCALIFORNIA INSTITUTE OF TECHNOLOGY
CALIFORNIA INSTITUTE OF TECHNOLOGY EARTHQUAKE ENGINEERING RESEARCH LABORATORY NEW BAYESIAN UPDATING METHODOLOGY FOR MODEL VALIDATION AND ROBUST PREDICTIONS BASED ON DATA FROM HIERARCHICAL SUBSYSTEM TESTS
More informationCSci 8980: Advanced Topics in Graphical Models Gaussian Processes
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian
More informationBayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2014
Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2014 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several
More informationBayesian Inference. Chris Mathys Wellcome Trust Centre for Neuroimaging UCL. London SPM Course
Bayesian Inference Chris Mathys Wellcome Trust Centre for Neuroimaging UCL London SPM Course Thanks to Jean Daunizeau and Jérémie Mattout for previous versions of this talk A spectacular piece of information
More informationForward Problems and their Inverse Solutions
Forward Problems and their Inverse Solutions Sarah Zedler 1,2 1 King Abdullah University of Science and Technology 2 University of Texas at Austin February, 2013 Outline 1 Forward Problem Example Weather
More informationBayesian model selection for computer model validation via mixture model estimation
Bayesian model selection for computer model validation via mixture model estimation Kaniav Kamary ATER, CNAM Joint work with É. Parent, P. Barbillon, M. Keller and N. Bousquet Outline Computer model validation
More informationStochastic Collocation Methods for Polynomial Chaos: Analysis and Applications
Stochastic Collocation Methods for Polynomial Chaos: Analysis and Applications Dongbin Xiu Department of Mathematics, Purdue University Support: AFOSR FA955-8-1-353 (Computational Math) SF CAREER DMS-64535
More informationPenalized Loss functions for Bayesian Model Choice
Penalized Loss functions for Bayesian Model Choice Martyn International Agency for Research on Cancer Lyon, France 13 November 2009 The pure approach For a Bayesian purist, all uncertainty is represented
More informationarxiv: v1 [stat.co] 23 Apr 2018
Bayesian Updating and Uncertainty Quantification using Sequential Tempered MCMC with the Rank-One Modified Metropolis Algorithm Thomas A. Catanach and James L. Beck arxiv:1804.08738v1 [stat.co] 23 Apr
More informationBayesian inference J. Daunizeau
Bayesian inference J. Daunizeau Brain and Spine Institute, Paris, France Wellcome Trust Centre for Neuroimaging, London, UK Overview of the talk 1 Probabilistic modelling and representation of uncertainty
More informationSRNDNA Model Fitting in RL Workshop
SRNDNA Model Fitting in RL Workshop yael@princeton.edu Topics: 1. trial-by-trial model fitting (morning; Yael) 2. model comparison (morning; Yael) 3. advanced topics (hierarchical fitting etc.) (afternoon;
More informationIntroduction to Bayesian Data Analysis
Introduction to Bayesian Data Analysis Phil Gregory University of British Columbia March 2010 Hardback (ISBN-10: 052184150X ISBN-13: 9780521841504) Resources and solutions This title has free Mathematica
More informationWhen one of things that you don t know is the number of things that you don t know
When one of things that you don t know is the number of things that you don t know Malcolm Sambridge Research School of Earth Sciences, Australian National University Canberra, Australia. Sambridge. M.,
More informationMCMC Sampling for Bayesian Inference using L1-type Priors
MÜNSTER MCMC Sampling for Bayesian Inference using L1-type Priors (what I do whenever the ill-posedness of EEG/MEG is just not frustrating enough!) AG Imaging Seminar Felix Lucka 26.06.2012 , MÜNSTER Sampling
More informationPATTERN RECOGNITION AND MACHINE LEARNING
PATTERN RECOGNITION AND MACHINE LEARNING Chapter 1. Introduction Shuai Huang April 21, 2014 Outline 1 What is Machine Learning? 2 Curve Fitting 3 Probability Theory 4 Model Selection 5 The curse of dimensionality
More informationDevelopment of Stochastic Artificial Neural Networks for Hydrological Prediction
Development of Stochastic Artificial Neural Networks for Hydrological Prediction G. B. Kingston, M. F. Lambert and H. R. Maier Centre for Applied Modelling in Water Engineering, School of Civil and Environmental
More informationApproximate Bayesian Computation: a simulation based approach to inference
Approximate Bayesian Computation: a simulation based approach to inference Richard Wilkinson Simon Tavaré 2 Department of Probability and Statistics University of Sheffield 2 Department of Applied Mathematics
More informationBayesian Hidden Markov Models and Extensions
Bayesian Hidden Markov Models and Extensions Zoubin Ghahramani Department of Engineering University of Cambridge joint work with Matt Beal, Jurgen van Gael, Yunus Saatci, Tom Stepleton, Yee Whye Teh Modeling
More informationSGN Advanced Signal Processing: Lecture 8 Parameter estimation for AR and MA models. Model order selection
SG 21006 Advanced Signal Processing: Lecture 8 Parameter estimation for AR and MA models. Model order selection Ioan Tabus Department of Signal Processing Tampere University of Technology Finland 1 / 28
More informationNeutron inverse kinetics via Gaussian Processes
Neutron inverse kinetics via Gaussian Processes P. Picca Politecnico di Torino, Torino, Italy R. Furfaro University of Arizona, Tucson, Arizona Outline Introduction Review of inverse kinetics techniques
More information7. Estimation and hypothesis testing. Objective. Recommended reading
7. Estimation and hypothesis testing Objective In this chapter, we show how the election of estimators can be represented as a decision problem. Secondly, we consider the problem of hypothesis testing
More informationTransdimensional Markov Chain Monte Carlo Methods. Jesse Kolb, Vedran Lekić (Univ. of MD) Supervisor: Kris Innanen
Transdimensional Markov Chain Monte Carlo Methods Jesse Kolb, Vedran Lekić (Univ. of MD) Supervisor: Kris Innanen Motivation for Different Inversion Technique Inversion techniques typically provide a single
More informationShould all Machine Learning be Bayesian? Should all Bayesian models be non-parametric?
Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/
More informationIntroduction to Bayesian methods in inverse problems
Introduction to Bayesian methods in inverse problems Ville Kolehmainen 1 1 Department of Applied Physics, University of Eastern Finland, Kuopio, Finland March 4 2013 Manchester, UK. Contents Introduction
More informationMarkov Chain Monte Carlo methods
Markov Chain Monte Carlo methods Tomas McKelvey and Lennart Svensson Signal Processing Group Department of Signals and Systems Chalmers University of Technology, Sweden November 26, 2012 Today s learning
More informationUtilizing Adjoint-Based Techniques to Improve the Accuracy and Reliability in Uncertainty Quantification
Utilizing Adjoint-Based Techniques to Improve the Accuracy and Reliability in Uncertainty Quantification Tim Wildey Sandia National Laboratories Center for Computing Research (CCR) Collaborators: E. Cyr,
More informationParameter Estimation. William H. Jefferys University of Texas at Austin Parameter Estimation 7/26/05 1
Parameter Estimation William H. Jefferys University of Texas at Austin bill@bayesrules.net Parameter Estimation 7/26/05 1 Elements of Inference Inference problems contain two indispensable elements: Data
More informationWhich model to use? How can we deal with these decisions automatically? Note flailing in data gaps and beyond ends for high M
Which model to use? Microlens modellers face many dilemmas: Blending or no blending? Keep original error bars, or resacle them? Simple scale factor, or more elaborate model. Point source or extended source?
More informationIntroduction to Machine Learning
Introduction to Machine Learning Brown University CSCI 1950-F, Spring 2012 Prof. Erik Sudderth Lecture 25: Markov Chain Monte Carlo (MCMC) Course Review and Advanced Topics Many figures courtesy Kevin
More informationPattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
More informationBayesian methods in economics and finance
1/26 Bayesian methods in economics and finance Linear regression: Bayesian model selection and sparsity priors Linear Regression 2/26 Linear regression Model for relationship between (several) independent
More informationStochastic Spectral Approaches to Bayesian Inference
Stochastic Spectral Approaches to Bayesian Inference Prof. Nathan L. Gibson Department of Mathematics Applied Mathematics and Computation Seminar March 4, 2011 Prof. Gibson (OSU) Spectral Approaches to
More informationBayesian rules of probability as principles of logic [Cox] Notation: pr(x I) is the probability (or pdf) of x being true given information I
Bayesian rules of probability as principles of logic [Cox] Notation: pr(x I) is the probability (or pdf) of x being true given information I 1 Sum rule: If set {x i } is exhaustive and exclusive, pr(x
More informationContents. Part I: Fundamentals of Bayesian Inference 1
Contents Preface xiii Part I: Fundamentals of Bayesian Inference 1 1 Probability and inference 3 1.1 The three steps of Bayesian data analysis 3 1.2 General notation for statistical inference 4 1.3 Bayesian
More informationOverview. Bayesian assimilation of experimental data into simulation (for Goland wing flutter) Why not uncertainty quantification?
Delft University of Technology Overview Bayesian assimilation of experimental data into simulation (for Goland wing flutter), Simao Marques 1. Why not uncertainty quantification? 2. Why uncertainty quantification?
More informationMarkov chain Monte Carlo methods for visual tracking
Markov chain Monte Carlo methods for visual tracking Ray Luo rluo@cory.eecs.berkeley.edu Department of Electrical Engineering and Computer Sciences University of California, Berkeley Berkeley, CA 94720
More informationBayesian inference J. Daunizeau
Bayesian inference J. Daunizeau Brain and Spine Institute, Paris, France Wellcome Trust Centre for Neuroimaging, London, UK Overview of the talk 1 Probabilistic modelling and representation of uncertainty
More informationF denotes cumulative density. denotes probability density function; (.)
BAYESIAN ANALYSIS: FOREWORDS Notation. System means the real thing and a model is an assumed mathematical form for the system.. he probability model class M contains the set of the all admissible models
More information(Extended) Kalman Filter
(Extended) Kalman Filter Brian Hunt 7 June 2013 Goals of Data Assimilation (DA) Estimate the state of a system based on both current and all past observations of the system, using a model for the system
More informationRecent Advances in Bayesian Inference for Inverse Problems
Recent Advances in Bayesian Inference for Inverse Problems Felix Lucka University College London, UK f.lucka@ucl.ac.uk Applied Inverse Problems Helsinki, May 25, 2015 Bayesian Inference for Inverse Problems
More informationPART I INTRODUCTION The meaning of probability Basic definitions for frequentist statistics and Bayesian inference Bayesian inference Combinatorics
Table of Preface page xi PART I INTRODUCTION 1 1 The meaning of probability 3 1.1 Classical definition of probability 3 1.2 Statistical definition of probability 9 1.3 Bayesian understanding of probability
More informationInference Control and Driving of Natural Systems
Inference Control and Driving of Natural Systems MSci/MSc/MRes nick.jones@imperial.ac.uk The fields at play We will be drawing on ideas from Bayesian Cognitive Science (psychology and neuroscience), Biological
More informationThe University of Auckland Applied Mathematics Bayesian Methods for Inverse Problems : why and how Colin Fox Tiangang Cui, Mike O Sullivan (Auckland),
The University of Auckland Applied Mathematics Bayesian Methods for Inverse Problems : why and how Colin Fox Tiangang Cui, Mike O Sullivan (Auckland), Geoff Nicholls (Statistics, Oxford) fox@math.auckland.ac.nz
More informationLearning Bayesian belief networks
Lecture 4 Learning Bayesian belief networks Milos Hauskrecht milos@cs.pitt.edu 5329 Sennott Square Administration Midterm: Monday, March 7, 2003 In class Closed book Material covered by Wednesday, March
More informationLarge Scale Bayesian Inference
Large Scale Bayesian I in Cosmology Jens Jasche Garching, 11 September 2012 Introduction Cosmography 3D density and velocity fields Power-spectra, bi-spectra Dark Energy, Dark Matter, Gravity Cosmological
More informationVariational Bayesian Inference Techniques
Advanced Signal Processing 2, SE Variational Bayesian Inference Techniques Johann Steiner 1 Outline Introduction Sparse Signal Reconstruction Sparsity Priors Benefits of Sparse Bayesian Inference Variational
More informationComputer Vision Group Prof. Daniel Cremers. 10a. Markov Chain Monte Carlo
Group Prof. Daniel Cremers 10a. Markov Chain Monte Carlo Markov Chain Monte Carlo In high-dimensional spaces, rejection sampling and importance sampling are very inefficient An alternative is Markov Chain
More informationPart 1: Expectation Propagation
Chalmers Machine Learning Summer School Approximate message passing and biomedicine Part 1: Expectation Propagation Tom Heskes Machine Learning Group, Institute for Computing and Information Sciences Radboud
More informationBayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
More informationPredictive Engineering and Computational Sciences. Research Challenges in VUQ. Robert Moser. The University of Texas at Austin.
PECOS Predictive Engineering and Computational Sciences Research Challenges in VUQ Robert Moser The University of Texas at Austin March 2012 Robert Moser 1 / 9 Justifying Extrapolitive Predictions Need
More informationA Statistical Input Pruning Method for Artificial Neural Networks Used in Environmental Modelling
A Statistical Input Pruning Method for Artificial Neural Networks Used in Environmental Modelling G. B. Kingston, H. R. Maier and M. F. Lambert Centre for Applied Modelling in Water Engineering, School
More informationOutline Introduction OLS Design of experiments Regression. Metamodeling. ME598/494 Lecture. Max Yi Ren
1 / 34 Metamodeling ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 1, 2015 2 / 34 1. preliminaries 1.1 motivation 1.2 ordinary least square 1.3 information
More informationBayesian Regression Linear and Logistic Regression
When we want more than point estimates Bayesian Regression Linear and Logistic Regression Nicole Beckage Ordinary Least Squares Regression and Lasso Regression return only point estimates But what if we
More informationRegularized Regression A Bayesian point of view
Regularized Regression A Bayesian point of view Vincent MICHEL Director : Gilles Celeux Supervisor : Bertrand Thirion Parietal Team, INRIA Saclay Ile-de-France LRI, Université Paris Sud CEA, DSV, I2BM,
More informationBayesian Methods and Uncertainty Quantification for Nonlinear Inverse Problems
Bayesian Methods and Uncertainty Quantification for Nonlinear Inverse Problems John Bardsley, University of Montana Collaborators: H. Haario, J. Kaipio, M. Laine, Y. Marzouk, A. Seppänen, A. Solonen, Z.
More informationNew Advances in Uncertainty Analysis and Estimation
New Advances in Uncertainty Analysis and Estimation Overview: Both sensor observation data and mathematical models are used to assist in the understanding of physical dynamic systems. However, observational
More informationCalibrating Environmental Engineering Models and Uncertainty Analysis
Models and Cornell University Oct 14, 2008 Project Team Christine Shoemaker, co-pi, Professor of Civil and works in applied optimization, co-pi Nikolai Blizniouk, PhD student in Operations Research now
More informationy(x) = x w + ε(x), (1)
Linear regression We are ready to consider our first machine-learning problem: linear regression. Suppose that e are interested in the values of a function y(x): R d R, here x is a d-dimensional vector-valued
More informationAn introduction to Bayesian inference and model comparison J. Daunizeau
An introduction to Bayesian inference and model comparison J. Daunizeau ICM, Paris, France TNU, Zurich, Switzerland Overview of the talk An introduction to probabilistic modelling Bayesian model comparison
More informationBayesian linear regression
Bayesian linear regression Linear regression is the basis of most statistical modeling. The model is Y i = X T i β + ε i, where Y i is the continuous response X i = (X i1,..., X ip ) T is the corresponding
More informationINTRODUCTION TO PATTERN RECOGNITION
INTRODUCTION TO PATTERN RECOGNITION INSTRUCTOR: WEI DING 1 Pattern Recognition Automatic discovery of regularities in data through the use of computer algorithms With the use of these regularities to take
More informationBayesian Inverse Problems
Bayesian Inverse Problems Jonas Latz Input/Output: www.latz.io Technical University of Munich Department of Mathematics, Chair for Numerical Analysis Email: jonas.latz@tum.de Garching, July 10 2018 Guest
More informationRecursive Deviance Information Criterion for the Hidden Markov Model
International Journal of Statistics and Probability; Vol. 5, No. 1; 2016 ISSN 1927-7032 E-ISSN 1927-7040 Published by Canadian Center of Science and Education Recursive Deviance Information Criterion for
More informationBayesian Inference and MCMC
Bayesian Inference and MCMC Aryan Arbabi Partly based on MCMC slides from CSC412 Fall 2018 1 / 18 Bayesian Inference - Motivation Consider we have a data set D = {x 1,..., x n }. E.g each x i can be the
More informationFunctional Estimation in Systems Defined by Differential Equation using Bayesian Smoothing Methods
Université Catholique de Louvain Institut de Statistique, Biostatistique et Sciences Actuarielles Functional Estimation in Systems Defined by Differential Equation using Bayesian Smoothing Methods 19th
More informationRecent advances in cosmological Bayesian model comparison
Recent advances in cosmological Bayesian model comparison Astrophysics, Imperial College London www.robertotrotta.com 1. What is model comparison? 2. The Bayesian model comparison framework 3. Cosmological
More informationNonparmeteric Bayes & Gaussian Processes. Baback Moghaddam Machine Learning Group
Nonparmeteric Bayes & Gaussian Processes Baback Moghaddam baback@jpl.nasa.gov Machine Learning Group Outline Bayesian Inference Hierarchical Models Model Selection Parametric vs. Nonparametric Gaussian
More informationDoing Bayesian Integrals
ASTR509-13 Doing Bayesian Integrals The Reverend Thomas Bayes (c.1702 1761) Philosopher, theologian, mathematician Presbyterian (non-conformist) minister Tunbridge Wells, UK Elected FRS, perhaps due to
More informationMiscellany : Long Run Behavior of Bayesian Methods; Bayesian Experimental Design (Lecture 4)
Miscellany : Long Run Behavior of Bayesian Methods; Bayesian Experimental Design (Lecture 4) Tom Loredo Dept. of Astronomy, Cornell University http://www.astro.cornell.edu/staff/loredo/bayes/ Bayesian
More informationAdvanced Statistical Methods. Lecture 6
Advanced Statistical Methods Lecture 6 Convergence distribution of M.-H. MCMC We denote the PDF estimated by the MCMC as. It has the property Convergence distribution After some time, the distribution
More informationRecent Advances in Bayesian Inference Techniques
Recent Advances in Bayesian Inference Techniques Christopher M. Bishop Microsoft Research, Cambridge, U.K. research.microsoft.com/~cmbishop SIAM Conference on Data Mining, April 2004 Abstract Bayesian
More informationCS Homework 3. October 15, 2009
CS 294 - Homework 3 October 15, 2009 If you have questions, contact Alexandre Bouchard (bouchard@cs.berkeley.edu) for part 1 and Alex Simma (asimma@eecs.berkeley.edu) for part 2. Also check the class website
More informationDIC, AIC, BIC, PPL, MSPE Residuals Predictive residuals
DIC, AIC, BIC, PPL, MSPE Residuals Predictive residuals Overall Measures of GOF Deviance: this measures the overall likelihood of the model given a parameter vector D( θ) = 2 log L( θ) This can be averaged
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning Empirical Bayes, Hierarchical Bayes Mark Schmidt University of British Columbia Winter 2017 Admin Assignment 5: Due April 10. Project description on Piazza. Final details coming
More informationMODULE -4 BAYEIAN LEARNING
MODULE -4 BAYEIAN LEARNING CONTENT Introduction Bayes theorem Bayes theorem and concept learning Maximum likelihood and Least Squared Error Hypothesis Maximum likelihood Hypotheses for predicting probabilities
More informationDefault Priors and Effcient Posterior Computation in Bayesian
Default Priors and Effcient Posterior Computation in Bayesian Factor Analysis January 16, 2010 Presented by Eric Wang, Duke University Background and Motivation A Brief Review of Parameter Expansion Literature
More informationTesting Restrictions and Comparing Models
Econ. 513, Time Series Econometrics Fall 00 Chris Sims Testing Restrictions and Comparing Models 1. THE PROBLEM We consider here the problem of comparing two parametric models for the data X, defined by
More informationBayesian Inference for Discretely Sampled Diffusion Processes: A New MCMC Based Approach to Inference
Bayesian Inference for Discretely Sampled Diffusion Processes: A New MCMC Based Approach to Inference Osnat Stramer 1 and Matthew Bognar 1 Department of Statistics and Actuarial Science, University of
More informationOutlier detection in ARIMA and seasonal ARIMA models by. Bayesian Information Type Criteria
Outlier detection in ARIMA and seasonal ARIMA models by Bayesian Information Type Criteria Pedro Galeano and Daniel Peña Departamento de Estadística Universidad Carlos III de Madrid 1 Introduction The
More informationLarge-scale Ordinal Collaborative Filtering
Large-scale Ordinal Collaborative Filtering Ulrich Paquet, Blaise Thomson, and Ole Winther Microsoft Research Cambridge, University of Cambridge, Technical University of Denmark ulripa@microsoft.com,brmt2@cam.ac.uk,owi@imm.dtu.dk
More informationA Bayesian perspective on GMM and IV
A Bayesian perspective on GMM and IV Christopher A. Sims Princeton University sims@princeton.edu November 26, 2013 What is a Bayesian perspective? A Bayesian perspective on scientific reporting views all
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationProbabilistic machine learning group, Aalto University Bayesian theory and methods, approximative integration, model
Aki Vehtari, Aalto University, Finland Probabilistic machine learning group, Aalto University http://research.cs.aalto.fi/pml/ Bayesian theory and methods, approximative integration, model assessment and
More informationMultimodal Nested Sampling
Multimodal Nested Sampling Farhan Feroz Astrophysics Group, Cavendish Lab, Cambridge Inverse Problems & Cosmology Most obvious example: standard CMB data analysis pipeline But many others: object detection,
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationBAYESIAN METHODS FOR VARIABLE SELECTION WITH APPLICATIONS TO HIGH-DIMENSIONAL DATA
BAYESIAN METHODS FOR VARIABLE SELECTION WITH APPLICATIONS TO HIGH-DIMENSIONAL DATA Intro: Course Outline and Brief Intro to Marina Vannucci Rice University, USA PASI-CIMAT 04/28-30/2010 Marina Vannucci
More information