Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother
|
|
- Deborah Barton
- 5 years ago
- Views:
Transcription
1 Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother J. E. Griffin and M. F. J. Steel University of Warwick Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 1/23
2 Introduction Nonparametric regression offers flexibility that many real applications require Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 2/23
3 Introduction Nonparametric regression offers flexibility that many real applications require Nonlinear relationships with minimal assumptions Interest could be in various aspects (cond. location, cond. spread,... ) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 2/23
4 Introduction Nonparametric regression offers flexibility that many real applications require Nonlinear relationships with minimal assumptions Interest could be in various aspects (cond. location, cond. spread,... ) Existing Bayesian approaches: flexible location modelling (Gaussian Processes, splines, wavelets) and local modelling (partial exchangeability, BPM) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 2/23
5 Introduction Nonparametric regression offers flexibility that many real applications require Nonlinear relationships with minimal assumptions Interest could be in various aspects (cond. location, cond. spread,... ) Existing Bayesian approaches: flexible location modelling (Gaussian Processes, splines, wavelets) and local modelling (partial exchangeability, BPM) Here we attempt to combine Bayesian NP function estimation and density estimation We also want to allow for centring over parametric models Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 2/23
6 Bayesian Nonparametric Modelling Usual hierarchical Bayesian model for y 1,...,y n : y i k(ψ i ), ψ i F, F Π, where k( ) is pdf and Π is flexible class Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 3/23
7 Bayesian Nonparametric Modelling Usual hierarchical Bayesian model for y 1,...,y n : y i k(ψ i ), ψ i F, F Π, where k( ) is pdf and Π is flexible class Here concentrate on stick-breaking class F d = i=1 p i δ θi, δ θ Dirac measure at θ and p i = V i j<i (1 V j) V 1,V 2,V 3,... independent with V i Be(a i,b i ) θ 1,θ 2,θ 3,... i.i.d. from centring distribution H Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 3/23
8 Bayesian Nonparametric Modelling Usual hierarchical Bayesian model for y 1,...,y n : y i k(ψ i ), ψ i F, F Π, where k( ) is pdf and Π is flexible class Here concentrate on stick-breaking class F d = i=1 p i δ θi, δ θ Dirac measure at θ and p i = V i j<i (1 V j) V 1,V 2,V 3,... independent with V i Be(a i,b i ) θ 1,θ 2,θ 3,... i.i.d. from centring distribution H Dirichlet process when a i = 1 and b i = M for all i Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 3/23
9 Bayesian Nonparametric Regression For pairs (x 1,y 1 ), (x 2,y 2 ),...,(x n,y n ) a natural extension is y i k(ψ i ), ψ i F x, F x d = j=1 p j (x)δ θj (x). Covers existing processes: Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 4/23
10 Bayesian Nonparametric Regression For pairs (x 1,y 1 ), (x 2,y 2 ),...,(x n,y n ) a natural extension is y i k(ψ i ), ψ i F x, F x d = j=1 p j (x)δ θj (x). Covers existing processes: p i (x) = p i : single p DDP (MacEachern, 2001; De Iorio et al., 2004; Gelfand et al., 2005) θ i (x) = θ i : e.g. πddp (Griffin and Steel, 2006) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 4/23
11 Bayesian Nonparametric Regression Here focus on models with θ i (x) = θ i Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 5/23
12 Bayesian Nonparametric Regression Here focus on models with θ i (x) = θ i Often undersmooth posterior mean (piecewise constant) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 5/23
13 Bayesian Nonparametric Regression Here focus on models with θ i (x) = θ i Often undersmooth posterior mean (piecewise constant) So consider: y i g(x i ) m(x i ) k(ψ i ), ψ F x, F x d = i=1 p i (x)δ θi, conditional regression function: parametric part g(x) and a nonparametric part m(x) For m(x): Gaussian process prior with mean 0 and covariance σ 2 0 ρ(x i,x j ) where ρ(x i,x j ) is a Matèrn correlation function Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 5/23
14 Bayesian Density Smoother Stick-breaking prior: consider the atoms and their ordering at each x R p Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 6/23
15 Bayesian Density Smoother Stick-breaking prior: consider the atoms and their ordering at each x R p Define closed, convex sets in R p, say I 1,I 2,... and construct F(x) by only considering {(V j,θ j ) x I j } Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 6/23
16 Bayesian Density Smoother Stick-breaking prior: consider the atoms and their ordering at each x R p Define closed, convex sets in R p, say I 1,I 2,... and construct F(x) by only considering {(V j,θ j ) x I j } Ordering determined by associated t j > 0 (smallest first) So prior is defined by (V 1,θ 1,I 1,t 1 ), (V 2,θ 2,I 2,t 2 ),... Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 6/23
17 Bayesian Density Smoother Stick-breaking prior: consider the atoms and their ordering at each x R p Define closed, convex sets in R p, say I 1,I 2,... and construct F(x) by only considering {(V j,θ j ) x I j } Ordering determined by associated t j > 0 (smallest first) So prior is defined by (V 1,θ 1,I 1,t 1 ), (V 2,θ 2,I 2,t 2 ),... If p s,w = P(s,w I k s I k or w I k ) is given, then Corr(F s,f w ) = 2(M + 1)p s,w 2 + M(1 + p s,w ) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 6/23
18 Bayesian Density Smoother For I k choose a ball of radius r k around C k Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 7/23
19 Bayesian Density Smoother For I k choose a ball of radius r k around C k (C 1,r 1,t 1 ), (C 2,r 1,t 1 ),...: Poisson process on R p R 2 + with intensity p(r) (pdf on R + ) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 7/23
20 Bayesian Density Smoother For I k choose a ball of radius r k around C k (C 1,r 1,t 1 ), (C 2,r 1,t 1 ),...: Poisson process on R p R 2 + with intensity p(r) (pdf on R + ) Some results for case where x R: p s,s+u = 2µ 2 ui 4µ 2µ 2 + ui where µ = E[r], I = u/2 p(r)dr and µ 2 = u/2 rp(r)dr If r Ga(α, ), F x is mean square differentiable of order q = 1, 2,... if and only if α 2q 1. Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 7/23
21 Dirichlet Process Regression Smoother Definition: Let (t i,c i,r i ) be a Poisson process with intensity βα Γ(α) rα 1 i exp{ βr i } on R + R p R + with associated marks (V i,θ i ) which are i.i.d. from Be(1,M) and H. If F x = (1 V j )δ θi {i x B ri (C i )} V i {j x B rj (C j ),t j <t i } then {F x x R p } follows a Dirichlet Process Regression Smoother (DPRS), represented as DPRS(M, H, α, β) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 8/23
22 Centring over Models Centre nonparametric model over nontrivial parametric model: Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 9/23
23 Centring over Models Centre nonparametric model over nontrivial parametric model: nonparametric model can indicate flaws in common parametric models can aid interpretation and prior elicitation Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 9/23
24 Centring over Models Centre nonparametric model over nontrivial parametric model: nonparametric model can indicate flaws in common parametric models can aid interpretation and prior elicitation Regression errors: ǫ i = y i g(x i ) All models centred over ǫ i N(0,σ 2 ) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 9/23
25 Centring over Models Model 1(a): ǫ i N(µ i,aσ 2 ), µ i F xi, F x DPRS(M,H,α,β), H N(0, (1 a)σ 2 ), 0 < a < 1 a close to zero: nonparametric modelling crucial Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 10/23
26 Centring over Models Model 1(a): ǫ i N(µ i,aσ 2 ), µ i F xi, F x DPRS(M,H,α,β), H N(0, (1 a)σ 2 ), 0 < a < 1 a close to zero: nonparametric modelling crucial Model 1(b): ǫ i m(x i ) N(µ i,aσ 2 ), µ i F xi, F x DPRS(M,H,α,β), H N(0, (1 a)σ 2 ), with σ 2 = σ 2 + σ0 2 and b = σ2 0 /σ2 b indicates relative importance m(x) (GP) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 10/23
27 Centring over Models Large a, small b: nonparametric modelling less critical, and g(x) is a good parametric model Interpretation of g(x) nonstandard (given F x ), so consider fixing median Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 11/23
28 Centring over Models Large a, small b: nonparametric modelling less critical, and g(x) is a good parametric model Interpretation of g(x) nonstandard (given F x ), so consider fixing median Model 2: ǫ i m(x i ) U ( σ ui,σ ui ), u i F xi, F x DPRS(M,H,α,β), which leads to symmetric error distributions Choose H Ga(3/2, 1/2) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 11/23
29 Computational Issues DPRS allows for much simpler MCMC sampling scheme than in Griffin and Steel (2006) Adapt Retrospective sampling methods from Papaspiliopoulos and Roberts (2004) (no truncation) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 12/23
30 Examples Example 1: Sine wave 100 observations from y i = sin(2πx i ) + ǫ i with x i U(0, 1) and Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 13/23
31 Examples Example 1: Sine wave 100 observations from y i = sin(2πx i ) + ǫ i with x i U(0, 1) and Error 1: ǫ i is t with 2.5 d.f. and conditional variance σ 2 (x) = (x 1 2 )2 Error 2: p(ǫ i x i ) = 0.3N(0.3, 0.01) + 0.7N( x i, 0.01) Bimodal at x i = 0 and unimodal (and normal) at x i = 1 Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 13/23
32 Examples Example 1: Sine wave 100 observations from y i = sin(2πx i ) + ǫ i with x i U(0, 1) and Error 1: ǫ i is t with 2.5 d.f. and conditional variance σ 2 (x) = (x 1 2 )2 Error 2: p(ǫ i x i ) = 0.3N(0.3, 0.01) + 0.7N( x i, 0.01) Bimodal at x i = 0 and unimodal (and normal) at x i = 1 Take g(x) = 0 throughout Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 13/23
33 Example: Sine wave, error 1 Model 1(a) Model 1(b) Model p(y x) Cond. var Figure 1: posterior predictive and σ 2 (x) (true value dashed) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 14/23
34 Example: Sine wave Results with Error 1: Small values of a indicate lack of normality Model 1(a): blocky as expected Models with GP regression function do better Cond. variance reasonably captured by latter models Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 15/23
35 Example: Sine wave, error 2 Predictive p(y x) Posterior of m(x) Predictive error Model 1(b) Model Figure 2: posterior predictive density, m(x) with data (dots), and posterior predictive error distribution Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 16/23
36 Example: Sine wave Results with Error 2: Model 1(b) can deal with bimodality Model 2 can not, by construction Small a: nonnormality Large b: constant centring model is poor Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 17/23
37 Examples Example 2: Scale economies Cost function for electricity distribution tc = f(cust) + β 1 wage + β 2 pcap + β 3 PUC + β 4 kwh + β 5 life + β 6 lf + β 7 kmwire + ǫ, tc: log total cost per customer cust: log number of customers Data: 81 municipal distributors in Ontario, Canada during 1993 Interest: effect of cust and other regressors on tc Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 18/23
38 Example: Scale economies DPRS model with cust as the covariate for ǫ and the GP Centre the model over two parametric regression models by choosing f(cust) to be: Parametric 1: γ 1 + γ 2 cust Parametric 2: γ 1 + γ 2 cust + γ 3 cust 2 Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 19/23
39 Example: Scale economies Results with Parametric 1: Inference on β, γ quite different for parametric and nonparametric models Small a for Model 1(a), much larger for Model 1(b) Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 20/23
40 Example: Scale economies Results with Parametric 1: Inference on β, γ quite different for parametric and nonparametric models Small a for Model 1(a), much larger for Model 1(b) Results with Parametric 2: Inference on β, γ similar for parametric and nonparametric models Small a for Model 1(a), much larger for Model 1(b) Now b smaller than with Parametric 1 Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 20/23
41 Example: Scale economies Model 1(a) Model 1(b) Model Lin Quadr Figure 3: Posterior mean of the nonparametric component(s) of the model Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 21/23
42 Example: Scale economies Nonparametric correction to parametric fit: linear model suggests problems; quadratic is better Conclusion: Quadratic parametric model is not bad; linear is inappropriate Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 22/23
43 Discussion Combine Bayesian NP density estimation and regression modelling Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 23/23
44 Discussion Combine Bayesian NP density estimation and regression modelling Separate modelling of components: NP smoother needs to do less Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 23/23
45 Discussion Combine Bayesian NP density estimation and regression modelling Separate modelling of components: NP smoother needs to do less Centring over parametric models: More structured approach Can identify specific problems of parametric models Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 23/23
46 Discussion Combine Bayesian NP density estimation and regression modelling Separate modelling of components: NP smoother needs to do less Centring over parametric models: More structured approach Can identify specific problems of parametric models Modelling ideas can be used in combination with any NP prior that allows for dependence on covariates Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother p. 23/23
Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother
Bayesian Nonparametric Modelling with the Dirichlet Process Regression Smoother J.E. Griffin and M. F. J. Steel University of Kent and University of Warwick Abstract In this paper we discuss implementing
More informationBAYESIAN NONPARAMETRIC MODELLING WITH THE DIRICHLET PROCESS REGRESSION SMOOTHER
Statistica Sinica 20 (2010), 1507-1527 BAYESIAN NONPARAMETRIC MODELLING WITH THE DIRICHLET PROCESS REGRESSION SMOOTHER J. E. Griffin and M. F. J. Steel University of Kent and University of Warwick Abstract:
More informationNormalized kernel-weighted random measures
Normalized kernel-weighted random measures Jim Griffin University of Kent 1 August 27 Outline 1 Introduction 2 Ornstein-Uhlenbeck DP 3 Generalisations Bayesian Density Regression We observe data (x 1,
More informationBayesian Nonparametric Regression through Mixture Models
Bayesian Nonparametric Regression through Mixture Models Sara Wade Bocconi University Advisor: Sonia Petrone October 7, 2013 Outline 1 Introduction 2 Enriched Dirichlet Process 3 EDP Mixtures for Regression
More informationNonparametric Bayesian Methods - Lecture I
Nonparametric Bayesian Methods - Lecture I Harry van Zanten Korteweg-de Vries Institute for Mathematics CRiSM Masterclass, April 4-6, 2016 Overview of the lectures I Intro to nonparametric Bayesian statistics
More informationNonparametric Bayesian modeling for dynamic ordinal regression relationships
Nonparametric Bayesian modeling for dynamic ordinal regression relationships Athanasios Kottas Department of Applied Mathematics and Statistics, University of California, Santa Cruz Joint work with Maria
More informationCompound Random Measures
Compound Random Measures Jim Griffin (joint work with Fabrizio Leisen) University of Kent Introduction: Two clinical studies 3 CALGB8881 3 CALGB916 2 2 β 1 1 β 1 1 1 5 5 β 1 5 5 β Infinite mixture models
More informationA Nonparametric Model for Stationary Time Series
A Nonparametric Model for Stationary Time Series Isadora Antoniano-Villalobos Bocconi University, Milan, Italy. isadora.antoniano@unibocconi.it Stephen G. Walker University of Texas at Austin, USA. s.g.walker@math.utexas.edu
More informationNon-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
More informationStatistics & Data Sciences: First Year Prelim Exam May 2018
Statistics & Data Sciences: First Year Prelim Exam May 2018 Instructions: 1. Do not turn this page until instructed to do so. 2. Start each new question on a new sheet of paper. 3. This is a closed book
More informationLecture: Gaussian Process Regression. STAT 6474 Instructor: Hongxiao Zhu
Lecture: Gaussian Process Regression STAT 6474 Instructor: Hongxiao Zhu Motivation Reference: Marc Deisenroth s tutorial on Robot Learning. 2 Fast Learning for Autonomous Robots with Gaussian Processes
More informationDefault priors for density estimation with mixture models
Bayesian Analysis ) 5, Number, pp. 45 64 Default priors for density estimation with mixture models J.E. Griffin Abstract. The infinite mixture of normals model has become a popular method for density estimation
More informationSTAT Advanced Bayesian Inference
1 / 32 STAT 625 - Advanced Bayesian Inference Meng Li Department of Statistics Jan 23, 218 The Dirichlet distribution 2 / 32 θ Dirichlet(a 1,...,a k ) with density p(θ 1,θ 2,...,θ k ) = k j=1 Γ(a j) Γ(
More informationSlice Sampling Mixture Models
Slice Sampling Mixture Models Maria Kalli, Jim E. Griffin & Stephen G. Walker Centre for Health Services Studies, University of Kent Institute of Mathematics, Statistics & Actuarial Science, University
More informationHierarchical Modeling for Univariate Spatial Data
Hierarchical Modeling for Univariate Spatial Data Geography 890, Hierarchical Bayesian Models for Environmental Spatial Data Analysis February 15, 2011 1 Spatial Domain 2 Geography 890 Spatial Domain This
More informationICML Scalable Bayesian Inference on Point processes. with Gaussian Processes. Yves-Laurent Kom Samo & Stephen Roberts
ICML 2015 Scalable Nonparametric Bayesian Inference on Point Processes with Gaussian Processes Machine Learning Research Group and Oxford-Man Institute University of Oxford July 8, 2015 Point Processes
More informationPattern Recognition and Machine Learning. Bishop Chapter 2: Probability Distributions
Pattern Recognition and Machine Learning Chapter 2: Probability Distributions Cécile Amblard Alex Kläser Jakob Verbeek October 11, 27 Probability Distributions: General Density Estimation: given a finite
More informationBayesian Nonparametrics: Dirichlet Process
Bayesian Nonparametrics: Dirichlet Process Yee Whye Teh Gatsby Computational Neuroscience Unit, UCL http://www.gatsby.ucl.ac.uk/~ywteh/teaching/npbayes2012 Dirichlet Process Cornerstone of modern Bayesian
More informationA comparative review of variable selection techniques for covariate dependent Dirichlet process mixture models
A comparative review of variable selection techniques for covariate dependent Dirichlet process mixture models William Barcella 1, Maria De Iorio 1 and Gianluca Baio 1 1 Department of Statistical Science,
More informationBayesian non-parametric model to longitudinally predict churn
Bayesian non-parametric model to longitudinally predict churn Bruno Scarpa Università di Padova Conference of European Statistics Stakeholders Methodologists, Producers and Users of European Statistics
More informationBayesian Nonparametrics
Bayesian Nonparametrics Lorenzo Rosasco 9.520 Class 18 April 11, 2011 About this class Goal To give an overview of some of the basic concepts in Bayesian Nonparametrics. In particular, to discuss Dirichelet
More informationBayesian inference for sample surveys. Roderick Little Module 2: Bayesian models for simple random samples
Bayesian inference for sample surveys Roderick Little Module : Bayesian models for simple random samples Superpopulation Modeling: Estimating parameters Various principles: least squares, method of moments,
More informationFlexible Regression Modeling using Bayesian Nonparametric Mixtures
Flexible Regression Modeling using Bayesian Nonparametric Mixtures Athanasios Kottas Department of Applied Mathematics and Statistics University of California, Santa Cruz Department of Statistics Brigham
More informationLecture 3a: Dirichlet processes
Lecture 3a: Dirichlet processes Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London c.archambeau@cs.ucl.ac.uk Advanced Topics
More informationADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING. Non-linear regression techniques Part - II
1 Non-linear regression techniques Part - II Regression Algorithms in this Course Support Vector Machine Relevance Vector Machine Support vector regression Boosting random projections Relevance vector
More informationAdditive Isotonic Regression
Additive Isotonic Regression Enno Mammen and Kyusang Yu 11. July 2006 INTRODUCTION: We have i.i.d. random vectors (Y 1, X 1 ),..., (Y n, X n ) with X i = (X1 i,..., X d i ) and we consider the additive
More informationBayesian Regression with Heteroscedastic Error Density and Parametric Mean Function
Bayesian Regression with Heteroscedastic Error Density and Parametric Mean Function Justinas Pelenis pelenis@ihs.ac.at Institute for Advanced Studies, Vienna May 8, 2013 Abstract This paper considers a
More informationNonparmeteric Bayes & Gaussian Processes. Baback Moghaddam Machine Learning Group
Nonparmeteric Bayes & Gaussian Processes Baback Moghaddam baback@jpl.nasa.gov Machine Learning Group Outline Bayesian Inference Hierarchical Models Model Selection Parametric vs. Nonparametric Gaussian
More informationDirichlet Processes: Tutorial and Practical Course
Dirichlet Processes: Tutorial and Practical Course (updated) Yee Whye Teh Gatsby Computational Neuroscience Unit University College London August 2007 / MLSS Yee Whye Teh (Gatsby) DP August 2007 / MLSS
More informationOutline. Binomial, Multinomial, Normal, Beta, Dirichlet. Posterior mean, MAP, credible interval, posterior distribution
Outline A short review on Bayesian analysis. Binomial, Multinomial, Normal, Beta, Dirichlet Posterior mean, MAP, credible interval, posterior distribution Gibbs sampling Revisit the Gaussian mixture model
More informationVariational Bayesian Inference for Parametric and Non-Parametric Regression with Missing Predictor Data
for Parametric and Non-Parametric Regression with Missing Predictor Data August 23, 2010 Introduction Bayesian inference For parametric regression: long history (e.g. Box and Tiao, 1973; Gelman, Carlin,
More informationNonparametric Methods
Nonparametric Methods Michael R. Roberts Department of Finance The Wharton School University of Pennsylvania July 28, 2009 Michael R. Roberts Nonparametric Methods 1/42 Overview Great for data analysis
More informationGaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
More informationBridging the two cultures: Latent variable statistical modeling with boosted regression trees
Bridging the two cultures: Latent variable statistical modeling with boosted regression trees Thomas G. Dietterich and Rebecca Hutchinson Oregon State University Corvallis, Oregon, USA 1 A Species Distribution
More informationChapter 2: Fundamentals of Statistics Lecture 15: Models and statistics
Chapter 2: Fundamentals of Statistics Lecture 15: Models and statistics Data from one or a series of random experiments are collected. Planning experiments and collecting data (not discussed here). Analysis:
More informationClassical and Bayesian inference
Classical and Bayesian inference AMS 132 January 18, 2018 Claudia Wehrhahn (UCSC) Classical and Bayesian inference January 18, 2018 1 / 9 Sampling from a Bernoulli Distribution Theorem (Beta-Bernoulli
More informationBayesian spatial quantile regression
Brian J. Reich and Montserrat Fuentes North Carolina State University and David B. Dunson Duke University E-mail:reich@stat.ncsu.edu Tropospheric ozone Tropospheric ozone has been linked with several adverse
More informationOn the Support of MacEachern s Dependent Dirichlet Processes and Extensions
Bayesian Analysis (2012) 7, Number 2, pp. 277 310 On the Support of MacEachern s Dependent Dirichlet Processes and Extensions Andrés F. Barrientos, Alejandro Jara and Fernando A. Quintana Abstract. We
More informationLearning Bayesian network : Given structure and completely observed data
Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani Learning problem Target: true distribution
More informationModel-free prediction intervals for regression and autoregression. Dimitris N. Politis University of California, San Diego
Model-free prediction intervals for regression and autoregression Dimitris N. Politis University of California, San Diego To explain or to predict? Models are indispensable for exploring/utilizing relationships
More informationGaussian processes for inference in stochastic differential equations
Gaussian processes for inference in stochastic differential equations Manfred Opper, AI group, TU Berlin November 6, 2017 Manfred Opper, AI group, TU Berlin (TU Berlin) inference in SDE November 6, 2017
More informationIntroduction to Smoothing spline ANOVA models (metamodelling)
Introduction to Smoothing spline ANOVA models (metamodelling) M. Ratto DYNARE Summer School, Paris, June 215. Joint Research Centre www.jrc.ec.europa.eu Serving society Stimulating innovation Supporting
More informationContents. Part I: Fundamentals of Bayesian Inference 1
Contents Preface xiii Part I: Fundamentals of Bayesian Inference 1 1 Probability and inference 3 1.1 The three steps of Bayesian data analysis 3 1.2 General notation for statistical inference 4 1.3 Bayesian
More informationGAUSSIAN PROCESS REGRESSION
GAUSSIAN PROCESS REGRESSION CSE 515T Spring 2015 1. BACKGROUND The kernel trick again... The Kernel Trick Consider again the linear regression model: y(x) = φ(x) w + ε, with prior p(w) = N (w; 0, Σ). The
More informationBayesian Statistics. Debdeep Pati Florida State University. April 3, 2017
Bayesian Statistics Debdeep Pati Florida State University April 3, 2017 Finite mixture model The finite mixture of normals can be equivalently expressed as y i N(µ Si ; τ 1 S i ), S i k π h δ h h=1 δ h
More informationGaussian Processes (10/16/13)
STA561: Probabilistic machine learning Gaussian Processes (10/16/13) Lecturer: Barbara Engelhardt Scribes: Changwei Hu, Di Jin, Mengdi Wang 1 Introduction In supervised learning, we observe some inputs
More informationA Simple Proof of the Stick-Breaking Construction of the Dirichlet Process
A Simple Proof of the Stick-Breaking Construction of the Dirichlet Process John Paisley Department of Computer Science Princeton University, Princeton, NJ jpaisley@princeton.edu Abstract We give a simple
More informationIntroduction to Gaussian Processes
Introduction to Gaussian Processes Iain Murray murray@cs.toronto.edu CSC255, Introduction to Machine Learning, Fall 28 Dept. Computer Science, University of Toronto The problem Learn scalar function of
More informationEcon 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines
Econ 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines Maximilian Kasy Department of Economics, Harvard University 1 / 37 Agenda 6 equivalent representations of the
More informationNormalising constants and maximum likelihood inference
Normalising constants and maximum likelihood inference Jakob G. Rasmussen Department of Mathematics Aalborg University Denmark March 9, 2011 1/14 Today Normalising constants Approximation of normalising
More informationBayesian isotonic density regression
Bayesian isotonic density regression Lianming Wang and David B. Dunson Biostatistics Branch, MD A3-3 National Institute of Environmental Health Sciences U.S. National Institutes of Health P.O. Box 33,
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond Department of Biomedical Engineering and Computational Science Aalto University January 26, 2012 Contents 1 Batch and Recursive Estimation
More informationThe Jeffreys Prior. Yingbo Li MATH Clemson University. Yingbo Li (Clemson) The Jeffreys Prior MATH / 13
The Jeffreys Prior Yingbo Li Clemson University MATH 9810 Yingbo Li (Clemson) The Jeffreys Prior MATH 9810 1 / 13 Sir Harold Jeffreys English mathematician, statistician, geophysicist, and astronomer His
More informationBayesian Analysis of Risk for Data Mining Based on Empirical Likelihood
1 / 29 Bayesian Analysis of Risk for Data Mining Based on Empirical Likelihood Yuan Liao Wenxin Jiang Northwestern University Presented at: Department of Statistics and Biostatistics Rutgers University
More informationWavelet-Based Nonparametric Modeling of Hierarchical Functions in Colon Carcinogenesis
Wavelet-Based Nonparametric Modeling of Hierarchical Functions in Colon Carcinogenesis Jeffrey S. Morris University of Texas, MD Anderson Cancer Center Joint wor with Marina Vannucci, Philip J. Brown,
More informationBayesian Inference on Joint Mixture Models for Survival-Longitudinal Data with Multiple Features. Yangxin Huang
Bayesian Inference on Joint Mixture Models for Survival-Longitudinal Data with Multiple Features Yangxin Huang Department of Epidemiology and Biostatistics, COPH, USF, Tampa, FL yhuang@health.usf.edu January
More informationFirst Year Examination Department of Statistics, University of Florida
First Year Examination Department of Statistics, University of Florida August 19, 010, 8:00 am - 1:00 noon Instructions: 1. You have four hours to answer questions in this examination.. You must show your
More informationSTAT 518 Intro Student Presentation
STAT 518 Intro Student Presentation Wen Wei Loh April 11, 2013 Title of paper Radford M. Neal [1999] Bayesian Statistics, 6: 475-501, 1999 What the paper is about Regression and Classification Flexible
More informationBayesian Modeling of Conditional Distributions
Bayesian Modeling of Conditional Distributions John Geweke University of Iowa Indiana University Department of Economics February 27, 2007 Outline Motivation Model description Methods of inference Earnings
More informationThe Bayesian approach to inverse problems
The Bayesian approach to inverse problems Youssef Marzouk Department of Aeronautics and Astronautics Center for Computational Engineering Massachusetts Institute of Technology ymarz@mit.edu, http://uqgroup.mit.edu
More informationInformation geometry for bivariate distribution control
Information geometry for bivariate distribution control C.T.J.Dodson + Hong Wang Mathematics + Control Systems Centre, University of Manchester Institute of Science and Technology Optimal control of stochastic
More informationColouring and breaking sticks, pairwise coincidence losses, and clustering expression profiles
Colouring and breaking sticks, pairwise coincidence losses, and clustering expression profiles Peter Green and John Lau University of Bristol P.J.Green@bristol.ac.uk Isaac Newton Institute, 11 December
More informationGibbs Sampling for (Coupled) Infinite Mixture Models in the Stick Breaking Representation
Gibbs Sampling for (Coupled) Infinite Mixture Models in the Stick Breaking Representation Ian Porteous, Alex Ihler, Padhraic Smyth, Max Welling Department of Computer Science UC Irvine, Irvine CA 92697-3425
More informationDirichlet Process Mixtures of Generalized Linear Models
Lauren A. Hannah David M. Blei Warren B. Powell Department of Computer Science, Princeton University Department of Operations Research and Financial Engineering, Princeton University Department of Operations
More informationIntroduction to the regression problem. Luca Martino
Introduction to the regression problem Luca Martino 2017 2018 1 / 30 Approximated outline of the course 1. Very basic introduction to regression 2. Gaussian Processes (GPs) and Relevant Vector Machines
More informationCOMP 551 Applied Machine Learning Lecture 20: Gaussian processes
COMP 55 Applied Machine Learning Lecture 2: Gaussian processes Instructor: Ryan Lowe (ryan.lowe@cs.mcgill.ca) Slides mostly by: (herke.vanhoof@mcgill.ca) Class web page: www.cs.mcgill.ca/~hvanho2/comp55
More informationWrapped Gaussian processes: a short review and some new results
Wrapped Gaussian processes: a short review and some new results Giovanna Jona Lasinio 1, Gianluca Mastrantonio 2 and Alan Gelfand 3 1-Università Sapienza di Roma 2- Università RomaTRE 3- Duke University
More informationPenalized Loss functions for Bayesian Model Choice
Penalized Loss functions for Bayesian Model Choice Martyn International Agency for Research on Cancer Lyon, France 13 November 2009 The pure approach For a Bayesian purist, all uncertainty is represented
More informationChapter 8.8.1: A factorization theorem
LECTURE 14 Chapter 8.8.1: A factorization theorem The characterization of a sufficient statistic in terms of the conditional distribution of the data given the statistic can be difficult to work with.
More informationA Brief Overview of Nonparametric Bayesian Models
A Brief Overview of Nonparametric Bayesian Models Eurandom Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin Also at Machine
More informationUncertainty Quantification for Inverse Problems. November 7, 2011
Uncertainty Quantification for Inverse Problems November 7, 2011 Outline UQ and inverse problems Review: least-squares Review: Gaussian Bayesian linear model Parametric reductions for IP Bias, variance
More informationA Process over all Stationary Covariance Kernels
A Process over all Stationary Covariance Kernels Andrew Gordon Wilson June 9, 0 Abstract I define a process over all stationary covariance kernels. I show how one might be able to perform inference that
More informationModeling Real Estate Data using Quantile Regression
Modeling Real Estate Data using Semiparametric Quantile Regression Department of Statistics University of Innsbruck September 9th, 2011 Overview 1 Application: 2 3 4 Hedonic regression data for house prices
More informationBrief Review on Estimation Theory
Brief Review on Estimation Theory K. Abed-Meraim ENST PARIS, Signal and Image Processing Dept. abed@tsi.enst.fr This presentation is essentially based on the course BASTA by E. Moulines Brief review on
More informationModelling geoadditive survival data
Modelling geoadditive survival data Thomas Kneib & Ludwig Fahrmeir Department of Statistics, Ludwig-Maximilians-University Munich 1. Leukemia survival data 2. Structured hazard regression 3. Mixed model
More informationHierarchical Modelling for Univariate Spatial Data
Hierarchical Modelling for Univariate Spatial Data Sudipto Banerjee 1 and Andrew O. Finley 2 1 Biostatistics, School of Public Health, University of Minnesota, Minneapolis, Minnesota, U.S.A. 2 Department
More informationPart 6: Multivariate Normal and Linear Models
Part 6: Multivariate Normal and Linear Models 1 Multiple measurements Up until now all of our statistical models have been univariate models models for a single measurement on each member of a sample of
More informationSTATISTICS 3A03. Applied Regression Analysis with SAS. Angelo J. Canty
STATISTICS 3A03 Applied Regression Analysis with SAS Angelo J. Canty Office : Hamilton Hall 209 Phone : (905) 525-9140 extn 27079 E-mail : cantya@mcmaster.ca SAS Labs : L1 Friday 11:30 in BSB 249 L2 Tuesday
More informationLecture 16-17: Bayesian Nonparametrics I. STAT 6474 Instructor: Hongxiao Zhu
Lecture 16-17: Bayesian Nonparametrics I STAT 6474 Instructor: Hongxiao Zhu Plan for today Why Bayesian Nonparametrics? Dirichlet Distribution and Dirichlet Processes. 2 Parameter and Patterns Reference:
More informationModeling and Predicting Healthcare Claims
Bayesian Nonparametric Regression Models for Modeling and Predicting Healthcare Claims Robert Richardson Department of Statistics, Brigham Young University Brian Hartman Department of Statistics, Brigham
More informationMeasurement Error and Linear Regression of Astronomical Data. Brandon Kelly Penn State Summer School in Astrostatistics, June 2007
Measurement Error and Linear Regression of Astronomical Data Brandon Kelly Penn State Summer School in Astrostatistics, June 2007 Classical Regression Model Collect n data points, denote i th pair as (η
More informationGaussian processes for spatial modelling in environmental health: parameterizing for flexibility vs. computational efficiency
Gaussian processes for spatial modelling in environmental health: parameterizing for flexibility vs. computational efficiency Chris Paciorek March 11, 2005 Department of Biostatistics Harvard School of
More information5 Operations on Multiple Random Variables
EE360 Random Signal analysis Chapter 5: Operations on Multiple Random Variables 5 Operations on Multiple Random Variables Expected value of a function of r.v. s Two r.v. s: ḡ = E[g(X, Y )] = g(x, y)f X,Y
More informationBayesian Nonparametric Autoregressive Models via Latent Variable Representation
Bayesian Nonparametric Autoregressive Models via Latent Variable Representation Maria De Iorio Yale-NUS College Dept of Statistical Science, University College London Collaborators: Lifeng Ye (UCL, London,
More informationFoundations of Nonparametric Bayesian Methods
1 / 27 Foundations of Nonparametric Bayesian Methods Part II: Models on the Simplex Peter Orbanz http://mlg.eng.cam.ac.uk/porbanz/npb-tutorial.html 2 / 27 Tutorial Overview Part I: Basics Part II: Models
More informationTutorial on Gaussian Processes and the Gaussian Process Latent Variable Model
Tutorial on Gaussian Processes and the Gaussian Process Latent Variable Model (& discussion on the GPLVM tech. report by Prof. N. Lawrence, 06) Andreas Damianou Department of Neuro- and Computer Science,
More informationGaussian Processes in Machine Learning
Gaussian Processes in Machine Learning November 17, 2011 CharmGil Hong Agenda Motivation GP : How does it make sense? Prior : Defining a GP More about Mean and Covariance Functions Posterior : Conditioning
More informationA Bayesian Nonparametric Approach to Monotone Missing Data in Longitudinal Studies with Informative Missingness
A Bayesian Nonparametric Approach to Monotone Missing Data in Longitudinal Studies with Informative Missingness A. Linero and M. Daniels UF, UT-Austin SRC 2014, Galveston, TX 1 Background 2 Working model
More informationConstruction of Dependent Dirichlet Processes based on Poisson Processes
Construction of Dependent Dirichlet Processes based on Poisson Processes Dahua Lin CSAIL, MIT dhlin@mit.edu Eric Grimson CSAIL, MIT welg@csail.mit.edu John Fisher CSAIL, MIT fisher@csail.mit.edu Abstract
More informationMotivation Scale Mixutres of Normals Finite Gaussian Mixtures Skew-Normal Models. Mixture Models. Econ 690. Purdue University
Econ 690 Purdue University In virtually all of the previous lectures, our models have made use of normality assumptions. From a computational point of view, the reason for this assumption is clear: combined
More informationNon-parametric Clustering with Dirichlet Processes
Non-parametric Clustering with Dirichlet Processes Timothy Burns SUNY at Buffalo Mar. 31 2009 T. Burns (SUNY at Buffalo) Non-parametric Clustering with Dirichlet Processes Mar. 31 2009 1 / 24 Introduction
More informationPrior Choice, Summarizing the Posterior
Prior Choice, Summarizing the Posterior Statistics 220 Spring 2005 Copyright c 2005 by Mark E. Irwin Informative Priors Binomial Model: y π Bin(n, π) π is the success probability. Need prior p(π) Bayes
More informationDensity Modeling and Clustering Using Dirichlet Diffusion Trees
p. 1/3 Density Modeling and Clustering Using Dirichlet Diffusion Trees Radford M. Neal Bayesian Statistics 7, 2003, pp. 619-629. Presenter: Ivo D. Shterev p. 2/3 Outline Motivation. Data points generation.
More informationBayesian estimation of the discrepancy with misspecified parametric models
Bayesian estimation of the discrepancy with misspecified parametric models Pierpaolo De Blasi University of Torino & Collegio Carlo Alberto Bayesian Nonparametrics workshop ICERM, 17-21 September 2012
More informationEstimation Theory. as Θ = (Θ 1,Θ 2,...,Θ m ) T. An estimator
Estimation Theory Estimation theory deals with finding numerical values of interesting parameters from given set of data. We start with formulating a family of models that could describe how the data were
More informationGibbs Sampling in Endogenous Variables Models
Gibbs Sampling in Endogenous Variables Models Econ 690 Purdue University Outline 1 Motivation 2 Identification Issues 3 Posterior Simulation #1 4 Posterior Simulation #2 Motivation In this lecture we take
More informationStatistics: Learning models from data
DS-GA 1002 Lecture notes 5 October 19, 2015 Statistics: Learning models from data Learning models from data that are assumed to be generated probabilistically from a certain unknown distribution is a crucial
More informationA Bayesian Treatment of Linear Gaussian Regression
A Bayesian Treatment of Linear Gaussian Regression Frank Wood December 3, 2009 Bayesian Approach to Classical Linear Regression In classical linear regression we have the following model y β, σ 2, X N(Xβ,
More informationPractical Bayesian Optimization of Machine Learning. Learning Algorithms
Practical Bayesian Optimization of Machine Learning Algorithms CS 294 University of California, Berkeley Tuesday, April 20, 2016 Motivation Machine Learning Algorithms (MLA s) have hyperparameters that
More information