Nonparametric Bayesian Methods - Lecture I
|
|
- Donald Young
- 6 years ago
- Views:
Transcription
1 Nonparametric Bayesian Methods - Lecture I Harry van Zanten Korteweg-de Vries Institute for Mathematics CRiSM Masterclass, April 4-6, 2016
2 Overview of the lectures I Intro to nonparametric Bayesian statistics II Consistency and contraction rates III Contraction rates for Gaussian process priors IV Rate-adaptive BNP, Challenges,... 2 / 50
3 Overview of Lecture I Bayesian statistics Nonparametric Bayesian statistics Nonparametric priors Dirichlet processes distribution function estimation Gaussian processes nonparametric regression Conditionally Gaussian processes Dirichlet mixtures nonparametric density estimation Some more examples Concluding remarks 3 / 50
4 Bayesian statistics 4 / 50
5 Bayesian vs. frequentist statistics Mathematical statistics: Have data X, possible distributions {P θ : θ Θ}. Want to make inference about θ on the basis of X. Paradigms in mathematical statistics: Classical /frequentist paradigm: There is a true value θ 0 Θ. Assume X P θ0. Bayesian paradigm: Think of data as being generated in steps as follows: Parameter is random: θ Π. Terminology Π: prior. Data given parameter: X θ P θ. Can then consider θ X : posterior distribution. 5 / 50
6 Bayesian vs. frequentist statistics Mathematical statistics: Have data X, possible distributions {P θ : θ Θ}. Want to make inference about θ on the basis of X. Paradigms in mathematical statistics: Classical /frequentist paradigm: There is a true value θ 0 Θ. Assume X P θ0. Bayesian paradigm: Think of data as being generated in steps as follows: Parameter is random: θ Π. Terminology Π: prior. Data given parameter: X θ P θ. Can then consider θ X : posterior distribution. 5 / 50
7 Bayesian vs. frequentist statistics Mathematical statistics: Have data X, possible distributions {P θ : θ Θ}. Want to make inference about θ on the basis of X. Paradigms in mathematical statistics: Classical /frequentist paradigm: There is a true value θ 0 Θ. Assume X P θ0. Bayesian paradigm: Think of data as being generated in steps as follows: Parameter is random: θ Π. Terminology Π: prior. Data given parameter: X θ P θ. Can then consider θ X : posterior distribution. 5 / 50
8 Bayes example - 1 [Bayes, Price (1763)] Suppose we have a coin that has probability p of turning up heads. We do 50 independent tosses and observe 42 heads. What can we say about p? Here we have an observation (the number 42) from a binomial distribution with parameters 50 and p and want to estimate p. Standard frequentist solution: take the estimate 42/50 = / 50
9 Bayes example - 1 [Bayes, Price (1763)] Suppose we have a coin that has probability p of turning up heads. We do 50 independent tosses and observe 42 heads. What can we say about p? Here we have an observation (the number 42) from a binomial distribution with parameters 50 and p and want to estimate p. Standard frequentist solution: take the estimate 42/50 = / 50
10 Bayes Example - 2 Bayesian approach: choose a prior distribution on p, say uniform on [0, 1]. Compute the posterior: beta(43, 9)-distribution (mode is at 42/50 = 0.84) p p prior data posterior 7 / 50
11 Bayes rule Observations X take values in sample space X. Model {P θ : θ Θ}. All P θ dominated: P θ µ, density p θ = dp θ /dµ. Prior distribution Π on the parameter θ. For the Bayesian: θ Π and X θ P θ. Hence, the pair (θ, X ) has density (θ, x) p θ (x) relative to Π µ. Then X has marginal density x p θ (x) Π(dθ), Θ and hence the conditional distribution of θ given X = x, i.e. the posterior, has density relative to the prior Π. θ p θ (x) Θ p θ(x) Π(dθ) 8 / 50
12 Bayes example again Have X Bin(n, θ), θ (0, 1). Likelihood: ( ) n p θ (X ) = θ X (1 θ) n X. X Prior: uniform distribution on (0, 1). By Bayes rule, posterior density proportional to θ θ X (1 θ) n X. Hence, posterior is Beta(X + 1, n X + 1). 9 / 50
13 Bayesian nonparametrics 10 / 50
14 Bayesian nonparametrics Challenges lie in particular in the area of high-dimensional or nonparametric models. Illustration 1: parametric vs. nonparametric regression Y i = f (t i ) + error i t t 11 / 50
15 Bayesian nonparametrics Illustration 2: parametric vs. nonparametric density estimation X 1,..., X n f Frequency Frequency x waiting 12 / 50
16 Bayesian nonparametrics In nonparametric problems, the parameter of interest is typically a function: e.g. a density, regression function, distribution function, hazard rate,..., or some other infinite-dimensional object. Bayesian approach is not at all fundamentally restricted to the parametric case, but: How do we construct priors on infinite-dimensional (function) spaces? How do we compute posteriors, or generate draws? What is the fundamental performance of procedures? 13 / 50
17 Nonparametric priors 14 / 50
18 Nonparametric priors - first remarks Often enough to describe how realizations are generated Possible ways to construct priors on an infinite-dimensional space Θ: Discrete priors: Consider (random) points θ 1, θ 2,..., in Θ and (random) probability weights w 1, w 2,... and define Π = w j δ θj. Stochastic Process approach: If Θ is a function space, use machinery for constructing stochastic processes Random series approach: If Θ is a function space, consider series expansions, put priors on coefficients / 50
19 Nonparametric priors - Dirichlet process 16 / 50
20 Dirichlet process - 1 Step 1: prior on simplex of probability vectors of length k: k 1 = {(y 1,..., y k ) R k : y 1,..., y k 0, y i = 1}. For α = (α 1,..., α k ) (0, ) k, define f α (y 1,..., y k 1 ) = C α k i=0 y α i 1 i 1 (y1,...,y k ) k 1 on R k 1, where y k = 1 y 1 y k 1 and C α is the appropriate normalizing constant. A random vector (Y 1,..., Y k ) in R k is said to have a Dirichlet distribution with parameter α = (α 1,..., α k ) if (Y 1,..., Y k 1 ) has density f α and Y k = 1 Y Y k / 50
21 Dirichlet process - 2 Step 2: definition of DP: Let α be a finite measure on R. A random probability measure P on R is called a Dirichlet Process with parameter α if for every partition A 1,..., A k of R, the vector (P(A 1 ),..., P(A k )) has a Dirichlet distribution with parameter (α(a 1 ),..., α(a k )). Notation: P DP(α). 18 / 50
22 Dirichlet process - 3 Step 3: Prove that DP exists! Theorem. For any finite measure α on R, the Dirichlet process with parameter α exists. Proof. For instance: Use Kolmogorov s consistency theorem to show a process P = (P(A) : A B(R)) with the right fdd s. Prove there exists a version of P such that every realization is a measure. 19 / 50
23 Dirichlet process Ten realizations from Dirichlet process with parameter 25 N(0, 1) 20 / 50
24 Dirichlet process - 5 Draws from the DP are discrete measures on R: Theorem. Let α be a finite measure, define M = α(r) and ᾱ = α/m. If we have independent θ 1, θ 2,... ᾱ and Y 1, Y 2,... Beta(1, M) and V j = Y j 1 j l=1 (1 Y l), then j=1 V jδ θj DP(α). This is the stick-breaking representation. 21 / 50
25 Distribution function estimation The DP is a conjugate prior for full distribution estimation: if P DP(α) and X 1,..., X n P P, then P X 1,..., X n DP(α + n i=1 δ X i ) Simulated data: 500 draws from a N(1, 1)-distribution, prior: Dirichlet process with parameter 25 N(0, 1). Left: 10 draws from the prior. Right: 10 draws from the posterior. 22 / 50
26 Nonparametric priors - Gaussian processes 23 / 50
27 Gaussian process priors - 1 A stochastic process W = (W t : t T ) is called Gaussian if for all n N and t 1,..., t n T, the vector (W t1,..., W tn ) has an n-dimensional Gaussian distribution. Associated functions: mean function: m(t) = EW t, covariance function: r(s, t) = Cov(W s, W t ). The GP is called centered, or zero-mean if m(t) = 0 for all t T. 24 / 50
28 Gaussian process priors - 2 For a 1,..., a n R and t 1,..., t n T, ( ) a i a j r(t i, t j ) = Var ai W ti 0, i j hence r is a positive definite, symmetric function on T T. Theorem. Let T be a set, m : T R a function and r : T T R a positive definite, symmetric function. Then there exists a Gaussian process with mean function m and covariance function r. Proof. Kolmogorov s consistency theorem. 25 / 50
29 Gaussian process priors: examples - 1 Brownian motion: m(t) = 0, r(s, t) = s t Regularity: 1/ / 50
30 Gaussian process priors: examples - 2 Integrated Brownian motion: t 0 W s ds, for W a Brownian motion. m(t) = 0, r(s, t) = s 2 t/2 t 3 / Regularity: 3/ / 50
31 Gaussian process priors: examples - 3 By Fubini and integration by parts, t tn t2 1 W t1 dt 1 dt 2 dt n = 0 0 (n 1)! 0 = 1 n! t 0 t 0 (t s) n 1 W s ds (t s) n dw s. The Riemann-Liouville process with parameter α > 0: W α t = Process has regularity α. t 0 (t s) α 1/2 dw s. 28 / 50
32 Gaussian process priors: examples - 4 Consider a centered Gaussian process W = (W t : t T ), with T R d, such that EW s W t = r(t s), s, t T, for a continuous r : R d R. Such a process is called stationary, or homogenous. By Bochner s theorem: r(t) = e i λ,t µ(dλ), R d for a finite Borel measure µ, called the spectral measure of the process. 29 / 50
33 Gaussian process priors: examples - 5 The squared exponential process: r(s, t) = exp( t s 2 ) Spectral measure: 2 d π d/2 exp( λ 2 /4) dλ Regularity: / 50
34 Gaussian process priors: examples - 6 The Matérn process: µ(dλ) (1 + λ 2 ) (α+d/2) dλ, α > 0. Covariance function: r(s, t) = 21 α Γ(α) t s α K α ( t s ), where K α is the modified Bessel function of the second kind of order α. Regularity: α. For d = 1, α = 1/2, get the Ornstein-Uhlenbeck process. 31 / 50
35 Gaussian process priors: examples - 6 The Matérn process: µ(dλ) (1 + λ 2 ) (α+d/2) dλ, α > 0. Covariance function: r(s, t) = 21 α Γ(α) t s α K α ( t s ), where K α is the modified Bessel function of the second kind of order α. Regularity: α. For d = 1, α = 1/2, get the Ornstein-Uhlenbeck process. 31 / 50
36 Gaussian process regression - 1 Observations: X i = f (t i ) + ε i, t i [0, 1] fixed ε i independent N(0, 1). Prior on f : law of a centered GP with covariance function r. Posterior: this prior is conjugate for this model: (f (t 1 ),..., f (t n )) X 1,..., X n N n ((I + Σ 1 ) 1 X, (I + Σ 1 ) 1 ), where Σ the is matrix with Σ ij = r(t i, t j ). 32 / 50
37 Gaussian process regression - 2 Data: 200 simulated data points. Prior: multiple of integrated Brownian motion Left: 10 draws from the prior. Right: 10 draws from the posterior. 33 / 50
38 Nonparametric priors - Conditionally Gaussian processes 34 / 50
39 CGP s - 1 Observation about GP s: Families of GP s typically depend on auxiliary parameters: hyper parameters. Performance can heavily depend on tuning of parameters. How to choose values of hyper parameters? 35 / 50
40 CGP s - 2 Regression with a squared exponential GP with covariance (x, y) exp( (x y) 2 /l 2 ), for different length scale hyper parameters l. l too small: l correct: y y x x 36 / 50
41 CGP s - 3 Q: How to choose the best values of hyper parameters? A: Let the data decide! Possible approaches: Put a prior on the hyper parameters as well: full Bayes Estimate hyper parameters : empirical Bayes 37 / 50
42 CGP s - 3 Q: How to choose the best values of hyper parameters? A: Let the data decide! Possible approaches: Put a prior on the hyper parameters as well: full Bayes Estimate hyper parameters : empirical Bayes 37 / 50
43 CGP s - 4 Squared exponential GP with gamma length scale: l Γ(a, b) f l GP with cov (x, y) exp( (x y) 2 /l 2 ) Example of a hierarchical prior Prior is only conditionally Gaussian Q: does this solve the bias-variance issue? 38 / 50
44 CGP s - 4 Squared exponential GP with gamma length scale: l Γ(a, b) f l GP with cov (x, y) exp( (x y) 2 /l 2 ) Example of a hierarchical prior Prior is only conditionally Gaussian Q: does this solve the bias-variance issue? 38 / 50
45 Nonparametric priors - Dirichlet mixtures 39 / 50
46 DP mixture priors - 1 Idea: Consider location/scale mixtures of Gaussians of the form p G (x) = ϕ σ (x µ) G(dµ, dσ), where ϕ σ ( µ) G is the N(µ, σ 2 )-density is a probability measure (mixing measure). Construct a prior on densities by making G random. 40 / 50
47 DP mixture priors - 2 Draw g from a Gaussian DP mixture prior: G DP(G 0 ) (G 0 often N IW ) p G p G Another example of a hierarchical prior 41 / 50
48 DP mixture density estimation Data: 272 waiting times between geyser eruptions Prior: DP mixture of normals Posterior mean: Density waiting 42 / 50
49 Some more examples 43 / 50
50 Estimating the drift of a diffusion Observation model: dx t = b(x t ) dt + dw t. Goal: estimate b. Prior: s IG(a, b) J Ps(λ) b s, J s J j=1 j 2 Z j e j e j : Fourier basis, Z j N(0, 1) posterior mean marginal 0.68 cred. posterior mean PPRS marginal 0.68 cred [Van der Meulen, Schauer, vz. (2014)] Figure 14: Comparison of the estimate of drift using the Butane Dihedral Angle data. Red solid: A Fourier prior with = 1.5. Blue dashed: Results of Papaspiliopoulos et al. (2012). The posterior mean with 68% credible bands is pictured. Right: Histogram of the data. 44 / 50
51 Nonparametric estimation of a Poisson intensity Observation model: counts from an inhomogenous Poisson process with periodic intensity λ. Goal: estimate λ. Prior: B-spline expansion with priors on knots and coefficients again with a thinned out dataset. We randomly removed counts, retaining about 1, 000 counts. The same analysis then leads to the posterior plot given in Figure 3. In this case, the uncertainty in the posterior distribution becomes clearly visible x 104 Posterior mean and point wise quantiles (10000 states) Posterior mean Point wise 95% credible intervals 12 Intensity :00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 24:00 Time (hours) :00 1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 23:00 24:00 Time (hours) Figure 3: Top panel: posterior distribution of the intensity function based on the thinned data. (Blue: posterior mean, red: point-wise 95% credible intervals). Lower panel: posterior distribution of the knot locations (Histogram). [Belitser, Serra, vz. (2015)] We find that the prior that we defined in Section 2.2 is a computationally feasible choice for nonparametric Bayesian intensity smoothing in the context of this kind of periodic 45 / 50
52 Binary prediction on a graph Observation model: P(Y i = 1) = Ψ(f (i)), for f : G R. Prior: Conditionally Gaussian with precision L p, L: graph Laplacian [Hartog, vz. (in prep.)] 46 / 50
53 Concluding remarks 47 / 50
54 Take home from Lecture I Within the Bayesian paradigm it is perfectly possible and natural to deal with nonparametric statistical problems. Many nonparametric priors have been proposed and studied: DP s, GP s, DP mixtures, series expansion,... Numerical techniques have been developed to sample from the corresponding posteriors In a variety of statistical settings, the results can be quite satisfactory. Some (theoretical) questions: So do these procedures do what we expect them to do? Why/why not? Do they have desirable properties like consistency? Can we say something more about performance, e.g. about (optimal) convergence rates? 48 / 50
55 Take home from Lecture I Within the Bayesian paradigm it is perfectly possible and natural to deal with nonparametric statistical problems. Many nonparametric priors have been proposed and studied: DP s, GP s, DP mixtures, series expansion,... Numerical techniques have been developed to sample from the corresponding posteriors In a variety of statistical settings, the results can be quite satisfactory. Some (theoretical) questions: So do these procedures do what we expect them to do? Why/why not? Do they have desirable properties like consistency? Can we say something more about performance, e.g. about (optimal) convergence rates? 48 / 50
56 DP: Some references for Lecture I - 1 Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. Annals of Statistics, 1, DP mixtures: Ferguson, T. S. (1983). Bayesian density estimation by mixtures of normal distributions. In Recent Advances in Statistics, ed. M. Rizvi et al., Escobar, M. and West, M. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90, MacEachern, S. N. and Muller, P. (1998) Estimating mixture of Dirichlet Process Models. Journal of Computational and Graphical Statistics, 7 (2), Neal, R. M. (2000). Markov Chain sampling methods for Dirichlet process mixture models. Journal of Computational and Graphical Statistics, 9, / 50
57 Some references for Lecture I - 2 GP priors: Lenk, P. J. (1988). The logistic normal distribution for Bayesian, nonparametric, predictive densities. J. Amer. Statist. Assoc Lenk, P. J. (1991). Towards a practicable Bayesian nonparametric density estimator. Biometrika Rasmussen, C. E. and Williams, C. K. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA. General text: Hjort, N.L., et al., eds. Bayesian nonparametrics. Vol. 28. Cambridge University Press, / 50
Bayesian Regularization
Bayesian Regularization Aad van der Vaart Vrije Universiteit Amsterdam International Congress of Mathematicians Hyderabad, August 2010 Contents Introduction Abstract result Gaussian process priors Co-authors
More informationBayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
More informationFoundations of Nonparametric Bayesian Methods
1 / 27 Foundations of Nonparametric Bayesian Methods Part II: Models on the Simplex Peter Orbanz http://mlg.eng.cam.ac.uk/porbanz/npb-tutorial.html 2 / 27 Tutorial Overview Part I: Basics Part II: Models
More informationNon-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
More informationBayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
More informationSTAT 518 Intro Student Presentation
STAT 518 Intro Student Presentation Wen Wei Loh April 11, 2013 Title of paper Radford M. Neal [1999] Bayesian Statistics, 6: 475-501, 1999 What the paper is about Regression and Classification Flexible
More informationSTAT Advanced Bayesian Inference
1 / 32 STAT 625 - Advanced Bayesian Inference Meng Li Department of Statistics Jan 23, 218 The Dirichlet distribution 2 / 32 θ Dirichlet(a 1,...,a k ) with density p(θ 1,θ 2,...,θ k ) = k j=1 Γ(a j) Γ(
More informationPATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS Parametric Distributions Basic building blocks: Need to determine given Representation: or? Recall Curve Fitting Binary Variables
More informationA Process over all Stationary Covariance Kernels
A Process over all Stationary Covariance Kernels Andrew Gordon Wilson June 9, 0 Abstract I define a process over all stationary covariance kernels. I show how one might be able to perform inference that
More informationNonparametric Bayesian Methods (Gaussian Processes)
[70240413 Statistical Machine Learning, Spring, 2015] Nonparametric Bayesian Methods (Gaussian Processes) Jun Zhu dcszj@mail.tsinghua.edu.cn http://bigml.cs.tsinghua.edu.cn/~jun State Key Lab of Intelligent
More informationBayesian Nonparametrics
Bayesian Nonparametrics Lorenzo Rosasco 9.520 Class 18 April 11, 2011 About this class Goal To give an overview of some of the basic concepts in Bayesian Nonparametrics. In particular, to discuss Dirichelet
More informationICML Scalable Bayesian Inference on Point processes. with Gaussian Processes. Yves-Laurent Kom Samo & Stephen Roberts
ICML 2015 Scalable Nonparametric Bayesian Inference on Point Processes with Gaussian Processes Machine Learning Research Group and Oxford-Man Institute University of Oxford July 8, 2015 Point Processes
More informationBayesian estimation of the discrepancy with misspecified parametric models
Bayesian estimation of the discrepancy with misspecified parametric models Pierpaolo De Blasi University of Torino & Collegio Carlo Alberto Bayesian Nonparametrics workshop ICERM, 17-21 September 2012
More informationGaussian Process Regression
Gaussian Process Regression 4F1 Pattern Recognition, 21 Carl Edward Rasmussen Department of Engineering, University of Cambridge November 11th - 16th, 21 Rasmussen (Engineering, Cambridge) Gaussian Process
More informationIntroduction to Probabilistic Machine Learning
Introduction to Probabilistic Machine Learning Piyush Rai Dept. of CSE, IIT Kanpur (Mini-course 1) Nov 03, 2015 Piyush Rai (IIT Kanpur) Introduction to Probabilistic Machine Learning 1 Machine Learning
More informationGaussian Processes. Le Song. Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012
Gaussian Processes Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 01 Pictorial view of embedding distribution Transform the entire distribution to expected features Feature space Feature
More informationPattern Recognition and Machine Learning. Bishop Chapter 2: Probability Distributions
Pattern Recognition and Machine Learning Chapter 2: Probability Distributions Cécile Amblard Alex Kläser Jakob Verbeek October 11, 27 Probability Distributions: General Density Estimation: given a finite
More informationBayesian Nonparametrics
Bayesian Nonparametrics Peter Orbanz Columbia University PARAMETERS AND PATTERNS Parameters P(X θ) = Probability[data pattern] 3 2 1 0 1 2 3 5 0 5 Inference idea data = underlying pattern + independent
More informationOutline. Binomial, Multinomial, Normal, Beta, Dirichlet. Posterior mean, MAP, credible interval, posterior distribution
Outline A short review on Bayesian analysis. Binomial, Multinomial, Normal, Beta, Dirichlet Posterior mean, MAP, credible interval, posterior distribution Gibbs sampling Revisit the Gaussian mixture model
More informationA Brief Overview of Nonparametric Bayesian Models
A Brief Overview of Nonparametric Bayesian Models Eurandom Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin Also at Machine
More informationSTA414/2104. Lecture 11: Gaussian Processes. Department of Statistics
STA414/2104 Lecture 11: Gaussian Processes Department of Statistics www.utstat.utoronto.ca Delivered by Mark Ebden with thanks to Russ Salakhutdinov Outline Gaussian Processes Exam review Course evaluations
More informationHyperparameter estimation in Dirichlet process mixture models
Hyperparameter estimation in Dirichlet process mixture models By MIKE WEST Institute of Statistics and Decision Sciences Duke University, Durham NC 27706, USA. SUMMARY In Bayesian density estimation and
More informationCPSC 540: Machine Learning
CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is
More informationReview. DS GA 1002 Statistical and Mathematical Models. Carlos Fernandez-Granda
Review DS GA 1002 Statistical and Mathematical Models http://www.cims.nyu.edu/~cfgranda/pages/dsga1002_fall16 Carlos Fernandez-Granda Probability and statistics Probability: Framework for dealing with
More informationMotivation Scale Mixutres of Normals Finite Gaussian Mixtures Skew-Normal Models. Mixture Models. Econ 690. Purdue University
Econ 690 Purdue University In virtually all of the previous lectures, our models have made use of normality assumptions. From a computational point of view, the reason for this assumption is clear: combined
More informationMore Spectral Clustering and an Introduction to Conjugacy
CS8B/Stat4B: Advanced Topics in Learning & Decision Making More Spectral Clustering and an Introduction to Conjugacy Lecturer: Michael I. Jordan Scribe: Marco Barreno Monday, April 5, 004. Back to spectral
More informationStatistics & Data Sciences: First Year Prelim Exam May 2018
Statistics & Data Sciences: First Year Prelim Exam May 2018 Instructions: 1. Do not turn this page until instructed to do so. 2. Start each new question on a new sheet of paper. 3. This is a closed book
More informationBayesian RL Seminar. Chris Mansley September 9, 2008
Bayesian RL Seminar Chris Mansley September 9, 2008 Bayes Basic Probability One of the basic principles of probability theory, the chain rule, will allow us to derive most of the background material in
More informationA Nonparametric Model for Stationary Time Series
A Nonparametric Model for Stationary Time Series Isadora Antoniano-Villalobos Bocconi University, Milan, Italy. isadora.antoniano@unibocconi.it Stephen G. Walker University of Texas at Austin, USA. s.g.walker@math.utexas.edu
More informationVariational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures
17th Europ. Conf. on Machine Learning, Berlin, Germany, 2006. Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures Shipeng Yu 1,2, Kai Yu 2, Volker Tresp 2, and Hans-Peter
More informationPROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS
PROBABILITY: LIMIT THEOREMS II, SPRING 15. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please
More informationLearning Bayesian network : Given structure and completely observed data
Learning Bayesian network : Given structure and completely observed data Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani Learning problem Target: true distribution
More informationDirichlet Processes: Tutorial and Practical Course
Dirichlet Processes: Tutorial and Practical Course (updated) Yee Whye Teh Gatsby Computational Neuroscience Unit University College London August 2007 / MLSS Yee Whye Teh (Gatsby) DP August 2007 / MLSS
More informationNonparametric Bayesian Uncertainty Quantification
Nonparametric Bayesian Uncertainty Quantification Lecture 1: Introduction to Nonparametric Bayes Aad van der Vaart Universiteit Leiden, Netherlands YES, Eindhoven, January 2017 Contents Introduction Recovery
More informationProbability. Machine Learning and Pattern Recognition. Chris Williams. School of Informatics, University of Edinburgh. August 2014
Probability Machine Learning and Pattern Recognition Chris Williams School of Informatics, University of Edinburgh August 2014 (All of the slides in this course have been adapted from previous versions
More informationLecture 3a: Dirichlet processes
Lecture 3a: Dirichlet processes Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London c.archambeau@cs.ucl.ac.uk Advanced Topics
More informationFoundations of Statistical Inference
Foundations of Statistical Inference Jonathan Marchini Department of Statistics University of Oxford MT 2013 Jonathan Marchini (University of Oxford) BS2a MT 2013 1 / 27 Course arrangements Lectures M.2
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 3 Linear
More informationBayesian nonparametric estimation of finite population quantities in absence of design information on nonsampled units
Bayesian nonparametric estimation of finite population quantities in absence of design information on nonsampled units Sahar Z Zangeneh Robert W. Keener Roderick J.A. Little Abstract In Probability proportional
More informationBayesian Nonparametrics: Dirichlet Process
Bayesian Nonparametrics: Dirichlet Process Yee Whye Teh Gatsby Computational Neuroscience Unit, UCL http://www.gatsby.ucl.ac.uk/~ywteh/teaching/npbayes2012 Dirichlet Process Cornerstone of modern Bayesian
More informationLikelihood-free MCMC
Bayesian inference for stable distributions with applications in finance Department of Mathematics University of Leicester September 2, 2011 MSc project final presentation Outline 1 2 3 4 Classical Monte
More informationBayesian Nonparametric Regression for Diabetes Deaths
Bayesian Nonparametric Regression for Diabetes Deaths Brian M. Hartman PhD Student, 2010 Texas A&M University College Station, TX, USA David B. Dahl Assistant Professor Texas A&M University College Station,
More informationOptimality of Poisson Processes Intensity Learning with Gaussian Processes
Journal of Machine Learning Research 16 (2015) 2909-2919 Submitted 9/14; Revised 3/15; Published 12/15 Optimality of Poisson Processes Intensity Learning with Gaussian Processes Alisa Kirichenko Harry
More informationBayesian Models in Machine Learning
Bayesian Models in Machine Learning Lukáš Burget Escuela de Ciencias Informáticas 2017 Buenos Aires, July 24-29 2017 Frequentist vs. Bayesian Frequentist point of view: Probability is the frequency of
More informationNonparmeteric Bayes & Gaussian Processes. Baback Moghaddam Machine Learning Group
Nonparmeteric Bayes & Gaussian Processes Baback Moghaddam baback@jpl.nasa.gov Machine Learning Group Outline Bayesian Inference Hierarchical Models Model Selection Parametric vs. Nonparametric Gaussian
More informationPMR Learning as Inference
Outline PMR Learning as Inference Probabilistic Modelling and Reasoning Amos Storkey Modelling 2 The Exponential Family 3 Bayesian Sets School of Informatics, University of Edinburgh Amos Storkey PMR Learning
More informationBayesian Nonparametric Models
Bayesian Nonparametric Models David M. Blei Columbia University December 15, 2015 Introduction We have been looking at models that posit latent structure in high dimensional data. We use the posterior
More informationIntroduction. Chapter 1
Chapter 1 Introduction In this book we will be concerned with supervised learning, which is the problem of learning input-output mappings from empirical data (the training dataset). Depending on the characteristics
More informationSpatial Bayesian Nonparametrics for Natural Image Segmentation
Spatial Bayesian Nonparametrics for Natural Image Segmentation Erik Sudderth Brown University Joint work with Michael Jordan University of California Soumya Ghosh Brown University Parsing Visual Scenes
More informationGaussian processes. Chuong B. Do (updated by Honglak Lee) November 22, 2008
Gaussian processes Chuong B Do (updated by Honglak Lee) November 22, 2008 Many of the classical machine learning algorithms that we talked about during the first half of this course fit the following pattern:
More informationSTATS 200: Introduction to Statistical Inference. Lecture 29: Course review
STATS 200: Introduction to Statistical Inference Lecture 29: Course review Course review We started in Lecture 1 with a fundamental assumption: Data is a realization of a random process. The goal throughout
More informationDensity Estimation. Seungjin Choi
Density Estimation Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr http://mlg.postech.ac.kr/
More informationBayesian Nonparametrics for Speech and Signal Processing
Bayesian Nonparametrics for Speech and Signal Processing Michael I. Jordan University of California, Berkeley June 28, 2011 Acknowledgments: Emily Fox, Erik Sudderth, Yee Whye Teh, and Romain Thibaux Computer
More informationLecture 16-17: Bayesian Nonparametrics I. STAT 6474 Instructor: Hongxiao Zhu
Lecture 16-17: Bayesian Nonparametrics I STAT 6474 Instructor: Hongxiao Zhu Plan for today Why Bayesian Nonparametrics? Dirichlet Distribution and Dirichlet Processes. 2 Parameter and Patterns Reference:
More informationLecture 2: Priors and Conjugacy
Lecture 2: Priors and Conjugacy Melih Kandemir melih.kandemir@iwr.uni-heidelberg.de May 6, 2014 Some nice courses Fred A. Hamprecht (Heidelberg U.) https://www.youtube.com/watch?v=j66rrnzzkow Michael I.
More informationState Space Representation of Gaussian Processes
State Space Representation of Gaussian Processes Simo Särkkä Department of Biomedical Engineering and Computational Science (BECS) Aalto University, Espoo, Finland June 12th, 2013 Simo Särkkä (Aalto University)
More informationLog Gaussian Cox Processes. Chi Group Meeting February 23, 2016
Log Gaussian Cox Processes Chi Group Meeting February 23, 2016 Outline Typical motivating application Introduction to LGCP model Brief overview of inference Applications in my work just getting started
More informationStochastic Processes, Kernel Regression, Infinite Mixture Models
Stochastic Processes, Kernel Regression, Infinite Mixture Models Gabriel Huang (TA for Simon Lacoste-Julien) IFT 6269 : Probabilistic Graphical Models - Fall 2018 Stochastic Process = Random Function 2
More informationA Simple Proof of the Stick-Breaking Construction of the Dirichlet Process
A Simple Proof of the Stick-Breaking Construction of the Dirichlet Process John Paisley Department of Computer Science Princeton University, Princeton, NJ jpaisley@princeton.edu Abstract We give a simple
More informationUniversität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Bayesian Learning. Tobias Scheffer, Niels Landwehr
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning Tobias Scheffer, Niels Landwehr Remember: Normal Distribution Distribution over x. Density function with parameters
More informationMFM Practitioner Module: Quantitative Risk Management. John Dodson. September 23, 2015
MFM Practitioner Module: Quantitative Risk Management September 23, 2015 Mixtures Mixtures Mixtures Definitions For our purposes, A random variable is a quantity whose value is not known to us right now
More informationarxiv: v1 [stat.me] 6 Nov 2013
Electronic Journal of Statistics Vol. 0 (0000) ISSN: 1935-7524 DOI: 10.1214/154957804100000000 A Generalized Savage-Dickey Ratio Ewan Cameron e-mail: dr.ewan.cameron@gmail.com url: astrostatistics.wordpress.com
More informationCSci 8980: Advanced Topics in Graphical Models Gaussian Processes
CSci 8980: Advanced Topics in Graphical Models Gaussian Processes Instructor: Arindam Banerjee November 15, 2007 Gaussian Processes Outline Gaussian Processes Outline Parametric Bayesian Regression Gaussian
More informationAnnouncements. Proposals graded
Announcements Proposals graded Kevin Jamieson 2018 1 Bayesian Methods Machine Learning CSE546 Kevin Jamieson University of Washington November 1, 2018 2018 Kevin Jamieson 2 MLE Recap - coin flips Data:
More informationDirichlet Process. Yee Whye Teh, University College London
Dirichlet Process Yee Whye Teh, University College London Related keywords: Bayesian nonparametrics, stochastic processes, clustering, infinite mixture model, Blackwell-MacQueen urn scheme, Chinese restaurant
More informationContents. Part I: Fundamentals of Bayesian Inference 1
Contents Preface xiii Part I: Fundamentals of Bayesian Inference 1 1 Probability and inference 3 1.1 The three steps of Bayesian data analysis 3 1.2 General notation for statistical inference 4 1.3 Bayesian
More informationStat 5101 Lecture Notes
Stat 5101 Lecture Notes Charles J. Geyer Copyright 1998, 1999, 2000, 2001 by Charles J. Geyer May 7, 2001 ii Stat 5101 (Geyer) Course Notes Contents 1 Random Variables and Change of Variables 1 1.1 Random
More informationNaïve Bayes classification
Naïve Bayes classification 1 Probability theory Random variable: a variable whose possible values are numerical outcomes of a random phenomenon. Examples: A person s height, the outcome of a coin toss
More informationGenerative Clustering, Topic Modeling, & Bayesian Inference
Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 12-14, 2017 Prof. Michael Paul Unsupervised Naïve Bayes Last week
More informationBayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
More informationIntroduction: MLE, MAP, Bayesian reasoning (28/8/13)
STA561: Probabilistic machine learning Introduction: MLE, MAP, Bayesian reasoning (28/8/13) Lecturer: Barbara Engelhardt Scribes: K. Ulrich, J. Subramanian, N. Raval, J. O Hollaren 1 Classifiers In this
More informationECE521 Tutorial 11. Topic Review. ECE521 Winter Credits to Alireza Makhzani, Alex Schwing, Rich Zemel and TAs for slides. ECE521 Tutorial 11 / 4
ECE52 Tutorial Topic Review ECE52 Winter 206 Credits to Alireza Makhzani, Alex Schwing, Rich Zemel and TAs for slides ECE52 Tutorial ECE52 Winter 206 Credits to Alireza / 4 Outline K-means, PCA 2 Bayesian
More informationStatistics: Learning models from data
DS-GA 1002 Lecture notes 5 October 19, 2015 Statistics: Learning models from data Learning models from data that are assumed to be generated probabilistically from a certain unknown distribution is a crucial
More informationModel Based Clustering of Count Processes Data
Model Based Clustering of Count Processes Data Tin Lok James Ng, Brendan Murphy Insight Centre for Data Analytics School of Mathematics and Statistics May 15, 2017 Tin Lok James Ng, Brendan Murphy (Insight)
More informationThe Bayesian approach to inverse problems
The Bayesian approach to inverse problems Youssef Marzouk Department of Aeronautics and Astronautics Center for Computational Engineering Massachusetts Institute of Technology ymarz@mit.edu, http://uqgroup.mit.edu
More informationNPFL108 Bayesian inference. Introduction. Filip Jurčíček. Institute of Formal and Applied Linguistics Charles University in Prague Czech Republic
NPFL108 Bayesian inference Introduction Filip Jurčíček Institute of Formal and Applied Linguistics Charles University in Prague Czech Republic Home page: http://ufal.mff.cuni.cz/~jurcicek Version: 21/02/2014
More informationBayesian linear regression
Bayesian linear regression Linear regression is the basis of most statistical modeling. The model is Y i = X T i β + ε i, where Y i is the continuous response X i = (X i1,..., X ip ) T is the corresponding
More informationIntroduction to Bayesian Methods. Introduction to Bayesian Methods p.1/??
to Bayesian Methods Introduction to Bayesian Methods p.1/?? We develop the Bayesian paradigm for parametric inference. To this end, suppose we conduct (or wish to design) a study, in which the parameter
More informationADVANCED MACHINE LEARNING ADVANCED MACHINE LEARNING. Non-linear regression techniques Part - II
1 Non-linear regression techniques Part - II Regression Algorithms in this Course Support Vector Machine Relevance Vector Machine Support vector regression Boosting random projections Relevance vector
More informationProbabilistic modeling. The slides are closely adapted from Subhransu Maji s slides
Probabilistic modeling The slides are closely adapted from Subhransu Maji s slides Overview So far the models and algorithms you have learned about are relatively disconnected Probabilistic modeling framework
More informationComputer Emulation With Density Estimation
Computer Emulation With Density Estimation Jake Coleman, Robert Wolpert May 8, 2017 Jake Coleman, Robert Wolpert Emulation and Density Estimation May 8, 2017 1 / 17 Computer Emulation Motivation Expensive
More informationBayesian non-parametric model to longitudinally predict churn
Bayesian non-parametric model to longitudinally predict churn Bruno Scarpa Università di Padova Conference of European Statistics Stakeholders Methodologists, Producers and Users of European Statistics
More informationIntroduction to Probabilistic Graphical Models
Introduction to Probabilistic Graphical Models Sargur Srihari srihari@cedar.buffalo.edu 1 Topics 1. What are probabilistic graphical models (PGMs) 2. Use of PGMs Engineering and AI 3. Directionality in
More informationReview of Probabilities and Basic Statistics
Alex Smola Barnabas Poczos TA: Ina Fiterau 4 th year PhD student MLD Review of Probabilities and Basic Statistics 10-701 Recitations 1/25/2013 Recitation 1: Statistics Intro 1 Overview Introduction to
More informationGaussian processes for inference in stochastic differential equations
Gaussian processes for inference in stochastic differential equations Manfred Opper, AI group, TU Berlin November 6, 2017 Manfred Opper, AI group, TU Berlin (TU Berlin) inference in SDE November 6, 2017
More informationHierarchical Models & Bayesian Model Selection
Hierarchical Models & Bayesian Model Selection Geoffrey Roeder Departments of Computer Science and Statistics University of British Columbia Jan. 20, 2016 Contact information Please report any typos or
More informationIntroduction to Machine Learning. Lecture 2
Introduction to Machine Learning Lecturer: Eran Halperin Lecture 2 Fall Semester Scribe: Yishay Mansour Some of the material was not presented in class (and is marked with a side line) and is given for
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistical Sciences! rsalakhu@cs.toronto.edu! h0p://www.cs.utoronto.ca/~rsalakhu/ Lecture 7 Approximate
More informationNew Bayesian methods for model comparison
Back to the future New Bayesian methods for model comparison Murray Aitkin murray.aitkin@unimelb.edu.au Department of Mathematics and Statistics The University of Melbourne Australia Bayesian Model Comparison
More informationEcon 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines
Econ 2148, fall 2017 Gaussian process priors, reproducing kernel Hilbert spaces, and Splines Maximilian Kasy Department of Economics, Harvard University 1 / 37 Agenda 6 equivalent representations of the
More informationSTA 4273H: Sta-s-cal Machine Learning
STA 4273H: Sta-s-cal Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistical Sciences! rsalakhu@cs.toronto.edu! h0p://www.cs.utoronto.ca/~rsalakhu/ Lecture 2 In our
More informationPROBABILITY DISTRIBUTIONS. J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
PROBABILITY DISTRIBUTIONS Credits 2 These slides were sourced and/or modified from: Christopher Bishop, Microsoft UK Parametric Distributions 3 Basic building blocks: Need to determine given Representation:
More informationNormalized kernel-weighted random measures
Normalized kernel-weighted random measures Jim Griffin University of Kent 1 August 27 Outline 1 Introduction 2 Ornstein-Uhlenbeck DP 3 Generalisations Bayesian Density Regression We observe data (x 1,
More informationA short introduction to INLA and R-INLA
A short introduction to INLA and R-INLA Integrated Nested Laplace Approximation Thomas Opitz, BioSP, INRA Avignon Workshop: Theory and practice of INLA and SPDE November 7, 2018 2/21 Plan for this talk
More informationBayesian inference. Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark. April 10, 2017
Bayesian inference Rasmus Waagepetersen Department of Mathematics Aalborg University Denmark April 10, 2017 1 / 22 Outline for today A genetic example Bayes theorem Examples Priors Posterior summaries
More informationLecture 4: Introduction to stochastic processes and stochastic calculus
Lecture 4: Introduction to stochastic processes and stochastic calculus Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London
More informationINTRODUCTION TO BAYESIAN INFERENCE PART 2 CHRIS BISHOP
INTRODUCTION TO BAYESIAN INFERENCE PART 2 CHRIS BISHOP Personal Healthcare Revolution Electronic health records (CFH) Personal genomics (DeCode, Navigenics, 23andMe) X-prize: first $10k human genome technology
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing
More informationCOMP 551 Applied Machine Learning Lecture 19: Bayesian Inference
COMP 551 Applied Machine Learning Lecture 19: Bayesian Inference Associate Instructor: (herke.vanhoof@mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551 Unless otherwise noted, all material posted
More informationQuantifying the Price of Uncertainty in Bayesian Models
Provided by the author(s) and NUI Galway in accordance with publisher policies. Please cite the published version when available. Title Quantifying the Price of Uncertainty in Bayesian Models Author(s)
More information