MAD-Bayes: MAP-based Asymptotic Derivations from Bayes
|
|
|
- Oscar Lawson
- 5 years ago
- Views:
Transcription
1 MAD-Bayes: MAP-based Asymptotic Derivations from Bayes Tamara Broderick Brian Kulis Michael I. Jordan
2 Cat Clusters Mouse clusters Dog 1
3 Cat Clusters Dog Mouse Lizard Sheep Picture 1 Picture 2 Picture 3 Picture 4 Picture 5 Picture 6 Picture 7 2
4 Cat Features Dog Mouse Lizard Sheep Picture 1 Picture 2 Picture 3 Picture 4 Picture 5 Picture 6 Picture 7 3
5 Cat Features Dog Mouse Lizard Sheep Picture 1 Picture 2 Picture 3 Picture 4 Picture 5 Many other possible latent structures in data Picture 6 Picture 7 3
6 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters 4
7 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters 4
8 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters 4
9 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters 4
10 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters 4
11 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters 4
12 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters Nonparametric Bayes! Modular (general latent structure)! Flexible (K can grow as data grows)! Coherent treatment of uncertainty But...! E.g., Silicon Valley: can have petabytes of data 4! Practitioners turn to what runs
13 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters Nonparametric Bayes! Modular (general latent structure)! Flexible (K can grow as data grows)! Coherent treatment of uncertainty But...! E.g., Silicon Valley: can have petabytes of data 4! Practitioners turn to what runs
14 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters Nonparametric Bayes! Modular (general latent structure)! Flexible (K can grow as data grows)! Coherent treatment of uncertainty But...! E.g., Silicon Valley: can have petabytes of data 4! Practitioners turn to what runs
15 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters Nonparametric Bayes! Modular (general latent structure)! Flexible (K can grow as data grows)! Coherent treatment of uncertainty But...! E.g., Silicon Valley: can have petabytes of data 4! Practitioners turn to what runs
16 How do we learn latent structure? K-means! Fast! Can parallelize! Straightforward! Only works for K clusters Nonparametric Bayes! Modular (general latent structure)! Flexible (K can grow as data grows)! Coherent treatment of uncertainty But...! E.g., Silicon Valley: can have petabytes of data 4! Practitioners turn to what runs
17 MAD-Bayes Perspectives! Bayesian nonparametrics assists the optimizationbased inference community " New, modular, flexible, nonparametric objectives & regularizers " Alternative perspective: fast initialization Inspiration! Consider a finite Gaussian mixture model! The steps of the EM algorithm limit to the steps of the K-means algorithm as the Gaussian variance is taken to 0 5
18 MAD-Bayes Perspectives! Bayesian nonparametrics assists the optimizationbased inference community " New, modular, flexible, nonparametric objectives & regularizers " Alternative perspective: fast initialization Inspiration! Consider a finite Gaussian mixture model! The steps of the EM algorithm limit to the steps of the K-means algorithm as the Gaussian variance is taken to 0 5
19 MAD-Bayes Perspectives! Bayesian nonparametrics assists the optimizationbased inference community " New, modular, flexible, nonparametric objectives & regularizers " Alternative perspective: fast initialization Inspiration! Consider a finite Gaussian mixture model! The steps of the EM algorithm limit to the steps of the K-means algorithm as the Gaussian variance is taken to 0 5
20 MAD-Bayes Perspectives! Bayesian nonparametrics assists the optimizationbased inference community " New, modular, flexible, nonparametric objectives & regularizers " Alternative perspective: fast initialization Inspiration! Consider a finite Gaussian mixture model! The steps of the EM algorithm limit to the steps of the K-means algorithm as the Gaussian variance is taken to 0 5
21 MAD-Bayes Perspectives! Bayesian nonparametrics assists the optimizationbased inference community " New, modular, flexible, nonparametric objectives & regularizers " Alternative perspective: fast initialization Inspiration! Consider a finite Gaussian mixture model! The steps of the EM algorithm limit to the steps of the K-means algorithm as the Gaussian variance is taken to 0 5
22 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective 6
23 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective 6
24 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective 6
25 K-means K-means clustering problem 7
26 K-means K-means clustering problem minimize (sum of square distances from data points to cluster centers) 7
27 K-means K-means clustering problem minimize (sum of square distances from data points to cluster centers) 7
28 K-means K-means clustering problem minimize N n=1 x n center n 2 8
29 K-means K-means clustering problem minimize N n=1 x n center n 2 8
30 K-means K-means clustering problem minimize N n=1 x n center n 2 8
31 K-means K-means objective minimize N n=1 x n center n 2 8
32 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
33 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
34 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
35 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
36 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
37 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
38 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
39 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
40 Lloyd s algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to a cluster 2. Update cluster means 9
41 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective
42 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective
43 Bayesian model 10
44 Bayesian model 10
45 Bayesian model 10
46 Bayesian model 11
47 Bayesian model Nonparametric! number of parameters can grow with the number of data points 11
48 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective
49 MAD-Bayes The MAD-Bayes idea! Start with nonparametric Bayes model! Take a similar limit to get a K-means-like objective
50 MAD-Bayes 12
51 MAD-Bayes! Maximum a Posteriori (MAP) is an optimization problem argmax parameters P(parameters data)! We take a limit of the objective (posterior) and get one like K-means " Small-variance asymptotics 12
52 MAD-Bayes! Maximum a Posteriori (MAP) is an optimization problem argmax parameters P(parameters data)! We take a limit of the objective (posterior) and get one like K-means " Small-variance asymptotics 12
53 MAD-Bayes! Maximum a Posteriori (MAP) is an optimization problem argmax parameters P(parameters data)! We take a limit of the objective (posterior) and get one like K-means " Small-variance asymptotics 12
54 MAD-Bayes Bayesian posterior K-means-like objectives 13
55 MAD-Bayes Bayesian posterior K-means-like objectives Mixture of K Gaussians K-means 13
56 MAD-Bayes Bayesian posterior K-means-like objectives Mixture of K Gaussians K-means Dirichlet process mixture Unbounded number of clusters 13
57 MAD-Bayes Bayesian posterior K-means-like objectives Mixture of K Gaussians K-means Dirichlet process mixture Unbounded number of clusters Hierarchical Dirichlet process Multiple data sets share cluster centers 13
58 MAD-Bayes Bayesian posterior K-means-like objectives Mixture of K Gaussians K-means Dirichlet process mixture Unbounded number of clusters Hierarchical Dirichlet process Multiple data sets share cluster centers
59 MAD-Bayes Bayesian posterior K-means-like objectives Mixture of K Gaussians K-means Dirichlet process mixture Unbounded number of clusters Hierarchical Dirichlet process Multiple data sets share cluster centers Beta process Features 13
60 Features Z Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 Point 1 Point 2 Point 3 Point 4 Point 5 Point 6 Point 7 14
61 Features Z Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 Point 1 Point 2 A Point 3 Point 4 Point 5 Point 6 Point 7 14
62 Features Z Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 Point 1 Point 2 A Point 3 Point 4 Point 5 Point 6 Point 7 14
63 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
64 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
65 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
66 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
67 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
68 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
69 MAD-Bayes Bayesian posterior P(Z, A X) 1 (2 2 ) ND/2 exp 1 2 K + exp N n=1 n 2 tr((x ZA) (X ZA)) K + H h=1 K h! k=1 1 (2 2 ) K+ D/2 exp 1 tr(a A) 2 2 (S N,k 1)!(N S N,k )!. N! 15
70 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K
71 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K
72 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K
73 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K + 2. BP-means algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to features! Create a new feature if it lowers the objective 2. Update feature means A (Z Z) 1 Z X 16
74 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K + 2. BP-means algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to features! Create a new feature if it lowers the objective 2. Update feature means A (Z Z) 1 Z X 16
75 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K + 2. BP-means algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to features! Create a new feature if it lowers the objective 2. Update feature means A (Z Z) 1 Z X 16
76 MAD-Bayes BP-means objective argmin K +,Z,Atr[(X ZA) (X ZA)] + K + 2. BP-means algorithm Iterate until no changes: 1. For n = 1,..., N! Assign point n to features! Create a new feature if it lowers the objective 2. Update feature means A (Z Z) 1 Z X 16
77 MAD-Bayes Griffiths & Ghahramani (2006) computer vision problem tabletop data Bayesian posterior Gibbs sampler BP-means algorithm 8.5 * 10 3 sec 0.36 sec Still faster by order of magnitude if restart 1000 times 17
78 MAD-Bayes Parallelism and optimistic concurrency control DP-means alg. BP-means alg. # data points 134M 8M time per iteration 5.5 min 4.3 min 18
79 MAD-Bayes Bayesian posterior K-means-like objectives Mixture of K Gaussians K-means Dirichlet process mixture Unbounded number of clusters Hierarchical Dirichlet process Multiple data sets share cluster centers Beta process Features 19
80 MAD-Bayes conclusions 20
81 MAD-Bayes conclusions! We provide new optimization objectives and regularizers " In fact, general means of obtaining more " Straightforward, fast algorithms! Can the promise of nonparametric Bayes be fulfilled? 20
82 MAD-Bayes conclusions! We provide new optimization objectives and regularizers " In fact, general means of obtaining more " Straightforward, fast algorithms! Can the promise of nonparametric Bayes be fulfilled? 20
83 MAD-Bayes conclusions! We provide new optimization objectives and regularizers " In fact, general means of obtaining more " Straightforward, fast algorithms! Can the promise of nonparametric Bayes be fulfilled? 20
84 References T. Broderick, B. Kulis, and M. I. Jordan. MAD- Bayes: MAP-based asymptotic derivations from Bayes. In International Conference on Machine Learning, X. Pan, J. E. Gonzales, S. Jegelka, T. Broderick, and M. I. Jordan. Optimistic concurrency control for distributed unsupervised learning. In Neural Information Processing Systems, T. Broderick, N. Boyd, A. Wibisono, A. C. Wilson, and M. I. Jordan. Streaming variational Bayes. In Neural Information Processing Systems,
85 Further References T. Griffiths and Z. Ghahramani. Infinite latent feature models and the Indian buffet process. In Neural Information Processing Systems, N. L. Hjort. Nonparametric Bayes estimators based on beta processes in models for life history data. Annals of Statistics, 18(3): , J. F. C. Kingman. The representation of partition structures. Journal of the London Mathematical Society, 2(2):374, B. Kulis and M. I. Jordan. Revisiting k-means: New algorithms via Bayesian nonparametrics. In International Conference on Machine Learning, J. Pitman. Exchangeable and partially exchangeable random partitions. Probability Theory and Related Fields, 102(2): , R. Thibaux and M. I. Jordan. Hierarchical beta processes and the Indian buffet process. In International Conference on Artificial Intelligence and Statistics, 2007.
Nonparametric Bayesian Methods: Models, Algorithms, and Applications (Day 5)
Nonparametric Bayesian Methods: Models, Algorithms, and Applications (Day 5) Tamara Broderick ITT Career Development Assistant Professor Electrical Engineering & Computer Science MIT Bayes Foundations
A Brief Overview of Nonparametric Bayesian Models
A Brief Overview of Nonparametric Bayesian Models Eurandom Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin Also at Machine
Non-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
Infinite latent feature models and the Indian Buffet Process
p.1 Infinite latent feature models and the Indian Buffet Process Tom Griffiths Cognitive and Linguistic Sciences Brown University Joint work with Zoubin Ghahramani p.2 Beyond latent classes Unsupervised
Bayesian non parametric approaches: an introduction
Introduction Latent class models Latent feature models Conclusion & Perspectives Bayesian non parametric approaches: an introduction Pierre CHAINAIS Bordeaux - nov. 2012 Trajectory 1 Bayesian non parametric
Infinite Latent Feature Models and the Indian Buffet Process
Infinite Latent Feature Models and the Indian Buffet Process Thomas L. Griffiths Cognitive and Linguistic Sciences Brown University, Providence RI 292 tom griffiths@brown.edu Zoubin Ghahramani Gatsby Computational
Bayesian Nonparametrics for Speech and Signal Processing
Bayesian Nonparametrics for Speech and Signal Processing Michael I. Jordan University of California, Berkeley June 28, 2011 Acknowledgments: Emily Fox, Erik Sudderth, Yee Whye Teh, and Romain Thibaux Computer
Bayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
Fast Approximate MAP Inference for Bayesian Nonparametrics
Fast Approximate MAP Inference for Bayesian Nonparametrics Y. Raykov A. Boukouvalas M.A. Little Department of Mathematics Aston University 10th Conference on Bayesian Nonparametrics, 2015 1 Iterated Conditional
Bayesian nonparametric latent feature models
Bayesian nonparametric latent feature models Indian Buffet process, beta process, and related models François Caron Department of Statistics, Oxford Applied Bayesian Statistics Summer School Como, Italy
Part IV: Monte Carlo and nonparametric Bayes
Part IV: Monte Carlo and nonparametric Bayes Outline Monte Carlo methods Nonparametric Bayesian models Outline Monte Carlo methods Nonparametric Bayesian models The Monte Carlo principle The expectation
Bayesian nonparametric models for bipartite graphs
Bayesian nonparametric models for bipartite graphs François Caron Department of Statistics, Oxford Statistics Colloquium, Harvard University November 11, 2013 F. Caron 1 / 27 Bipartite networks Readers/Customers
Small-variance Asymptotics for Dirichlet Process Mixtures of SVMs
Small-variance Asymptotics for Dirichlet Process Mixtures of SVMs Yining Wang Jun Zhu Tsinghua University July, 2014 Y. Wang and J. Zhu (Tsinghua University) Max-Margin DP-means July, 2014 1 / 25 Outline
Bayesian Nonparametrics
Bayesian Nonparametrics Peter Orbanz Columbia University PARAMETERS AND PATTERNS Parameters P(X θ) = Probability[data pattern] 3 2 1 0 1 2 3 5 0 5 Inference idea data = underlying pattern + independent
Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures
17th Europ. Conf. on Machine Learning, Berlin, Germany, 2006. Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures Shipeng Yu 1,2, Kai Yu 2, Volker Tresp 2, and Hans-Peter
Bayesian Nonparametric Models
Bayesian Nonparametric Models David M. Blei Columbia University December 15, 2015 Introduction We have been looking at models that posit latent structure in high dimensional data. We use the posterior
Approximate Decentralized Bayesian Inference
Approximate Decentralized Bayesian Inference Trevor Campbell LIDS, MIT Cambridge, MA 02139 tdc@mit.edu Jonathan P. How LIDS, MIT Cambridge, MA 02139 how@mit.edu Abstract This paper presents an approximate
Lecture 3a: Dirichlet processes
Lecture 3a: Dirichlet processes Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London c.archambeau@cs.ucl.ac.uk Advanced Topics
Bayesian nonparametric latent feature models
Bayesian nonparametric latent feature models François Caron UBC October 2, 2007 / MLRG François Caron (UBC) Bayes. nonparametric latent feature models October 2, 2007 / MLRG 1 / 29 Overview 1 Introduction
Truncation error of a superposed gamma process in a decreasing order representation
Truncation error of a superposed gamma process in a decreasing order representation B julyan.arbel@inria.fr Í www.julyanarbel.com Inria, Mistis, Grenoble, France Joint work with Igor Pru nster (Bocconi
Outline. Binomial, Multinomial, Normal, Beta, Dirichlet. Posterior mean, MAP, credible interval, posterior distribution
Outline A short review on Bayesian analysis. Binomial, Multinomial, Normal, Beta, Dirichlet Posterior mean, MAP, credible interval, posterior distribution Gibbs sampling Revisit the Gaussian mixture model
Bayesian Nonparametric Learning of Complex Dynamical Phenomena
Duke University Department of Statistical Science Bayesian Nonparametric Learning of Complex Dynamical Phenomena Emily Fox Joint work with Erik Sudderth (Brown University), Michael Jordan (UC Berkeley),
Probabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Infinite Feature Models: The Indian Buffet Process Eric Xing Lecture 21, April 2, 214 Acknowledgement: slides first drafted by Sinead Williamson
Nonparametric Factor Analysis with Beta Process Priors
Nonparametric Factor Analysis with Beta Process Priors John Paisley Lawrence Carin Department of Electrical & Computer Engineering Duke University, Durham, NC 7708 jwp4@ee.duke.edu lcarin@ee.duke.edu Abstract
Haupthseminar: Machine Learning. Chinese Restaurant Process, Indian Buffet Process
Haupthseminar: Machine Learning Chinese Restaurant Process, Indian Buffet Process Agenda Motivation Chinese Restaurant Process- CRP Dirichlet Process Interlude on CRP Infinite and CRP mixture model Estimation
Bayesian Mixtures of Bernoulli Distributions
Bayesian Mixtures of Bernoulli Distributions Laurens van der Maaten Department of Computer Science and Engineering University of California, San Diego Introduction The mixture of Bernoulli distributions
Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric?
Should all Machine Learning be Bayesian? Should all Bayesian models be non-parametric? Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/
19 : Bayesian Nonparametrics: The Indian Buffet Process. 1 Latent Variable Models and the Indian Buffet Process
10-708: Probabilistic Graphical Models, Spring 2015 19 : Bayesian Nonparametrics: The Indian Buffet Process Lecturer: Avinava Dubey Scribes: Rishav Das, Adam Brodie, and Hemank Lamba 1 Latent Variable
Priors for Random Count Matrices with Random or Fixed Row Sums
Priors for Random Count Matrices with Random or Fixed Row Sums Mingyuan Zhou Joint work with Oscar Madrid and James Scott IROM Department, McCombs School of Business Department of Statistics and Data Sciences
Bayesian Nonparametric Models on Decomposable Graphs
Bayesian Nonparametric Models on Decomposable Graphs François Caron INRIA Bordeaux Sud Ouest Institut de Mathématiques de Bordeaux University of Bordeaux, France francois.caron@inria.fr Arnaud Doucet Departments
Streaming Variational Bayes
Streaming Variational Bayes Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C. Wilson, Michael I. Jordan UC Berkeley Discussion led by Miao Liu September 13, 2013 Introduction The SDA-Bayes Framework
Bayesian Machine Learning
Bayesian Machine Learning Andrew Gordon Wilson ORIE 6741 Lecture 2: Bayesian Basics https://people.orie.cornell.edu/andrew/orie6741 Cornell University August 25, 2016 1 / 17 Canonical Machine Learning
Advanced Machine Learning
Advanced Machine Learning Nonparametric Bayesian Models --Learning/Reasoning in Open Possible Worlds Eric Xing Lecture 7, August 4, 2009 Reading: Eric Xing Eric Xing @ CMU, 2006-2009 Clustering Eric Xing
Introduction to Machine Learning. Maximum Likelihood and Bayesian Inference. Lecturers: Eran Halperin, Lior Wolf
1 Introduction to Machine Learning Maximum Likelihood and Bayesian Inference Lecturers: Eran Halperin, Lior Wolf 2014-15 We know that X ~ B(n,p), but we do not know p. We get a random sample from X, a
Bayesian Hidden Markov Models and Extensions
Bayesian Hidden Markov Models and Extensions Zoubin Ghahramani Department of Engineering University of Cambridge joint work with Matt Beal, Jurgen van Gael, Yunus Saatci, Tom Stepleton, Yee Whye Teh Modeling
Gentle Introduction to Infinite Gaussian Mixture Modeling
Gentle Introduction to Infinite Gaussian Mixture Modeling with an application in neuroscience By Frank Wood Rasmussen, NIPS 1999 Neuroscience Application: Spike Sorting Important in neuroscience and for
An Alternative Prior Process for Nonparametric Bayesian Clustering
An Alternative Prior Process for Nonparametric Bayesian Clustering Hanna Wallach (UMass Amherst) Shane Jensen (UPenn) Lee Dicker (Harvard) Katherine Heller (Cambridge) Nonparametric Bayesian Clustering
Hierarchical Models, Nested Models and Completely Random Measures
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/238729763 Hierarchical Models, Nested Models and Completely Random Measures Article March 2012
Bayesian Nonparametrics: Dirichlet Process
Bayesian Nonparametrics: Dirichlet Process Yee Whye Teh Gatsby Computational Neuroscience Unit, UCL http://www.gatsby.ucl.ac.uk/~ywteh/teaching/npbayes2012 Dirichlet Process Cornerstone of modern Bayesian
Introduction to Machine Learning. Maximum Likelihood and Bayesian Inference. Lecturers: Eran Halperin, Yishay Mansour, Lior Wolf
1 Introduction to Machine Learning Maximum Likelihood and Bayesian Inference Lecturers: Eran Halperin, Yishay Mansour, Lior Wolf 2013-14 We know that X ~ B(n,p), but we do not know p. We get a random sample
Introduction to Probabilistic Machine Learning
Introduction to Probabilistic Machine Learning Piyush Rai Dept. of CSE, IIT Kanpur (Mini-course 1) Nov 03, 2015 Piyush Rai (IIT Kanpur) Introduction to Probabilistic Machine Learning 1 Machine Learning
9/12/17. Types of learning. Modeling data. Supervised learning: Classification. Supervised learning: Regression. Unsupervised learning: Clustering
Types of learning Modeling data Supervised: we know input and targets Goal is to learn a model that, given input data, accurately predicts target data Unsupervised: we know the input only and want to make
The Indian Buffet Process: An Introduction and Review
Journal of Machine Learning Research 12 (2011) 1185-1224 Submitted 3/10; Revised 3/11; Published 4/11 The Indian Buffet Process: An Introduction and Review Thomas L. Griffiths Department of Psychology
Parametric Unsupervised Learning Expectation Maximization (EM) Lecture 20.a
Parametric Unsupervised Learning Expectation Maximization (EM) Lecture 20.a Some slides are due to Christopher Bishop Limitations of K-means Hard assignments of data points to clusters small shift of a
Image segmentation combining Markov Random Fields and Dirichlet Processes
Image segmentation combining Markov Random Fields and Dirichlet Processes Jessica SODJO IMS, Groupe Signal Image, Talence Encadrants : A. Giremus, J.-F. Giovannelli, F. Caron, N. Dobigeon Jessica SODJO
MACHINE LEARNING AND PATTERN RECOGNITION Fall 2006, Lecture 8: Latent Variables, EM Yann LeCun
Y. LeCun: Machine Learning and Pattern Recognition p. 1/? MACHINE LEARNING AND PATTERN RECOGNITION Fall 2006, Lecture 8: Latent Variables, EM Yann LeCun The Courant Institute, New York University http://yann.lecun.com
Distance dependent Chinese restaurant processes
David M. Blei Department of Computer Science, Princeton University 35 Olden St., Princeton, NJ 08540 Peter Frazier Department of Operations Research and Information Engineering, Cornell University 232
Feature Allocations, Probability Functions, and Paintboxes
Feature Allocations, Probability Functions, and Paintboxes Tamara Broderick, Jim Pitman, Michael I. Jordan Abstract The problem of inferring a clustering of a data set has been the subject of much research
Collapsed Variational Bayesian Inference for Hidden Markov Models
Collapsed Variational Bayesian Inference for Hidden Markov Models Pengyu Wang, Phil Blunsom Department of Computer Science, University of Oxford International Conference on Artificial Intelligence and
Clustering using Mixture Models
Clustering using Mixture Models The full posterior of the Gaussian Mixture Model is p(x, Z, µ,, ) =p(x Z, µ, )p(z )p( )p(µ, ) data likelihood (Gaussian) correspondence prob. (Multinomial) mixture prior
Generative Clustering, Topic Modeling, & Bayesian Inference
Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 12-14, 2017 Prof. Michael Paul Unsupervised Naïve Bayes Last week
ICML Scalable Bayesian Inference on Point processes. with Gaussian Processes. Yves-Laurent Kom Samo & Stephen Roberts
ICML 2015 Scalable Nonparametric Bayesian Inference on Point Processes with Gaussian Processes Machine Learning Research Group and Oxford-Man Institute University of Oxford July 8, 2015 Point Processes
Truncation-free Stochastic Variational Inference for Bayesian Nonparametric Models
Truncation-free Stochastic Variational Inference for Bayesian Nonparametric Models Chong Wang Machine Learning Department Carnegie Mellon University chongw@cs.cmu.edu David M. Blei Computer Science Department
Bayesian Nonparametric Regression for Diabetes Deaths
Bayesian Nonparametric Regression for Diabetes Deaths Brian M. Hartman PhD Student, 2010 Texas A&M University College Station, TX, USA David B. Dahl Assistant Professor Texas A&M University College Station,
Bagging During Markov Chain Monte Carlo for Smoother Predictions
Bagging During Markov Chain Monte Carlo for Smoother Predictions Herbert K. H. Lee University of California, Santa Cruz Abstract: Making good predictions from noisy data is a challenging problem. Methods
Nonparametric Bayesian Models for Sparse Matrices and Covariances
Nonparametric Bayesian Models for Sparse Matrices and Covariances Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Bayes
Gaussian Mixture Models
Gaussian Mixture Models Pradeep Ravikumar Co-instructor: Manuela Veloso Machine Learning 10-701 Some slides courtesy of Eric Xing, Carlos Guestrin (One) bad case for K- means Clusters may overlap Some
Bayesian Learning (II)
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Bayesian Learning (II) Niels Landwehr Overview Probabilities, expected values, variance Basic concepts of Bayesian learning MAP
Collapsed Variational Dirichlet Process Mixture Models
Collapsed Variational Dirichlet Process Mixture Models Kenichi Kurihara Dept. of Computer Science Tokyo Institute of Technology, Japan kurihara@mi.cs.titech.ac.jp Max Welling Dept. of Computer Science
Dirichlet Enhanced Latent Semantic Analysis
Dirichlet Enhanced Latent Semantic Analysis Kai Yu Siemens Corporate Technology D-81730 Munich, Germany Kai.Yu@siemens.com Shipeng Yu Institute for Computer Science University of Munich D-80538 Munich,
Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models
Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models Tutorial at CVPR 2012 Erik Sudderth Brown University Work by E. Fox, E. Sudderth, M. Jordan, & A. Willsky AOAS 2011: A Sticky HDP-HMM with
Unsupervised Learning
Unsupervised Learning Bayesian Model Comparison Zoubin Ghahramani zoubin@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc in Intelligent Systems, Dept Computer Science University College
Interpretable Latent Variable Models
Interpretable Latent Variable Models Fernando Perez-Cruz Bell Labs (Nokia) Department of Signal Theory and Communications, University Carlos III in Madrid 1 / 24 Outline 1 Introduction to Machine Learning
Spatial Bayesian Nonparametrics for Natural Image Segmentation
Spatial Bayesian Nonparametrics for Natural Image Segmentation Erik Sudderth Brown University Joint work with Michael Jordan University of California Soumya Ghosh Brown University Parsing Visual Scenes
Pattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
Pattern Recognition and Machine Learning. Bishop Chapter 9: Mixture Models and EM
Pattern Recognition and Machine Learning Chapter 9: Mixture Models and EM Thomas Mensink Jakob Verbeek October 11, 27 Le Menu 9.1 K-means clustering Getting the idea with a simple example 9.2 Mixtures
Introduction to Machine Learning
Introduction to Machine Learning Brown University CSCI 1950-F, Spring 2012 Prof. Erik Sudderth Lecture 25: Markov Chain Monte Carlo (MCMC) Course Review and Advanced Topics Many figures courtesy Kevin
Online Bayesian Passive-Agressive Learning
Online Bayesian Passive-Agressive Learning International Conference on Machine Learning, 2014 Tianlin Shi Jun Zhu Tsinghua University, China 21 August 2015 Presented by: Kyle Ulrich Introduction Online
CSCI 5822 Probabilistic Model of Human and Machine Learning. Mike Mozer University of Colorado
CSCI 5822 Probabilistic Model of Human and Machine Learning Mike Mozer University of Colorado Topics Language modeling Hierarchical processes Pitman-Yor processes Based on work of Teh (2006), A hierarchical
Coresets for Bayesian Logistic Regression
Coresets for Bayesian Logistic Regression Tamara Broderick ITT Career Development Assistant Professor, MIT With: Jonathan H. Huggins, Trevor Campbell 1 Bayesian inference Bayesian inference Complex, modular
Model Averaging (Bayesian Learning)
Model Averaging (Bayesian Learning) We want to predict the output Y of a new case that has input X = x given the training examples e: p(y x e) = m M P(Y m x e) = m M P(Y m x e)p(m x e) = m M P(Y m x)p(m
Sparse Bayesian Nonparametric Regression
François Caron caronfr@cs.ubc.ca Arnaud Doucet arnaud@cs.ubc.ca Departments of Computer Science and Statistics, University of British Columbia, Vancouver, Canada Abstract One of the most common problems
Non-parametric Bayesian Modeling and Fusion of Spatio-temporal Information Sources
th International Conference on Information Fusion Chicago, Illinois, USA, July -8, Non-parametric Bayesian Modeling and Fusion of Spatio-temporal Information Sources Priyadip Ray Department of Electrical
Expectation Propagation Algorithm
Expectation Propagation Algorithm 1 Shuang Wang School of Electrical and Computer Engineering University of Oklahoma, Tulsa, OK, 74135 Email: {shuangwang}@ou.edu This note contains three parts. First,
Bayesian Nonparametrics: Models Based on the Dirichlet Process
Bayesian Nonparametrics: Models Based on the Dirichlet Process Alessandro Panella Department of Computer Science University of Illinois at Chicago Machine Learning Seminar Series February 18, 2013 Alessandro
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS Parametric Distributions Basic building blocks: Need to determine given Representation: or? Recall Curve Fitting Binary Variables
Understanding Covariance Estimates in Expectation Propagation
Understanding Covariance Estimates in Expectation Propagation William Stephenson Department of EECS Massachusetts Institute of Technology Cambridge, MA 019 wtstephe@csail.mit.edu Tamara Broderick Department
Bayesian nonparametric models for bipartite graphs
Bayesian nonparametric models for bipartite graphs François Caron INRIA IMB - University of Bordeaux Talence, France Francois.Caron@inria.fr Abstract We develop a novel Bayesian nonparametric model for
Infinite Hierarchical Hidden Markov Models
Katherine A. Heller Engineering Department University of Cambridge Cambridge, UK heller@gatsby.ucl.ac.uk Yee Whye Teh and Dilan Görür Gatsby Unit University College London London, UK {ywteh,dilan}@gatsby.ucl.ac.uk
Bayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework
HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for
Dirichlet Processes and other non-parametric Bayesian models
Dirichlet Processes and other non-parametric Bayesian models Zoubin Ghahramani http://learning.eng.cam.ac.uk/zoubin/ zoubin@cs.cmu.edu Statistical Machine Learning CMU 10-702 / 36-702 Spring 2008 Model
Tree-Based Inference for Dirichlet Process Mixtures
Yang Xu Machine Learning Department School of Computer Science Carnegie Mellon University Pittsburgh, USA Katherine A. Heller Department of Engineering University of Cambridge Cambridge, UK Zoubin Ghahramani
U-Likelihood and U-Updating Algorithms: Statistical Inference in Latent Variable Models
U-Likelihood and U-Updating Algorithms: Statistical Inference in Latent Variable Models Jaemo Sung 1, Sung-Yang Bang 1, Seungjin Choi 1, and Zoubin Ghahramani 2 1 Department of Computer Science, POSTECH,
CPSC 540: Machine Learning
CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is
Bayesian non-parametric model to longitudinally predict churn
Bayesian non-parametric model to longitudinally predict churn Bruno Scarpa Università di Padova Conference of European Statistics Stakeholders Methodologists, Producers and Users of European Statistics
Probabilistic Time Series Classification
Probabilistic Time Series Classification Y. Cem Sübakan Boğaziçi University 25.06.2013 Y. Cem Sübakan (Boğaziçi University) M.Sc. Thesis Defense 25.06.2013 1 / 54 Problem Statement The goal is to assign
Latent Dirichlet Alloca/on
Latent Dirichlet Alloca/on Blei, Ng and Jordan ( 2002 ) Presented by Deepak Santhanam What is Latent Dirichlet Alloca/on? Genera/ve Model for collec/ons of discrete data Data generated by parameters which
Lecture 13 : Variational Inference: Mean Field Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2017 Lecture 13 : Variational Inference: Mean Field Approximation Lecturer: Willie Neiswanger Scribes: Xupeng Tong, Minxing Liu 1 Problem Setup 1.1
CS6220: DATA MINING TECHNIQUES
CS6220: DATA MINING TECHNIQUES Matrix Data: Clustering: Part 2 Instructor: Yizhou Sun yzsun@ccs.neu.edu October 19, 2014 Methods to Learn Matrix Data Set Data Sequence Data Time Series Graph & Network
Nonparametric Probabilistic Modelling
Nonparametric Probabilistic Modelling Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Signal processing and inference
Dirichlet Process. Yee Whye Teh, University College London
Dirichlet Process Yee Whye Teh, University College London Related keywords: Bayesian nonparametrics, stochastic processes, clustering, infinite mixture model, Blackwell-MacQueen urn scheme, Chinese restaurant
Latent Variable Models and EM Algorithm
SC4/SM8 Advanced Topics in Statistical Machine Learning Latent Variable Models and EM Algorithm Dino Sejdinovic Department of Statistics Oxford Slides and other materials available at: http://www.stats.ox.ac.uk/~sejdinov/atsml/
arxiv: v2 [stat.ml] 10 Sep 2012
![arxiv: v2 [stat.ml] 10 Sep 2012](/thumbs/86/93201787.jpg "arxiv: v2 [stat.ml] 10 Sep 2012")
Distance Dependent Infinite Latent Feature Models arxiv:1110.5454v2 [stat.ml] 10 Sep 2012 Samuel J. Gershman 1, Peter I. Frazier 2 and David M. Blei 3 1 Department of Psychology and Princeton Neuroscience
Bayesian estimation of the discrepancy with misspecified parametric models
Bayesian estimation of the discrepancy with misspecified parametric models Pierpaolo De Blasi University of Torino & Collegio Carlo Alberto Bayesian Nonparametrics workshop ICERM, 17-21 September 2012
STA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 7 Approximate
Intelligent Systems:
Intelligent Systems: Undirected Graphical models (Factor Graphs) (2 lectures) Carsten Rother 15/01/2015 Intelligent Systems: Probabilistic Inference in DGM and UGM Roadmap for next two lectures Definition
arxiv: v2 [stat.ml] 4 Aug 2011
![arxiv: v2 [stat.ml] 4 Aug 2011](/thumbs/87/95597257.jpg "arxiv: v2 [stat.ml] 4 Aug 2011")
A Tutorial on Bayesian Nonparametric Models Samuel J. Gershman 1 and David M. Blei 2 1 Department of Psychology and Neuroscience Institute, Princeton University 2 Department of Computer Science, Princeton
A Stick-Breaking Construction of the Beta Process
John Paisley 1 jwp4@ee.duke.edu Aimee Zaas 2 aimee.zaas@duke.edu Christopher W. Woods 2 woods004@mc.duke.edu Geoffrey S. Ginsburg 2 ginsb005@duke.edu Lawrence Carin 1 lcarin@ee.duke.edu 1 Department of
Spatial Normalized Gamma Process
Spatial Normalized Gamma Process Vinayak Rao Yee Whye Teh Presented at NIPS 2009 Discussion and Slides by Eric Wang June 23, 2010 Outline Introduction Motivation The Gamma Process Spatial Normalized Gamma
The Kernel Beta Process
The Kernel Beta Process Lu Ren Electrical & Computer Engineering Dept. Duke University Durham, NC 27708 lr22@duke.edu David Dunson Department of Statistical Science Duke University Durham, NC 27708 dunson@stat.duke.edu