Bayesian Nonparametric Learning of Complex Dynamical Phenomena
|
|
|
- Phillip Branden Allison
- 5 years ago
- Views:
Transcription
1 Duke University Department of Statistical Science Bayesian Nonparametric Learning of Complex Dynamical Phenomena Emily Fox Joint work with Erik Sudderth (Brown University), Michael Jordan (UC Berkeley), and Alan Willsky (MIT)
2 Temporal Complexity of Time Series COMPLEXITY Autoregressive ARM A Linear state space Non-linear state space a.k.a. dynamic linear model (statistics), linear Trade-off dynamical between system expressivity (control) of model, and computational tractability Bayesian nonparametrics: state observations Data-driven flexibility in model complexity, computationally attractive properties
3 Complex, but Patterned Conference AudioFOCUS ON Wooters and Huijbregts, LNCS, 2007 SINGLE TIME SERIES Honey Oh et. al., Bee IJCV, Dances 2007 Pavlovic Human et. al., Motion NIPS, 2001 Carvalho and Lopes, Trends Comp. in Stock Stat. Data & Data Anal., 2006
4 Discovering Common Behaviors Time Series 1 Time Series 2 Time Series 3 Jointly model multiple time-series Transfer knowledge between related time-series Find interesting relationships
5 MODE Hidden Markov Model Model for single time series with patterned behaviors Transition distributions jumping jacks squats side twists Mode sequence TIME jumping jacks squats side twists Observations (e.g., body position) Tutorial: Rabiner, Proc. IEEE 1989
6 Dirichlet Process Mixture Model Stick-breaking construction for: DP(aH) DPMM Stick of unit probability mass
7 MODE HDP-HMM: Teh, et. al., JASA 2006 TIME Dirichlet process (DP): Mode space of unbounded size Model complexity adapts to observations Hierarchical: Ties mode transition distributions Shared sparsity
8 Sticky HDP-HMM HDP-HMM of Teh, et. al., JASA 2006 special case when. Global transition distribution: Mode-specific transition distributions: Hyperpriors on extra mass on component corresponding to self-transition sparsity of b is shared Related self-transition parameter: Beal, et.al., NIPS 2002
9 Samples From the Prior MODE SEQUENCES HDP-HMM Sticky HDP-HMM
10 Speaker Diarization dim 0-10 MFCCs TIME x 10 4 Bob John J i l l B o b Jane John TIME
11 MoG Emissions modes mixture components observations Speaker 1 Speaker 2 Speaker 3 Speaker 4
12 Results: 21 meetings Overall DER Best DER Worst DER Sticky HDP-HMM 17.84% 1.26% 34.29% Non-Sticky HDP-HMM 23.91% 6.26% 46.95% ICSI 18.37% 4.39% 32.23% DER = Diarization Error Rate
13 Results: Meeting 1 Sticky DER = 1.26% ICSI DER = 7.56%
14 Results: Meeting % Sticky DER = 20.48% ICSI DER = 22.00%
15 Outline Background: HMM, DP Temporal Persistence: Sticky HDP-HMM Conditionally Linear Dynamics HDP-AR-HMM and HDP-SLDS Relating Multiple Time Series
16 Switching Dynamical Process Each dynamical mode has conditionally linear dynamics Captures more complex temporal dependencies HDP-AR-HMM Switching linear dynamical systems HDP-SLDS Switching AR processes Gaussian HMMs Conditionally vector autoregressive Conditionally linear state space models
17 Sparsity-Inducing Prior - ARD HDP-SLDS: decompose on columns HDP-AR-HMM: decompose on lag blocks Inverse Wishart prior on covariance of additive noise terms
18 IBOVESPA Stochastic Volatility Data: IBOVESPA daily returns Goal: detect changes in volatility Daily Returns HDP-SLDS allows for unbounded number of volatility regimes 10 key world events cited in: Carvalho and Lopes, Comp. Stat. & Data Anal., 2006 Two-regime Markov switching stochastic volatility (MSSV) model, Particle filter for online inference
19 IBOVESPA Stochastic Volatility Posterior probability of HDP-SLDS regime changepoint at each date 10 key world events shown in red Hong Kong stock index falls 10.4% HDP-SLDS non-sticky HDP-SLDS
20 IBOVESPA Stochastic Volatility ROC curves for: ROC HDP-SLDS Non-sticky HDP-SLDS HDP-AR(1)-HMM HDP-AR(2)-HMM Declare a detection if inferred changepoint is within small window around world event
21 Dancing Honey Bees
22 Honey Bee Observations Sequence 1 Sequence 2 Sequence 3 3 bee dance sequences with expert labeled dances: Turn right (green) Waggle (red) Turn left (blue) x-pos y-pos Observation vector: Head angle (cosq, sinq) x-y body position Oh et. al., IJCV, 2007 sinq cosq Time
![1% SLDS [Oh]: 93.](/docs-images/83/87632992/images/23-5.jpg "4% HDP-AR-HMM: 92.")
23 Honey Bee Results: HDP-AR-HMM Sequence 1 Sequence 2 Sequence 3 HDP-AR-HMM: 88.1% SLDS [Oh]: 93.4% HDP-AR-HMM: 92.5% SLDS [Oh]: 90.2% HDP-AR-HMM: 88.2% SLDS [Oh]: 90.4%
24 Log-Likelihood Predictive Likelihood ARD prior useful in regularizing higher order models Compare AR(1), AR(2), and AR(7) MNIW models to AR(7) ARD model Train on first half of each bee sequence, test on second half Turning dances well-approximated by AR(1), waggle dance by AR(2) 95% HPD Intervals of Predictive Likelihood
25 Outline Background: HMM, DP Temporal Persistence: Sticky HDP-HMM Conditionally Linear Dynamics: HDP-AR-HMM, HDP-SLDS Relating Multiple Time Series Beta process Bernoulli process featural model BP-AR-HMM
26 Discovering Common Behaviors Time Series 1 Time Series 2 Time Series 3 z z 2 1 z 5 z 3 z 4 z 5 z 7 z 6 z z 9 8 z 10 z z 2 1 z 5 z z 4 3 z 12 z 13 z 7 z 6 z z 9 8 z 11 z 10 z z 2 1 z 5 z z 4 3 z 7 z 6 z z 9 8 z 14 z 10 z 15 COMPLETELY GOAL: IDENTICAL RELATED DISJOINT STATE BUT NOT STATE SPACES IDENTICAL SPACES
27 Discovering Common Behaviors Time Series 1 Time Series 2 Time Series 3 Model each time series as an AR-HMM Related time series Tie together under a common prior Infinitely many possible behaviors Bayesian nonparametrics
28 Discovering Common Behaviors Hierarchical Dirichlet Process HDP z z 2 1 z 5 z 3 z 4 z 5 z 7 z 6 z z 9 8 z 10 z z 2 1 z 3 z 4 z 5 z 6 z 12 z 8 z 13 z 7 z 9 z 10 z 11 z z 2 1 z 5 z z 4 3 z 7 z 6 z z 9 8 z 14 z 10 z 15 Time Series 1 Time Series 2 Time Series 3 HDP-AR-HMM Utilize beta process prior Allows for an Objects unbounded can exhibit state space unique behaviors Assumes all Two time objects series exhibit can switch the same between behaviors the same in same way set of behaviors in different manners
29 Beta Process Prior for Featural Models Beta process-bernoulli Process Equivalently Points generated from Poisson process on: Beta process: Hjort, Annals of Statistics, 1990 Beta process Bernoulli process: Thibaux and Jordan, AISTATS, 2007
30 Indian Buffet Process (IBP) Marginalize beta process measure Indian buffet process (IBP) Shared features: Dirichlet Process Chinese Restaurant Process Unique features: Beta- Bernoulli Process Indian Buffet Process Griffiths and Ghahramani, TR, 2005
31 Indian Buffet Process (IBP) Marginalize beta process measure Indian buffet process (IBP) Shared features: Unique features: Previously sampled Griffiths and Ghahramani, TR, 2005
32 Indian Buffet Process (IBP) Marginalize beta process measure Indian buffet process (IBP) Shared features: Unique features: Previously sampled Griffiths and Ghahramani, TR, 2005
33 Time Customers Series BP-AR-HMM Features indicating behaviors of object i Transition patterns for object i Shared library of behaviors Features constrain dynamics to chosen behaviors Dynamic Dishes Behavior
34 BP-AR-HMM Features indicating behaviors of object i Transition patterns for object i Shared library of behaviors Features constrain dynamics to chosen behaviors
35 BP-AR-HMM Features indicating behaviors of object i Transition patterns for object i Shared library of behaviors mode evolution transition distributions observations
36 BP-AR-HMM Features indicating behaviors of object i Transition patterns for object i Beta process prior Shared library of behaviors Beta process realization Bernoulli process realization Beta process prior: Encourages sharing + allows variability
37 Time Series BP-AR-HMM Features indicating behaviors of object i Transition patterns for object i Beta Conditioned on the feature matrix, process model reduces to collection of finite AR-HMMs prior Shared library of behaviors Efficient MCMC computations Dynamic Behavior Likelihood: based on induced collection of finite AR-HMMs (marginalize mode sequence via FFBS) Prior: IBP Forward-Backward Algorithm Model not conjugate birth-death RJMCMC Generic to Models with Discrete State Spaces (e.g. HMM, SLDS)
38 Motion Capture 62-dim observations Chose 12 gross motor components Ran MCMC sampler for 20,000 iters Ran 25 chains of sampler CMU MoCap: DISCOVER COMMON BEHAVIORS
39 Library of Discovered MoCap Behaviors Chosen MCMC sample minimizes an expected Hamming distance criterion
40 Jumping Jacks
41 Side Twists
42 Arm Circles
43 Squats
44 Unique Behaviors Bend Over Punching Toe Touch 1 Toe Touch 2 Identified collection of behaviors appearing in only one movie
45 Split Motions: Knee Raises
46 Split Motions: Running in Place
47 Library of Discovered MoCap Behaviors
48 Learned MoCap Feature Matrices Learned feature matrices for four different models:
49 Conclusion Examined Bayesian nonparametric Markov switching processes: HMM, AR-HMM, SLDS Developed method of relating multiple time series Demonstrated utility on challenging datasets including speaker diarization, stock volatility, and motion capture analysis
Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models
Applied Bayesian Nonparametrics 3. Infinite Hidden Markov Models Tutorial at CVPR 2012 Erik Sudderth Brown University Work by E. Fox, E. Sudderth, M. Jordan, & A. Willsky AOAS 2011: A Sticky HDP-HMM with
Bayesian Nonparametrics for Speech and Signal Processing
Bayesian Nonparametrics for Speech and Signal Processing Michael I. Jordan University of California, Berkeley June 28, 2011 Acknowledgments: Emily Fox, Erik Sudderth, Yee Whye Teh, and Romain Thibaux Computer
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 59, NO. 4, APRIL
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 59, NO. 4, APRIL 2011 1569 Bayesian Nonparametric Inference of Switching Dynamic Linear Models Emily Fox, Member, IEEE, Erik B. Sudderth, Member, IEEE, Michael
Mixed Membership Models for Time Series
20 Mixed Membership Models for Time Series Emily B. Fox Department of Statistics, University of Washington, Seattle, WA 98195, USA Michael I. Jordan Computer Science Division and Department of Statistics,
Bayesian Nonparametric Learning of Complex Dynamical Phenomena
Bayesian Nonparametric Learning of Complex Dynamical Phenomena by Emily B. Fox S.B., Electrical Engineering, Massachusetts Institute of Technology, 24 M.Eng., Elect. Eng. and Comp. Sci., Massachusetts
Bayesian Nonparametric Methods for Learning Markov Switching Processes
IEEE SIGNAL PROCESSING MAGAZINE SPECIAL ISSUE 1 Bayesian Nonparametric Methods for Learning Markov Switching Processes Emily B. Fox, Member, IEEE, Erik B. Sudderth, Member, IEEE, Michael I. Jordan, Fellow,
Non-Parametric Bayes
Non-Parametric Bayes Mark Schmidt UBC Machine Learning Reading Group January 2016 Current Hot Topics in Machine Learning Bayesian learning includes: Gaussian processes. Approximate inference. Bayesian
Bayesian Nonparametric Methods for Learning Markov Switching Processes
Bayesian Nonparametric Methods for Learning Markov Switching Processes The MIT Faculty has made this article openly available Please share how this access benefits you Your story matters Citation As Published
A Brief Overview of Nonparametric Bayesian Models
A Brief Overview of Nonparametric Bayesian Models Eurandom Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin Also at Machine
Probabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Infinite Feature Models: The Indian Buffet Process Eric Xing Lecture 21, April 2, 214 Acknowledgement: slides first drafted by Sinead Williamson
Bayesian non parametric approaches: an introduction
Introduction Latent class models Latent feature models Conclusion & Perspectives Bayesian non parametric approaches: an introduction Pierre CHAINAIS Bordeaux - nov. 2012 Trajectory 1 Bayesian non parametric
Spatial Bayesian Nonparametrics for Natural Image Segmentation
Spatial Bayesian Nonparametrics for Natural Image Segmentation Erik Sudderth Brown University Joint work with Michael Jordan University of California Soumya Ghosh Brown University Parsing Visual Scenes
JOINT MODELING OF MULTIPLE TIME SERIES VIA THE BETA PROCESS WITH APPLICATION TO MOTION CAPTURE SEGMENTATION
The Annals of Applied Statistics 2014, Vol. 8, No. 3, 1281 1313 DOI: 10.1214/14-AOAS742 Institute of Mathematical Statistics, 2014 JOINT MODELING OF MULTIPLE TIME SERIES VIA THE BETA PROCESS WITH APPLICATION
Inference in Explicit Duration Hidden Markov Models
Inference in Explicit Duration Hidden Markov Models Frank Wood Joint work with Chris Wiggins, Mike Dewar Columbia University November, 2011 Wood (Columbia University) EDHMM Inference November, 2011 1 /
Nonparametric Bayesian Models for Sparse Matrices and Covariances
Nonparametric Bayesian Models for Sparse Matrices and Covariances Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Bayes
Bayesian Hidden Markov Models and Extensions
Bayesian Hidden Markov Models and Extensions Zoubin Ghahramani Department of Engineering University of Cambridge joint work with Matt Beal, Jurgen van Gael, Yunus Saatci, Tom Stepleton, Yee Whye Teh Modeling
Infinite-State Markov-switching for Dynamic. Volatility Models : Web Appendix
Infinite-State Markov-switching for Dynamic Volatility Models : Web Appendix Arnaud Dufays 1 Centre de Recherche en Economie et Statistique March 19, 2014 1 Comparison of the two MS-GARCH approximations
Gentle Introduction to Infinite Gaussian Mixture Modeling
Gentle Introduction to Infinite Gaussian Mixture Modeling with an application in neuroscience By Frank Wood Rasmussen, NIPS 1999 Neuroscience Application: Spike Sorting Important in neuroscience and for
MAD-Bayes: MAP-based Asymptotic Derivations from Bayes
MAD-Bayes: MAP-based Asymptotic Derivations from Bayes Tamara Broderick Brian Kulis Michael I. Jordan Cat Clusters Mouse clusters Dog 1 Cat Clusters Dog Mouse Lizard Sheep Picture 1 Picture 2 Picture 3
Bayesian nonparametric models for bipartite graphs
Bayesian nonparametric models for bipartite graphs François Caron Department of Statistics, Oxford Statistics Colloquium, Harvard University November 11, 2013 F. Caron 1 / 27 Bipartite networks Readers/Customers
Bayesian Nonparametrics: Models Based on the Dirichlet Process
Bayesian Nonparametrics: Models Based on the Dirichlet Process Alessandro Panella Department of Computer Science University of Illinois at Chicago Machine Learning Seminar Series February 18, 2013 Alessandro
Lecture 3a: Dirichlet processes
Lecture 3a: Dirichlet processes Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London c.archambeau@cs.ucl.ac.uk Advanced Topics
The Infinite Factorial Hidden Markov Model
The Infinite Factorial Hidden Markov Model Jurgen Van Gael Department of Engineering University of Cambridge, UK jv279@cam.ac.uk Yee Whye Teh Gatsby Unit University College London, UK ywteh@gatsby.ucl.ac.uk
STA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 11 Project
arxiv: v3 [stat.me] 13 Nov 2014
![arxiv: v3 [stat.me] 13 Nov 2014](/thumbs/73/68922790.jpg "arxiv: v3 [stat.me] 13 Nov 2014")
The Annals of Applied Statistics 2014, Vol. 8, No. 3, 1281 1313 DOI: 10.1214/14-AOAS742 c Institute of Mathematical Statistics, 2014 arxiv:1308.4747v3 [stat.me] 13 Nov 2014 JOINT MODELING OF MULTIPLE TIME
Non-parametric Clustering with Dirichlet Processes
Non-parametric Clustering with Dirichlet Processes Timothy Burns SUNY at Buffalo Mar. 31 2009 T. Burns (SUNY at Buffalo) Non-parametric Clustering with Dirichlet Processes Mar. 31 2009 1 / 24 Introduction
Hierarchical Models, Nested Models and Completely Random Measures
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/238729763 Hierarchical Models, Nested Models and Completely Random Measures Article March 2012
19 : Bayesian Nonparametrics: The Indian Buffet Process. 1 Latent Variable Models and the Indian Buffet Process
10-708: Probabilistic Graphical Models, Spring 2015 19 : Bayesian Nonparametrics: The Indian Buffet Process Lecturer: Avinava Dubey Scribes: Rishav Das, Adam Brodie, and Hemank Lamba 1 Latent Variable
Bayesian Nonparametric Models
Bayesian Nonparametric Models David M. Blei Columbia University December 15, 2015 Introduction We have been looking at models that posit latent structure in high dimensional data. We use the posterior
Bayesian Methods for Machine Learning
Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),
STA 414/2104: Machine Learning
STA 414/2104: Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistics! rsalakhu@cs.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 9 Sequential Data So far
Bayesian nonparametric latent feature models
Bayesian nonparametric latent feature models Indian Buffet process, beta process, and related models François Caron Department of Statistics, Oxford Applied Bayesian Statistics Summer School Como, Italy
Bayesian Nonparametrics: Dirichlet Process
Bayesian Nonparametrics: Dirichlet Process Yee Whye Teh Gatsby Computational Neuroscience Unit, UCL http://www.gatsby.ucl.ac.uk/~ywteh/teaching/npbayes2012 Dirichlet Process Cornerstone of modern Bayesian
Graphical Models for Query-driven Analysis of Multimodal Data
Graphical Models for Query-driven Analysis of Multimodal Data John Fisher Sensing, Learning, & Inference Group Computer Science & Artificial Intelligence Laboratory Massachusetts Institute of Technology
Hidden Markov models: from the beginning to the state of the art
Hidden Markov models: from the beginning to the state of the art Frank Wood Columbia University November, 2011 Wood (Columbia University) HMMs November, 2011 1 / 44 Outline Overview of hidden Markov models
Infinite latent feature models and the Indian Buffet Process
p.1 Infinite latent feature models and the Indian Buffet Process Tom Griffiths Cognitive and Linguistic Sciences Brown University Joint work with Zoubin Ghahramani p.2 Beyond latent classes Unsupervised
Lecture 16-17: Bayesian Nonparametrics I. STAT 6474 Instructor: Hongxiao Zhu
Lecture 16-17: Bayesian Nonparametrics I STAT 6474 Instructor: Hongxiao Zhu Plan for today Why Bayesian Nonparametrics? Dirichlet Distribution and Dirichlet Processes. 2 Parameter and Patterns Reference:
Bayesian Mixtures of Bernoulli Distributions
Bayesian Mixtures of Bernoulli Distributions Laurens van der Maaten Department of Computer Science and Engineering University of California, San Diego Introduction The mixture of Bernoulli distributions
Priors for Random Count Matrices with Random or Fixed Row Sums
Priors for Random Count Matrices with Random or Fixed Row Sums Mingyuan Zhou Joint work with Oscar Madrid and James Scott IROM Department, McCombs School of Business Department of Statistics and Data Sciences
arxiv: v2 [stat.ml] 10 Sep 2012
![arxiv: v2 [stat.ml] 10 Sep 2012](/thumbs/86/93201787.jpg "arxiv: v2 [stat.ml] 10 Sep 2012")
Distance Dependent Infinite Latent Feature Models arxiv:1110.5454v2 [stat.ml] 10 Sep 2012 Samuel J. Gershman 1, Peter I. Frazier 2 and David M. Blei 3 1 Department of Psychology and Princeton Neuroscience
arxiv: v2 [stat.ml] 4 Aug 2011
![arxiv: v2 [stat.ml] 4 Aug 2011](/thumbs/87/95597257.jpg "arxiv: v2 [stat.ml] 4 Aug 2011")
A Tutorial on Bayesian Nonparametric Models Samuel J. Gershman 1 and David M. Blei 2 1 Department of Psychology and Neuroscience Institute, Princeton University 2 Department of Computer Science, Princeton
Applied Nonparametric Bayes
Applied Nonparametric Bayes Michael I. Jordan Department of Electrical Engineering and Computer Science Department of Statistics University of California, Berkeley http://www.cs.berkeley.edu/ jordan Acknowledgments:
Infinite Latent Feature Models and the Indian Buffet Process
Infinite Latent Feature Models and the Indian Buffet Process Thomas L. Griffiths Cognitive and Linguistic Sciences Brown University, Providence RI 292 tom griffiths@brown.edu Zoubin Ghahramani Gatsby Computational
Stochastic Variational Inference for the HDP-HMM
Stochastic Variational Inference for the HDP-HMM Aonan Zhang San Gultekin John Paisley Department of Electrical Engineering & Data Science Institute Columbia University, New York, NY Abstract We derive
Dirichlet Processes and other non-parametric Bayesian models
Dirichlet Processes and other non-parametric Bayesian models Zoubin Ghahramani http://learning.eng.cam.ac.uk/zoubin/ zoubin@cs.cmu.edu Statistical Machine Learning CMU 10-702 / 36-702 Spring 2008 Model
Haupthseminar: Machine Learning. Chinese Restaurant Process, Indian Buffet Process
Haupthseminar: Machine Learning Chinese Restaurant Process, Indian Buffet Process Agenda Motivation Chinese Restaurant Process- CRP Dirichlet Process Interlude on CRP Infinite and CRP mixture model Estimation
Hierarchical Dirichlet Processes
Hierarchical Dirichlet Processes Yee Whye Teh, Michael I. Jordan, Matthew J. Beal and David M. Blei Computer Science Div., Dept. of Statistics Dept. of Computer Science University of California at Berkeley
Dirichlet Processes: Tutorial and Practical Course
Dirichlet Processes: Tutorial and Practical Course (updated) Yee Whye Teh Gatsby Computational Neuroscience Unit University College London August 2007 / MLSS Yee Whye Teh (Gatsby) DP August 2007 / MLSS
Variational Inference for Hierarchical Dirichlet Process Based Nonparametric Models
1 INTRODUCTION 1 Variational Inference for Hierarchical Dirichlet Process Based Nonparametric Models Will Stephenson Advisor: Erik Sudderth, Reader: Ben Raphael Abstract We examine two popular statistical
Shared Segmentation of Natural Scenes. Dependent Pitman-Yor Processes
Shared Segmentation of Natural Scenes using Dependent Pitman-Yor Processes Erik Sudderth & Michael Jordan University of California, Berkeley Parsing Visual Scenes sky skyscraper sky dome buildings trees
Hidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 22 April 2, 2018 1 Reminders Homework
Lecture 10. Announcement. Mixture Models II. Topics of This Lecture. This Lecture: Advanced Machine Learning. Recap: GMMs as Latent Variable Models
Advanced Machine Learning Lecture 10 Mixture Models II 30.11.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ Announcement Exercise sheet 2 online Sampling Rejection Sampling Importance
A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models
A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models Jeff A. Bilmes (bilmes@cs.berkeley.edu) International Computer Science Institute
Hierarchical Dirichlet Processes with Random Effects
Hierarchical Dirichlet Processes with Random Effects Seyoung Kim Department of Computer Science University of California, Irvine Irvine, CA 92697-34 sykim@ics.uci.edu Padhraic Smyth Department of Computer
Nonparametric Bayesian Methods: Models, Algorithms, and Applications (Day 5)
Nonparametric Bayesian Methods: Models, Algorithms, and Applications (Day 5) Tamara Broderick ITT Career Development Assistant Professor Electrical Engineering & Computer Science MIT Bayes Foundations
Nonparmeteric Bayes & Gaussian Processes. Baback Moghaddam Machine Learning Group
Nonparmeteric Bayes & Gaussian Processes Baback Moghaddam baback@jpl.nasa.gov Machine Learning Group Outline Bayesian Inference Hierarchical Models Model Selection Parametric vs. Nonparametric Gaussian
Nonparametric Probabilistic Modelling
Nonparametric Probabilistic Modelling Zoubin Ghahramani Department of Engineering University of Cambridge, UK zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Signal processing and inference
Nonparametric Factor Analysis with Beta Process Priors
Nonparametric Factor Analysis with Beta Process Priors John Paisley Lawrence Carin Department of Electrical & Computer Engineering Duke University, Durham, NC 7708 jwp4@ee.duke.edu lcarin@ee.duke.edu Abstract
Sharing Clusters Among Related Groups: Hierarchical Dirichlet Processes
Sharing Clusters Among Related Groups: Hierarchical Dirichlet Processes Yee Whye Teh (1), Michael I. Jordan (1,2), Matthew J. Beal (3) and David M. Blei (1) (1) Computer Science Div., (2) Dept. of Statistics
Chapter 4 Dynamic Bayesian Networks Fall Jin Gu, Michael Zhang
Chapter 4 Dynamic Bayesian Networks 2016 Fall Jin Gu, Michael Zhang Reviews: BN Representation Basic steps for BN representations Define variables Define the preliminary relations between variables Check
Spatial Normalized Gamma Process
Spatial Normalized Gamma Process Vinayak Rao Yee Whye Teh Presented at NIPS 2009 Discussion and Slides by Eric Wang June 23, 2010 Outline Introduction Motivation The Gamma Process Spatial Normalized Gamma
Stochastic Processes, Kernel Regression, Infinite Mixture Models
Stochastic Processes, Kernel Regression, Infinite Mixture Models Gabriel Huang (TA for Simon Lacoste-Julien) IFT 6269 : Probabilistic Graphical Models - Fall 2018 Stochastic Process = Random Function 2
Bayesian Nonparametrics
Bayesian Nonparametrics Peter Orbanz Columbia University PARAMETERS AND PATTERNS Parameters P(X θ) = Probability[data pattern] 3 2 1 0 1 2 3 5 0 5 Inference idea data = underlying pattern + independent
CPSC 540: Machine Learning
CPSC 540: Machine Learning MCMC and Non-Parametric Bayes Mark Schmidt University of British Columbia Winter 2016 Admin I went through project proposals: Some of you got a message on Piazza. No news is
Dynamic Probabilistic Models for Latent Feature Propagation in Social Networks
Dynamic Probabilistic Models for Latent Feature Propagation in Social Networks Creighton Heaukulani and Zoubin Ghahramani University of Cambridge TU Denmark, June 2013 1 A Network Dynamic network data
Infinite Hierarchical Hidden Markov Models
Katherine A. Heller Engineering Department University of Cambridge Cambridge, UK heller@gatsby.ucl.ac.uk Yee Whye Teh and Dilan Görür Gatsby Unit University College London London, UK {ywteh,dilan}@gatsby.ucl.ac.uk
Bayesian Nonparametrics
Bayesian Nonparametrics Lorenzo Rosasco 9.520 Class 18 April 11, 2011 About this class Goal To give an overview of some of the basic concepts in Bayesian Nonparametrics. In particular, to discuss Dirichelet
Steven L. Scott. Presented by Ahmet Engin Ural
Steven L. Scott Presented by Ahmet Engin Ural Overview of HMM Evaluating likelihoods The Likelihood Recursion The Forward-Backward Recursion Sampling HMM DG and FB samplers Autocovariance of samplers Some
Recent Advances in Bayesian Inference Techniques
Recent Advances in Bayesian Inference Techniques Christopher M. Bishop Microsoft Research, Cambridge, U.K. research.microsoft.com/~cmbishop SIAM Conference on Data Mining, April 2004 Abstract Bayesian
Construction of Dependent Dirichlet Processes based on Poisson Processes
1 / 31 Construction of Dependent Dirichlet Processes based on Poisson Processes Dahua Lin Eric Grimson John Fisher CSAIL MIT NIPS 2010 Outstanding Student Paper Award Presented by Shouyuan Chen Outline
Interpretable Latent Variable Models
Interpretable Latent Variable Models Fernando Perez-Cruz Bell Labs (Nokia) Department of Signal Theory and Communications, University Carlos III in Madrid 1 / 24 Outline 1 Introduction to Machine Learning
Towards inference for skewed alpha stable Levy processes
Towards inference for skewed alpha stable Levy processes Simon Godsill and Tatjana Lemke Signal Processing and Communications Lab. University of Cambridge www-sigproc.eng.cam.ac.uk/~sjg Overview Motivation
Part IV: Monte Carlo and nonparametric Bayes
Part IV: Monte Carlo and nonparametric Bayes Outline Monte Carlo methods Nonparametric Bayesian models Outline Monte Carlo methods Nonparametric Bayesian models The Monte Carlo principle The expectation
Collapsed Variational Dirichlet Process Mixture Models
Collapsed Variational Dirichlet Process Mixture Models Kenichi Kurihara Dept. of Computer Science Tokyo Institute of Technology, Japan kurihara@mi.cs.titech.ac.jp Max Welling Dept. of Computer Science
Bayesian Nonparametric Models on Decomposable Graphs
Bayesian Nonparametric Models on Decomposable Graphs François Caron INRIA Bordeaux Sud Ouest Institut de Mathématiques de Bordeaux University of Bordeaux, France francois.caron@inria.fr Arnaud Doucet Departments
Lecture 9. Time series prediction
Lecture 9 Time series prediction Prediction is about function fitting To predict we need to model There are a bewildering number of models for data we look at some of the major approaches in this lecture
Supplementary Notes: Segment Parameter Labelling in MCMC Change Detection
Supplementary Notes: Segment Parameter Labelling in MCMC Change Detection Alireza Ahrabian 1 arxiv:1901.0545v1 [eess.sp] 16 Jan 019 Abstract This work addresses the problem of segmentation in time series
Image segmentation combining Markov Random Fields and Dirichlet Processes
Image segmentation combining Markov Random Fields and Dirichlet Processes Jessica SODJO IMS, Groupe Signal Image, Talence Encadrants : A. Giremus, J.-F. Giovannelli, F. Caron, N. Dobigeon Jessica SODJO
Non-parametric Bayesian Methods
Non-parametric Bayesian Methods Uncertainty in Artificial Intelligence Tutorial July 25 Zoubin Ghahramani Gatsby Computational Neuroscience Unit University College London, UK Center for Automated Learning
Outline. Binomial, Multinomial, Normal, Beta, Dirichlet. Posterior mean, MAP, credible interval, posterior distribution
Outline A short review on Bayesian analysis. Binomial, Multinomial, Normal, Beta, Dirichlet Posterior mean, MAP, credible interval, posterior distribution Gibbs sampling Revisit the Gaussian mixture model
Bayesian nonparametric latent feature models
Bayesian nonparametric latent feature models François Caron UBC October 2, 2007 / MLRG François Caron (UBC) Bayes. nonparametric latent feature models October 2, 2007 / MLRG 1 / 29 Overview 1 Introduction
An Introduction to Bayesian Machine Learning
1 An Introduction to Bayesian Machine Learning José Miguel Hernández-Lobato Department of Engineering, Cambridge University April 8, 2013 2 What is Machine Learning? The design of computational systems
The Kernel Beta Process
The Kernel Beta Process Lu Ren Electrical & Computer Engineering Dept. Duke University Durham, NC 27708 lr22@duke.edu David Dunson Department of Statistical Science Duke University Durham, NC 27708 dunson@stat.duke.edu
ICML Scalable Bayesian Inference on Point processes. with Gaussian Processes. Yves-Laurent Kom Samo & Stephen Roberts
ICML 2015 Scalable Nonparametric Bayesian Inference on Point Processes with Gaussian Processes Machine Learning Research Group and Oxford-Man Institute University of Oxford July 8, 2015 Point Processes
Hidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 19 Nov. 5, 2018 1 Reminders Homework
Modelling Genetic Variations with Fragmentation-Coagulation Processes
Modelling Genetic Variations with Fragmentation-Coagulation Processes Yee Whye Teh, Charles Blundell, Lloyd Elliott Gatsby Computational Neuroscience Unit, UCL Genetic Variations in Populations Inferring
This work has been accepted in 2018 IEEE 88th Vehicular Technology Conference (VTC2018-Fall) August 2018, Chicago, USA. IEEE Copyright Notice:
This work has been accepted in 2018 IEEE 88th Vehicular Technology Conference (VTC2018-Fall) 27 30 August 2018, Chicago, USA IEEE Copyright Notice: 2018 IEEE. Personal use of this material is permitted.
Infinite Support Vector Machines in Speech Recognition
Infinite Support Vector Machines in Speech Recognition Jingzhou Yang, Rogier C. van Dalen and Mark Gales Department of Engineering, University of Cambridge Trumpington Street, Cambridge, CB2 1PZ, UK {jy308,rcv25,mjfg}@eng.cam.ac.uk
Linear Dynamical Systems
Linear Dynamical Systems Sargur N. srihari@cedar.buffalo.edu Machine Learning Course: http://www.cedar.buffalo.edu/~srihari/cse574/index.html Two Models Described by Same Graph Latent variables Observations
Non-parametric Bayesian Modeling and Fusion of Spatio-temporal Information Sources
th International Conference on Information Fusion Chicago, Illinois, USA, July -8, Non-parametric Bayesian Modeling and Fusion of Spatio-temporal Information Sources Priyadip Ray Department of Electrical
STA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 7 Approximate
27 : Distributed Monte Carlo Markov Chain. 1 Recap of MCMC and Naive Parallel Gibbs Sampling
10-708: Probabilistic Graphical Models 10-708, Spring 2014 27 : Distributed Monte Carlo Markov Chain Lecturer: Eric P. Xing Scribes: Pengtao Xie, Khoa Luu In this scribe, we are going to review the Parallel
Pattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
The Infinite Latent Events Model
The Infinite Latent Events Model David Wingate, Noah D. Goodman, Daniel M. Roy and Joshua B. Tenenbaum Brain and Cognitive Sciences Massachusetts Institute of Technology Cambridge, MA 02139 Abstract We
Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures
17th Europ. Conf. on Machine Learning, Berlin, Germany, 2006. Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures Shipeng Yu 1,2, Kai Yu 2, Volker Tresp 2, and Hans-Peter
Probabilistic Graphical Models
Probabilistic Graphical Models Brown University CSCI 295-P, Spring 213 Prof. Erik Sudderth Lecture 11: Inference & Learning Overview, Gaussian Graphical Models Some figures courtesy Michael Jordan s draft
Bayesian Nonparametric Hidden Semi-Markov Models
Journal of Machine Learning Research 14 (2013) 673-701 Submitted 12/11; Revised 9/12; Published 2/13 Bayesian Nonparametric Hidden Semi-Markov Models Matthew J. Johnson Alan S. Willsky Laboratory for Information
Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems
Bayesian Learning and Inference in Recurrent Switching Linear Dynamical Systems Scott W. Linderman Matthew J. Johnson Andrew C. Miller Columbia University Harvard and Google Brain Harvard University Ryan
Nonparametric Bayesian Methods - Lecture I
Nonparametric Bayesian Methods - Lecture I Harry van Zanten Korteweg-de Vries Institute for Mathematics CRiSM Masterclass, April 4-6, 2016 Overview of the lectures I Intro to nonparametric Bayesian statistics
Lecture 13 : Variational Inference: Mean Field Approximation
10-708: Probabilistic Graphical Models 10-708, Spring 2017 Lecture 13 : Variational Inference: Mean Field Approximation Lecturer: Willie Neiswanger Scribes: Xupeng Tong, Minxing Liu 1 Problem Setup 1.1