# Study Notes on the Latent Dirichlet Allocation

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Study Notes on the Latent Dirichlet Allocation Xugang Ye 1. Model Framework A word is an element of dictionary {1,,}. A document is represented by a sequence of words: =(,, ), {1,,}. A corpus is a collection of documents: ={ (),, () }. {,} Dirichlet mixture prior Dirichlet prior Poisson prior ξ Word Word frequency belief (-D) Graphical representation of two-level model Document ={ 1,, } Poisson prior ξ Topic Word Document ={ 1,, } Topic belief (-D) Discrete distribution Dirichlet priors Graphical representation of LDA In two-level generative model, the probability of seeing an -words document is parameterized by hyper parameters {,} of the Dirichlet mixture prior, where contains mixture parameters and contains -dimensional pseudo-count vectors. The probability can be evaluated via conditioning on the word frequency belief as follows (,) = (,,)(,) = ( )(,) = ( ) (,) Γ( = ), (1) Γ( ) Γ( ()) Γ( ) 1

2 which is an explicit formula and where is the -th mixture parameter, is the pseudo-count of the -th keyword of the -th component, =, () is the count of the -th keyword, and Γ stands for gamma function. The two-level model does not explicitly consider the topic issue. Each Dirichlet component may represent a mix of topics. Latent Dirichlet allocation (LDA) aggressively considers the topic issue by incorporating the distribution of the word over topic, that is (, ), where the matrix =[ ] parameterizes this distribution and stands for the probability of seeing the -th keyword under the -th topic. LDA is a three-level generative model in which there is a topic level between the word level and the belief level and in LDA, becomes topic belief. LDA assumes that whenever a key word is observed, there is an associated topic, hence an -words document is associated with a topic sequence of length. By conditioning on and conditional independence, the parameterized probability (,) of seeing can be written as (,) = (,,)(,) = (, )( ). (2) By chain rule and conditional independence, we have (, ) =(,,,,,) =(,,,,,, )(,,,,,) =(, )(,,,,,) = =(,) (,). (3) Plugging (3) into (2) yields (,) = (,) ( ). (4) By conditioning on and conditional independence, we have ( ) =,( ) = ( ) = ( ). (5) Plugging (5) into (4) and exchanging the order of integration and summation yields 2

3 (,) = (,) ( ) Note that = (,) ( ) = (,) ( ) = (,) ( ). (6) (,) = (,) (,) = Plugging (7) into (6) yields the final simplified form (,) = (,) ( ),, = (,). (7) = (,) ( ), (8) which unfortunately does not have explicit formula (the integral cannot be removed because is latent). Note that (,) = (, ), we have (,) = (,) ( ), (9) which can be understood as a form of two level model. But in LDA, there is no direct arc from to, it has to go through the intermediate latent topic level and apply the distribution of word over topic. The derivation of the probability (8) shows that each document can exhibit multiple topics and each word can also associate with multiple topics. This flexibility gives LDA strong power to model a large collection of documents, as compared with some old models like unigram and mixture of unigram. Compared with the plsi, LDA has natural way of assigning probability to a previously unseen document. To estimate, from a corpus ={ (),, () }, one needs to maximize the log likelihood of data: ln (,) = ln (),, (10) 3

4 which is computationally intractable. However, the variational inference method can provide a computationally tractable lower bound. Given a set of estimated parameters {,}, one way to test it is to calculate the perplexity of a test set of documents = { (),, () }. The perplexity has the form perplexity ( ) = exp (),, (11) where is the total number of keywords in -th document. In order to evaluate (11), one has to compute ln (), for each. Let () = (, ). By (8), we have ln (,) =ln()( ) = ln (()). An estimator is ln (() ), where all () are drawn from the Dirichlet distribution Dir(). 2. Variational Inference The key of the variational inference method is to apply the Jensen s inequality to obtain a lower bound of ln (,). Let (, ) be a joint probability density function of,, applying the idea of importance sampling yields ln (,) =ln,,, =ln,,, (, ) (,) (, )ln,,, =(,), (12) (,) where the last step is by Jensen s inequality. Note that (,) = (, )ln,,, (,) Hence, = (, )ln,,,(,) (,) = (, )ln,,, +ln(,) = (, )ln (,) (,),,, + (, )ln(, ) = (, ),,, + ln(, ). 4

5 ln (,) =(,) +(, ),,,, (13) which says ln (,) is the sum of the variational lower bound and the KL-distance between the variational posterior and the true posterior. In variational method, ones wants to find a (,) that is as much close to ln (,) as possible. And this is equivalent to find a (parameterized) (, ) that has as small KL-distance to,,, as possible. Using the fact (factorization by conditional independence) that,,, =,,,, we have =,,, =,,,,, =(, ), (14) (,) = (, )ln (,) (,) = (ln (,)) + ln + ln ln (, ). (15) Now, let (, ) be parameterized: () Graphical representation of variational distribution (,), =, (),, (), =, (),, (), (),, (), = (),, () =,, (),, () =,,, (),, (),, (),, () 5

6 = (),, (),, () = = (), (16) where is Dirichlet parameter and () is multinomial parameter. Applying this variational distribution to (15) yields the expansion of each term as (ln (, )) = (, )ln(, ) = () ln(, ) = () ln(, ) = () ln(, ) = () (ln (,) ) = () ( ln (,) ) = () ln, = () ln, + +ln, = ln, () + + ln, () = () ln, \ () = () ln, = () ln, = () ln,, (17) ln = (, )ln = () ln = () ln = () ln = () ln 6

7 = () ln + +ln = ln () = ln () () = ln () () = ln = () ln. = () ln = () Ψ( ) Ψ. (18) ln = (, )ln = () ln = ln () = ln \ = (ln Γ( ) ln Γ( ) + ( 1)ln ) =lnγ( ) ln Γ( ) + ( 1) ln =lnγ( ) ln Γ( ) + ( 1)Ψ( ) Ψ, (19) ln (, ) = ln () = ln + ln (), ln = ln Γ( ) ln Γ( ) + ( 1)Ψ( ) Ψ, (20) ln () = (, )ln () = () ln () = () ln () = () ln () 7

8 = () ln () () = () ln () = () ln () = () () ln, (21) where Ψ is digamma function. Therefore, (,) =; (),, (), ;, = () ln, + () Ψ( ) Ψ \ + ln Γ( ) ln Γ( ) + ( 1)Ψ( ) Ψ lnγ( )+ ln Γ( ) ( 1)Ψ( ) Ψ () () ln. (22) 3 EM Algorithm The purpose of the EM algorithm is to maximize (10) by maximizing its variational lower bound. Since ln (), () ; (,),, (,), () ;,, we have ln (,) = ln (), () ; (,),, (,), () ;, =(, ;,), (23) which is overall variational lower bound and where = (,) is a tensor and =[ () ] is a matrix. The idea of the EM algorithm is to start from an initial (,), and iteratively improve the estimate via the following alternating E-step and M-step: 8

9 E-step: Given (,), find (, ) to maximize (, ;,), M-step: Given (, ) found from the E-step, find (,) to maximize (, ;,). The two steps are repeated until the value of (, ;,) converges. The update equations can be derived (using Lagrange multipliers) from the overall variational lower bound (, ;,) = () ; (,),, (,), () ;, = (,) ln, () + (,) Ψ () () Ψ + ln Γ( ) ln Γ( ) + ( 1)Ψ () () Ψ () ln Γ( ) ln Γ( () ) + ( () 1)Ψ () () Ψ (,) (,) ln (24) (,) and the constraints = 1, =1,,; =1, =1,,, =1,..,. In E-step, the update equations for and are (,), ()exp Ψ () () Ψ, (25) () = +. (26) (,) And in order for the variational bound converges, it requires alternating between these two equations. Once and are obtained in E-step, the update equation for can be determined by using Lagrange multipliers, it s (,) ( () =), (27) where is indicator function. Finally, there is no analytical form of the update equation for in the M-step, the Newton s method can be employed for obtaining updated by maximizing = ln Γ( ) ln Γ( ) + ( 1)Ψ () () Ψ. (28) Once the EM algorithm converges, it can return two target distributions:, which is word distribution over topic and /( ), which is topic distribution. 9

10 4 Gibbs Sampling Given the data ={ (),, () }, let ={ (),, () } be the corresponding latent variables such that () is associated with (). The central quantity of the Gibbs sampling methods for learning a LDA model is the full conditional posterior distribution ( () \ (),). Note that (,) () \ (), = (). (29) (\,) And most Gibbs sampling methods put a prior on so that (29) is parameterized as (,,) () \ (),,, =. (30) () (\,,) The nominator of right hand side of (30) is (,,) =(,,)(,) =(, )( ). (31) To evaluate (31), one may refer to the conditional independence represented by the following graph: Dirichlet prior Poisson prior Topic ξ Word () () () Topic belief (-D) Discrete distribution Dirichlet priors Document () ={ () () 1,, } Graphical representation of LDA for corpus By the conditional independence for and, we have ( ) = (, )( ) ( ={ (),, () }) = ( )( ) = ( () () ) (() ) () () 10

11 = ( () () ) (() ) () () = (() () ) () = () () (() ) () = () () ( ) = (), (32) () where () is count of topic in in document, and Β is Beta function, (, ) = (,, )(, ) = (, )( ) = ( () (),) ( ) = () (), ( ) = ( ) = (() () ) () () = (() () )() = () () () () = (), (33) () where is count of topic to word in all documents. Plugging (32) and (33) into (31) yields (,,) = () (). (34) () () For the denominator of the right hand side of (30), note that 11

12 \ (),, = (),\ (),\ (), = () \ (),\ (),,\ (),\ (), = () () (),( () ) \ (),\ (), \ (),\ (), = () () () (), (35) where and () respectively correspond to and (),with count associated with () excluded. Let = () and = (), then (30) is reduced to () =\ (), () =,\ (),, () () () () = () () () () () () () () = () () = () () () () () () () () () () () () (). (36) Given this full conditional posterior, the Gibbs sampling procedure is as straightforward as follows: Step 0: Initialize,, Step 1: Iteratively update by drawing () according to (36) Step 2: Update, by maximizing the joint likelihood function (34). Go to step (1) The procedure above is generic. The step 1 usually requires huge number of iterations. The objective function in step 2 is tractable compared with (10) and there is no need for a variational lower bound. 12

13 One can use Newton s method to maximize the log of this likelihood function. Once the burn-out phase is reached, the two target distributions can be estimated as (() )= () + / +, which represents the word distribution over topic and (() )=+ () () / +, which represents the topic distribution in document. () As a special case, is fixed, then the full conditional posterior (36) can be reduced to () =\ (), () =,\ (),, + () 1. (37) In fact, by reducing (34) and (35), we have (,,) =(,,)(,) =(, )( ) () () = \ (),, = (),\ (),\ (),, (38) = () \ (),\ (),,\ (),\ (), = () () (),( () ) \ (),\ (), \ (),\ (), = and dividing (38) by (39) yields (37). () (), (39) Compared with the variational inference method, which is deterministic, the Gibbs sampling is stochastic (in step 1), hence, it s easier for Gibbs sampling s iterations to jump out of local optima. Moreover, it s easier for the Gibbs sampling algorithm to incorporate the uncertainty on. The variational inference method (together with its EM algorithm) introduced in sections 2, 3 assumes that is fixed, whereas it s easy for the Gibbs sampling algorithm to relax this assumption (as the full conditional posterior formula (36) has shown). And as is fixed, the full conditional posterior formula has simpler form. Another advantage is that the Gibbs sampling algorithm relaxes the assumption that documents are independent. In variational inference method, the likelihood of seeing the corpus is the product of the likelihoods of seeing individual documents, however no such assumption is made in the inference based on the Gibbs sampling method. Given the hyperparameters,, the step 1 of the Gibbs sampling procedure can be applied to a corpus = { (),, () } to calculate the perplexity. The burn-out phases of multiple runs can be averaged to 13

14 obtain the estimates of and ={ (),, () }. Given,, the joint likelihood of seeing is simply expressed as (, ) = (,, )(, ) = (, )( ) () = () (), () () = (), () () = (), () () () () (), () () / (), () () () (),\ () () (), () () = () () (), () \ () = (), () () =, () () \ () = (), () () () / (), () () () / (), () () () = (), (40) () where () is number of -th keyword in document. This joint likelihood of corpus leads to a new perplexity formula as perplexity () = exp () Particularly, () () (). (41) =( (), () ). Although this perplexity formula assumes the independence between documents, it does not deal with the integral over. However, the estimation method is still based on sampling. To obtain the expected perplexity, the multiple runs of drawing () are required for obtaining the expected, corresponding to the target corpus. There is a very elegant toy example for testing any implementation of an LDA learning algorithm. Suppose there are 10 topics and 25 words, and the ground truth topic-to-word distribution (10 25 matrix) is represented by the following figure 14

15 The ground truth of (10 25 matrix) Also suppose =1, which means each topic is expected to have equal chance to appear. After generating 2000 documents with each has length 100, it is expected to recover the ground truth of from the artificially generated data. The following figure shows that it s well recovered by the Gibbs sampling procedure. Gibbs sampling results for learning 15

16 5 Application 5.1 Classification For the classification purpose, suppose an LDA model (), () is trained from a corpus that is labeled, then for an unlabeled document, the posterior probability that it belongs to class is ( ) = ( )() = (), () () ( )() (), () (), (42) where (), () is the -th conditional likelihood that can be evaluated according to (8), and () is the portion of documents that is labeled. Practically, due to the small scale of the joint probability of the keywords sequence of a document, only ln (), () is available for each. Let =ln (), () +ln() and = max, then ( ) = ( ) ( ), (43) which is an usual formula for avoiding numerical overflow. Since the size of the dictionary could be very large, the problem of feature selection arises. An ingenious idea that David Blei proposed is to take all the labeled and unlabelled documents as a single corpus and apply the variational inference method for the purpose of feature selection. Note that the objective function is (, ;,) = () ; (,),, (,), () ;,. Upon the termination of the algorithm, we will have () for each. And () is a document-specific parameter vector of the Dirichlet posterior of the topic belief. Since () has much lower dimension than (), we can use () as the new representation of the document. Compared with the usual feature selection methods like mutual information and statistics, which are basically greedy selection methods that represent each feature with a single word, each feature generated by LDA's variational inference process is represented by a topic that is related to many words. This implies that the LDA's feature vector of dimensions carry more information than selecting words. Without the worry about which key-word is informative for whether assigning a document to a class, the LDA features are theoretically superior. Moreover, this feature selection idea can also be implemented when the Gibbs sampling algorithm is used. That is, the Gibbs sampling algorithm can be applied on all the labeled and unlabelled documents. consequently, for each document, we can estimate the expected topic composition (() ) as () =+ () () / +, where () is counted from () that is generated by the Gibbs sampling. We can use () as the new dimension-reduced representation of the document for the purpose of document classification. Sometimes it's interesting to estimate ( ) where is a topic hidden in unlabelled documents. By conditioning on unlabelled document, we have ( ) = (,)( ) 16

17 = ( )( ) = ( )( ) () () = ( )( ). (44) () Practically there are only limited number of unlabelled documents, say ={ (),, () }. But one estimator of (44) can be built as ( ) = () () () = () ( () ) ( () ) = () () ( ) ( () ) = () (),, (45) ( () ) where () can be computed using (43) and ( () ) can be computed as ( () )= (), () (), () (). One may need special care for computing = in order to ( () ) avoid numerical overflow. Let =ln (), = ln (), and =ln( () ), then = ( ) =exp( ( ) ). Another method for computing (45) is to apply the Gibbs sampling to infer the latent topics sequence () for (). Given (), () can be estimated as the portion of the topic in (). And finally () can be estimated as the portion of the topic in (),..., (). Since () = ( )() =( ), one estimator of () can be (), which yields ( ) = () () () () () = (), (46) which is the weighted average of the ( ) over (),, (). It may need the model averaging in burn-out phase of the Gibbs sampling procedure to obtain averaged estimate of (). 5.2 Similarity The similarity between two documents () and () can be measured by the distance between (() ) and (() ), each of which is the expected topic composition. Given the LDA hyper- 17

18 parameters {,}, they are evaluated as () =+ () () / + and () = + () () / +, where () and () are counted from () and () that are generated from Gibbs sampling. One can calculate the similarity measure of () and (). If the variational inference method is used, then one can calculate the similarity measure of () / () and () () /. 5.3 Supervised LDA Dirichlet prior, response value Topic belief (-D) Poisson prior ξ Topic Discrete distribution Word Dirichlet priors Document ={ 1,, } Graphical representation of slda,,,, =,,,,,,,, =,, (, )( ) =,, (, ) =,, (, ) =,, (,). (47),, ~,, (48) where is the counts vector of.,,,, =,,,,. (49) (,) 18

19 It's not quite clear how this method (by David Blei and Jon McAliffe, 2007) is superior to the method of first finding the LDA feature representations and then do regression. Also, compared with (8), the summation Σ in (47) cannot be simplified. However, the experimental results reported in the 2007 paper shows better results than the regression using LDA features and Lasso. (to be continued ) 19

### Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003. Following slides borrowed ant then heavily modified from: Jonathan Huang

### Text Mining for Economics and Finance Latent Dirichlet Allocation

Text Mining for Economics and Finance Latent Dirichlet Allocation Stephen Hansen Text Mining Lecture 5 1 / 45 Introduction Recall we are interested in mixed-membership modeling, but that the plsi model

### 13: Variational inference II

10-708: Probabilistic Graphical Models, Spring 2015 13: Variational inference II Lecturer: Eric P. Xing Scribes: Ronghuo Zheng, Zhiting Hu, Yuntian Deng 1 Introduction We started to talk about variational

### Another Walkthrough of Variational Bayes. Bevan Jones Machine Learning Reading Group Macquarie University

Another Walkthrough of Variational Bayes Bevan Jones Machine Learning Reading Group Macquarie University 2 Variational Bayes? Bayes Bayes Theorem But the integral is intractable! Sampling Gibbs, Metropolis

### Latent Dirichlet Allocation Introduction/Overview

Latent Dirichlet Allocation Introduction/Overview David Meyer 03.10.2016 David Meyer http://www.1-4-5.net/~dmm/ml/lda_intro.pdf 03.10.2016 Agenda What is Topic Modeling? Parametric vs. Non-Parametric Models

### Lecture 13 : Variational Inference: Mean Field Approximation

10-708: Probabilistic Graphical Models 10-708, Spring 2017 Lecture 13 : Variational Inference: Mean Field Approximation Lecturer: Willie Neiswanger Scribes: Xupeng Tong, Minxing Liu 1 Problem Setup 1.1

### Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) A review of topic modeling and customer interactions application 3/11/2015 1 Agenda Agenda Items 1 What is topic modeling? Intro Text Mining & Pre-Processing Natural Language

### Dirichlet Enhanced Latent Semantic Analysis

Dirichlet Enhanced Latent Semantic Analysis Kai Yu Siemens Corporate Technology D-81730 Munich, Germany Kai.Yu@siemens.com Shipeng Yu Institute for Computer Science University of Munich D-80538 Munich,

### Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures

17th Europ. Conf. on Machine Learning, Berlin, Germany, 2006. Variational Bayesian Dirichlet-Multinomial Allocation for Exponential Family Mixtures Shipeng Yu 1,2, Kai Yu 2, Volker Tresp 2, and Hans-Peter

### Recent Advances in Bayesian Inference Techniques

Recent Advances in Bayesian Inference Techniques Christopher M. Bishop Microsoft Research, Cambridge, U.K. research.microsoft.com/~cmbishop SIAM Conference on Data Mining, April 2004 Abstract Bayesian

### Document and Topic Models: plsa and LDA

Document and Topic Models: plsa and LDA Andrew Levandoski and Jonathan Lobo CS 3750 Advanced Topics in Machine Learning 2 October 2018 Outline Topic Models plsa LSA Model Fitting via EM phits: link analysis

### Evaluation Methods for Topic Models

University of Massachusetts Amherst wallach@cs.umass.edu April 13, 2009 Joint work with Iain Murray, Ruslan Salakhutdinov and David Mimno Statistical Topic Models Useful for analyzing large, unstructured

### CS Lecture 18. Topic Models and LDA

CS 6347 Lecture 18 Topic Models and LDA (some slides by David Blei) Generative vs. Discriminative Models Recall that, in Bayesian networks, there could be many different, but equivalent models of the same

### Latent Dirichlet Alloca/on

Latent Dirichlet Alloca/on Blei, Ng and Jordan ( 2002 ) Presented by Deepak Santhanam What is Latent Dirichlet Alloca/on? Genera/ve Model for collec/ons of discrete data Data generated by parameters which

### Note 1: Varitional Methods for Latent Dirichlet Allocation

Technical Note Series Spring 2013 Note 1: Varitional Methods for Latent Dirichlet Allocation Version 1.0 Wayne Xin Zhao batmanfly@gmail.com Disclaimer: The focus of this note was to reorganie the content

### LDA with Amortized Inference

LDA with Amortied Inference Nanbo Sun Abstract This report describes how to frame Latent Dirichlet Allocation LDA as a Variational Auto- Encoder VAE and use the Amortied Variational Inference AVI to optimie

### A graph contains a set of nodes (vertices) connected by links (edges or arcs)

BOLTZMANN MACHINES Generative Models Graphical Models A graph contains a set of nodes (vertices) connected by links (edges or arcs) In a probabilistic graphical model, each node represents a random variable,

### Topic Models. Brandon Malone. February 20, Latent Dirichlet Allocation Success Stories Wrap-up

Much of this material is adapted from Blei 2003. Many of the images were taken from the Internet February 20, 2014 Suppose we have a large number of books. Each is about several unknown topics. How can

### Topic Modelling and Latent Dirichlet Allocation

Topic Modelling and Latent Dirichlet Allocation Stephen Clark (with thanks to Mark Gales for some of the slides) Lent 2013 Machine Learning for Language Processing: Lecture 7 MPhil in Advanced Computer

### Bayesian Methods for Machine Learning

Bayesian Methods for Machine Learning CS 584: Big Data Analytics Material adapted from Radford Neal s tutorial (http://ftp.cs.utoronto.ca/pub/radford/bayes-tut.pdf), Zoubin Ghahramni (http://hunch.net/~coms-4771/zoubin_ghahramani_bayesian_learning.pdf),

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 7 Approximate

### Replicated Softmax: an Undirected Topic Model. Stephen Turner

Replicated Softmax: an Undirected Topic Model Stephen Turner 1. Introduction 2. Replicated Softmax: A Generative Model of Word Counts 3. Evaluating Replicated Softmax as a Generative Model 4. Experimental

### Topic Models. Charles Elkan November 20, 2008

Topic Models Charles Elan elan@cs.ucsd.edu November 20, 2008 Suppose that we have a collection of documents, and we want to find an organization for these, i.e. we want to do unsupervised learning. One

### Fast Inference and Learning for Modeling Documents with a Deep Boltzmann Machine

Fast Inference and Learning for Modeling Documents with a Deep Boltzmann Machine Nitish Srivastava nitish@cs.toronto.edu Ruslan Salahutdinov rsalahu@cs.toronto.edu Geoffrey Hinton hinton@cs.toronto.edu

### Infering the Number of State Clusters in Hidden Markov Model and its Extension

Infering the Number of State Clusters in Hidden Markov Model and its Extension Xugang Ye Department of Applied Mathematics and Statistics, Johns Hopkins University Elements of a Hidden Markov Model (HMM)

### Information retrieval LSI, plsi and LDA. Jian-Yun Nie

Information retrieval LSI, plsi and LDA Jian-Yun Nie Basics: Eigenvector, Eigenvalue Ref: http://en.wikipedia.org/wiki/eigenvector For a square matrix A: Ax = λx where x is a vector (eigenvector), and

### Gibbs Sampling. Héctor Corrada Bravo. University of Maryland, College Park, USA CMSC 644:

Gibbs Sampling Héctor Corrada Bravo University of Maryland, College Park, USA CMSC 644: 2019 03 27 Latent semantic analysis Documents as mixtures of topics (Hoffman 1999) 1 / 60 Latent semantic analysis

### Introduction to Machine Learning

Introduction to Machine Learning Brown University CSCI 1950-F, Spring 2012 Prof. Erik Sudderth Lecture 20: Expectation Maximization Algorithm EM for Mixture Models Many figures courtesy Kevin Murphy s

### Time-Sensitive Dirichlet Process Mixture Models

Time-Sensitive Dirichlet Process Mixture Models Xiaojin Zhu Zoubin Ghahramani John Lafferty May 25 CMU-CALD-5-4 School of Computer Science Carnegie Mellon University Pittsburgh, PA 523 Abstract We introduce

### Bayesian Learning. HT2015: SC4 Statistical Data Mining and Machine Learning. Maximum Likelihood Principle. The Bayesian Learning Framework

HT5: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford http://www.stats.ox.ac.uk/~sejdinov/sdmml.html Maximum Likelihood Principle A generative model for

### Collapsed Variational Bayesian Inference for Hidden Markov Models

Collapsed Variational Bayesian Inference for Hidden Markov Models Pengyu Wang, Phil Blunsom Department of Computer Science, University of Oxford International Conference on Artificial Intelligence and

### Lecture 8: Graphical models for Text

Lecture 8: Graphical models for Text 4F13: Machine Learning Joaquin Quiñonero-Candela and Carl Edward Rasmussen Department of Engineering University of Cambridge http://mlg.eng.cam.ac.uk/teaching/4f13/

### CS839: Probabilistic Graphical Models. Lecture 7: Learning Fully Observed BNs. Theo Rekatsinas

CS839: Probabilistic Graphical Models Lecture 7: Learning Fully Observed BNs Theo Rekatsinas 1 Exponential family: a basic building block For a numeric random variable X p(x ) =h(x)exp T T (x) A( ) = 1

### Note for plsa and LDA-Version 1.1

Note for plsa and LDA-Version 1.1 Wayne Xin Zhao March 2, 2011 1 Disclaimer In this part of PLSA, I refer to [4, 5, 1]. In LDA part, I refer to [3, 2]. Due to the limit of my English ability, in some place,

### Latent Dirichlet Bayesian Co-Clustering

Latent Dirichlet Bayesian Co-Clustering Pu Wang 1, Carlotta Domeniconi 1, and athryn Blackmond Laskey 1 Department of Computer Science Department of Systems Engineering and Operations Research George Mason

### Variational Inference (11/04/13)

STA561: Probabilistic machine learning Variational Inference (11/04/13) Lecturer: Barbara Engelhardt Scribes: Matt Dickenson, Alireza Samany, Tracy Schifeling 1 Introduction In this lecture we will further

### VIBES: A Variational Inference Engine for Bayesian Networks

VIBES: A Variational Inference Engine for Bayesian Networks Christopher M. Bishop Microsoft Research Cambridge, CB3 0FB, U.K. research.microsoft.com/ cmbishop David Spiegelhalter MRC Biostatistics Unit

### Posterior Regularization

Posterior Regularization 1 Introduction One of the key challenges in probabilistic structured learning, is the intractability of the posterior distribution, for fast inference. There are numerous methods

### Probabilistic Time Series Classification

Probabilistic Time Series Classification Y. Cem Sübakan Boğaziçi University 25.06.2013 Y. Cem Sübakan (Boğaziçi University) M.Sc. Thesis Defense 25.06.2013 1 / 54 Problem Statement The goal is to assign

### Gentle Introduction to Infinite Gaussian Mixture Modeling

Gentle Introduction to Infinite Gaussian Mixture Modeling with an application in neuroscience By Frank Wood Rasmussen, NIPS 1999 Neuroscience Application: Spike Sorting Important in neuroscience and for

### Lecture 16 Deep Neural Generative Models

Lecture 16 Deep Neural Generative Models CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago May 22, 2017 Approach so far: We have considered simple models and then constructed

### Gaussian Mixture Model

Case Study : Document Retrieval MAP EM, Latent Dirichlet Allocation, Gibbs Sampling Machine Learning/Statistics for Big Data CSE599C/STAT59, University of Washington Emily Fox 0 Emily Fox February 5 th,

### Lecture 21: Spectral Learning for Graphical Models

10-708: Probabilistic Graphical Models 10-708, Spring 2016 Lecture 21: Spectral Learning for Graphical Models Lecturer: Eric P. Xing Scribes: Maruan Al-Shedivat, Wei-Cheng Chang, Frederick Liu 1 Motivation

### Two Useful Bounds for Variational Inference

Two Useful Bounds for Variational Inference John Paisley Department of Computer Science Princeton University, Princeton, NJ jpaisley@princeton.edu Abstract We review and derive two lower bounds on the

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistical Sciences! rsalakhu@cs.toronto.edu! h0p://www.cs.utoronto.ca/~rsalakhu/ Lecture 7 Approximate

### Mixtures of Gaussians. Sargur Srihari

Mixtures of Gaussians Sargur srihari@cedar.buffalo.edu 1 9. Mixture Models and EM 0. Mixture Models Overview 1. K-Means Clustering 2. Mixtures of Gaussians 3. An Alternative View of EM 4. The EM Algorithm

### CS6220: DATA MINING TECHNIQUES

CS6220: DATA MINING TECHNIQUES Matrix Data: Clustering: Part 2 Instructor: Yizhou Sun yzsun@ccs.neu.edu October 19, 2014 Methods to Learn Matrix Data Set Data Sequence Data Time Series Graph & Network

### Quantitative Biology II Lecture 4: Variational Methods

10 th March 2015 Quantitative Biology II Lecture 4: Variational Methods Gurinder Singh Mickey Atwal Center for Quantitative Biology Cold Spring Harbor Laboratory Image credit: Mike West Summary Approximate

### Large-Scale Feature Learning with Spike-and-Slab Sparse Coding

Large-Scale Feature Learning with Spike-and-Slab Sparse Coding Ian J. Goodfellow, Aaron Courville, Yoshua Bengio ICML 2012 Presented by Xin Yuan January 17, 2013 1 Outline Contributions Spike-and-Slab

### An overview of Bayesian analysis

An overview of Bayesian analysis Benjamin Letham Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA bletham@mit.edu May 2012 1 Introduction and Notation This work provides

### Outline. Binomial, Multinomial, Normal, Beta, Dirichlet. Posterior mean, MAP, credible interval, posterior distribution

Outline A short review on Bayesian analysis. Binomial, Multinomial, Normal, Beta, Dirichlet Posterior mean, MAP, credible interval, posterior distribution Gibbs sampling Revisit the Gaussian mixture model

### Generative Clustering, Topic Modeling, & Bayesian Inference

Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 12-14, 2017 Prof. Michael Paul Unsupervised Naïve Bayes Last week

### Mixture Models and Expectation-Maximization

Mixture Models and Expectation-Maximiation David M. Blei March 9, 2012 EM for mixtures of multinomials The graphical model for a mixture of multinomials π d x dn N D θ k K How should we fit the parameters?

### Latent Variable View of EM. Sargur Srihari

Latent Variable View of EM Sargur srihari@cedar.buffalo.edu 1 Examples of latent variables 1. Mixture Model Joint distribution is p(x,z) We don t have values for z 2. Hidden Markov Model A single time

### Pattern Recognition and Machine Learning

Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability

### Topic Modeling: Beyond Bag-of-Words

University of Cambridge hmw26@cam.ac.uk June 26, 2006 Generative Probabilistic Models of Text Used in text compression, predictive text entry, information retrieval Estimate probability of a word in a

### Time-Sensitive Dirichlet Process Mixture Models

Time-Sensitive Dirichlet Process Mixture Models Xiaojin Zhu Zoubin Ghahramani John Lafferty May 25 CMU-CALD-5-4 School of Computer Science Carnegie Mellon University Pittsburgh, PA 523 Abstract We introduce

### Learning MN Parameters with Approximation. Sargur Srihari

Learning MN Parameters with Approximation Sargur srihari@cedar.buffalo.edu 1 Topics Iterative exact learning of MN parameters Difficulty with exact methods Approximate methods Approximate Inference Belief

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 11 Project

### Latent Dirichlet Allocation

Latent Dirichlet Allocation 1 Directed Graphical Models William W. Cohen Machine Learning 10-601 2 DGMs: The Burglar Alarm example Node ~ random variable Burglar Earthquake Arcs define form of probability

### Lecture 6: Graphical Models: Learning

Lecture 6: Graphical Models: Learning 4F13: Machine Learning Zoubin Ghahramani and Carl Edward Rasmussen Department of Engineering, University of Cambridge February 3rd, 2010 Ghahramani & Rasmussen (CUED)

### Collapsed Variational Inference for Sum-Product Networks

for Sum-Product Networks Han Zhao 1, Tameem Adel 2, Geoff Gordon 1, Brandon Amos 1 Presented by: Han Zhao Carnegie Mellon University 1, University of Amsterdam 2 June. 20th, 2016 1 / 26 Outline Background

### Cheng Soon Ong & Christian Walder. Canberra February June 2018

Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 218 Outlines Overview Introduction Linear Algebra Probability Linear Regression 1

### CSC2535: Computation in Neural Networks Lecture 7: Variational Bayesian Learning & Model Selection

CSC2535: Computation in Neural Networks Lecture 7: Variational Bayesian Learning & Model Selection (non-examinable material) Matthew J. Beal February 27, 2004 www.variational-bayes.org Bayesian Model Selection

### Sequence labeling. Taking collective a set of interrelated instances x 1,, x T and jointly labeling them

HMM, MEMM and CRF 40-957 Special opics in Artificial Intelligence: Probabilistic Graphical Models Sharif University of echnology Soleymani Spring 2014 Sequence labeling aking collective a set of interrelated

### Latent Dirichlet Allocation

Outlines Advanced Artificial Intelligence October 1, 2009 Outlines Part I: Theoretical Background Part II: Application and Results 1 Motive Previous Research Exchangeability 2 Notation and Terminology

### Modeling User Rating Profiles For Collaborative Filtering

Modeling User Rating Profiles For Collaborative Filtering Benjamin Marlin Department of Computer Science University of Toronto Toronto, ON, M5S 3H5, CANADA marlin@cs.toronto.edu Abstract In this paper

### Lab 12: Structured Prediction

December 4, 2014 Lecture plan structured perceptron application: confused messages application: dependency parsing structured SVM Class review: from modelization to classification What does learning mean?

### CS145: INTRODUCTION TO DATA MINING

CS145: INTRODUCTION TO DATA MINING Text Data: Topic Model Instructor: Yizhou Sun yzsun@cs.ucla.edu December 4, 2017 Methods to be Learnt Vector Data Set Data Sequence Data Text Data Classification Clustering

### Basic math for biology

Basic math for biology Lei Li Florida State University, Feb 6, 2002 The EM algorithm: setup Parametric models: {P θ }. Data: full data (Y, X); partial data Y. Missing data: X. Likelihood and maximum likelihood

### Unsupervised Learning

Unsupervised Learning Bayesian Model Comparison Zoubin Ghahramani zoubin@gatsby.ucl.ac.uk Gatsby Computational Neuroscience Unit, and MSc in Intelligent Systems, Dept Computer Science University College

### Variational Scoring of Graphical Model Structures

Variational Scoring of Graphical Model Structures Matthew J. Beal Work with Zoubin Ghahramani & Carl Rasmussen, Toronto. 15th September 2003 Overview Bayesian model selection Approximations using Variational

### ECE 5984: Introduction to Machine Learning

ECE 5984: Introduction to Machine Learning Topics: (Finish) Expectation Maximization Principal Component Analysis (PCA) Readings: Barber 15.1-15.4 Dhruv Batra Virginia Tech Administrativia Poster Presentation:

### Statistical Pattern Recognition

Statistical Pattern Recognition Expectation Maximization (EM) and Mixture Models Hamid R. Rabiee Jafar Muhammadi, Mohammad J. Hosseini Spring 2014 http://ce.sharif.edu/courses/92-93/2/ce725-2 Agenda Expectation-maximization

### Machine Learning Summer School

Machine Learning Summer School Lecture 3: Learning parameters and structure Zoubin Ghahramani zoubin@eng.cam.ac.uk http://learning.eng.cam.ac.uk/zoubin/ Department of Engineering University of Cambridge,

### Bayesian Nonparametrics for Speech and Signal Processing

Bayesian Nonparametrics for Speech and Signal Processing Michael I. Jordan University of California, Berkeley June 28, 2011 Acknowledgments: Emily Fox, Erik Sudderth, Yee Whye Teh, and Romain Thibaux Computer

### Content-based Recommendation

Content-based Recommendation Suthee Chaidaroon June 13, 2016 Contents 1 Introduction 1 1.1 Matrix Factorization......................... 2 2 slda 2 2.1 Model................................. 3 3 flda 3

### Machine Learning Techniques for Computer Vision

Machine Learning Techniques for Computer Vision Part 2: Unsupervised Learning Microsoft Research Cambridge x 3 1 0.5 0.2 0 0.5 0.3 0 0.5 1 ECCV 2004, Prague x 2 x 1 Overview of Part 2 Mixture models EM

### Lecture : Probabilistic Machine Learning

Lecture : Probabilistic Machine Learning Riashat Islam Reasoning and Learning Lab McGill University September 11, 2018 ML : Many Methods with Many Links Modelling Views of Machine Learning Machine Learning

### Introduction to Bayesian inference

Introduction to Bayesian inference Thomas Alexander Brouwer University of Cambridge tab43@cam.ac.uk 17 November 2015 Probabilistic models Describe how data was generated using probability distributions

### 16 : Approximate Inference: Markov Chain Monte Carlo

10-708: Probabilistic Graphical Models 10-708, Spring 2017 16 : Approximate Inference: Markov Chain Monte Carlo Lecturer: Eric P. Xing Scribes: Yuan Yang, Chao-Ming Yen 1 Introduction As the target distribution

### Bayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2016

Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2016 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several

### Bayesian Networks: Construction, Inference, Learning and Causal Interpretation. Volker Tresp Summer 2015

Bayesian Networks: Construction, Inference, Learning and Causal Interpretation Volker Tresp Summer 2015 1 Introduction So far we were mostly concerned with supervised learning: we predicted one or several

### Deep Learning Srihari. Deep Belief Nets. Sargur N. Srihari

Deep Belief Nets Sargur N. Srihari srihari@cedar.buffalo.edu Topics 1. Boltzmann machines 2. Restricted Boltzmann machines 3. Deep Belief Networks 4. Deep Boltzmann machines 5. Boltzmann machines for continuous

### Part 1: Expectation Propagation

Chalmers Machine Learning Summer School Approximate message passing and biomedicine Part 1: Expectation Propagation Tom Heskes Machine Learning Group, Institute for Computing and Information Sciences Radboud

### STA 414/2104: Machine Learning

STA 414/2104: Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistics! rsalakhu@cs.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 9 Sequential Data So far

### Expectation Maximization (EM)

Expectation Maximization (EM) The EM algorithm is used to train models involving latent variables using training data in which the latent variables are not observed (unlabeled data). This is to be contrasted

### Online Bayesian Passive-Agressive Learning

Online Bayesian Passive-Agressive Learning International Conference on Machine Learning, 2014 Tianlin Shi Jun Zhu Tsinghua University, China 21 August 2015 Presented by: Kyle Ulrich Introduction Online

### Variational Message Passing. By John Winn, Christopher M. Bishop Presented by Andy Miller

Variational Message Passing By John Winn, Christopher M. Bishop Presented by Andy Miller Overview Background Variational Inference Conjugate-Exponential Models Variational Message Passing Messages Univariate

### Generative Clustering, Topic Modeling, & Bayesian Inference

Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 11-13, 2018 Prof. Michael Paul Unsupervised Naïve Bayes Last week

### STATS 306B: Unsupervised Learning Spring Lecture 2 April 2

STATS 306B: Unsupervised Learning Spring 2014 Lecture 2 April 2 Lecturer: Lester Mackey Scribe: Junyang Qian, Minzhe Wang 2.1 Recap In the last lecture, we formulated our working definition of unsupervised

### Statistical Debugging with Latent Topic Models

Statistical Debugging with Latent Topic Models David Andrzejewski, Anne Mulhern, Ben Liblit, Xiaojin Zhu Department of Computer Sciences University of Wisconsin Madison European Conference on Machine Learning,

### Variational Principal Components

Variational Principal Components Christopher M. Bishop Microsoft Research 7 J. J. Thomson Avenue, Cambridge, CB3 0FB, U.K. cmbishop@microsoft.com http://research.microsoft.com/ cmbishop In Proceedings

### Benchmarking and Improving Recovery of Number of Topics in Latent Dirichlet Allocation Models

Benchmarking and Improving Recovery of Number of Topics in Latent Dirichlet Allocation Models Jason Hou-Liu January 4, 2018 Abstract Latent Dirichlet Allocation (LDA) is a generative model describing the

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 2: PROBABILITY DISTRIBUTIONS Parametric Distributions Basic building blocks: Need to determine given Representation: or? Recall Curve Fitting Binary Variables

### CS6220 Data Mining Techniques Hidden Markov Models, Exponential Families, and the Forward-backward Algorithm

CS6220 Data Mining Techniques Hidden Markov Models, Exponential Families, and the Forward-backward Algorithm Jan-Willem van de Meent, 19 November 2016 1 Hidden Markov Models A hidden Markov model (HMM)

### STA 4273H: Statistical Machine Learning

STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 9 Project