Web Search and Text Mining. Lecture 16: Topics and Communities

Size: px
Start display at page:

Download "Web Search and Text Mining. Lecture 16: Topics and Communities"

Transcription

1 Web Search and Tet Mining Lecture 16: Topics and Communities

2 Outline Latent Dirichlet Allocation (LDA) Graphical models for social netorks

3 Eploration, discovery, and query-ansering in the contet of the relationships of authors/actors, topics, communities, roles for large document collections and the associated social netorks. 1) analysis of topic trend 2) find the authors ho are most likely to rite on a topic 3) find the unusual paper ritten by an author 4) discover organizational structure via communication patterns 5) roles/communities membership of social actors

4 Topics Models Discovering the topics in a tet corpus. Latent semantic indeing (discovered dimension hard to interpret, not natural at all from a generative model viepoint) Probabilistic latent semantic indeing (not a complete generative model) Latent Dirichlet allocation: topics are multinomial distributions over ords (Blei et. al., 2003)

5 Representation of Topics WORD PROB. WORD PROB. WORD PROB. WORD PROB. LIGHT.0306 RECOGNITION.0500 KERNEL.0547 SOURCE.0389 RESPONSE.0282 CHARACTER.0334 VECTOR.0293 INDEPENDENT.0376 INTENSITY.0252 TANGENT.0246 SUPPORT.0293 SOURCES.0344 RETINA.0241 CHARACTERS.0232 MARGIN.0239 SEPARATION.0322 OPTICAL.0233 DISTANCE.0197 SVM.0196 INFORMATION.0319 KOCH.0190 HANDWRITTEN.0166 DATA.0165 ICA.0276 Each topic is a represented as a multinomial distribution over the set of ords in a vocabulary. φ t = [φ 1t,..., φ V t ] T, V i=1 φ it = 1 Conceptually, each document is a miture of topics.

6 Some Models Unigra mmodel The ords of every document are dran independently from a single multinomial distribution, Graphical representation, p() = N BLEI, NG, AND JORDAN i=1 p( i ) N (a) unigram M

7 Miture of unigrams Each document is generated by 1) choosing a topic z; 2) generating N ords independently from the conditional multinomial p( z) p() = z p(z) N i=1 p( i z) A document belongs to a single topic.

8 Graphical representation, N (a) unigram M z N (b) miture of unigrams M

9 plsi A document label d and a ord i are conditionally independent given an unobserved topic z, p(d, i ) = z p(z)p(d, i z) = z p(z)p(d z)p( i z) = = p(d) z p(z d)p( i z) A document can have multiple topics ith p(z d) as the miture eights.

10 ment the unigram model ith a discrete random topic variable z (Figure 3b), e z N M Graphical representation, (b) miture of unigrams d z N M But drabacks: (c) plsi/aspect model Figure 3: Graphical model representation of different models of discrete data. 1) p(z d) for documents in training set, no generalization 2) number parameters linear in number of documents kv + km, prune to overfitting ure of unigrams

11 LDA Key difference from plsi: The topic miture eights modeled by a k-parameter hidden random variable rather than a large set of individual parameters hich are eplicitly linked to the training set (p(z d), d M). The hidden variable θ Dirichlet(α)

12 Topic (LDA) Author α θ a d α z β φ β φ β T N d D A N d D (a) (b) The corpus is generated by Figure 10: Different generative models for docume 1) a distribution over topics 6.1 A is sampled Simple Topic from a (LDA) Dirichlet Model dist. θ Dirichlet(α) 2) for each ord in theas document, mentioned earliera single the topic paper, is chosen there have according been a number to this of other earli distribution. z M ultinomial(θ) document content are based on the idea that the probability distribution can be epressed as a miture of topics, here each topic is a probabili 3) each ord is sampled [Blei from et al., a 2003, multinomial Hofmann, distribution 1999, Uedaover and Saito, ords2003, specific Iyer and Osten focus on one such model Latent Dirichlet Allocation [LDA; Blei et to the sampled topic. generation M ultinomial(p( z)) of a corpus is a three step process. First, for each document, is sampled from a Dirichlet distribution. Second, for each ord in the d chosen according to this distribution. Finally, each ord is sampled from over ords specific to the sampled topic. The parameters of this model are similar to those of the author top distribution over ords for each topic, and Θ represents a distribution over

13 Given the parameters α and β, the joint distribution of a topic miture θ, a set of N topics z, and a set of N ords is given by Sum over θ, z, p(θ, z, α, β) = p(θ α) p( α, β) = p(θ α) N i=1 N i=1 p(z n θ)p( n z n, β) p(z n θ)p( n z n, β) dθ z n

14 The plsi model posits that each ord of a training document comes from a randomly chosen topic. The topics are themselves dran from a document-specific distribution over topics, i.e., a point on the topic simple. There is one such distribution for each document; the set of BLEI, NG, AND JORDAN Geometric Picture topic 1 topic simple topic 2 ord simple topic 3 Figure 4: The topic simple for three topics embedded in the ord simple for three ords. The corners of the ord simple correspond to the three distributions here each ord (respectively) has probability one. The three points of the topic simple correspond to three different distributions over ords. The miture of unigrams places each document at one of the corners of the topic simple. The plsi model induces an empirical distribution on the topic simple denoted by. LDA places a smooth distribution on the topic simple denoted by the contour lines. Word simple: (V 1)-D simple.

15 1. Unigram: one point in the ord simple, all ords from the same dist The other models using latent topics, form a (K 1)-subsimple using K points in the ord simple. 2. Miture of unigrams: one of the K points is chosen randomly 3. plsi: one point in topic comple for each document in training 4. LDA: both training doc and unseen doc from a continuous dist on topic simple

16 Author Models Documents are ritten by authors, model based on the interests of authors Topic (LDA) epressed as author-ord Author distributions (McCallum, 1999). Author-Topic a d α θ a d α θ A z z β φ β φ β φ T N d D A N d D T N d D (a) (b) (c) Figure 10: Different generative models for documents. 6.1 A Simple Topic (LDA) Model

17 Author Author-Topic Author-Topic Models a d a d α θ A z β φ β φ A N d D T N d D (b) (c) 0: Different generative models for documents. DA) Model Each author is a multinomial distribution over topics. (Rosen-Zvi et. al., 2005) per, there have been a number of other earlier approaches to modeling n the idea that the probability distribution over ords in a document e of topics, here each topic is a probability distribution over ords 999, Ueda and Saito, 2003, Iyer and Ostendorf, 1999]. Here e ill atent Dirichlet Allocation [LDA; Blei et al., 2003]. 3 In LDA, the ree step process. First, for each document, a distribution over topics istribution. Second, for each ord in the document, a single topic is bution. Finally, each ord is sampled from a multinomial distribution pled topic. odel are similar to those of the author topic model: Φ represents a h topic, and Θ represents a distribution over topics for each document. 1) Author chosen uniformly from author list a dist over topics. 2) A topic chosen from that distribution, and a ord sampled.

18 Figure 3: Graphical model for the author topic model. Under this generative process, each topic is dran independently hen conditioned on Θ, and each ord is dran independently hen conditioned on Φ and z. The probability of the corpus, conditioned on Θ and Φ (and implicitly on a fied number of topics T ), is Probabilities of ords given topics: Φ, W T matri. D Probabilities of ords given P ( Θ, topic Φ, A) t: = φ t, WP ( -dimensional d Θ, Φ, a d ). vector. (1) d=1 Probabilities of topics given authors: Θ, T A matri. We can obtain the probability of the ords in each document, d by summing over the latent variables Probabilities and z, toofgive topics given author a: θ a, T -dimensional vector. P ( d Θ, Φ, A) = = = = N d i=1 N d P ( i Θ, Φ, a d ) A i=1 a=1 t=1 N d A i=1 a=1 t=1 N d i=1 1 A d a a d T P ( i, z i = t, i = a Θ, Φ, a d )) T P ( i z i = t, φ t )P (z i = t i = a, θ a )P ( i = a a d ) T φ i tθ ta, (2) t=1 here the factorization in the third line makes use of the independence assumptions of the model. The last line in the equations above epresses the probability of the ords in terms the entries of the parameter matrices Φ and Θ introduced earlier. The probability distribution over author assignments, P ( i = a a d ), is assumed to be uniform over the elements of a d, and deterministic if

19 Social Netork Analysis In a document corpus, authors interact through co-authorship, citations etc. In an corpus, users can be authors and recipients of messages. The interactions among the authors/users give rise to a social netork. We can etract communities from the netork and discover roles played by the social actors. Eploring communication patterns as ell as semantics.

20 ! a d " z $ # T z N Author-Recipient-Topic Models Eplicit modeling D authors as ell as recipient D in s (McCallumAuthor-Topic et. al., 2005). Model Etensions: Author-Recipient-Topic eplicit modeling Model of roles: RART (AT) model. [Rosen-Zvi, Griffiths, Steyvers, Smyth 2004] d $ # A N d (ART) [This paper] a d a d r d! " A z! " A,A z $ # T N d $ # T N d D D Figure 1: Three related models, and the ART model. In all models, each observed ord,, is generated from a multinomial ord distribution, φ z, specific to a particular topic/author, z, hoever topics are selected differently in each of the models.

21 Community-Author-Topic Models Discovering e-communities through topic similarity as ell as communication patterns (Ding et. al., WWW 2006). β φ ure 12: Community similarity comparisons U α γ θ ψ C T α di ω N d D topic z i and the author i responsible for this assigned based on the posterior probability cond all other variables: P (z i, i ω i, z i, i, i, a d i denote the topic and author assigned to ω i, and i are all other assignments of topic and a cluding current α di instance. i represents other ords in the document set and a d is the observ set for this document. A key issue in using Gibbs sampling for di approimation ω is the evaluation of conditional probability. In Author-Topic N d model, given T top ords, P (z i, i ω i, Dz i, i, i, a d ) is estimated UT 2 by definingfigure a community 5: Modeling as no more community than a ith Figuretopics 14: Modeling community ith topics and of users. users P (z i = j, i = k ω i = m, z i, i, i, a d also test the similarity among topics(users) for the sider the conditional probability P (c, u, z ω), a ord ω associates three variables: community, user and topic. Our C W T AT P (ω i = m i = k)p ( i = k z i = j topics) hich are discovered as a community by CUT 1 2). Typically the topics(users) associated ith the changes hen both factors are simultaneously considered. mj + β Ckj + α topics) in a community interpretation represent of the high semantic similarities. meaning One of P ould (c, u, z ω) epect isne communities to emerge. The Σ m model C W T m the probability that ord ω is generated by user u under j + V β Σ j C AT kj + ample, in Fig. 8, Topic 5 and Topic 12 that conike.grigsby are in Fig. 14 constrains the community as a joint distribution topic both z, in contained community in the c. topic set of over topic and users. Hoever, here m such nonlinear m and j generative j, α and β are prior p onoho, ho is Unfortunately, the community this companion conditional of Mike probability models cannot require be computed directly. To get P (c, u, z ω),e have: more efficient yet approimate of times solutions. that ord It ould ω i = also mbe is assigned to top larger computational for ord andresources, topic Dirichlets, asking for Cmj W T represents th y. C AT represents the number of times that author γ α β ψ θ φ C C T,U

Topic Modelling and Latent Dirichlet Allocation

Topic Modelling and Latent Dirichlet Allocation Topic Modelling and Latent Dirichlet Allocation Stephen Clark (with thanks to Mark Gales for some of the slides) Lent 2013 Machine Learning for Language Processing: Lecture 7 MPhil in Advanced Computer

More information

Document and Topic Models: plsa and LDA

Document and Topic Models: plsa and LDA Document and Topic Models: plsa and LDA Andrew Levandoski and Jonathan Lobo CS 3750 Advanced Topics in Machine Learning 2 October 2018 Outline Topic Models plsa LSA Model Fitting via EM phits: link analysis

More information

Latent Dirichlet Allocation

Latent Dirichlet Allocation Journal of Machine Learning Research 3 (2003) 993-1022 Submitted 2/02; Published 1/03 Latent Dirichlet Allocation David M. Blei Computer Science Division University of California Bereley, CA 94720, USA

More information

Information retrieval LSI, plsi and LDA. Jian-Yun Nie

Information retrieval LSI, plsi and LDA. Jian-Yun Nie Information retrieval LSI, plsi and LDA Jian-Yun Nie Basics: Eigenvector, Eigenvalue Ref: http://en.wikipedia.org/wiki/eigenvector For a square matrix A: Ax = λx where x is a vector (eigenvector), and

More information

Applying Latent Dirichlet Allocation to Group Discovery in Large Graphs

Applying Latent Dirichlet Allocation to Group Discovery in Large Graphs Lawrence Livermore National Laboratory Applying Latent Dirichlet Allocation to Group Discovery in Large Graphs Keith Henderson and Tina Eliassi-Rad keith@llnl.gov and eliassi@llnl.gov This work was performed

More information

Markov Topic Models. Bo Thiesson, Christopher Meek Microsoft Research One Microsoft Way Redmond, WA 98052

Markov Topic Models. Bo Thiesson, Christopher Meek Microsoft Research One Microsoft Way Redmond, WA 98052 Chong Wang Computer Science Dept. Princeton University Princeton, NJ 08540 Bo Thiesson, Christopher Meek Microsoft Research One Microsoft Way Redmond, WA 9805 David Blei Computer Science Dept. Princeton

More information

Latent Dirichlet Allocation

Latent Dirichlet Allocation Journal of Machine Learning Research 3 (2003) 993-1022 Submitted 2/02; Published 1/03 Latent Dirichlet Allocation David M. Blei Computer Science Division University of California Bereley, CA 94720, USA

More information

Hybrid Models for Text and Graphs. 10/23/2012 Analysis of Social Media

Hybrid Models for Text and Graphs. 10/23/2012 Analysis of Social Media Hybrid Models for Text and Graphs 10/23/2012 Analysis of Social Media Newswire Text Formal Primary purpose: Inform typical reader about recent events Broad audience: Explicitly establish shared context

More information

Latent Dirichlet Allocation

Latent Dirichlet Allocation Outlines Advanced Artificial Intelligence October 1, 2009 Outlines Part I: Theoretical Background Part II: Application and Results 1 Motive Previous Research Exchangeability 2 Notation and Terminology

More information

Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) Latent Dirichlet Allocation (LDA) D. Blei, A. Ng, and M. Jordan. Journal of Machine Learning Research, 3:993-1022, January 2003. Following slides borrowed ant then heavily modified from: Jonathan Huang

More information

CS Lecture 18. Topic Models and LDA

CS Lecture 18. Topic Models and LDA CS 6347 Lecture 18 Topic Models and LDA (some slides by David Blei) Generative vs. Discriminative Models Recall that, in Bayesian networks, there could be many different, but equivalent models of the same

More information

Topic Models and Applications to Short Documents

Topic Models and Applications to Short Documents Topic Models and Applications to Short Documents Dieu-Thu Le Email: dieuthu.le@unitn.it Trento University April 6, 2011 1 / 43 Outline Introduction Latent Dirichlet Allocation Gibbs Sampling Short Text

More information

Text Mining for Economics and Finance Latent Dirichlet Allocation

Text Mining for Economics and Finance Latent Dirichlet Allocation Text Mining for Economics and Finance Latent Dirichlet Allocation Stephen Hansen Text Mining Lecture 5 1 / 45 Introduction Recall we are interested in mixed-membership modeling, but that the plsi model

More information

Study Notes on the Latent Dirichlet Allocation

Study Notes on the Latent Dirichlet Allocation Study Notes on the Latent Dirichlet Allocation Xugang Ye 1. Model Framework A word is an element of dictionary {1,,}. A document is represented by a sequence of words: =(,, ), {1,,}. A corpus is a collection

More information

CS145: INTRODUCTION TO DATA MINING

CS145: INTRODUCTION TO DATA MINING CS145: INTRODUCTION TO DATA MINING Text Data: Topic Model Instructor: Yizhou Sun yzsun@cs.ucla.edu December 4, 2017 Methods to be Learnt Vector Data Set Data Sequence Data Text Data Classification Clustering

More information

Topic Models. Advanced Machine Learning for NLP Jordan Boyd-Graber OVERVIEW. Advanced Machine Learning for NLP Boyd-Graber Topic Models 1 of 1

Topic Models. Advanced Machine Learning for NLP Jordan Boyd-Graber OVERVIEW. Advanced Machine Learning for NLP Boyd-Graber Topic Models 1 of 1 Topic Models Advanced Machine Learning for NLP Jordan Boyd-Graber OVERVIEW Advanced Machine Learning for NLP Boyd-Graber Topic Models 1 of 1 Low-Dimensional Space for Documents Last time: embedding space

More information

A Continuous-Time Model of Topic Co-occurrence Trends

A Continuous-Time Model of Topic Co-occurrence Trends A Continuous-Time Model of Topic Co-occurrence Trends Wei Li, Xuerui Wang and Andrew McCallum Department of Computer Science University of Massachusetts 140 Governors Drive Amherst, MA 01003-9264 Abstract

More information

topic modeling hanna m. wallach

topic modeling hanna m. wallach university of massachusetts amherst wallach@cs.umass.edu Ramona Blei-Gantz Helen Moss (Dave's Grandma) The Next 30 Minutes Motivations and a brief history: Latent semantic analysis Probabilistic latent

More information

Topic Models. Charles Elkan November 20, 2008

Topic Models. Charles Elkan November 20, 2008 Topic Models Charles Elan elan@cs.ucsd.edu November 20, 2008 Suppose that we have a collection of documents, and we want to find an organization for these, i.e. we want to do unsupervised learning. One

More information

Applying LDA topic model to a corpus of Italian Supreme Court decisions

Applying LDA topic model to a corpus of Italian Supreme Court decisions Applying LDA topic model to a corpus of Italian Supreme Court decisions Paolo Fantini Statistical Service of the Ministry of Justice - Italy CESS Conference - Rome - November 25, 2014 Our goal finding

More information

Dynamic Mixture Models for Multiple Time Series

Dynamic Mixture Models for Multiple Time Series Dynamic Mixture Models for Multiple Time Series Xing Wei Computer Science Department Univeristy of Massachusetts Amherst, MA 01003 xei@cs.umass.edu Jimeng Sun Computer Science Department Carnegie Mellon

More information

Knowledge Discovery and Data Mining 1 (VO) ( )

Knowledge Discovery and Data Mining 1 (VO) ( ) Knowledge Discovery and Data Mining 1 (VO) (707.003) Probabilistic Latent Semantic Analysis Denis Helic KTI, TU Graz Jan 16, 2014 Denis Helic (KTI, TU Graz) KDDM1 Jan 16, 2014 1 / 47 Big picture: KDDM

More information

Generative Clustering, Topic Modeling, & Bayesian Inference

Generative Clustering, Topic Modeling, & Bayesian Inference Generative Clustering, Topic Modeling, & Bayesian Inference INFO-4604, Applied Machine Learning University of Colorado Boulder December 12-14, 2017 Prof. Michael Paul Unsupervised Naïve Bayes Last week

More information

Latent Dirichlet Allocation Introduction/Overview

Latent Dirichlet Allocation Introduction/Overview Latent Dirichlet Allocation Introduction/Overview David Meyer 03.10.2016 David Meyer http://www.1-4-5.net/~dmm/ml/lda_intro.pdf 03.10.2016 Agenda What is Topic Modeling? Parametric vs. Non-Parametric Models

More information

Pachinko Allocation: DAG-Structured Mixture Models of Topic Correlations

Pachinko Allocation: DAG-Structured Mixture Models of Topic Correlations : DAG-Structured Mixture Models of Topic Correlations Wei Li and Andrew McCallum University of Massachusetts, Dept. of Computer Science {weili,mccallum}@cs.umass.edu Abstract Latent Dirichlet allocation

More information

Probabilistic Dyadic Data Analysis with Local and Global Consistency

Probabilistic Dyadic Data Analysis with Local and Global Consistency Deng Cai DENGCAI@CAD.ZJU.EDU.CN Xuanhui Wang XWANG20@CS.UIUC.EDU Xiaofei He XIAOFEIHE@CAD.ZJU.EDU.CN State Key Lab of CAD&CG, College of Computer Science, Zhejiang University, 100 Zijinggang Road, 310058,

More information

Latent Dirichlet Allocation (LDA)

Latent Dirichlet Allocation (LDA) Latent Dirichlet Allocation (LDA) A review of topic modeling and customer interactions application 3/11/2015 1 Agenda Agenda Items 1 What is topic modeling? Intro Text Mining & Pre-Processing Natural Language

More information

Fast Inference and Learning for Modeling Documents with a Deep Boltzmann Machine

Fast Inference and Learning for Modeling Documents with a Deep Boltzmann Machine Fast Inference and Learning for Modeling Documents with a Deep Boltzmann Machine Nitish Srivastava nitish@cs.toronto.edu Ruslan Salahutdinov rsalahu@cs.toronto.edu Geoffrey Hinton hinton@cs.toronto.edu

More information

Note for plsa and LDA-Version 1.1

Note for plsa and LDA-Version 1.1 Note for plsa and LDA-Version 1.1 Wayne Xin Zhao March 2, 2011 1 Disclaimer In this part of PLSA, I refer to [4, 5, 1]. In LDA part, I refer to [3, 2]. Due to the limit of my English ability, in some place,

More information

Probabilistic Graphical Models

Probabilistic Graphical Models Probabilistic Graphical Models Lecture Notes Fall 2009 November, 2009 Byoung-Ta Zhang School of Computer Science and Engineering & Cognitive Science, Brain Science, and Bioinformatics Seoul National University

More information

Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation

Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation Indraneel Muherjee David M. Blei Department of Computer Science Princeton University 3 Olden Street Princeton, NJ

More information

Latent Dirichlet Alloca/on

Latent Dirichlet Alloca/on Latent Dirichlet Alloca/on Blei, Ng and Jordan ( 2002 ) Presented by Deepak Santhanam What is Latent Dirichlet Alloca/on? Genera/ve Model for collec/ons of discrete data Data generated by parameters which

More information

Collaborative User Clustering for Short Text Streams

Collaborative User Clustering for Short Text Streams Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17) Collaborative User Clustering for Short Text Streams Shangsong Liang, Zhaochun Ren, Emine Yilmaz, and Evangelos Kanoulas

More information

Topic Models. Brandon Malone. February 20, Latent Dirichlet Allocation Success Stories Wrap-up

Topic Models. Brandon Malone. February 20, Latent Dirichlet Allocation Success Stories Wrap-up Much of this material is adapted from Blei 2003. Many of the images were taken from the Internet February 20, 2014 Suppose we have a large number of books. Each is about several unknown topics. How can

More information

PROBABILISTIC LATENT SEMANTIC ANALYSIS

PROBABILISTIC LATENT SEMANTIC ANALYSIS PROBABILISTIC LATENT SEMANTIC ANALYSIS Lingjia Deng Revised from slides of Shuguang Wang Outline Review of previous notes PCA/SVD HITS Latent Semantic Analysis Probabilistic Latent Semantic Analysis Applications

More information

Web-Mining Agents Topic Analysis: plsi and LDA. Tanya Braun Ralf Möller Universität zu Lübeck Institut für Informationssysteme

Web-Mining Agents Topic Analysis: plsi and LDA. Tanya Braun Ralf Möller Universität zu Lübeck Institut für Informationssysteme Web-Mining Agents Topic Analysis: plsi and LDA Tanya Braun Ralf Möller Universität zu Lübeck Institut für Informationssysteme Acknowledgments Pilfered from: Ramesh M. Nallapati Machine Learning applied

More information

Lecture 3: Pattern Classification. Pattern classification

Lecture 3: Pattern Classification. Pattern classification EE E68: Speech & Audio Processing & Recognition Lecture 3: Pattern Classification 3 4 5 The problem of classification Linear and nonlinear classifiers Probabilistic classification Gaussians, mitures and

More information

Dirichlet Enhanced Latent Semantic Analysis

Dirichlet Enhanced Latent Semantic Analysis Dirichlet Enhanced Latent Semantic Analysis Kai Yu Siemens Corporate Technology D-81730 Munich, Germany Kai.Yu@siemens.com Shipeng Yu Institute for Computer Science University of Munich D-80538 Munich,

More information

Latent Dirichlet Bayesian Co-Clustering

Latent Dirichlet Bayesian Co-Clustering Latent Dirichlet Bayesian Co-Clustering Pu Wang 1, Carlotta Domeniconi 1, and athryn Blackmond Laskey 1 Department of Computer Science Department of Systems Engineering and Operations Research George Mason

More information

Machine Learning

Machine Learning Machine Learning 10-701 Tom M. Mitchell Machine Learning Department Carnegie Mellon University April 5, 2011 Today: Latent Dirichlet Allocation topic models Social network analysis based on latent probabilistic

More information

Distributed Gibbs Sampling of Latent Topic Models: The Gritty Details THIS IS AN EARLY DRAFT. YOUR FEEDBACKS ARE HIGHLY APPRECIATED.

Distributed Gibbs Sampling of Latent Topic Models: The Gritty Details THIS IS AN EARLY DRAFT. YOUR FEEDBACKS ARE HIGHLY APPRECIATED. Distributed Gibbs Sampling of Latent Topic Models: The Gritty Details THIS IS AN EARLY DRAFT. YOUR FEEDBACKS ARE HIGHLY APPRECIATED. Yi Wang yi.wang.2005@gmail.com August 2008 Contents Preface 2 2 Latent

More information

Probabilistic Latent Semantic Analysis

Probabilistic Latent Semantic Analysis Probabilistic Latent Semantic Analysis Yuriy Sverchkov Intelligent Systems Program University of Pittsburgh October 6, 2011 Outline Latent Semantic Analysis (LSA) A quick review Probabilistic LSA (plsa)

More information

Kernel Density Topic Models: Visual Topics Without Visual Words

Kernel Density Topic Models: Visual Topics Without Visual Words Kernel Density Topic Models: Visual Topics Without Visual Words Konstantinos Rematas K.U. Leuven ESAT-iMinds krematas@esat.kuleuven.be Mario Fritz Max Planck Institute for Informatics mfrtiz@mpi-inf.mpg.de

More information

Learning the Semantic Correlation: An Alternative Way to Gain from Unlabeled Text

Learning the Semantic Correlation: An Alternative Way to Gain from Unlabeled Text Learning the Semantic Correlation: An Alternative Way to Gain from Unlabeled Text Yi Zhang Machine Learning Department Carnegie Mellon University yizhang1@cs.cmu.edu Jeff Schneider The Robotics Institute

More information

Parametric Embedding for Class Visualization

Parametric Embedding for Class Visualization Parametric Embedding for Class Visualization Tomoharu Iwata Kazumi Saito Naonori Ueda NTT Communication Science Laboratories, Japan Sean Stromsten BAE Systems Advanced Information Technologies, USA Thomas

More information

Chapter 8 PROBABILISTIC MODELS FOR TEXT MINING. Yizhou Sun Department of Computer Science University of Illinois at Urbana-Champaign

Chapter 8 PROBABILISTIC MODELS FOR TEXT MINING. Yizhou Sun Department of Computer Science University of Illinois at Urbana-Champaign Chapter 8 PROBABILISTIC MODELS FOR TEXT MINING Yizhou Sun Department of Computer Science University of Illinois at Urbana-Champaign sun22@illinois.edu Hongbo Deng Department of Computer Science University

More information

Statistical Debugging with Latent Topic Models

Statistical Debugging with Latent Topic Models Statistical Debugging with Latent Topic Models David Andrzejewski, Anne Mulhern, Ben Liblit, Xiaojin Zhu Department of Computer Sciences University of Wisconsin Madison European Conference on Machine Learning,

More information

RETRIEVAL MODELS. Dr. Gjergji Kasneci Introduction to Information Retrieval WS

RETRIEVAL MODELS. Dr. Gjergji Kasneci Introduction to Information Retrieval WS RETRIEVAL MODELS Dr. Gjergji Kasneci Introduction to Information Retrieval WS 2012-13 1 Outline Intro Basics of probability and information theory Retrieval models Boolean model Vector space model Probabilistic

More information

Recent Advances in Bayesian Inference Techniques

Recent Advances in Bayesian Inference Techniques Recent Advances in Bayesian Inference Techniques Christopher M. Bishop Microsoft Research, Cambridge, U.K. research.microsoft.com/~cmbishop SIAM Conference on Data Mining, April 2004 Abstract Bayesian

More information

Probabilistic Latent Semantic Analysis

Probabilistic Latent Semantic Analysis Probabilistic Latent Semantic Analysis Dan Oneaţă 1 Introduction Probabilistic Latent Semantic Analysis (plsa) is a technique from the category of topic models. Its main goal is to model cooccurrence information

More information

16 : Approximate Inference: Markov Chain Monte Carlo

16 : Approximate Inference: Markov Chain Monte Carlo 10-708: Probabilistic Graphical Models 10-708, Spring 2017 16 : Approximate Inference: Markov Chain Monte Carlo Lecturer: Eric P. Xing Scribes: Yuan Yang, Chao-Ming Yen 1 Introduction As the target distribution

More information

Parallelized Variational EM for Latent Dirichlet Allocation: An Experimental Evaluation of Speed and Scalability

Parallelized Variational EM for Latent Dirichlet Allocation: An Experimental Evaluation of Speed and Scalability Parallelized Variational EM for Latent Dirichlet Allocation: An Experimental Evaluation of Speed and Scalability Ramesh Nallapati, William Cohen and John Lafferty Machine Learning Department Carnegie Mellon

More information

Understanding Comments Submitted to FCC on Net Neutrality. Kevin (Junhui) Mao, Jing Xia, Dennis (Woncheol) Jeong December 12, 2014

Understanding Comments Submitted to FCC on Net Neutrality. Kevin (Junhui) Mao, Jing Xia, Dennis (Woncheol) Jeong December 12, 2014 Understanding Comments Submitted to FCC on Net Neutrality Kevin (Junhui) Mao, Jing Xia, Dennis (Woncheol) Jeong December 12, 2014 Abstract We aim to understand and summarize themes in the 1.65 million

More information

Topic modeling with more confidence: a theory and some algorithms

Topic modeling with more confidence: a theory and some algorithms Topic modeling with more confidence: a theory and some algorithms Long Nguyen Department of Statistics Department of EECS University of Michigan, Ann Arbor Pacific-Asia Knowledge Discovery and Data Mining,

More information

Topic Discovery Project Report

Topic Discovery Project Report Topic Discovery Project Report Shunyu Yao and Xingjiang Yu IIIS, Tsinghua University {yao-sy15, yu-xj15}@mails.tsinghua.edu.cn Abstract In this report we present our implementations of topic discovery

More information

Mixtures of Multinomials

Mixtures of Multinomials Mixtures of Multinomials Jason D. M. Rennie jrennie@gmail.com September, 25 Abstract We consider two different types of multinomial mixtures, () a wordlevel mixture, and (2) a document-level mixture. We

More information

Gaussian Mixture Model

Gaussian Mixture Model Case Study : Document Retrieval MAP EM, Latent Dirichlet Allocation, Gibbs Sampling Machine Learning/Statistics for Big Data CSE599C/STAT59, University of Washington Emily Fox 0 Emily Fox February 5 th,

More information

Topic Models. Material adapted from David Mimno University of Maryland INTRODUCTION. Material adapted from David Mimno UMD Topic Models 1 / 51

Topic Models. Material adapted from David Mimno University of Maryland INTRODUCTION. Material adapted from David Mimno UMD Topic Models 1 / 51 Topic Models Material adapted from David Mimno University of Maryland INTRODUCTION Material adapted from David Mimno UMD Topic Models 1 / 51 Why topic models? Suppose you have a huge number of documents

More information

Applying hlda to Practical Topic Modeling

Applying hlda to Practical Topic Modeling Joseph Heng lengerfulluse@gmail.com CIST Lab of BUPT March 17, 2013 Outline 1 HLDA Discussion 2 the nested CRP GEM Distribution Dirichlet Distribution Posterior Inference Outline 1 HLDA Discussion 2 the

More information

Latent Dirichlet Allocation: Stability and Applications to Studies of User-Generated Content

Latent Dirichlet Allocation: Stability and Applications to Studies of User-Generated Content Latent Dirichlet Allocation: Stability and Applications to Studies of User-Generated Content Sergei Koltcov ul Soyuza Pechatniov, 27 St Petersburg, Russia soltsov@hseru Olessia Koltsova ul Soyuza Pechatniov,

More information

Topic Modeling for Personalized Recommendation of Volatile Items

Topic Modeling for Personalized Recommendation of Volatile Items Topic Modeling for Personalized Recommendation of Volatile Items Maks Ovsjanikov 1 and Ye Chen 2 1 Stanford University, maks@stanford.edu 2 Microsoft Corporation, yec@microsoft.com Abstract. One of the

More information

Lecture 19, November 19, 2012

Lecture 19, November 19, 2012 Machine Learning 0-70/5-78, Fall 0 Latent Space Analysis SVD and Topic Models Eric Xing Lecture 9, November 9, 0 Reading: Tutorial on Topic Model @ ACL Eric Xing @ CMU, 006-0 We are inundated with data

More information

Modeling Environment

Modeling Environment Topic Model Modeling Environment What does it mean to understand/ your environment? Ability to predict Two approaches to ing environment of words and text Latent Semantic Analysis (LSA) Topic Model LSA

More information

Classification of Text Documents and Extraction of Semantically Related Words using Hierarchical Latent Dirichlet Allocation.

Classification of Text Documents and Extraction of Semantically Related Words using Hierarchical Latent Dirichlet Allocation. Classification of Text Documents and Extraction of Semantically Related Words using Hierarchical Latent Dirichlet Allocation BY Imane Chatri A thesis submitted to the Concordia Institute for Information

More information

Bayesian Semi-supervised Learning with Deep Generative Models

Bayesian Semi-supervised Learning with Deep Generative Models Bayesian Semi-supervised Learning with Deep Generative Models Jonathan Gordon Department of Engineering Cambridge University jg801@cam.ac.uk José Miguel Hernández-Lobato Department of Engineering Cambridge

More information

A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation

A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation THIS IS A DRAFT VERSION. FINAL VERSION TO BE PUBLISHED AT NIPS 06 A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation Yee Whye Teh School of Computing National University

More information

Title Document Clustering. Issue Date Right.

Title Document Clustering. Issue Date Right. NAOSITE: Nagasaki University's Ac Title Author(s) Comparing LDA with plsi as a Dimens Document Clustering Masada, Tomonari; Kiyasu, Senya; Mi Citation Lecture Notes in Computer Science, Issue Date 2008-03

More information

CS6220: DATA MINING TECHNIQUES

CS6220: DATA MINING TECHNIQUES CS6220: DATA MINING TECHNIQUES Matrix Data: Clustering: Part 2 Instructor: Yizhou Sun yzsun@ccs.neu.edu October 19, 2014 Methods to Learn Matrix Data Set Data Sequence Data Time Series Graph & Network

More information

Multilayer Neural Networks

Multilayer Neural Networks Pattern Recognition Lecture 4 Multilayer Neural Netors Prof. Daniel Yeung School of Computer Science and Engineering South China University of Technology Lec4: Multilayer Neural Netors Outline Introduction

More information

Distributed ML for DOSNs: giving power back to users

Distributed ML for DOSNs: giving power back to users Distributed ML for DOSNs: giving power back to users Amira Soliman KTH isocial Marie Curie Initial Training Networks Part1 Agenda DOSNs and Machine Learning DIVa: Decentralized Identity Validation for

More information

Dynamic Topic Models. Abstract. 1. Introduction

Dynamic Topic Models. Abstract. 1. Introduction David M. Blei Computer Science Department, Princeton University, Princeton, NJ 08544, USA John D. Lafferty School of Computer Science, Carnegie Mellon University, Pittsburgh PA 15213, USA BLEI@CS.PRINCETON.EDU

More information

Probabilistic Latent Semantic Analysis

Probabilistic Latent Semantic Analysis Probabilistic Latent Semantic Analysis Seungjin Choi Department of Computer Science and Engineering Pohang University of Science and Technology 77 Cheongam-ro, Nam-gu, Pohang 37673, Korea seungjin@postech.ac.kr

More information

Gaussian Models

Gaussian Models Gaussian Models ddebarr@uw.edu 2016-04-28 Agenda Introduction Gaussian Discriminant Analysis Inference Linear Gaussian Systems The Wishart Distribution Inferring Parameters Introduction Gaussian Density

More information

AN INTRODUCTION TO TOPIC MODELS

AN INTRODUCTION TO TOPIC MODELS AN INTRODUCTION TO TOPIC MODELS Michael Paul December 4, 2013 600.465 Natural Language Processing Johns Hopkins University Prof. Jason Eisner Making sense of text Suppose you want to learn something about

More information

CS 572: Information Retrieval

CS 572: Information Retrieval CS 572: Information Retrieval Lecture 11: Topic Models Acknowledgments: Some slides were adapted from Chris Manning, and from Thomas Hoffman 1 Plan for next few weeks Project 1: done (submit by Friday).

More information

arxiv: v6 [stat.ml] 11 Apr 2017

arxiv: v6 [stat.ml] 11 Apr 2017 Improved Gibbs Sampling Parameter Estimators for LDA Dense Distributions from Sparse Samples: Improved Gibbs Sampling Parameter Estimators for LDA arxiv:1505.02065v6 [stat.ml] 11 Apr 2017 Yannis Papanikolaou

More information

Note 1: Varitional Methods for Latent Dirichlet Allocation

Note 1: Varitional Methods for Latent Dirichlet Allocation Technical Note Series Spring 2013 Note 1: Varitional Methods for Latent Dirichlet Allocation Version 1.0 Wayne Xin Zhao batmanfly@gmail.com Disclaimer: The focus of this note was to reorganie the content

More information

Lecture 13 : Variational Inference: Mean Field Approximation

Lecture 13 : Variational Inference: Mean Field Approximation 10-708: Probabilistic Graphical Models 10-708, Spring 2017 Lecture 13 : Variational Inference: Mean Field Approximation Lecturer: Willie Neiswanger Scribes: Xupeng Tong, Minxing Liu 1 Problem Setup 1.1

More information

Information Retrieval and Web Search

Information Retrieval and Web Search Information Retrieval and Web Search IR models: Vector Space Model IR Models Set Theoretic Classic Models Fuzzy Extended Boolean U s e r T a s k Retrieval: Adhoc Filtering Brosing boolean vector probabilistic

More information

Query-document Relevance Topic Models

Query-document Relevance Topic Models Query-document Relevance Topic Models Meng-Sung Wu, Chia-Ping Chen and Hsin-Min Wang Industrial Technology Research Institute, Hsinchu, Taiwan National Sun Yat-Sen University, Kaohsiung, Taiwan Institute

More information

Evaluation Methods for Topic Models

Evaluation Methods for Topic Models University of Massachusetts Amherst wallach@cs.umass.edu April 13, 2009 Joint work with Iain Murray, Ruslan Salakhutdinov and David Mimno Statistical Topic Models Useful for analyzing large, unstructured

More information

A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation

A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation Yee Whye Teh Gatsby Computational Neuroscience Unit University College London 17 Queen Square, London WC1N 3AR, UK ywteh@gatsby.ucl.ac.uk

More information

Mixture Models in Text Mining Tools in R

Mixture Models in Text Mining Tools in R Text Mining Mixture Models in Text Mining Tools in R Text is the most common vehicle for the formal exchange of information. Learn meaningful information from natural language text. Availability of large

More information

Inference Methods for Latent Dirichlet Allocation

Inference Methods for Latent Dirichlet Allocation Inference Methods for Latent Dirichlet Allocation Chase Geigle University of Illinois at Urbana-Champaign Department of Computer Science geigle1@illinois.edu October 15, 2016 Abstract Latent Dirichlet

More information

A Probabilistic Clustering-Projection Model for Discrete Data

A Probabilistic Clustering-Projection Model for Discrete Data Proc. PKDD, Porto, Portugal, 2005 A Probabilistic Clustering-Projection Model for Discrete Data Shipeng Yu 12, Kai Yu 2, Volker Tresp 2, and Hans-Peter Kriegel 1 1 Institute for Computer Science, University

More information

A Unified Posterior Regularized Topic Model with Maximum Margin for Learning-to-Rank

A Unified Posterior Regularized Topic Model with Maximum Margin for Learning-to-Rank A Unified Posterior Regularized Topic Model with Maximum Margin for Learning-to-Rank Shoaib Jameel Shoaib Jameel 1, Wai Lam 2, Steven Schockaert 1, and Lidong Bing 3 1 School of Computer Science and Informatics,

More information

Probabilistic Topic Models Tutorial: COMAD 2011

Probabilistic Topic Models Tutorial: COMAD 2011 Probabilistic Topic Models Tutorial: COMAD 2011 Indrajit Bhattacharya Assistant Professor Dept of Computer Sc. & Automation Indian Institute Of Science, Bangalore My Background Interests Topic Models Probabilistic

More information

Probablistic Graphical Models, Spring 2007 Homework 4 Due at the beginning of class on 11/26/07

Probablistic Graphical Models, Spring 2007 Homework 4 Due at the beginning of class on 11/26/07 Probablistic Graphical odels, Spring 2007 Homework 4 Due at the beginning of class on 11/26/07 Instructions There are four questions in this homework. The last question involves some programming which

More information

Introduction To Machine Learning

Introduction To Machine Learning Introduction To Machine Learning David Sontag New York University Lecture 21, April 14, 2016 David Sontag (NYU) Introduction To Machine Learning Lecture 21, April 14, 2016 1 / 14 Expectation maximization

More information

Introduction to Bayesian inference

Introduction to Bayesian inference Introduction to Bayesian inference Thomas Alexander Brouwer University of Cambridge tab43@cam.ac.uk 17 November 2015 Probabilistic models Describe how data was generated using probability distributions

More information

State Space and Hidden Markov Models

State Space and Hidden Markov Models State Space and Hidden Markov Models Kunsch H.R. State Space and Hidden Markov Models. ETH- Zurich Zurich; Aliaksandr Hubin Oslo 2014 Contents 1. Introduction 2. Markov Chains 3. Hidden Markov and State

More information

SUPERVISED MULTI-MODAL TOPIC MODEL FOR IMAGE ANNOTATION

SUPERVISED MULTI-MODAL TOPIC MODEL FOR IMAGE ANNOTATION SUPERVISE MULTI-MOAL TOPIC MOEL FOR IMAGE AOTATIO Thu Hoai Tran 2 and Seungjin Choi 12 1 epartment of Computer Science and Engineering POSTECH Korea 2 ivision of IT Convergence Engineering POSTECH Korea

More information

6.867 Machine learning

6.867 Machine learning 6.867 Machine learning Mid-term eam October 8, 6 ( points) Your name and MIT ID: .5.5 y.5 y.5 a).5.5 b).5.5.5.5 y.5 y.5 c).5.5 d).5.5 Figure : Plots of linear regression results with different types of

More information

Modeling User Rating Profiles For Collaborative Filtering

Modeling User Rating Profiles For Collaborative Filtering Modeling User Rating Profiles For Collaborative Filtering Benjamin Marlin Department of Computer Science University of Toronto Toronto, ON, M5S 3H5, CANADA marlin@cs.toronto.edu Abstract In this paper

More information

LDA with Amortized Inference

LDA with Amortized Inference LDA with Amortied Inference Nanbo Sun Abstract This report describes how to frame Latent Dirichlet Allocation LDA as a Variational Auto- Encoder VAE and use the Amortied Variational Inference AVI to optimie

More information

Bayesian Approach 2. CSC412 Probabilistic Learning & Reasoning

Bayesian Approach 2. CSC412 Probabilistic Learning & Reasoning CSC412 Probabilistic Learning & Reasoning Lecture 12: Bayesian Parameter Estimation February 27, 2006 Sam Roweis Bayesian Approach 2 The Bayesian programme (after Rev. Thomas Bayes) treats all unnown quantities

More information

Lecture 8 Learning Sequence Motif Models Using Expectation Maximization (EM) Colin Dewey February 14, 2008

Lecture 8 Learning Sequence Motif Models Using Expectation Maximization (EM) Colin Dewey February 14, 2008 Lecture 8 Learning Sequence Motif Models Using Expectation Maximization (EM) Colin Dewey February 14, 2008 1 Sequence Motifs what is a sequence motif? a sequence pattern of biological significance typically

More information

Latent Topic Models for Hypertext

Latent Topic Models for Hypertext Latent Topic Models for Hypertext Amit Gruber School of CS and Eng. The Hebrew University Jerusalem 9194 Israel amitg@cs.huji.ac.il Michal Rosen-Zvi IBM Research Laboratory in Haifa Haifa University, Mount

More information

Note on Algorithm Differences Between Nonnegative Matrix Factorization And Probabilistic Latent Semantic Indexing

Note on Algorithm Differences Between Nonnegative Matrix Factorization And Probabilistic Latent Semantic Indexing Note on Algorithm Differences Between Nonnegative Matrix Factorization And Probabilistic Latent Semantic Indexing 1 Zhong-Yuan Zhang, 2 Chris Ding, 3 Jie Tang *1, Corresponding Author School of Statistics,

More information

HIERARCHICAL RELATIONAL MODELS FOR DOCUMENT NETWORKS. BY JONATHAN CHANG 1 AND DAVID M. BLEI 2 Facebook and Princeton University

HIERARCHICAL RELATIONAL MODELS FOR DOCUMENT NETWORKS. BY JONATHAN CHANG 1 AND DAVID M. BLEI 2 Facebook and Princeton University The Annals of Applied Statistics 2010, Vol. 4, No. 1, 124 150 DOI: 10.1214/09-AOAS309 Institute of Mathematical Statistics, 2010 HIERARCHICAL RELATIONAL MODELS FOR DOCUMENT NETWORKS BY JONATHAN CHANG 1

More information