Complexity of Inference in Topic Models

Transcription

1 Complexiy of Inference in Topic Models David Sonag, Daniel M. Roy Massachuses Insiue of Technology Absrac We consider he compuaional complexiy of finding he MAP assignmen of opics o words in Laen Dirichle Allocaion. We show ha, when he effecive number of opics per documen is small, exac inference aes polynomial ime. In conras, we show ha, when a documen has a large number of opics, finding he MAP assignmen in LDA is NP-hard. Our resuls moivae furher sudy of he srucure in real-world opic models, and raise a number of quesions abou he requiremens for accurae inference during boh learning and es-ime use of opic models. 1 Inroducion Probabilisic models of ex and opics, nown as opic models, are powerful ools for exploring large daa ses and for maing inferences abou he conen of documens. Topic models are frequenly used for deriving low-dimensional represenaions of documens ha are hen used for informaion rerieval, documen summarizaion, and classificaion [Blei & McAuliffe, 2008; Lacose-Julien e al., 2009]. Almos all uses of opic models require inference. For example, unsupervised learning of opic models using Expecaion Maximizaion requires he repeaed compuaion of marginal probabiliies of wha opics are presen in he documens. For applicaions in informaion rerieval and classificaion, each new documen necessiaes inference o deermine wha opics are presen. Alhough here is a wealh of lieraure on approximae inference algorihms for opic models, such Gibbs sampling and variaional inference [Blei e al., 2003; Griffihs & Seyvers, 2004; Muherjee & Blei, 2009; Poreous e al., 2008; Teh e al., 2007], lile is nown abou he complexiy of exac inference. In his paper, we consider he compuaional complexiy of inference in opic models, beginning wih one of he simples and mos popular models, Laen Dirichle Allocaion (LDA) [Blei e al., 2003]. We chose o sudy LDA because we believe ha i capures he essence of wha maes inference easy or hard in opic models. Our hope is ha our resuls will moivae discussion of he following quesions, guiding research of boh new opic models and approximae inference for opic models: 1. Wha is he srucure of real-world LDA inference problems? Migh here be srucure in naural problem insances ha maes hem differen from hard insances (e.g., hose used in our reducions)? 2. How much does having accurae inference affec he resuls of learning? Wih a large raining se or sufficienly long documens, migh here be enough averaging for learning o succeed even wih somewha inaccurae inference? 3. Wha are he requiremens of applicaions ha use es-ime inference? How accurae does es-ime inference need o be? Wha quaniies are needed (e.g., marginals, lielihood, mos liely assignmen)? 1

2 2 MAP inference We will consider he inference problem for a single documen. The LDA model saes ha he documen, represened as a collecion of words w = (w 1, w 2,..., w N ), is generaed as follows: a disribuion over he T opics is sampled from a Dirichle disribuion, θ Dir(α); hen, for i = 1,..., N, we sample a opic z i Mulinomial(θ) and word w i Pr(w z i ). Assume ha hese word disribuions have been previously esimaed, and denoe l i = log Pr(w i z i = ) as he log probabiliy of he ih word being generaed from opic. Afer inegraing ou he opic disribuion vecor, he join disribuion of he opic assignmens is given by Pr(z 1,..., z N ) = Γ( α ) Γ(n + α ) Γ(α ) Γ( α + N) where n is he oal number of words assigned o opic. N Pr(w i z i ), (1) In his paper, we will focus on he inference problem of finding he mos liely assignmen of opics o words, i.e. he maximum a poseriori (MAP) assignmen. Taing he logarihm of Eq. 1 and ignoring consans, finding he MAP assignmen is seen o be equivalen o he following combinaorial opimizaion problem: Φ = max x i {0,1},n subjec o i=1 lg Γ(n + α ) + x i l i (2) i, x i = 1, x i = n, where he indicaor variable x i = I[z i = ] denoes he assignmen of word i o opic. 2.1 Exac maximizaion for small number of opics Suppose a documen only uses τ T opics. Tha is, T could be large, bu we are guaraneed ha he MAP assignmen for a documen uses a mos τ differen opics. In his secion, we show how we can use his nowledge o efficienly find a maximizing assignmen of words o opics. We firs observe ha, if we new he number of words assigned o each opic, finding he MAP assignmen is easy. For i {1,..., T }, le n i be he number of words assigned o opic i in he MAP assignmen. Then, he MAP assignmen x is found by solving he following opimizaion problem: x i l i (3) max x i {0,1} subjec o i, x i = 1, i x i = n, which is equivalen o weighed b-maching in a biparie graph (he words are on one side, he opics on he oher) and can be opimally solved in ime O(bm 3 ), where b = max n = O(N) and m = N + T [Schrijver, 2003]. We call (n 1,..., n T ) a valid pariion when n i 0 and n = N. Using weighed b- maching, we can find a MAP assignmen of words o opics by rying all ( ) T τ = Θ(T τ ) choices of τ opics and all possible valid pariions wih a mos τ non-zeros. for all subses A {1, 2,..., T } such ha A = τ do for all valid pariions n = (n 1, n 2,..., n T ) such ha n = 0 for A do Φ A, n Weighed-B-Maching(A, n, l) + lg Γ(n + α ) end for end for reurn arg max A, n Φ A, n i 2

3 w 1 w 2 w 3 w 4 w w Figure 1: (Lef) A LDA insance derived from a -se pacing insance. (Cener) Plo of F (n ) = lg Γ(n + α) for various values of α. The x-axis varies n, he number of words assigned o opic, and he y-axis shows F (n ). (Righ) Behavior of lg Γ(n + α) as α 0. The funcion is sable everywhere bu a zero, where he reward for sparsiy increases wihou bound. There are a mos N τ valid pariions wih τ non-zero couns. For each of hese, we solve he b-maching problem o find he mos liely assignmen of words o opics ha saisfies he cardinaliy consrains. Thus, he oal running ime is O((NT ) τ N(N + τ) 3 ). This is racable when he number of opics τ appearing in a documen is a consan. 2.2 Inference is NP-hard for large numbers of opics In his secion, we show ha probabilisic inference is NP-hard in he general seing where a documen may have a large number of opics in is MAP assignmen. Le MAX-LDA(α) denoe he decision problem of wheher Φ > V (see Eq. 2) for some V R, where he hyperparameers α = α for all opics. We consider boh α < 1 and α 1 because, as shown in Figure 1, he opimizaion problem is qualiaively differen in hese wo cases. Theorem 1. MAX-LDA(α) is NP-hard for all α > 0. Proof. Our proof is a sraighforward generalizaion of he approach used by Halperin & Karp [200] o show ha he minimum enropy se cover problem is hard o approximae. The proof is done by reducion from -se pacing (-SP), for 3. In -SP, we are given a collecion of -elemen ses over some universe of elemens Σ wih Σ = n. The goal is o find he larges collecion of disjoin ses. There exiss a consan c > 1 such ha i is NP-hard o decide wheher a -SP insance has (i) a soluion wih n/ disjoin ses covering all elemens (called a perfec maching), or (ii) a mos cn/ disjoin ses (called a (cn/)-maching). We now describe how o consruc a LDA inference problem from a -SP insance. This requires specifying he words in he documen, he number of opics, and he word log probabiliies l i. Le each elemen i Σ correspond o a word w i, and le each se correspond o one opic. The documen consiss of all of he words (i.e., Σ). We assign uniform probabiliy o he words in each opic, so ha Pr(w i z i = ) = 1 for i, and 0 oherwise. Figure 1 illusraes he resuling LDA model. The opics are on he op, and he words from he documen are on he boom. An edge is drawn beween a opic (se) and a word (elemen) if he corresponding se conains ha elemen. Wha remains is o show ha we can solve some -SP problem by using his reducion and solving a MAX-LDA(α) problem. For echnical reasons involving α > 1, we require ha is sufficienly large. We will use he following resul, proved in he Appendix. Lemma 2. Le P be a -SP insance for > (1 + α) 2, and le P be he derived MAX- LDA(α) insance. There exiss consans C U and C L < C U such ha, if here is a perfec maching in P, hen Φ C U. If, on he oher hand, here is a mos a (cn/)-maching in P, hen Φ < C L. Le P be a -SP insance for > (3 + α) 2, P be he derived MAX-LDA(α) insance, and C U and C L < C U be as in Lemma 2. Then, by esing Φ < C L and Φ > C U we can decide wheher P is a perfec maching or a bes a (cn/)-maching. Hence -SP reduces o MAX-LDA(α). 3

4 The bold lines in Figure 1 indicae he MAP assignmen, which for his example corresponds o a perfec maching for he original -se pacing insance. More realisic documens would have significanly more words han opics used. Alhough his is no possible while eeping = 3, since he MAP assignmen always has τ N/, we can insead reduce from a -se pacing problem wih 3. Lemma 2 shows ha his is hard as well. 3 Conclusion In his paper, we have shown ha he complexiy of inference in LDA srongly depends on he effecive number of opics per documen. When we can guaranee ha a documen is generaed from a small number of opics (regardless of he number of opics in he model), MAX-LDA can be solved in polynomial ime. On he oher hand, if a documen can use an arbirary number of opics, MAX-LDA is NP-hard. The choice of hyperparameers for he Dirichle does no affec our resuls. I would be ineresing o show analogous resuls for compuing marginals and he pariion funcion. I is sraighforward o exend boh our posiive and negaive resuls o relaed models, such as probabilisic laen semanic analysis (PLSA) [Hofmann, 1999] or correlaed opic models [Blei & Laffery, 2006]. Appendix: Proof of Lemma 2 Proof of Lemma 2. Assume here are T ses each having 3 elemens, and le Φ be he opimal LDA objecive. Define F (n) = log Γ(n + α). Since l i is consan across all opics, he linear erm in Eq. 2 will be a consan K. Firs, noe ha, if here is a perfec maching, Φ n F () + (T n ) F (0) + K. (4) The F (0) erm is he conribuion of unused opics. Oherwise, assume ha he bes pacing has γ cn/ ses, each wih elemens. Then, by he properies of he log-gamma funcion, Φ γf () + n γ 1 F ( 1) + (T n ) F (0) + K, () where we assume, conservaively, ha all of he remaining words are explained by opics assigned ( 1) words. Also, since here was no perfec maching, here were a mos T n unused opics. Using our bound on γ, we have where Φ cn F () + n cn 1 F ( 1) + (T n ) F (0) + K (6) = cn n(1 c) F () + 1 F ( 1) + (T n ) F (0) + K (7) = dn F () + (T n ) F (0) + K, (8) d := c + (1 c)β, for β := F ( 1) F () 1. (9) Noe ha F ()/ as. Along wih he convexiy of F, i follows ha here exiss a 0 such ha β < 1 for all > 0. Noe ha > (3 + α) 2 suffices. This implies ha d < 1, which shows ha here is a non-zero gap beween he possible values of Φ. We have ha 1 1 as. Therefore, β < 1 for some if and only if he slope of F exceeds one a some poin. Noe ha he maximum concenraion objecive, F (n) = n log n, saisfies he condiions on F and, in paricular, we have β < 1 for = 3. 4

5 References Blei, David, & Laffery, John Correlaed Topic Models. Pages of: Weiss, Y., Schölopf, B., & Pla, J. (eds), Advances in Neural Informaion Processing Sysems 18. Cambridge, MA: MIT Press. Blei, David, & McAuliffe, Jon Supervised Topic Models. Pages of: Pla, J.C., Koller, D., Singer, Y., & Roweis, S. (eds), Advances in Neural Informaion Processing Sysems 20. Cambridge, MA: MIT Press. Blei, David M., Ng, Andrew Y., & Jordan, Michael I Laen Dirichle allocaion. J. Mach. Learn. Res., 3, Griffihs, Thomas L., & Seyvers, Mar Finding scienific opics. Proceedings of he Naional Academy of Sciences of he Unied Saes of America, 101(Suppl 1), Halperin, Eran, & Karp, Richard M The minimum-enropy se cover problem. Theor. Compu. Sci., 348(2), Hofmann, Thomas Probabilisic laen semanic indexing. Pages 0 7 of: SIGIR 99: Proceedings of he 22nd annual inernaional ACM SIGIR conference on Research and developmen in informaion rerieval. New Yor, NY, USA: ACM. Lacose-Julien, Simon, Sha, Fei, & Jordan, Michael DiscLDA: Discriminaive Learning for Dimensionaliy Reducion and Classificaion. Pages of: Koller, D., Schuurmans, D., Bengio, Y., & Boou, L. (eds), Advances in Neural Informaion Processing Sysems 21. Muherjee, Indraneel, & Blei, David M Relaive Performance Guaranees for Approximae Inference in Laen Dirichle Allocaion. Pages of: Koller, D., Schuurmans, D., Bengio, Y., & Boou, L. (eds), Advances in Neural Informaion Processing Sysems 21. Poreous, Ian, Newman, David, Ihler, Alexander, Asuncion, Arhur, Smyh, Padhraic, & Welling, Max Fas collapsed gibbs sampling for laen dirichle allocaion. Pages of: KDD 08: Proceeding of he 14h ACM SIGKDD inernaional conference on Knowledge discovery and daa mining. New Yor, NY, USA: ACM. Schrijver, Alexander Combinaorial opimizaion. Polyhedra and efficiency. Vol. A. Algorihms and Combinaorics, vol. 24. Berlin: Springer-Verlag. Pahs, flows, machings, Chapers Teh, Y. W., Newman, D., & Welling, M A Collapsed Variaional Bayesian Inference Algorihm for Laen Dirichle Allocaion. In: Advances in Neural Informaion Processing Sysems, vol. 19.