Gibbs Sampler Derivation for Latent Dirichlet Allocation (Blei et al., 2003) Lecture Notes Arjun Mukherjee (UH)

Transcription

1 Gibbs Sampler Derivation for Latent Dirichlet Allocation (Blei et al., 2003) Lecture Notes Arjun Mukherjee (UH) I. Generative process, Plates, Notations II. The joint distribution: PP (WW, ZZ; αα; ββ) = PP (WW ZZ)PP (ZZ) = II 1 II 2 II 1 = PP (WW ZZ) = (PP (WW ZZ, Φ)PP (Φ))ddΦ II 2 = PP (ZZ) = PP (ZZ Θ)PP (Θ)ddΘ Let us solve for II 2 first II 2 = PP (ZZ) = PP (ZZ Θ)PP (Θ)ddΘ From definition, Θ = {θθ 1, θθ 2, θθ } < θθ dd dd {1 } > PP (Θ) = PP (θθ dd αα) Now, we know, θθ dd ~ (αα). Recall that θθ dd = {θθ dd,1, θθ dd,2, θθ dd, } < θθ dd,kk tt {1 } > Recall that XX =< xx 1, xx > ~(αα 1, αα ) has the following PDF: ff(xx 1, xx ; αα 1, αα ) = 1 (xx BB(αα) ii )αα ii 1 ii=1, where BB(αα) = Γ(αα ii ) Γ( αα ii ) Also since integrating the PDF over the simplex must equal 1 (from definition), we have

2 Thus, (xx BB(αα) ii )αα ii 1 ii=1 [ 1 ]dddd = 1 or [ ii=1 (xx ii ) αα ii 1 ]dddd = BB(αα) (Identity A) PP (Θ) = PP (θθ dd αα) = ( 1 ((θθ BB(αα) dd,kk )αα 1 ) ) (1) In our case, we assume each hyper-parameter of the K dimensional Dir, αα 1 = = αα = αα, i.e., the vector αα =< αα 1, αα > = < αα, αα > Recall that ZZ = {zz,, zz,=nndd,.. zz dd=,=nn } =< zz dd, dd {1 }, {1 } > PP (ZZ Θ) = ( pp(zz dd, θθ dd )) (2) Now since zz dd, ~ CCCCCC(θθ dd ) We have pp(zz dd, θθ dd ) = (θθ dd,kk ) xx kk where for a given word,, out of {xx, xx kk= } exactly one of xx kk = 1 and rest all are zero, i.e., xx kk = 0. This follows directly from the categorical distribution because we are sampling a single topic for the tth word in document dd. The sampled topic k has the probability of θθ dd,kk. Thus, NN dd pp(zz dd, θθ dd ) = ( (θθ dd,kk ) xx kk) = (θθ dd,kk ) xx kk (3) as yy aa yy bb = yy aa+bb where yy = θθ dd,kk Now, i.e., xx kk = # of words in document dd that were assigned to topic kk. Let us denote this count by CC(dd, kk) oooo CC dd kk, CC(dd, kk) = CC dd kk = xx kk Continuing from (3), we get NN dd pp(zz dd, θθ dd ). = (θθ dd,kk ) xx kk Substituting (4) in (2), we get = (θθ dd,kk ) CC(dd,kk) (4) PP (ZZ Θ) = ( pp(zz dd, θθ dd )) = ( (θθ dd,kk ) CC(dd,kk) ) (5) Using (5) and (1), we get II 2 = PP (ZZ) = PP (ZZ Θ)PP (Θ)ddΘ = [ ( (θθ dd,kk ) CC(dd,kk) )][ ( 1 ((θθ BB(αα) dd,kk )αα 1 ) )]ddθ (6) Since, the document topic distributions, θθ dd are independent of each other, we can group as follows: = 1 [ ( ((θθ BB(αα) dd,kk )αα 1+CC(dd,kk) ))ddθθ dd ] Using identity A, we get (7) ( ((θθ dd,kk ) αα 1+CC(dd,kk) ))ddθθ dd = BB(αα + CC dd ) where CC dd =< CC dd, CC kk=2 dd, CC kk= dd >

3 And BB(αα + CC dd ) = Γ(αα kk +CC dd kk ) Γ( (αα kk +CC kk dd )). Continuing, from (7), we get 1 II 2 = PP (ZZ) = PP (ZZ Θ)PP (Θ)ddΘ = BB(αα + CC BB(αα) dd ) (8) Now Simplifying II 1 II 1 = PP (WW ZZ) = (PP (WW ZZ, Φ)PP (Φ))ddΦ PP (Φ) = PP (φφ kk ββ) Since φφ kk ~ (ββ) PP (Φ) = ( 1 ((φφ BB(ββ) kk,vv )ββ 1 ) ) (9) PP (WW ZZ, Φ) = ( ( pp(ww dd, φφ kk ))) (10) Since ww dd, ~ CCCCCC(φφ kk ) pp(ww dd, φφ kk ) = (φφ kk,vv ) xx dd, vv where for a given word,, in doc dd out of {xx dd,, xx vv= } exactly one of xx dd, vv = 1 and rest all are zero, i.e., xx dd, vv = 0 as we are drawing a word vv from the topic word distribution, φφ kk with probability φφ kk,vv Continuing from (10) and substituting pp(ww dd, φφ kk ) = PP (WW ZZ, Φ) = ( ( (φφ kk,vv ) xx vv dd, = (φφ kk,vv ) dd xx vv dd, Now, xx dd vv CC(kk, vv) = CC kk vv = xx vv dd, dd Thus, we have dd, )) (φφ kk,vv ) xx dd, vv = # of times word vv was assigned to topic kk. Let this count be denoted by CC kk vv dd, PP (WW ZZ, Φ) = (φφ kk,vv ) CC(kk,vv) Using (9) and (11), we get (11) II 1 = PP (WW ZZ) = (PP (WW ZZ, Φ)PP (Φ))ddΦ = ( (φφ kk,vv ) CC(kk,vv) ) ( ( 1 BB(ββ) ((φφ kk,vv )ββ 1 ) )) ddφ Noting that topic distributions, φφ kk are independent and grouping the terms = 1 BB(ββ) ( (φφ kk,vv )ββ+ CC(kk,vv) 1 ddφφ kk ) The integral simplifies to BB(ββ + CC kk ) where CC kk =< CC kk, CC kk vv=2, CC kk vv= >. Thus,

4 II 1 = PP (WW ZZ) = (PP (WW ZZ, Φ)PP (Φ))ddΦ = 1 BB(ββ + CC BB(ββ) kk ) (12) Thus, using (8) and (12), the full joint distribution of the model can be written as PP (WW, ZZ; αα; ββ) = PP (WW ZZ)PP (ZZ) = II 1 II 2 = [ 1 BB(αα + CC BB(αα) dd ) ][ 1 BB(ββ + CC BB(ββ) kk ) ] (13) III. The Posterior on θθ and φφ (A) Computing the posterior distribution for θθ dd having observed topic assignments, zz dd,nn in document dd. Prior: θθ dd αα ~ (αα). Prior probability: PP (θθ dd ) = 1 ((θθ BB(αα) dd,kk )αα 1 ) Likelihood: zz dd, θθ dd ~ CCCCCC(θθ dd ). NN Likelihood of this document given θθ dd : PP (ZZ dd θθ dd ) dd PP (zz dd, θθ dd ) Posterior on θθ dd using Bayes theorem: i.e., Posterior on θθ dd ZZ dd ~ (αα + CC dd ) = (θθ dd,kk ) CC(dd,kk) PP (ZZ PP (θθ dd ZZ dd ) = dd θθ dd )PP (θθ dd ) (θθ (PP (ZZ dd θθ dd )PP (θθ dd ))ddθθ dd,kk ) CC(dd,kk)+αα 1 dd (using (4)) which is nothing but proportional to the hllllll(αα + CC dd ). Here we see conjugate priors in action. Since Dirichlet is the conjugate prior of Categorical distribution, the posterior also takes the form of the prior, i.e., another Dirichlet with added pseudocounts. In fact one can directly use the properties of conjugate priors to arrive at θθ dd ZZ dd ~ (αα + CC dd ). Using the properties of the Dirichlet distribution, one can easily obtain the expected value of the probability mass associated to each topic in the document d as follows: (B) Computing the posterior distribution for φφ kk EE[θθ dd,kk ] = θθ dd,kk = αα+cc(dd,kk) (αα+cc(dd,kk)) (14) Using the fact that the Dirichlet is conjugate to the categorical distribution used for word emission process, we can arrive at φφ kk WW kk ~ (ββ + CC kk ). Thus, EE[φφ kk,vv ] = φφ kk,vv = ββ+cc(kk,vv) (ββ+cc(kk,vv)) (15) IV. Gibbs sampler Let ZZ = {zz dd,nn } = {zz,, zz,=nn1, zz dd=,=nn } be a NN ii dimensional random vector, i.e., ZZ denotes the collection of all latent topic variables, zz dd,nn corresponding to all words in all documents. Also let us posit a Markov chain XX = < ZZ (0), ZZ (1), ZZ (2),. ZZ (NN IIIIIIII ) > over the data and the model, whose stationary distribution converges to the posterior on distribution of ZZ.

5 Also let ZZ = {zz dd,nn } = {zz ii } be denoted by single subscript of ease of notation. For a given token/word, ii = (dd, ), i.e., the word at document dd, the Gibbs conditional (sampling distribution) for its latent topic zz ii can be constructed as follows: PP (zz ii = kk ZZ, WW) = PP (ZZ,WW ) PP (ZZ,WW ) = PP (WW ZZ)PP (ZZ) PP (WW ZZ )PP (ZZ )pp(ww ii ) PP (WW ZZ)PP (ZZ) PP (WW ZZ )PP (ZZ ) (16) where ZZ = {zz ii, ZZ } and WW = {ww ii, WW }. Equation (16) gives us the probability that the latent topic variable zz ii at ii = (dd, ) is assigned to topic kk {1 } having observed all other topic assignments and words except ww ii. Also for subsequent steps the subscript denotes all counts/variables/functional values upon discounting (or not accounting) the token at ww ii Expanding the sampling distribution using (13) we get: PP (zz ii = kk ZZ, WW, ww ii ) [ BB(αα+CC dd ) BB(αα+CC dd ) ] [ BB(ββ+CC kk ) BB(ββ+CC kk ) PP(WW ZZ)PP(ZZ) PP(WW ZZ )PP(ZZ [( BB(αα+CC dd ) )( BB(ββ+CC kk ))] ) [( BB(αα+CC dd ))( BB(ββ+CC kk ) ] (17) Thus, the Gibbs sampling distribution for pp(zz ii = kk) says that it is proportional to the full joint distribution of the model divided by the joint considering the token/word, ww ii and its associated topical assignment did not exist in our data/model. Observing that αα remains fixed, and ii corresponds to the topic and words, {zz ii, ww ii = (dd, )} at some document dd and some position in that document, we can further simplify the first term in (17) as [ BB(αα+CC dd ) BB(αα+CC dd ) ] = [ BB(αα+CC dd=dd ) (BB(αα+CC dd=dd )) ] = BB(αα+CC dd=dd ) BB(αα+[CC dd=dd ] ) (18) As for all documents other than dd the numerator and denominator remain exactly the same and cancel out. Now let us see what changes happen in the count vector CC dd=dd with or without including the term/word at ww ii = (dd, ) whose latent topic assignment is zz ii = kk. We recall that CC dd =< CC dd, CC dd kk=2, CC dd kk= > and CC dd kk = CC(dd, kk) # of words in document dd that were assigned to topic kk. Hence, we can write CC(dd, kk) = { [CC(dd, kk)] ; kk kk [CC(dd, kk)] + 1 ; kk = kk (19) Expanding (18) using the definition of BB(αα) = Γ(αα ii ), we get Γ( αα ii ) BB(αα+CC dd=dd ) BB(αα+[CC dd=dd ] ) = Γ(αα+CC(dd,kk)) Γ(αα+CC(dd,kk) ) Γ( (αα+cc(dd,kk) )) Γ( (αα+cc(dd,kk))) Using the result in (19) this further simplifies to (upon canceling out all non kk terms) = Γ(αα + CC(dd, kk )) Γ(αα + CC(dd, kk ) ) Γ ( (αα + CC(dd, kk) \kk ) + [αα + CC(dd, kk ) ]) Γ ( (αα + CC(dd, kk)) + [αα + CC(dd, kk )]) \kk = Γ(αα + CC(dd, kk ) + 1) Γ(αα + CC(dd, kk ) ) )] Γ ( (αα + CC(dd, kk) ) + [αα + CC(dd, kk ) \kk ]) Γ ( (αα + CC(dd, kk) ) + [αα + CC(dd, kk ) \kk + 1])

6 Using the identity Γ(xx + 1) = xxγ(xx), we get BB(αα + CC dd=dd ) BB(αα + [CC dd=dd ] ) = αα + CC(dd, kk ) (αα + CC(dd, kk) ) In a similar way, one can also simply the second term in (17) as follows: BB(ββ + CC kk) BB(ββ + CC kk ) BB (αα + CC = kk=kk ) (BB (αα + CC kk=kk )) = ββ + CC (kk, vv ) (ββ + CC (kk, vv ) ) Where vv refers to the vocabulary token which is assigned to ww ii. We can thus simplify the full Gibbs conditionals as follows: PP (zz ii = kk αα + CC(dd, kk ) ZZ, WW, ww ii ) [ ββ + CC(kk, vv ) ] [ ] (αα + CC(dd, kk) ) (ββ + CC(kk, vv ) ) The full algorithm for Gibbs sampling is as follows (from Fig 8, [Heinrich, 2008]). There are some minor notation changes which are noted below. MM; CC dd kk nn mm (kk) ; CC kk vv nn kk (tt) ; nn mm = nn mm (kk) kk ; nn mm = nn kk (tt) tt