Latent Spaces and Matrix Factorization

Transcription

1 Compuaional Linguisics Laen Spaces and Marix Facorizaion Dierich Klakow FR 4.7 Allgemeine Linguisik (Compuerlinguisik) Universiä des Saarlandes Summer 0 Goal Goal: rea documen clusering and word clusering on he same fooing (same semanic space) find low dimensional represenaions

2 The word documen marix Clusering Documen clusering words describe each documen by a vecor conaining he frequencies of he Word clusering describe each word by a vecor conaining he frequencies of is occurance in differen documen

3 Join word and documen clusering The word documen marix: Ener frequency (or f-idf) for each word and documen in a square scheme of numbers (marix) Marices 3

4 Marices A marix is an array wih wo indices e.g. in a pyhon program his could be A[i][j] wih i=..n and j=...m Marix When wriing, ofen a subscrip noaion is used a i, j or a square scheme: & a, A =... % an, a... i, j... a a, M... N, M # " Specific example of a x3 marix & A = %' ' # ' 8" The ranspose of a marix The wo indices are swapped e.g. in a pyhon program his could be A[j][i]=A[i][j] for i=..n and j=...m Marix for he marices from he previous slide we have: A & a, =... % am, a... j, i... a a, N... M, N # " Specific example of a x3 marix & A = %' ' # ' 8" Wha is A 4

5 Produc of wo marices The elemens of a produc marix can be calculaed in a pyhon program by for i in range(,n+): for j in range(,m+): Marix for k in range(,k+): C[i][j] = A[i][k]*B[k][j] In mah noaion C = A B wih c i, j = K k = a i, k b k, j Example see black board Uni marix Uni marix: he elemen are he indicaor funcion Example: a i, j = i, j Marix & A = 0 % # 0 " Ofen he uni marix is denoed by a 5

6 Orhogonal marices a marix A is orhogonal if = A A Marix Is he following marix orhogonal: & 0.96 A = % 0.8 ' 0.8# 0.96 " Marix 6

7 Laen Semanic Analysis (LSA) This secion mosly follows Manning and Schüze Chaper 5 Singular Value Decomposiion Decompose A such ha Marix Wih and ~ A = TSD ~ A A minimal T T = D D = A a by d marix S a n by n marix T a by n marix D a d by n marix 7

8 8 Marix Is An SVD of " # % & ' ' ' ' = A An arificial Example of Singular Value Decomposiion " # % & ' = T " # % & ' ' = D ( ) = S Marix Decompose More realisic Example (from Manning and Schüze)

9 More realisic Example (from Manning and Schüze) Marix More realisic Example (from Manning and Schüze) Marix 9

10 Documen-Documen Similariy Rewrie A A = r r r ( d d... d ) Marix d j wih r a vecor wih word frequencies of he j- h documen Similariy of i-h documen wih j-h documen d i j All documen-documen similariies A d A d r r Documen-Documen Similariy Rewrie Marix ~ ~ A A = = ( TSD ) = = DS DS T SD = ( SD ) TSD TSD SD Measure similariy in subspace defined by SD 0

11 More realisic Example (from Manning and Schüze) Resul for SD Marix More realisic Example (from Manning and Schüze) Decompose A such ha Marix

12 An even more realisic example Term Documen Marix Srucure An even more realisic example Documen-Documen Similariy Term Documen Marix Srucure

13 Represenaion for Documens in dimensional Subspace Term-Term Similariy Rewrie Marix ~~ AA = ( TSD )( TSD ) = TSD DS = TS = ST = ( TS)( TS) T Measure similariy in subspace defined by TS 3

14 Task How does your programming language suppor SVD Do some inerne search (~0 minues) Repor your findings Marix Homework See shee Marix 4

15 LSA Performance LSA consisenly improves recall on sandard es collecions (precision/recall generally improved) Variable performance on larger TREC collecions Dimensionaliy of Laen Space a magic number seems o work fine no saisfacory way of assessing value. Compuaional cos high Applicaion (by Landauer e. Al) Rae essay by similariy o exising ones Measure correlaion wih human raing onclusion: drop he righ key-words and you are se 5

16 Probabilisic Laen Semanic Analysis (PLSA) Moivaion Does orhogonally maer? Wouldn a sound saisical foundaion be beer? 6

17 PLSA Likelihood of documen P(doc) = P(erm doc)p(erm doc)... P(erm Inroduce erm-frequency marix X L l= P(erm l doc)= T = P(erm doc) A(erm L,doc) doc) PLSA Inroduce hidden opic P(erm doc)= P(erm opic K k= k )P(opic k doc) Shorhand =erm_ P( doc)= K k= P( k)p(k doc) Relaion o LSA? Likelihood of documen P(doc) = T ( '" & K = k= P( % k)p(k doc) # A(,doc) 7

18 PLSA: raining Training objecive funcion N d= logp(d)= N T K A(,d) log d= = k= P( k)p(k d) which is o be maximised w. r.. parameers P( k) and hen also P(k d), subjec o he consrains ha T = P( k)= and K k= P(k d)=. PLSA: raining Updae erm-opic marix P(,k) " P(,k) P(,k) " T = N P(,k) K d= P(,k) k= A(,d) P(k,d) P(,k)P(k,d) Updae opic-documen marix P(k,d) " P(k,d) P(k,d) " K k= T P(k,d) K = P(k,d) k= A(,d) P(,k) P(,k)P(k,d) 8

19 PLSA P( k) for some opics Comparison LSA and PLSA From Th. Hofmann, 000 9

20 Non-negaive Marix Facorizaion See: NMF: idea Find space ha separaes clusers beer 0

21 NMF: he model Decomposion of a non-negaive marix X in wo marices W and H boh nonnegaive A=WH A: N x M daa marix W: N x R source marix H: R x M mixure marix NMF: he model Deermine W and H such ha he produc WH is as close as possible o A W and H are bound o be non-negaive values Possible merics Kullback-Leibler-Divergenz Frobenius-Norm ( A WH) D A WH

22 NMF: raining Updae H W ab ab = H =W ab ab ( W A) ab ( W WH) ab + ( AH ) ab ( WHH ) + ab Relaion o updae From PLSA? In case he denominaor vanishes, add a small number Homework Implemen NMF for he marix from he las lecure

23 Summary Ways o find laen semanic spaces: LSA PLSA NMF Similar facorizaions Dieren arge funcions and consrains 3