Latent Spaces and Matrix Factorization

Transcription

1 Compuaional Linguisics Laen Spaces and Marix Facorizaion Sefan Thaer & Dierich Klakow FR 4.7 Allgemeine Linguisik (Compuerlinguisik) Universiä des Saarlandes Summer 2013

2 Goal Goal: rea documen clusering and word clusering on he same fooing (same semanic space) find low dimensional represenaions

3 The word documen marix

4 Clusering Documen clusering words describe each documen by a vecor conaining he frequencies of he Word clusering describe each word by a vecor conaining he frequencies of is occurance in differen documen

5 Join word and documen clusering The word documen marix: Ener frequency (or f-idf) for each word and documen in a square scheme of numbers (marix)

6 Marices

7 Marices A marix is an array wih wo indices j=1...m e.g. in a pyhon program his could be A[i][j] wih i=1..n and Marix When wriing, ofen a subscrip noaion is used a... a A or a square scheme: a 1,1... N,1 a i, j... a 1, M... N, M a i, j Specific example of a 2x3 marix A

8 The ranspose of a marix The wo indices are swapped e.g. in a pyhon program his could be A[j][i]=A[i][j] for i=1..n and j=1...m Marix for he marices from he previous slide we have: A a a 1,1... M,1 a... j, i... a a 1, N... M, N Specific example of a 2x3 marix A Wha is A

9 Produc of wo marices The elemens of a produc marix can be calculaed in a pyhon program by for i in range(1,n+1): for j in range(1,m+1): Marix for k in range(1,k+1): C[i][j] = A[i][k]*B[k][j] In mah noaion C A B wih c i, j k K 1 a i, k b k, j Example see black board

10 Uni marix Uni marix: he elemen are he indicaor funcion a i, j i, j Example: Marix 1 A Ofen he uni marix is denoed by a 1

11 Orhogonal marices a marix A is orhogonal if 1 A A Marix Is he following marix orhogonal: A

12 Marices in pyhon See hp://wiki.scipy.org/tenaive_numpy_tuorial#head-a9063f71090f3d1fbbdae5397ccb4e882d2cf603

13

14 Laen Semanic Analysis (LSA) This secion mosly follows Manning and Schüze Chaper 15

15 Singular Value Decomposiion Decompose A such ha A ~ TSD Wih Marix ~ A A 2 minimal and T T 1 D D 1 A a by dmarix T a by n marix S a n by n marix D a dby n marix

16 An arificial Example of Singular Value Decomposiion Is T Marix S 2 2 D An SVD of 2 A

17 More realisic Example (from Manning and Schüze) Decompose Marix

18 More realisic Example (from Manning and Schüze) Marix

19 More realisic Example (from Manning and Schüze) Marix

20 Documen-Documen Similariy Rewrie A A d wih 1 Marix d j d 2... a vecor d wih word frequencies of d he j- h documen Similariy of i-h documen wih j-h documen d d i j All documen-documen similariies A A

21 Documen-Documen Similariy Rewrie Marix ~ A ~ A ( TSD DS DS ( SD T SD ) ) TSD TSD SD Measure similariy in subspace defined by SD

22 More realisic Example (from Manning and Schüze) Resul for SD Marix

23 More realisic Example (from Manning and Schüze) Decompose A such ha Marix

24 An even more realisic example Term Documen Marix Srucure

25 An even more realisic example Documen-Documen Similariy Term Documen Marix Srucure

26 Represenaion for Documens in 2 dimensional Subspace

27 Term-Term Similariy Rewrie Marix ~~ AA ( TSD TSD DS TS ( TS)( TS) ST )( TSD T ) Measure similariy in subspace defined by TS

28 Task How does your programming language suppor SVD Do some inerne search (~10 minues) Repor your findings Marix

29 Homework See shee Marix

30 LSA Performance LSA consisenly improves recall on sandard es collecions (precision/recall generally improved) Variable performance on larger TREC collecions Dimensionaliy of Laen Space a magic number seems o work fine no saisfacory way of assessing value. Compuaional cos high

31 Applicaion (by Landauer e. Al) Rae essay by similariy o exising ones Measure correlaion wih human raing Conclusion: drop he righ key-words and you are se

32 Probabilisic Laen Semanic Analysis (PLSA)

33 Moivaion Does orhogonally maer? Wouldn a sound saisical foundaion be beer?

34 PLSA Likelihood of documen P(doc) = P(erm 1 doc)p(erm 2 doc)... P(erm L doc) Inroduce erm-frequency marix X L l= 1 P(erm l doc)= T = 1 P(erm doc) A(erm,doc)

35 PLSA Inroduce hidden opic P(erm doc)= K k= 1 P(erm opic k )P(opic k doc) Shorhand =erm_ P( doc)= K k= 1 P( k)p(k doc) Relaion o LSA? Likelihood of documen P(doc) = T K = 1 k= 1 P( k)p(k doc) A(,doc)

36 PLSA: raining Training objecive funcion N d= 1 logp(d)= N T A(,d) log K d= 1 = 1 k= 1 P( k)p(k d) whichiso bemaximisedw. r.. parameersp( k) andhen alsop(k d), subjec o he consrains ha T = 1 P( k)= 1and K k= 1 P(k d)= 1.

37 PLSA: raining Updae erm-opic marix P1(,k) P1(,k) P1(,k) T = 1 N d= 1 P1(,k) P1(,k) K k= 1 A(,d) P1(,k)P2(k,d) P2(k,d) Updae opic-documen marix P2(k,d) P2(k,d) P2(k,d) K k= 1 T = 1 P2(k,d) P2(k,d) K k= 1 A(,d) P1(,k) P1(,k)P2(k,d)

38 PLSA P( k) for some opics

39 Comparison LSA and PLSA From Th. Hofmann, 2000

40 Non-negaive Marix Facorizaion See:

41 NMF: idea Find space ha separaes clusers beer

42 NMF: he model Decomposion of a non-negaive marix X in wo marices W and H boh non-negaive A=WH A: N x M daa marix W: N x R source marix H: R x M mixure marix

43 NMF: he model Deermine W and H such ha he produc WH is as close as possible o A W and H are bound o be non-negaive values Possible merics Kullback-Leibler-Divergenz Frobenius-Norm D A WH 1 2 A WH 2

44 NMF: raining Updae H W ab ab = H =W ab ab W W WH AH WHH A ab ab ab ab Relaion o updae From PLSA? In case he denominaor vanishes, add a small number

45 Homework Implemen NMF for he marix from he las lecure

46 Summary Ways o find laen semanic spaces: LSA PLSA NMF Similar facorizaions Differen arge funcions and consrains