Information Retrieval Advanced IR models. Luca Bondi

Transcription

1 Advanced IR models Luca Bondi

2 Advanced IR models 2 (LSI) Pobabilistic Latent Semantic Analysis (plsa)

3 Vecto Space Model 3 Stating point: Vecto Space Model Documents and queies epesented as vectos in the tems space Index tem s significance epesented by eal valued weights w ",$ 0 associated to the pai t ",d $ Each document is epesented by a vecto in a M-dimensional space, whee M is the numbe of index tems in the vocabulay d $ = w,,$, w -,$,, w /,$ 0

4 Vecto Space Model 4 Documents that ae close to each othe in the tems space ae simila to each othe Also queies ae epesented as vectos in the tems space q = w,,2, w -,2,, w /,2 0 The Vecto Space Model computes the Similaity Coefficient SC q, d $ between the quey and each document Eucledean distance Cosine similaity Jaccad s similaity

5 Vecto Space Model: limits 5 Vecto Space Model is based on matching tems in documents with those of the quey Lexical matching methods can be inaccuate Thee ae many ways to expess a given concept (synonymy) The liteal tems in a use s quey might not match those of a elevant document Most wods have multiple meanings (polysemy) Tems in a use s quey will liteally match tems in ielevant documents Neglect coelations that exist between diffeent tems Bette appoach: etieve infomation on the basis of a conceptual topic

6 Tem-document matix 6 In ode to implement (LSI), a matix of tems by documents must be ceated A = w ",$ R 9 / ; whee w ",$ is the tem-document weight used by the Vecto Space Model Each column of A is a document (N is the numbe of documents in the collection) A = d, d ; Each ow of A is elated to an index tem

7 Tem-document matix 7 LSI boils down to computing the Singula Value Decomposition (SVD) of the tem-document matix A = UΣV 0 M A = M U Σ V 0 N N A = M N tem-document matix V = N document-concept matix. Columns of V ae denoted as v " Σ = concept weighting diagonal matix U = M tem-concept matix. Columns of U ae denoted as u "

8 Tem-document matix 8 u " = tem-concept vectos Each vecto u " epesents a concept (and thee ae uncoelated concepts) Each element of the vecto u " epesents the contibution of a tem to the i-th concept v " = documents-concept vectos Each vecto v " epesents a concept Each element of the vecto v " epesents how much a document is descibed by the i-th concept M A = M U Σ V 0 N v " N u "

9 Singula Value Decomposition 9 Each column of V 0 epesents a document in the -dimensional concept space Notation: the j-th column of V 0 is denoted as V 0 $ Notation: the j-th document epesented in the concept space V 0 $ is also denoted as d $ Each document is descibed by a linea combination of uncoelated concepts d $ = u, σ, v $,, + u J σ J v $,J M A = M U Σ V 0 $ d $ V 0 N d $ N

10 Documents and queies 10 Documents can be expessed equivalently in the tems space and in the concepts space d $ = UΣ K, d $ = Σ K, U 0 d $ Queies can be expessed equivalently in the tems space and in the concepts space q = UΣ K, q = Σ K, U 0 q M = M U Σ d $ d $

11 Documents and queies 11 If SVD is appoximated up to the k-th concept k < the tansfomation of queies and documents fom the tems space to the concepts space is not-evesible d k $ = U PΣ P K, d $ = Σ P K, U P 0 d $ d Q $ = U P Σ P d k $ M = M U P U k k Σ P Σ k d k $ d Q $ k

12 Geometical intepetation 12 t 3 t1 = ' goal ' t2 = ' stadium ' t3 = ' math ' σ u 2 2 v j,2u2 t 2 u 1 d j Ud T k j σ 1 v j,1u1 t 1

13 Geometical intepetation 13 t 3 q d j Ud T k j T Uq k u 2 u 1 t 2 t 1

14 Documents intepetation 14 Conside two documents (j, and j - ). Intuitively they ae simila if they shae many tems in common lage cosine similaity d $S 0 d$t d $S d $T they do not shae common tems, but they shae tems that cooccu in many othe documents the cosine similaity is small, but they ae close in the dimensional concept space d $ S 0 d $ T d $ S d $ T

15 Tems intepetation 15 Tems contibute to uncoelated concepts t " : how much a tem descibes each document t ": how much a tem chaacteizes each concept t " 0 = t " 0 VΣ K, t " 0 M A = 0 t " M U Σ V 0 N N

16 Tems intepetation 16 Conside two tems ca and automobile (i, and i - ). Intuitively they ae simila if they co-occu in many documents lage cosine similaity t "S 0 t "T t "S t "T they do not co-occu, but they occu with many of the same wods (e.g. moto, model, vehicle, engine, etc ) the cosine similaity is small, but they ae close in the dimensional concept space t " S 0 t " T t " S t " T Synonymy is captued in the dimensional concept space: synonym tems mapped to the same concept

17 Example 17 Synthetically geneated copus: M = 2000 tems N = 1000 documents = 20 topics à 2000/20 = 100 tems/topic Each document has a single topic Each document has between 50 and 100 tems each tem in a document with p = 0.95 is a tem (sampled at andom) of a given topic with p = 0.05 is a tem (sampled at andom) fom the vocabulay

18 Example tems documents

19 Example 19 In pinciple, documents that belong to the same topic should have SC à 1 documents that belong to diffeent topics should have SC à 0 Document-to-document similaity matices: d $S 0 d $T d $ S 0 d $ T d $S d $T d $ S d $ T

20 Updating 20 Suppose that a database aleady exists and the SVD of the temdocument matix has aleady been computed New tems/documents might be added: Re-compute the SVD computationally expensive Folding-in computationally cheap, might deteioate pefomance SVD-updating

21 Folding-in 21 Based on existing SVD decomposition New documents/tems have no effect on the epesentation of peexisting documents/tems Folding-in is pefomed in the same way as quey epesentation Given P new document vectos D = d [\],, d [\], d [\],^ Compute D J = UΣ K, D Append the esults to the columns of V 0 M A D = M U Σ V 0 N + P D J N + P

22 Folding-in 22 Similaly, when adding new tems Given P new tem vectos T = t [\],, t [\], t [\],^ Compute T J` = T`VΣ K, Append the esults to the ows of U M + P A T` M + P = U T J` Σ V 0 N N

23 SVD updating 23 Let D denote the P new document vectos to pocess D is a M P spase matix appended to the columns the temdocument matix A B = A D Define SVD B = U e Σ e V e 0 Then A = UΣV 0 B = UΣV 0 D U 0 B = ΣV 0 U 0 D U 0 B V 0 0 I^ = Σ U 0 D

24 SVD updating 24 Define F = Σ U 0 D RJ J9^ SVD F = U h Σ h V0 h Then it follows U 0 B V 0 0 I^ = Σ U 0 D U 0 U e Σ e V e 0 V 0 0 I^ = U h Σ h V h 0 U e Σ e V e 0 = UU h Σ h V h 0 V T 0 0 I^ U e = UU h V e = V 0 0 I^ V h Σ e = Σ h Thus the SVD of the updated matix can be computed fom the SVD of F (which is much smalle) A simila agument hold when adding new tems Adding documents/tems might also alte weights. SVD updating including weights is discussed in [Bey et al., 1995]

25 Example Small database of book titles 25

26 Example 26 Tem-document matix fo simplicity, tem weighting is not used

27 Example 27 Retaining k = 2 concepts, we can plot both tems and documents in the concept space Documents and tems petaining diffeential equations ae clusteed aound the x-axis and the moe geneal tems and documents elated to algoithms and applications ae clusteed aound the y-axis

28 Example 28 Suppose we ae inteested in the documents that petain to applications and theoy q - T = q T U - Σ Ḵ,

29 Example 29 All documents whose cosine with the quey vecto is geate than 0.9 ae illustated in the shaded egion

30 Example 30 Suppose the following titles need to be added With folding-in the titles ae added to the database, but the coodinates of the oiginal titles stay fixed

31 Example 31 Re-computing the SVD leads to a diffeent epesentation Note that B19 and B20 ae now futhe apat, as they should be (obseve the diffeent use of the wod odinay ) Folding- in SVD e- computation

32 Example 32 The esult of SVD-updating is vey simila to e-computing the SVD SVD updating SVD e- computation

33 Advanced IR models Pobabilistic Latent Semantic Analysis 33 (LSI) Pobabilistic Latent Semantic Analysis (plsa)

34 Pobabilistic Latent Semantic Analysis Intoduction 34 The basic idea of Latent Semantic Analysis (LSA) is to map documents (and by symmety tems) to a vecto space of educed dimensionality, the latent semantic space The mapping imposed by LSA is esticted to be linea Each document vecto is epesented as a linea combination of topic vectos The SVD is an efficient computational tool used to lean such a mapping

35 Pobabilistic Latent Semantic Analysis Intoduction 35 Pobabilistic Latent Semantic Analysis defines a pobabilistic latent vaiable model that descibes the co-occuence of data (documents and tems) Each document belongs to one o moe (latent) topics Fo each topic, a pobability model descibes the distibution of tems Befoe consideing plsa, let us discuss simple latent vaiable models Unigam model Mixtue of unigams

36 Pobabilistic Latent Semantic Analysis Unigam model 36 The unigam model is completely descibed by the pobability mass function (p.m.f.) epesenting the pobability of obseving each tem t " ( ) Pt i m ti Such p.m.f. can be simply obtained fom the nomalized histogam of tem counts in the document collection

37 Pobabilistic Latent Semantic Analysis Unigam model 37 Unde the unigam model, wods of evey document ae dawn independently fom a single multinomial distibution The pobability of obseving document d in the collection is [ P d = k P t l lm, n is the numbe of (distinct) tems that appea in the document

38 Pobabilistic Latent Semantic Analysis Mixtue of unigams 38 We augment the model with a discete andom topic vaiable z Z = z,,, z q K topics Each topic is completely chaacteized by its own p.m.f. We obtain a mixtue of unigams model Unde this model, each document is geneated by fist choosing a single topic z P and then geneating n wods independently fom the conditional multinomial P t " z P ) The pobability of obseving document d in the collection is q [ P d = t P z P Pm, k P t l z P lm,

39 Pobabilistic Latent Semantic Analysis Mixtue of unigams 39 The unigam model is completely descibed by a set of K p.m.f. s epesenting the pobability of obseving each tem t ", fo each of the K latent topics Pt ( z) i 1 Pt ( z) i m ti t m i The p.m.f. epesenting the topic pobabilities Pz ( k ) zk K

40 Pobabilistic Latent Semantic Analysis Pobabilistic 40 plsi elaxes the assumption made in the mixtue of unigams model that each document is geneated fom only one topic The plsi assumes that a document d $ and a tem t " ae conditionally independent given the unobseved topic z P P d $, t " = t P(z P )P(d $, t " z P ) v w x = t P(z P )P(d $ z P ) P t " z P v w x = P d $ t P t " z P P(z P d $ ) v w x plsi captues the possibility that a document contains multiple topics since P(z P d $ ) seves as the mixtue weights of the topics fo a paticula document d $ The estimation of the pobabilities can be pefomed using an iteative expectation-maximization (EM) pocedue.

41 Pobabilistic Latent Semantic Analysis Geometic intepetation 41 Conside a n-dimensional vecto space A multinomial distibution ove the tems can be seen as a n dimensional vecto P t,,, P t [ 0 The vectos coesponding to any multinomial distibution lie on a n 1 dimensional sub-space (embedding simplex), since [ t P t " = 1 "m,

42 Pobabilistic Latent Semantic Analysis Geometic intepetation 42 Conside K topics. Fo each topic, define the multinomial distibution P t " z P, i = 1,, n Each topic defines a vecto in the embedding simplex P t z P = P t, z P,, P t [ z P 0

43 Pobabilistic Latent Semantic Analysis Geometic intepetation 43 Pt ( z) i 1 P t z P = P t, z P,, P t [ z P 0 2 i 1 3 t Pt ( 3) Pt ( 2) Pt ( 1)

44 Pobabilistic Latent Semantic Analysis Geometic intepetation 44 The modelling assumption expessed in Py t d $ = t P z P d $ v w x P t z P is that conditional distibutions P t d $ fo all documents ae appoximated by a multinomial epesentable as a convex combination of factos P t z P, whee the mixing weights - P z P d $ - uniquely detemine a point on the spanned simplex

45 Pobabilistic Latent Semantic Analysis Geometic intepetation 45 P( t d j ) Pˆ( t d j ) P( t z ) 3 P( t z ) 2 P( t z ) 1

46 Pobabilistic Latent Semantic Analysis Geometic intepetation 46 Since the dimensionality of the sub-simplex is <= K 1 as opposed to a maximum of n 1 fo the complete pobability simplex, this pefom a dimensionality eduction Each document is paameteized by the K weights P z P d $, k = 1,, K

47 Pobabilistic Latent Semantic Analysis Geometic intepetation 47 P t d - P t d, Py t d - Py t d, P( t z ) 3 P( t z ) 2 P( t z ) 1

48 Advanced IR models Refeences 48 [Bey et al., 1995] M.W. Bey, S.T. Dumais and G.W. O Bien, Using Linea Algeba fo Intelligent, SIAM Review, Volume 37, Issue 4, Decembe 1995, pages: , [Hofmann, 1999] T. Hofmann, Pobabilistic Latent Semantic Analysis, Uncetainty in Atificial Intelligence, Stockholm, 1999,