Chapter 10: Information Retrieval. See corresponding chapter in Manning&Schütze

Size: px

Start display at page:

Download "Chapter 10: Information Retrieval. See corresponding chapter in Manning&Schütze"

Philip Rodgers
5 years ago
Views:

1 Chapter 10: Information Retrieval See corresponding chapter in Manning&Schütze

2 Evaluation Metrics in IR 2

3 Goal In IR there is a much larger variety of possible metrics For different tasks, different metrics might be appropriate 3

4 4

5 Evaluation of IR Systems All documents Retrieved Relevant Retrieved and relevant Recall=#(retrieved and relevant)/#(relevant) Precision=#(retrieved and relevant)/#(retrieved) 5

6 Precision vs. Recall 6

7 Average precision Goal: don t focus on a specific recall level still get one number AvgP N P r1 N r1 r rel( r) rel( r) P( r ) : precision at rank r rel(r) : indicator function; 1 if document at rank r is relevant 7

8 Mean average precision Problem: average precision still specific to query MAP 1 Q Q q1 AvgP( q) Q: number of queries 8

9 Interpolated precision Recall levels for each query distinct from 11 standard recall levels Interpolation procedure is necessary Let r j be the j-th standard recall level with j=1,2,,10. Then, P r max j rj rrj1 Pr 9

10 Interpolated precision 10

11 Vector Space Model and tf-idf 11

12 Goal Introduce a simple and fast retrieval algorithm 12

13 Preprocessing Stemming Stop words Longer units ( New York ) 13

14 Vector-space-model 14

15 Vector-space-model Considering every document as vector The vector contains the weights of the index terms as components In case of t index terms the dimension of the vector-space is also t Similarity of querie to a document is the correlation between the their vectors Correlation quantified by cosine of the angle between the vectors 15

16 Vector-space-model Index term weights tf i, j with occurs idf and i with n max freq log i N freq l i, j i, j freq l, j the frequency that termi in document j N n i number of totalnumber of documents douments 16 that contain term i

17 Vector-space-model Index term weights The weight of a term in a document is then calculated as product of the tf factor and the idf factor w i, j tf i, j idf i Or for the query w i, q max l freq 17 i, q freq l, q idf i

18 Distance Metrics Pick an L-norm Angel/cosine between vectors cos( q, d) n i1 n i1 q 2 i q d i i n i1 d 2 i 18

19 Vector-space-model Advantages Improves retrieval performance as compared to Boolean retrieval Partial matching allowed Sort according to similarity Disadvantages Assumes that index terms are independent 19

20 Models of Term Distribution 20

21 Models for Term Distribution Goal: Understand the statistical properties of key words in a documents collection Assumptions Probability for a term is proportional to the length of the document Short text: each word occurs only once Two neighboring occurrences of the same term are statistically independent 21

22 Poission Distribution Probabiliity that the i-th term occurs k times in the document P i ( k) e i k i k! i parameter of the distribution k 22

23 Processes described by Poisson Distribution (Wikipedia) The number of cars that pass through a certain point on a road (sufficiently distant from traffic lights) during a given period of time. The number of spelling mistakes one makes while typing a single page. The number of phone calls at a call center per minute. The number of times a web server is accessed per minute. The number of roadkill (animals killed) found per unit length of road. The number of mutations in a given stretch of DNA after a certain amount of radiation. The number of unstable nuclei that decayed within a given period of time in a piece of radioactive substance. The radioactivity of the substance will weaken with time, so the total time interval used in the model should be significantly less than the mean lifetime of the substance. The number of pine trees per unit area of mixed forest. The number of stars in a given volume of space. The number of V2 rocket attacks per area in England, according to the fictionalized account in Thomas Pynchon's Gravity's Rainbow. The number of light bulbs that burn out in a certain amount of time. The number of viruses that can infect a cell in cell culture. The number of hematopoietic stem cells in a sample of unfractionated bone marrow cells. The inventivity of an inventor over their career. The number of particles that "scatter" off of a target in a nuclear or high energy physics experiment. 23

24 Check normalization and expectation value of k -> white board 24

25 Interpretation Let N be the number of documents in the corpus N E i ( k) N i :cf i (collection frequency) N ( 1 P (0)) : df i i (document frequency) 25

26 Experimental Test of Poisson Model Word cf i i N(1-P(0)) df i Overestimation follows transformed soviet students james freshly ,54 Model often works Some terms like soviet are bursty independence assumption is not valid 26

27 Probabilistic Retrieval 27

28 Goal Attempt to justify tf-idf for retrieval 28

29 Probabilistic Retrieval -> white board -> see corresponding section in Manning&Schütze 29

30 Language Model based Retrieval 30

31 Goal Practical way of using the probabilistic ideas for retrieval Reading see J. Ponte and B. Croft A language modeling approach to information retrieval SigIR

32 Language Modeling The probability that a query Q was generated by a probabilistic model based on a document. q q q q n d d d... d 1 2 m p( d q) p( q d)* p( d) Uni-gram model: P( q d) n i1 P( q i d) Ignore p(d) for now 32

33 Performance tf-idf vs LM (original Ponte&Croft-paper) Results on TREC 10 collection a LM outperforms tf-idf 33

34 Smoothing Methods Jelinek-mercer method: involves a linear interpolation of the ML model with the collection model. P ( w d) (1 ) Pml ( w d) P( w C) 34

35 Smoothing Methods Absolute discounting: decrease the probability of seen words by substracting a constant from their counts. P ( w s d) max( c( w; d),0) P( w C) * c( w ; d) w* V 35

36 36 Smoothing Methods Bayesian smoothing using Dirichlet priors: A multinomial distribution, for which the conjugate prior for bayesian analysis is the dirichlet distribution: V w d w c C w P d w c d w P * ) ; ( ) ( ) ; ( ) ( *

37 Comparing different smoothing methods: sentence retrieval in question answering 37

38 Improved Language Models Bigrams Class LMs Grammar Prior knowledge (document length) Other resources (e.g. WordNet) 38

39 Latent Semantic Analysis 39

40 Goal Overcome semantic mismatch between terms in the query and the documents (e.g. cosmonaut vs. astronaut) 40

41 Term Document Matrix Structure Idea: derive semantic relatedness from co-occurance in term document matrix 41

42 Term Document Matrix Structure Create artificially heterogeneous collection 100 documents from 3 distinct newsgroups Indexed using standard stop word list distinct terms Term Document Matrix ( ) 8% fill of sparse matrix Matrix of cosine similarity between documents Clear structure apparent 42

43 Theory of LSA Whiteboard 43

44 Latent Semantic Analysis Word usage defined by term and document cooccurrence matrix structure Latent structure / semantics in word usage Clustering documents or words Singular Value Decomposition Cubic Computational Scaling 44

45 Term Document Matrix Structure 45

46 46

47 LSA Performance LSA consistently improves recall on standard test collections (precision/recall generally improved) Variable performance on larger TREC collections Dimensionality of Latent Space a magic number seems to work fine Computational cost high 47

48 Toolkits 48

49 Software to use Lucene Lemur 49

50 50

51 Summary Evaluation measures Vector space model Models of term distribution Probabilistic retrieval Latent semantic analysis Language models for IR 51

Ranked Retrieval (2)

Text Technologies for Data Science INFR11145 Ranked Retrieval (2) Instructor: Walid Magdy 31-Oct-2017 Lecture Objectives Learn about Probabilistic models BM25 Learn about LM for IR 2 1 Recall: VSM & TFIDF