Introduction to Information Retrieval CS276: Information Retrieval and Web Search Christopher Manning and Prabhakar Raghavan Lecture 6: Scoring, Term Weighting and the Vector Space Model
This lecture; IIR Sections 6.2-6.4.3 Ranked retrieval Scoring documents Term frequency Collection statistics Weighting schemes Vector space scoring
Binary term-document incidence matrix Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 1 1 0 0 0 1 Brutus 1 1 0 1 0 0 Caesar 1 1 0 1 1 1 Calpurnia 0 1 0 0 0 0 Cleopatra 1 0 0 0 0 0 mercy 1 0 1 1 1 1 worser 1 0 1 1 1 0
Term-document count matrices Consider the number of occurrences of a term in a document: Each document is a count vector in N v : a column below Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 157 73 0 0 0 0 Brutus 4 157 0 1 0 0 Caesar 232 227 0 2 1 1 Calpurnia 0 10 0 0 0 0 Cleopatra 57 0 0 0 0 0 mercy 2 0 3 5 5 1 worser 2 0 1 1 1 0
tf-idf weighting The tf-idf weight of a term is the product of its tf weight and its idf weight. w (1 log tft, d ) log ( N / df t, d t ) Best known weighting scheme in information retrieval Note: the - in tf-idf is a hyphen, not a minus sign! Alternative names: tf.idf, tf x idf Increases with the number of occurrences within a document Increases with the rarity of the term in the collection
Binary count weight matrix Antony and Cleopatra Julius Caesar The Tempest Hamlet Othello Macbeth Antony 5.25 3.18 0 0 0 0.35 Brutus 1.21 6.1 0 1 0 0 Caesar 8.59 2.54 0 1.51 0.25 0 Calpurnia 0 1.54 0 0 0 0 Cleopatra 2.85 0 0 0 0 0 mercy 1.51 0 1.9 0.12 5.25 0.88 worser 1.37 0 0.11 4.15 0.25 1.95 R
Documents as vectors So we have a V -dimensional vector space Terms are axes of the space Documents are points or vectors in this space d j = (w 1j, w 2j,, w tj )
Graphic Representation Example: D 1 = 2T 1 + 3T 2 + 5T 3 T 3 D 2 = 3T 1 + 7T 2 + T 3 Q = 0T 1 + 0T 2 + 2T 3 5 D 1 = 2T 1 + 3T 2 + 5T 3 Q = 0T 1 + 0T 2 + 2T 3 2 3 T 1 D 2 = 3T 1 + 7T 2 + T 3 T 2 7 Is D 1 or D 2 more similar to Q? How to measure the degree of similarity? Distance? Angle? Projection? 8
Documents as vectors So we have a V -dimensional vector space Terms are axes of the space Documents are points or vectors in this space d j = (w 1j, w 2j,, w tj ) Very high-dimensional: tens of millions of dimensions when you apply this to a web search engine These are very sparse vectors - most entries are zero.
Document Collection A collection of n documents can be represented in the vector space model by a term-document matrix. An entry in the matrix corresponds to the weight of a term in the document; zero means the term has no significance in the document or it simply doesn t exist in the document. T 1 T 2. T t D 1 w 11 w 21 w t1 D 2 w 12 w 22 w t2 : : : : : : : : D n w 1n w 2n w tn 10
Queries as vectors Key idea 1: Do the same for queries: represent them as vectors in the space Key idea 2: Rank documents according to their proximity to the query in this space proximity = similarity of vectors proximity inverse of distance rank more relevant documents higher than less relevant documents It is possible to enforce a certain threshold so that the size of the retrieved set can be controlled.
Formalizing vector space proximity First cut: distance between two points ( = distance between the end points of the two vectors) Euclidean distance? Euclidean distance is a bad idea...... because Euclidean distance is large for vectors of different lengths.
Why distance is a bad idea The Euclidean distance between q and d 2 is large even though the distribution of terms in the query q and the distribution of terms in the document d 2 are very similar.
Use angle instead of distance Thought experiment: take a document d and append it to itself. Call this document d. Semantically d and d have the same content The Euclidean distance between the two documents can be quite large The angle between the two documents is 0, corresponding to maximal similarity. Key idea: Rank documents according to angle with query.
Similarity Measure - Inner Product Similarity between vectors for the document d i and query q can be computed as the vector inner product (a.k.a. dot product): sim(d j,q) = d j q = t w ij w i1 where w ij is the weight of term i in document j and w iq is the weight of term i in the query For binary vectors, the inner product is the number of matched query terms in the document (size of intersection). For weighted term vectors, it is the sum of the products of the weights of the matched terms. iq 15
Inner Product -- Examples Binary: D = 1, 1, 1, 0, 1, 1, 0 Q = 1, 0, 1, 0, 0, 1, 1 sim(d, Q) = 3 Size of vector = size of vocabulary = 7 0 means corresponding term not found in document or query Weighted: D 1 = 2T 1 + 3T 2 + 5T 3 D 2 = 3T 1 + 7T 2 + 1T 3 Q = 0T 1 + 0T 2 + 2T 3 sim(d 1, Q) = 2*0 + 3*0 + 5*2 = 10 sim(d 2, Q) = 3*0 + 7*0 + 1*2 = 2 16
Properties of Inner Product The inner product is unbounded. Favors long documents with a large number of unique terms. Measures how many terms matched but not how many terms are not matched. 17
From angles to cosines The following two notions are equivalent. Rank documents in increasing order of the angle between query and document Rank documents in decreasing order of cosine(angle between query,document) Cosine is a monotonically decreasing function for the interval [0 o, 180 o ]
From angles to cosines But how and why should we be computing cosines?
Length normalization A vector can be (length-) normalized by dividing each of its components by its length for this we use the L 2 norm: 2 x x Dividing a vector by its L 2 norm makes it a unit (length) vector 2 Effect on the two documents d and d (d appended to itself) from earlier slide: they have identical vectors after length-normalization. Long and short documents now have comparable weights i i
cosine(query,document) V i i V i i V i i i d q q d d d q q d q d q d q 1 2 1 2 1 ), cos( q i i d i i q,d q d q d
Cosine for length-normalized vectors For length-normalized vectors, cosine similarity is simply the dot product (or scalar product): cos(q,d ) q d V q d i1 i i for q, d length-normalized. 22
Cosine similarity illustrated 23
Cosine similarity amongst 3 documents How similar are the novels SaS: Sense and Sensibility PaP: Pride and Prejudice, and WH: Wuthering Heights term SaS PaP WH affection 115 58 20 jealous 10 7 11 gossip 2 0 6 wuthering 0 0 38
3 documents example contd. Log frequency weighting term SaS PaP WH affection 3.06 2.76 2.30 jealous 2.00 1.85 2.04 gossip 1.30 0 1.78 wuthering 0 0 2.58 After length normalization term SaS PaP WH affection 0.789 0.832 0.524 jealous 0.515 0.555 0.465 gossip 0.335 0 0.405 wuthering 0 0 0.588
Computing cosine scores
Summary vector space ranking Represent the query as a weighted tf-idf vector Represent each document as a weighted tf-idf vector Compute the cosine similarity score for the query vector and each document vector Rank documents with respect to the query by score Return the top K (e.g., K = 10) to the user
Comments on Vector Space Models Simple, mathematically based approach. Considers both local (tf) and global (idf) word occurrence frequencies. Provides partial matching and ranked results. Tends to work quite well in practice despite obvious weaknesses. Allows efficient implementation for large document collections. 28
Problems with Vector Space Model Missing semantic information (e.g. word sense). Missing syntactic information (e.g. phrase structure, word order, proximity information). Assumption of term independence (e.g. ignores synonomy). Lacks the control of a Boolean model (e.g., requiring a term to appear in a document). Given a two-term query A B, may prefer a document containing A frequently but not B, over a document that contains both A and B, but both less frequently. 29
Resources for today s lecture IIR 6.2 6.4.3 http://www.miislita.com/information-retrievaltutorial/cosine-similarity-tutorial.html Term weighting and cosine similarity tutorial for SEO folk!