Motivation. User. Retrieval Model Result: Query. Document Collection. Information Need. Information Retrieval / Chapter 3: Retrieval Models

Size: px

Start display at page:

Download "Motivation. User. Retrieval Model Result: Query. Document Collection. Information Need. Information Retrieval / Chapter 3: Retrieval Models"

Georgia Black
5 years ago
Views:

1 3. Retrieval Models

2 Motivation Information Need User Retrieval Model Result: Query Document Collection 2

3 Agenda 3.1 Boolean Retrieval 3.2 Vector Space Model 3.3 Probabilistic IR 3.4 Statistical Language Models 3.5 Relevance Feedback 3.6 Query Expansion 3.7 Novelty & Diversity 3

4 3.1 Boolean Retrieval Documents are interpreted as sets of terms, which can be understood as assignments to Boolean variables one variable per known term variable for a term is true if the term is present and false if it is not present in the document Queries are Boolean expressions combining variables with the operators AND ( ), OR ( ), and NOT ( ) gothenburg AND (amusement OR shopping) AND NOT museum 4

5 Boolean Retrieval A document is said to match a query if the corresponding Boolean expression evaluates to true given its assignment of truth values to variables Boolean retrieval has clear semantics, i.e., a document either matches a query or it does not All matching documents are considered equal, i.e., there is no ranking of documents 5

6 Term-Document Matrix Document collection seen as term-document matrix d 1 d 2 d 3 d 4 d 5 d 6 amusement park gothenburg sweden museum shopping liseberg art Matrix entries assume values 0 and 1 for Boolean Retrieval 6

7 Boolean Retrieval in Practice While limited, due to its lack of ranking, Boolean Retrieval is still used in practice, sometimes as a supplement to more advanced retrieval models Library search (e.g., 7

Boolean Retrieval in Practice Patent search (e.g., http://appft.uspto.

8 Boolean Retrieval in Practice Patent search (e.g., Modern search engines also provide Boolean operators in disguise (e.g., AND, OR, -, +) to filter the set of returned documents 8

9 Extensions of Boolean Retrieval Several extensions have been proposed to mitigate the lack of ranking in Boolean Retrieval Boolean Retrieval with fields (e.g., title, author, body) allows more precise specification of information need (e.g., author:knuth AND title:tex) and can yield limited ranking of results if fields are weighted Additional operators have been proposed, e.g.: apple NEAR(5) recipe matches documents where the terms apple and recipe occur within a window of five terms 9

3.2 Vector Space Model Idea: Represent documents and queries as vectors in a common high-dimensional vector space and use distance/similarity between document vectors and query vector to rank

10 3.2 Vector Space Model Idea: Represent documents and queries as vectors in a common high-dimensional vector space and use distance/similarity between document vectors and query vector to rank documents Historical Background: SMART Project at Cornell University during the 1960s led by Gerard Salton ACM SIGIR awards Gerard Salton Award every three years to people with significant contributions in IR Source: 10

11 Mathematical Background: Vectors Vectors are elements of a multidimensional space, e.g. the Euclidean plane R 2 or the k-dimensional space R k v = 5 v1 v 2 6 œ R 2 v = S W U v 1. v k T X V œ R k Vectors can be added to each other v + ų = S W U v 1. v k T X V + S W U u 1. u k T X V = S T v 1 + u 1 W X U. V v k + u k

12 Mathematical Background: Vectors Vectors can be multiplied with a scalar (real number) S T S T v 1 v 1 W X W X v = U V = U. V v k v k 1 2 Vectors can be multiplied with each other yielding a scalar 5 S T S T 1 v 1 u 1 W X W X kÿ 36 v ų = U. V U. V = (v i u i ) v k u i=1 k 16 12

13 Mathematical Background: Vectors Vectors have a length v = S T v 1 W X U. V - v k - = ˆ ıÿ Ù k i=1 v 2 i Ô 18 Cosine of the angle between two vectors cos(ų, v) = ų v ų v = ų ų v v = Ò qk q k i=1 (u i v i ) Ò qk i=1 v2 i i=1 u2 i 13

14 Documents and Queries as Vectors Documents and queries are represented as vectors in a vector space with one dimension per known term Idea 1: Binary term weighting vector component is 1 if the term is present in the document and 0 if the term is not present in the document commonly used to represent query Observations: The following do not play a role how often the terms occurs in a document how many documents contain the term 14

15 Term Weighting using tf.idf Idea 2: Term weighting using tf.idf Term frequency indicating how often the term v occurs in document d tf (v, d) Document frequency indicating how many documents from the document collection contain the term v df (v) 15

16 Term Weighting using tf.idf Intuitively, the vector of a document should have a high value for a term, if the term occurs often in the document and the term does not occur in many documents Inverse document frequency of term v with D as the cardinality of the document collection D, i.e., the total number of documents therein idf (v) = log D df (v) 16

17 Logarithmic Dampening idf dampening ohne Dämpfung mit Dämpfung no dampening D = df Base of logarithm (e.g., 2 or 10) does not play a role 17

18 Term Weighting using tf.idf d 1 d 2 d 3 d 4 d 5 d 6 amusement park gothenburg sweden museum shopping liseberg art df (v) idf (v) log(6/3) = 1.00 log(6/2) = 1.58 log(6/5) = 0.26 log(6/5) = 0.26 log(6/3) = 1.00 log(6/1) = 2.58 log(6/3) = 1.00 log(6/2) =

19 Term Weighting using tf.idf d 1 d2 d3 d4 d5 d6 amusement park gothenburg sweden museum shopping 2.58 liseberg art

20 Euclidean Distance How to measure the distance/similarity between vectors? Idea 1: Euclidean distance ˆ ıÿ d(q, d) = Ù k (q i d i ) 2 i=1 q d Problem: Euclidean distance depends on the length of the vectors, i.e., longer documents that contain more terms are penalized, whereas shorter documents profit 20

21 Cosine Similarity Idea 2: Cosine similarity measures the cosine of the angle between two vectors, which is independent of their lengths sim(q, d) = q d q d q k i=1 = (q i d i ) Ò qk Ò i=1 q qk i 2 i=1 d 2 i q d Observation: Document vector has similarity of 1 with query vector if both point in the same direction, i.e. both contain the same terms with the same proportions 21

22 Vector Space Model d 1 d2 d3 d4 d5 d6 amusement park gothenburg sweden museum shopping 2.58 liseberg art q Query is amusement park gotenburg 22

23 Vector Space Model d 1 d2 d3 d4 d5 d6 amusement park gothenburg sweden museum shopping 2.58 liseberg art q cos( q, d 1 )= Ô Ô

24 Vector Space Model d 1 d2 d3 d4 d5 d6 amusement park gothenburg sweden museum shopping 2.58 liseberg art q cos( q, d i ) Documents are ranked as d 2, d 5, d 1, d 4, d 6, d 3 24

25 Vector Space Model in Practice Cosine similarity can be computed more efficiently by normalizing document vectors upfront (cf. Chapter 3) Cosine similarity often implemented in simplified manner sim(q, d) = ÿ v œ q tf (v, d) idf (v) with query q and document d as bags of terms Many variations of tf.idf exist (e.g., normalizing term frequencies depending on length of document) 25

26 Summary Boolean Retrieval as a an early simple retrieval model that still plays a role in practice to this day Vector space model represents query and documents as vectors in a high-dimensional vector space Term weighting using tf.idf considers how often terms from the query occur in documents and in how many documents from the collection they occur Cosine similarity to determine a ranking of documents in response to a specific query 26

27 Literature [1] C. D. Manning, P. Raghavan, and H. Schütze: Introduction to Information Retrieval, Cambridge University Press, 2008 (Chapter 6) [2] W. B. Croft, D. Metzler, and T. Strohman: Search Engines Information Retrieval in Practice, Pearson Education, 2009 (Chapter 7) 27

Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 6 Scoring term weighting and the vector space model

Introduction to Information Retrieval (Manning, Raghavan, Schutze) Chapter 6 Scoring term weighting and the vector space model Ranked retrieval Thus far, our queries have all been Boolean. Documents either