Vector Space Scoring Introduction to Information Retrieval INF 141 Donald J. Patterson

Size: px

Start display at page:

Download "Vector Space Scoring Introduction to Information Retrieval INF 141 Donald J. Patterson"

Theodore Parrish
6 years ago
Views:

1 Vector Space Scoring Introduction to Information Retrieval INF 141 Donald J. Patterson Content adapted from Hinrich Schütze

3 Collection Frequency, cf Define: The total number of occurences of the term in the entire corpus

4 Collection Frequency, cf Define: The total number of occurences of the term in the entire corpus Document Frequency, df Define: The total number of documents which contain the term in the corpus

5 W ord Collection F requency Document F requency insurance try

6 W ord Collection F requency Document F requency insurance try This suggests that df is better at discriminating between documents

7 W ord Collection F requency Document F requency insurance try This suggests that df is better at discriminating between documents How do we use df?

9 Term-Frequency, Inverse Document Frequency Weights

10 Term-Frequency, Inverse Document Frequency Weights tf-idf

11 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency

12 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document

13 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document idf = inverse document frequency

14 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document idf = inverse document frequency a measure of the informativeness of a term

15 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document idf = inverse document frequency a measure of the informativeness of a term it s rarity across the corpus

16 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document idf = inverse document frequency a measure of the informativeness of a term it s rarity across the corpus could be just a count of documents with the term

17 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document idf = inverse document frequency a measure of the informativeness of a term it s rarity across the corpus could be just a count of documents with the term more commonly it is:

18 Term-Frequency, Inverse Document Frequency Weights tf-idf tf = term frequency some measure of term density in a document idf = inverse document frequency a measure of the informativeness of a term it s rarity across the corpus could be just a count of documents with the term more commonly it is: idf t = log ( ) corpus df t

19 TF-IDF Examples idf t = log ( corpus df t ) idf t = log 10 ( 1, 000, 000 df t ) term df t idf t calpurnia 1 animal 10 sunday 1000 fly 10, 000 under 100, 000 the 1, 000,

20 TF-IDF Summary Assign tf-idf weight for each term t in a document d: ( ) corpus tfidf(t, d) = (1 + log(tf t,d )) log df t,d Increases with number of occurrences of term in a doc. Increases with rarity of term across entire corpus Three different metrics term frequency document frequency collection/corpus frequency

21 Now, real-valued term-document matrices Bag of words model Each element of matrix is tf-idf value Antony and Julius T he T empest Hamlet Othello M acbeth Cleopatra Caesar Antony Brutus Caesar Calpurnia Cleopatra mercy worser

22 Vector Space Scoring That is a nice matrix, but How does it relate to scoring? Next, vector space scoring

23 Vector Space Scoring Vector Space Model Define: Vector Space Model Representing a set of documents as vectors in a common vector space. It is fundamental to many operations (query,document) pair scoring document classification document clustering Queries are represented as a document A short one, but mathematically equivalent

24 Vector Space Scoring Vector Space Model Define: Vector Space Model A document, d, is defined as a vector: One component for each term in the dictionary Assume the term is the tf-idf score V (d) t = (1 + log(tf t,d )) log V (d) ( ) corpus df t,d A corpus is many vectors together. A document can be thought of as a point in a multidimensional space, with axes related to terms.

25 Vector Space Scoring Vector Space Model Recall our Shakespeare Example: Antony and Julius T he T empest Hamlet Othello M acbeth Cleopatra Caesar Antony Brutus Caesar Calpurnia Cleopatra mercy worser

26 Vector Space Scoring Vector Space Model Recall our Shakespeare Example: V (d 1 ) Antony and Julius T he T empest Hamlet Othello M acbeth Cleopatra Caesar Antony Brutus Caesar Calpurnia Cleopatra mercy worser

27 Vector Space Scoring Vector Space Model Recall our Shakespeare Example: V (d 1 ) V (d2 ) V (d6 ) Antony and Julius T he T empest Hamlet Othello M acbeth Cleopatra Caesar Antony Brutus Caesar Calpurnia Cleopatra mercy worser

28 Vector Space Scoring Vector Space Model Recall our Shakespeare Example: V (d 1 ) V (d2 ) V (d6 ) Antony and Julius T he T empest Hamlet Othello M acbeth Cleopatra Caesar Antony Brutus Caesar Calpurnia Cleopatra mercy worser V (d 6 ) 7

Vector Space Scoring Introduction to Information Retrieval Informatics 141 / CS 121 Donald J. Patterson

Vector Space Scoring Introduction to Information Retrieval Informatics 141 / CS 121 Donald J. Patterson Content adapted from Hinrich Schütze http://www.informationretrieval.org Querying Corpus-wide statistics