Information Retrival Ranking. tf-idf. Eddie Aronovich. October 30, 2012

Size: px

Start display at page:

Download "Information Retrival Ranking. tf-idf. Eddie Aronovich. October 30, 2012"

Diana Lawrence
6 years ago
Views:

1 October 30, 2012

2 Table of contents 1 2 Term Frequency tf idf

3 Inforamation Retrival the techniques of storing and recovering and often disseminating recorded data especially through the use of a computerized system http : // webster.comdictionary/information retrieval

4 Characteristics

5 Characteristics Word frequency - The Automatic Creation of Literature Abstracts (Luhn 58)

6 Characteristics Word frequency - The Automatic Creation of Literature Abstracts (Luhn 58) Cue words, title/heading, structural - New methods in automatic extracting (Edmindson 1969)

7 Characteristics Word frequency - The Automatic Creation of Literature Abstracts (Luhn 58) Cue words, title/heading, structural - New methods in automatic extracting (Edmindson 1969) Cohesion - Adaptive method of automatic abstracting and indexing (EF Skorochodko 1971/2)

8 Characteristics Word frequency - The Automatic Creation of Literature Abstracts (Luhn 58) Cue words, title/heading, structural - New methods in automatic extracting (Edmindson 1969) Cohesion - Adaptive method of automatic abstracting and indexing (EF Skorochodko 1971/2) Structure based classification - Automatic condensation of electronic publications by sentence selection (Brandow 1995)

9 Characteristics Word frequency - The Automatic Creation of Literature Abstracts (Luhn 58) Cue words, title/heading, structural - New methods in automatic extracting (Edmindson 1969) Cohesion - Adaptive method of automatic abstracting and indexing (EF Skorochodko 1971/2) Structure based classification - Automatic condensation of electronic publications by sentence selection (Brandow 1995)

10 Similarity Let D be some document vector

11 Similarity Let D be some document vector Let Q be some query vector

12 Similarity Let D be some document vector Let Q be some query vector Cosine the angle represents the similarity between them

13 Similarity Let D be some document vector Let Q be some query vector Cosine the angle represents the similarity between them After normalization, we can use the dot-product instead: Sim( D, Q) = w ti,q w ti,d t i Q,D

14 Relevance documets by their relevance to a query

15 Relevance documets by their relevance to a query P(R D) - probability a document D is relevant P( R D) - probability a document D is non-relevant

16 Relevance documets by their relevance to a query P(R D) - probability a document D is relevant P( R D) - probability a document D is non-relevant Precision: P(R D) = P(R D) P(D)

17 Relevance documets by their relevance to a query P(R D) - probability a document D is relevant P( R D) - probability a document D is non-relevant Precision: P(R D) = P(R D) P(D) Sensitivity: P(D R) = P(R D) P(R)

18 Relevance documets by their relevance to a query P(R D) - probability a document D is relevant P( R D) - probability a document D is non-relevant Precision: P(R D) = P(R D) P(D) Sensitivity: P(D R) = P(R D) P(R)

19 How to classify a document? There is no right and wrong - but a list of methods including: Bayesian approach: log P(R D) P( R D) = log P(D R)P(R) P(D R)P( R)

20 How to classify a document? There is no right and wrong - but a list of methods including: Bayesian approach: log P(R D) P( R D) = log P(D R)P(R) P(D R)P( R) if P(R), P( R) are independent of D, then we get: log P(D R) P(D R)

21 How to classify a document? There is no right and wrong - but a list of methods including: Bayesian approach: log P(R D) P( R D) = log P(D R)P(R) P(D R)P( R) if P(R), P( R) are independent of D, then we get: log P(D R) P(D R) Additional methods exists based on precision and sensitivity.

22 Term Frequency tf idf Term Frequency (tf) Term Frequency (tf): tf (t, D) = F(t,D) max{f(w,d):w D} F() - some frequency function t - a specific term w - all the terms in the document The maximum value will get a word that appears many time in the document

23 Term Frequency tf idf Inverse Document Frequency (idf) Inverse Document Frequency (idf): idf (t, D) = log D D D:t D D - All the documents we have D,t - as before If term t appears ni many documents, this value is low. We look for terms that are rare between documents

24 Term Frequency tf idf tf idf tfidf (t, D, D) = tf (t, D) idf (t, D)

25 Thank You!

Chap 2: Classical models for information retrieval

Chap 2: Classical models for information retrieval Jean-Pierre Chevallet & Philippe Mulhem LIG-MRIM Sept 2016 Jean-Pierre Chevallet & Philippe Mulhem Models of IR 1 / 81 Outline Basic IR Models 1 Basic