Information Retrieval (Relevance Feedback & Query Expansion)

Transcription

1 Infomation Retieval (Relevance Feedback & Quey Epansion) Fabio Aiolli Dipatimento di Matematica Univesità di Padova Anno Accademico 1 Relevance feedback and quey epansion Goal: To efine the answe set by involving the use in the etieval pocess (feedback/inteaction) Local Methods (adjust the use queies) Relevance feedback Pseudo (o Blind) Relevance Feedback (Global) indiect Relevance Feedback Global Methods (independent of the queies and esults) Quey epansion/efomulation with a thesauus (WodNet) Quey epansion via automatic thesauus geneation Othe techniques (spelling coection,...) 2

2 Relevance feedback Basic Pocedue of RF 1. The use issues a simple quey 2. The system etuns an initial set of etieval esults 3. The use maks some of these documents as elevant/ielevant 4. The system computes a bette epesentation of the infomation need based on this feedback 5. The system displays a evised set of esults Repeat the pocedue one o moe times. This pocess helps the use to focalize its own infomation need as well. 3 Relevance Feedback: Eample Image seach engine 4

3 Results fo Initial Quey 5 Relevance Feedback step 6

4 Results afte Relevance Feedback 7 Rocchio Algoithm The Rocchio algoithm incopoates elevance feedback infomation into the vecto space model. Want to maimize sim (Q, C ) - sim (Q, C n ) The optimal quey vecto fo sepaating elevant and non-elevant documents (with cosine sim.): 1 1 Qopt = d j C N C d C j d C Q opt = optimal quey; C = set of el. doc vectos; N = collection size Unealistic: we don t know elevant documents. j d j 8

5 The Theoetically Best Quey Optimal quey o o o o o o non-elevant documents o elevant documents 9 Rocchio 1971 Algoithm (SMART) Used in pactice: 1 1 qm = αq D 0 + β d j γ D d D j q m = modified quey vecto; q 0 = oiginal quey vecto; α,β,γ: weights (hand-chosen o set empiically); D = set of known elevant doc vectos; D n = set of known ielevant doc vectos New quey moves towad elevant documents and away fom ielevant documents Tadeoff α vs. β and γ : If we have a lot of judged documents, we want a highe β and γ. Tem weight can go negative Negative tem weights ae ignoed Altenatively, weights can be nomalized in [0,1] n d D j d n j 10

6 Relevance feedback on initial quey Initial quey Revised quey o o o o o o known non-elevant documents o known elevant documents 11 Relevance Feedback in vecto spaces We can modify the quey based on elevance feedback and apply standad vecto space model. Use only the docs that wee maked. Relevance feedback can impove ecall and pecision Relevance feedback is most useful fo inceasing ecall in situations whee ecall is impotant 12

7 Positive vs Negative Feedback Usual choices fo paametes α, β nad γ β >> γ, i.e. geate impotance to the docs judged elevant than to the docs judged ielevant γ 0, as the docs maked ielevant ae typically nea-positive. Howeve, many systems only allow positive feedback (γ=0). α 0, in ode to pevent ovefitting, i.e. the ecessive influence of noisy chaacteistics of the docs maked (i)elevant on the esulting quey Reasonable values can be α=1, β=.75, γ=.15 The values of the paametes could be made dependent on the iteation, i.e. inceasing α and deceasing β and γ (late queies aleady incopoate the contibution of pevious feedback iteations) 13 Pobabilistic elevance feedback Rathe than eweighting in a vecto space If use has told us some elevant and ielevant documents, then we can poceed to build a classifie, such as a Naive Bayes model: P(t k R) = D k / D P(t k NR) = (N k - D k ) / (N - D ) t k = tem in document; D k = known elevant doc containing t k ; N k = total numbe of docs containing t k Moe on late lectues on pobabilistic classification This is effectively anothe way of changing the (implicit) quey tem weights But note: the above poposal peseves no memoy of the oiginal weights 14

8 Relevance Feedback: Assumptions A1: Use has sufficient knowledge fo initial quey. A2: Relevance pototypes ae well-behaved. Tem distibution in elevant documents will be simila Tem distibution in non-elevant documents will be diffeent fom those in elevant documents Eithe: All elevant documents ae tightly clusteed aound a single pototype. O: Thee ae diffeent pototypes, but they have significant vocabulay ovelap. Similaities between elevant and ielevant documents ae small 15 Violation of A1 Use does not have sufficient initial knowledge. Eamples: Misspellings (Bittany Spees). Coss-language infomation etieval (hígado). Mismatch of seache s vocabulay vs. collection vocabulay Cosmonaut/astonaut, laptop / notebook compute 16

9 Violation of A2 Thee ae seveal elevance pototypes. Eamples: Buma/Myanma/Bimania Pop stas that woked at Buge King Often: instances of a vey geneal concept 17 Relevance Feedback: Poblems Long queies ae inefficient fo typical IR engine. Long esponse times fo use. High cost fo etieval system. Patial solution: Only eweight cetain pominent tems Pehaps top 20 by tem fequency Uses ae often eluctant to povide eplicit feedback It s often hade to undestand why a paticula document was etieved afte apply elevance feedback 18

10 Relevance Feedback on the Web [in 2003: now less majo seach engines, but same geneal stoy] Some seach engines offe a simila/elated pages featue (this is a tivial fom of elevance feedback) Google (link-based) Altavista Stanfod WebBase But some don t because it s had to eplain to aveage use: Alltheweb msn Yahoo Ecite initially had tue elevance feedback, but abandoned it due to lack of use. Why? 19 Ecite Relevance Feedback Spink et al (about Ecite) Only about 4% of quey sessions fom a use used elevance feedback option Epessed as Moe like this link net to each esult But about 70% of uses only looked at fist page of esults and didn t pusue things futhe So 4% is about 1/8 of people etending seach Relevance feedback impoved esults about 2/3 of the time 20

11 Eecise Find pages like this one! What weight setting fo α,β,γ? 21 Pseudo Relevance Feedback Automatic local analysis Pseudo elevance feedback attempts to automate the manual pat of elevance feedback. Retieve an initial set of elevant documents. Assume that top m anked documents ae elevant. Do elevance feedback Mostly woks (pehaps bette than global analysis!) Found to impove pefomance in TREC ad-hoc task Dange of quey dift 22

12 Indiect elevance feedback On the web, DiectHit intoduced a fom of indiect elevance feedback. DiectHit anked documents highe that uses look at moe often. Clicked on links ae assumed likely to be elevant Assuming the displayed summaies ae good, etc. Globally: Not use o quey specific. This is the geneal aea of clicksteam mining (see Joachims wok). Applied in advetisment anking fo eample. 23 Relevance Feedback Summay Relevance feedback has been shown to be vey effective at impoving elevance of esults. Requies enough judged documents, othewise it s unstable ( 5 ecommended) Requies queies fo which the set of elevant documents is medium to lage Full elevance feedback is painful fo the use. Full elevance feedback is not vey efficient in most IR systems. Othe types of inteactive etieval may impove elevance by as much with less wok. 24

13 Quey Refomulation: Vocabulay Tools Feedback Infomation about stop lists, stemming, etc. Numbes of hits on each tem o phase Suggestions Thesauus Contolled vocabulay Bowse lists of tems in the inveted inde 25 Quey Epansion In elevance feedback, uses give additional input (elevant/non-elevant) on documents, which is used to eweight tems in the documents In quey epansion, uses give additional input (good/bad seach tem) on wods o phases. 26

14 Quey Epansion: Eample Also: see Types of Quey Epansion Global Analysis: Thesauus-based Contolled vocabulay Maintained by editos (e.g., medline, DD system) Manual thesauus E.g. MedLine: physician, syn: doc, docto, MD, medico Automatically deived thesauus (co-occuence statistics) Refinements based on quey log mining Common on the web Local Analysis: Analysis of documents in esult set 28

15 Contolled Vocabulay 29 Co-occuence Thesauus Simplest way to compute one is based on temtem similaities in C = AA T whee A is temdocument mati. w i,j = (nomalized) weighted count (t, d ) d n i j t i m d j With intege counts what do you get fo a boolean cooccuence mati? 30

16 Automatic Thesauus Geneation Eample 31 Quey Epansion: Summay Quey epansion is often effective in inceasing ecall. Not always with geneal thesaui Faily successful fo subject-specific collections In most cases, pecision is deceased, often significantly. Oveall, not as useful as elevance feedback; may be as good as pseudo-elevance feedback 32