Relevance feedback and query expansion. Goal: To refine the answer set by involving the user in the retrieval process (feedback/interaction)

Transcription

1 Relevance feedback and quey epansin Gal: T efine the answe set by invlving the use in the etieval pcess (feedback/inteactin) Lcal Methds (adust the use queies) Relevance feedback Pseud ( Blind) Relevance Feedback (Glbal) indiect Relevance Feedback Glbal Methds (independent f the queies and esults) Quey epansin/efmulatin with a thesauus (WdNet) Quey epansin via autmatic thesauus geneatin Othe techniques (spelling cectin,...) 1 Relevance feedback Basic Pcedue f RF 1. The use issues a simple quey 2. The system etuns an initial set f etieval esults 3. The use maks sme f these dcuments as elevant/ielevant 4. The system cmputes a bette epesentatin f the infmatin need based n this feedback 5. The system displays a evised set f esults Repeat the pcedue ne me times. This pcess helps the use t fcalize its wn infmatin need as well. 2

2 Relevance Feedback: Eample Image seach engine 3 Results f Initial Quey 4

3 Relevance Feedback step 5 Results afte Relevance Feedback 6

4 Rcchi Algithm The Rcchi algithm incpates elevance feedback infmatin int the vect space mdel. Want t maimize sim (Q, C ) - sim (Q, C n ) The ptimal quey vect f sepaating elevant and nn-elevant dcuments (with csine sim.): 1 1 Qpt = d C N C d C d C Q pt = ptimal quey; C = set f el. dc vects; N = cllectin size Unealistic: we dn t knw elevant dcuments. d 7 The Theetically Best Quey Optimal quey nn-elevant dcuments elevant dcuments 8

5 Rcchi 1971 Algithm (SMART) Used in pactice: 1 1 qm = αq0 + β d γ D D d D q m = mdified quey vect; q 0 = iginal quey vect; α,β,γ: weights (hand-chsen set empiically); D = set f knwn elevant dc vects; D n = set f knwn ielevant dc vects New quey mves twad elevant dcuments and away fm ielevant dcuments Tadeff α vs. β and γ : If we have a lt f udged dcuments, we want a highe β and γ. Tem weight can g negative Negative tem weights ae igned Altenatively, weights can be nmalized in [0,1] n d D d n 9 Relevance feedback n initial quey Initial quey Revised quey knwn nn-elevant dcuments knwn elevant dcuments 10

6 Relevance Feedback in vect spaces We can mdify the quey based n elevance feedback and apply standad vect space mdel. Use nly the dcs that wee maked. Relevance feedback can impve ecall and pecisin Relevance feedback is mst useful f inceasing ecall in situatins whee ecall is imptant 11 Psitive vs Negative Feedback Usual chices f paametes α, β nad γ β >> γ, i.e. geate imptance t the dcs udged elevant than t the dcs udged ielevant γ 0, as the dcs maked ielevant ae typically nea-psitive. Hweve, many systems nly allw psitive feedback (γ=0). α 0, in de t pevent vefitting, i.e. the ecessive influence f nisy chaacteistics f the dcs maked (i)elevant n the esulting quey Reasnable values can be α=1, β=.75, γ=.15 The values f the paametes culd be made dependent n the iteatin, i.e. inceasing α and deceasing β and γ (late queies aleady incpate the cntibutin f pevius feedback iteatins) 12

7 Pbabilistic elevance feedback Rathe than eweighting in a vect space If use has tld us sme elevant and ielevant dcuments, then we can pceed t build a classifie, such as a Naive Bayes mdel: P(t k R) = D k / D P(t k NR) = (N k - D k ) / (N - D ) t k = tem in dcument; D k = knwn elevant dc cntaining t k ; N k = ttal numbe f dcs cntaining t k Me n late lectues n pbabilistic classificatin This is effectively anthe way f changing the (implicit) quey tem weights But nte: the abve ppsal peseves n memy f the iginal weights 13 Relevance Feedback: Assumptins A1: Use has sufficient knwledge f initial quey. A2: Relevance pttypes ae well-behaved. Tem distibutin in elevant dcuments will be simila Tem distibutin in nn-elevant dcuments will be diffeent fm thse in elevant dcuments Eithe: All elevant dcuments ae tightly clusteed aund a single pttype. O: Thee ae diffeent pttypes, but they have significant vcabulay velap. Similaities between elevant and ielevant dcuments ae small 14

8 Vilatin f A1 Use des nt have sufficient initial knwledge. Eamples: Misspellings (Bittany Spees). Css-language infmatin etieval (hígad). Mismatch f seache s vcabulay vs. cllectin vcabulay Csmnaut/astnaut, laptp / ntebk cmpute 15 Vilatin f A2 Thee ae seveal elevance pttypes. Eamples: Buma/Myanma/Bimania Pp stas that wked at Buge King Often: instances f a vey geneal cncept 16

9 Relevance Feedback: Pblems Lng queies ae inefficient f typical IR engine. Lng espnse times f use. High cst f etieval system. Patial slutin: Only eweight cetain pminent tems Why? Pehaps tp 20 by tem fequency Uses ae ften eluctant t pvide eplicit feedback It s ften hade t undestand why a paticula dcument was etieved afte apply elevance feedback 17 Relevance Feedback n the Web [in 2003: nw less ma seach engines, but same geneal sty] Sme seach engines ffe a simila/elated pages featue (this is a tivial fm f elevance feedback) Ggle (link-based) Altavista Stanfd WebBase But sme dn t because it s had t eplain t aveage use: Alltheweb msn Yah Ecite initially had tue elevance feedback, but abandned it due t lack f use. Why? 18

10 Ecite Relevance Feedback Spink et al Only abut 4% f quey sessins fm a use used elevance feedback ptin Epessed as Me like this link net t each esult But abut 70% f uses nly lked at fist page f esults and didn t pusue things futhe S 4% is abut 1/8 f peple etending seach Relevance feedback impved esults abut 2/3 f the time 19 Pseud Relevance Feedback Autmatic lcal analysis Pseud elevance feedback attempts t autmate the manual pat f elevance feedback. Retieve an initial set f elevant dcuments. Assume that tp m anked dcuments ae elevant. D elevance feedback Mstly wks (pehaps bette than glbal analysis!) Fund t impve pefmance in TREC ad-hc task Dange f quey dift 20

11 Indiect elevance feedback On the web, DiectHit intduced a fm f indiect elevance feedback. DiectHit anked dcuments highe that uses lk at me ften. Clicked n links ae assumed likely t be elevant Assuming the displayed summaies ae gd, etc. Glbally: Nt use quey specific. This is the geneal aea f clicksteam mining 21 Relevance Feedback Summay Relevance feedback has been shwn t be vey effective at impving elevance f esults. Requies enugh udged dcuments, thewise it s unstable ( 5 ecmmended) Requies queies f which the set f elevant dcuments is medium t lage Full elevance feedback is painful f the use. Full elevance feedback is nt vey efficient in mst IR systems. Othe types f inteactive etieval may impve elevance by as much with less wk. 22

12 Quey Refmulatin: Vcabulay Tls Feedback Infmatin abut stp lists, stemming, etc. Numbes f hits n each tem phase Suggestins Thesauus Cntlled vcabulay Bwse lists f tems in the inveted inde 23 Quey Epansin In elevance feedback, uses give additinal input (elevant/nn-elevant) n dcuments, which is used t eweight tems in the dcuments In quey epansin, uses give additinal input (gd/bad seach tem) n wds phases. 24

13 Quey Epansin: Eample Als: see Types f Quey Epansin Glbal Analysis: Thesauus-based Cntlled vcabulay Maintained by edits (e.g., medline) Manual thesauus E.g. MedLine: physician, syn: dc, dct, MD, medic Autmatically deived thesauus (c-ccuence statistics) Refinements based n quey lg mining Cmmn n the web Lcal Analysis: Analysis f dcuments in esult set 26

14 Cntlled Vcabulay 27 C-ccuence Thesauus Simplest way t cmpute ne is based n temtem similaities in C = AA T whee A is temdcument mati. w i, = (nmalized) weighted cunt (t i, d ) t i m d n With intege cunts what d yu get f a blean cccuence mati? 28

15 Autmatic Thesauus Geneatin Eample 29 Quey Epansin: Summay Quey epansin is ften effective in inceasing ecall. Nt always with geneal thesaui Faily successful f subect-specific cllectins In mst cases, pecisin is deceased, ften significantly. Oveall, nt as useful as elevance feedback; may be as gd as pseud-elevance feedback 30