Query Expansion. Lecture Objectives. Text Technologies for Data Science INFR Learn about Query Expansion. Implement: 10/24/2017

Transcription

1 Tet Technlgies fr Data Science INFR11145 Query Epansin Instructr: Walid Magdy 24-Oct-2017 Lecture Objectives Learn abut Query Epansin Query epansin methds Relevance feedback in IR Rcchi s algrithm PRF Implement: Rchhi RF 2 1

2 Query Epansin Query: representatin f user s infrmatin need Many times it can be subptimal Different wrds can have the same meaning replacement, replace, replacing, replaced Stemming g, gne, went Lemmatisatin (NLP) car, vehicle, autmbile?? US, USA, the states, united states f America?? Stemming/Lemmatisatin culd be applied t nrmalise dcument and queries Research shw that n significant difference between bth Query Epansin (QE) add mre wrds f the same meaning t yur query fr better retrieval 3 Query Epansin: Methds Thesaurus Grup wrds int sets f synnyms (synsets) Typically gruping is n the wrd level (neglects cntet) Manually built: e.g. WrdNet NLTK wrdnet: Autmatically built: Wrds c-ccurence Parallel crpus f translatins Retrieved dcuments-based epansin Relevance feedback Pseud (Blind) relevance feedback Query lgs 4 2

3 Autmatic Thesaurus: c-ccurence Wrds c-ccurring in a dcument/paragraph are likely t be (in sme sense) similar r related in meaning Built using cllectin matri (term-dcument matri) Fr a cllectin matri A, where A t,d is the nrmalised weight f term t in dcument d, similarity matri culd be calculated as fllws: C = A.A T where, C u,v is the similarity scre between terms u and v. The higher the scre, the mre similar the terms Advantage: unsupervised Disadvantage: related wrds mre than real synnyms 5 Autmatic Thesaurus: c-ccurence Eample 6 3

4 Autmatic Thesaurus: parallel crpus Parallel crpus are the main training resurce fr machine translatin systems Nature: sets f tw parallel sentences in tw different languages (surce and target language) Idea: Mre than ne wrd in language X can be translated int the same wrd in language Y these wrds in language X culd be cnsidered synsets Requirement: the presence f parallel crpus (training data) supervised methd 7 Autmatic Thesaurus: parallel crpus English French Align Sentences Remve Stpwrds Stem Wrds Align Terms EN FR terms dic. FR EN terms dic. EN EN terms dic. Backff Alignment 8 4

5 Autmatic Thesaurus: parallel crpus Eample mtr weight travel clr link mtr 0.63 engin 0.36 weight 0.86 wt 0.14 travel 0.67 mve 0.19 displac 0.14 clr 0.56 clur 0.25 dye 0.19 link 0.4 cnnect 0.18 bnd 0.17 crsslink0.13 bind 0.12 clth tube area game play fabric 0.36 clth 0.3 garment 0.2 tissu 0.14 tube 0.88 pipe 0.12 area 0.4 zne 0.23 regin 0.2 surfac 0.17 set 0.6 game 0.4 set 0.3 play 0.24 read 0.17 game 0.16 reprduc0.1 9 Thesaurus-based QE Wrks fr very specific applicatins (e.g. medical dmain) Many times fails t imprve retrieval Smetimes reduces bth precisin and recall Hw? When it wrks, it is hard t get a cnsistent perfrmance ver all queries: Imprves sme, and reduces thers. Significant? Why it fails? Lack f cntet Current research: wrd embeddings N cnsistent imprvement still 10 5

6 Relevance Feedback Idea: let user give feedback t the IR system abut samples f what is relevant and what is nt. User feedback n relevance f dcs in initial results User issues a (shrt, simple) query The user marks sme results as relevant r nn-relevant. The system cmputes a better representatin f the infrmatin need based n feedback. Relevance feedback can g thrugh ne r mre iteratins Frm user perspective: it may be difficult t frmulate a gd query when yu dn t knw the cllectin well, BUT easier t judge particular dcuments 11 Eample 1: Image Search 12 6

7 Eample 2: Tet Search Initial query: New space satellite applicatins Initial Results 1. NASA Hasn t Scrapped Imaging Spectrmeter 2. NASA Scratches Envirnment Gear Frm Satellite Plan 3. Science Panel Backs NASA Satellite Plan, But Urges Launches f Smaller Prbes 4. A NASA Satellite Prject Accmplishes Incredible Feat: Staying Within Budget 5. Scientist Wh Epsed Glbal Warming Prpses Satellites fr Climate Research 6. Reprt Prvides Supprt fr the Critics Of Using Big Satellites t Study Climate 7. Arianespace Receives Satellite Launch Pact Frm Telesat Canada 8. Telecmmunicatins Tale f Tw Cmpanies User then marks relevant dcuments with + System learns new terms 13 New terms cmmn in selected dcs new space satellite applicatin nasa es launch aster instrument rianespace bundespst ss rcket scientist bradcast earth il measure 14 7

8 Adding new terms t the query 1. NASA Scratches Envirnment Gear Frm Satellite Plan 2. NASA Hasn t Scrapped Imaging Spectrmeter 3. When the Pentagn Launches a Secret Satellite, Space Sleuths D Sme Spy Wrk f Their Own 4. NASA Uses Warm Supercnductrs Fr Fast Circuit 5. Telecmmunicatins Tale f Tw Cmpanies 6. Sviets May Adapt Parts f SS-20 Missile Fr Cmmercial Use 7. Gaping Gap: Pentagn Lags in Race T Match the Sviets In Rcket Launchers 8. Rescue f Satellite By Space Agency T Cst $90 Millin Hpefully better results! 15 Theretical Optimal Query Fund clser t rel dcs and away frm irrel nes. Challenge: we dn t knw the truly relevant dcs Optimal Query Q nn-relevant dcuments relevant dcuments 16 8

9 Rcchi s Algrithm Key Cncept: Vectr Centrid Recall that, in VSM, we represent dcuments as pints in a high-dimensinal space The centrid is the centre mass f a set f pints Ԧμ C = 1 C ԦdεC Ԧd where C is a set f dcuments. Intrduced Rcchi Algrithm: thery Rcchi seeks the query Ԧq pt that maimizes Ԧq pt = argma[sim Ԧq, C rel sim Ԧq, C irrel ] q Fr Csine similarity Ԧq pt = 1 d C rel j d j C rel 1 C irrel d j C rel d j Ԧq pt = Ԧμ C rel Ԧμ C irrel 18 9

10 Rcchi Algrithm: in practice Only small set f dcs are knwn t be rel r irrel Ԧq m = α Ԧq 0 + β 1 d D rel j γ d j D rel 1 D irrel d j D irrel d j Ԧq 0 = riginal query vectr D rel = set f knwn relevant dc vectrs D irrel = set f knwn nn-relevant dc vectrs Ԧq m = mdified query vectr α = riginal query weights (hand-chsen r set empirically) β = psitive feedback weight γ = negative feedback weight New query mves tward relevant dcuments and away frm nn-relevant dcuments 19 Ntes abut setting weights: α, β, γ Values f β, γ cmpared t α are set high when large judged dcuments are available. In practice, +ve feedback is mre valuable than -ve feedback (usually, set β>γ) Many systems nly allw psitive feedback (γ=0). Or, use nly highest-ranked negative dcument. When γ>0, sme weights in query vectr can g -ve. In practice, tp n t terms in d j D rel are nly selected n = 5 50 Tp n t are identified using e.g. TFIDF 20 10

11 Effect f Relevance Feedback n Query Initial Query Mdified Query Q 0 Q m knwn nn-relevant dcuments knwn relevant dcuments 21 Effect f Relevance Feedback n Retrieval Relevance feedback can imprve recall and precisin In practice, relevance feedback is mst useful fr increasing recall in situatins where recall is imprtant. Empirically, ne rund f relevance feedback is ften very useful. Tw runds is smetimes marginally useful

12 Relevance Feedback: Issues Lng queries are inefficient fr typical IR engine. High cst fr retrieval system. Lng respnse times fr user. It s ften harder t understand why a particular dcument was retrieved after applying relevance feedback Users are ften reluctant t prvide eplicit feedback 23 Relevance Feedback: Evaluatin Assess n all dcuments in the cllectin Spectacular imprvements, but it s cheating! Use dcuments in residual cllectin (set f dcuments minus thse assessed relevant) fr secnd result set Measures usually then lwer than fr riginal query Relative perfrmance f RF variants can be validly cmpared Hard t cmpare with and withut RF Use tw cllectins each with their wn relevance assessments q 0 and user feedback frm first cllectin Bth q 0 and q m run n secnd cllectin and measured User Studies (time-based cmparisn) 24 12

13 Relevance Feedback: Evaluatin True evaluatin f usefulness must cmpare t ther methds taking the same amunt f time. Practically: User revises and resubmits query Users may prefer revisin/resubmissin t having t judge relevance f dcuments. Useful fr query suggestin t ther users Is there a way t apply relevance feedback withut user s input? 25 Pseud (Blind) Relevance Feedback Slves the prblem f users hate t prvide feedback Feedback is applied blindly (PRF) Autmates the manual part f true relevance feedback. Algrithm: Retrieve a ranked list f hits fr the user s query Assume that the tp k dcuments are relevant D relevance feedback (e.g. Rcchi) Typically applies nly psitive relevance feedback (γ=0) Mstly wrks Still can g hrribly wrng fr sme queries (when tp k dcs are nt relevant) Several iteratins can lead t query drift 26 13

14 PRF (BRF) Was prven t be useful fr many IR applicatins News search (learn names and entities) Scial media search (learn hashtags) Web search (implicit feedback is used mre = clicks) Sme dmains are mre challenging Patent search Tp dcuments are usually nt relevant Patent tet in general is unclear/cnfusing PRF is the mst basic QE methd fr IR Unsupervised Language independent Des nt require any kind f language resurces 27 PRF (BRF): Evaluatin In practice, different number f feedback dcs (n d ) and terms (n t ) are usually tested fr PRF n d : 1 50 n t : 5 50 Results f PRF are directly cmpared t baseline (with n PRF) It is nt cnsidered cheating in this case. Why? It is essential t shw that imprvement is significant, and preferred t shw the % f queries imprved vs degraded

15 Summary QE: autmatically add mre terms t user s query t better match relevant dcs QE via thesaurus Manual/autmatic thesaurus: useful fr specific applicatins Fail when cntet is imprtant Relevance feedback Get samples f rel/irrel dcs fr etracting QE useful terms Rcchi s is ne f the mst cmmn algrithms fr query mdificatin PRF Skips user s input fr the feedback prcess Fund t be useful in many applicatins 29 Resurces Tet bk 1: Intr t IR, Chapter 9 Tet bk 2: IR in Practice, Chapter 6.2, 6.3 Reading: Magdy W. and G. J. F. Jnes. A Study n Query Epansin Methds fr Patent Retrieval. PAIR CIKM