Anomalous State of Knowledge. Administrative. Relevance Feedback Query Expansion" computer use in class J hw3 out assignment 3 out later today

Transcription

1 Relevance Feedback Query Epansin" David Kauchak cs458 Fall 2012 adapted frm: Kevin Knight, Administrative Anmalus State f Knwledge n n n cmputer use in class J hw3 ut assignment 3 ut later tday n due date? Basic parad: Infrmatin needs arise because the user desn t knw smething Search systems are designed t satisfy these needs, but the user needs t knw what he is lking fr Hwever, if the user knws what he s lking fr, there may nt be a need t search in the first place

2 What shuld be returned? What is actually returned Similar pages Relevance feedback User prvides feedback n relevance f dcuments in the initial set f results: What did similar pages d? n n n n User issues a query The user marks sme results as relevant r nn-relevant The system cmputes a better results based n the feedback May iterate Des this slve ur prblem?

3 An eample Results fr initial query Image search engine: Relevance Feedback Results after Relevance Feedback

4 Ideas? Relevance feedback Fr ranked mdels we represent ur query as a vectr f weights, which we view as a pint in a high dimensinal space We want t bias the query twards dcuments that the user selected (the relevant dcuments ) We want t bias the query away frm dcuments that the user did nt select (the nn-relevant dcuments ) Initial query Δ knwn nn-relevant dcuments knwn relevant dcuments Relevance feedback Relevance feedback n initial query Initial query Δ Initial query Δ Δ knwn nn-relevant dcuments knwn relevant dcuments Revised query Hw can we mve the query? knwn nn-relevant dcuments knwn relevant dcuments

5 Rcchi Algrithm The Rcchi algrithm uses the vectr space mdel t pick a better query Rcchi seeks the query q pt that maimizes the difference between the query similarity with the relevant set f dcuments (C r ) vs. the nn-relevant set f dcuments (C nr )! q pt = argma[sim(!! q q,c r ) " sim( q!,c nr )] Centrid The centrid is the center f mass f a set f pints 1 µ( C) = C d C Where is the centrid? d Rcchi Algrithm Find the new query by mving it twards the centrid f the relevant queries and away frm the centrid f the nn-relevant queries Rcchi in actin query vectr = riginal query vectr + relevant vectr " nn " relevant vectr! q pt = 1! # d j $ 1! # C r C! nr! d j "C r d j "C nr d j Original query Relevant centrid Nn-relevant centrid (+) (-) New query ?

6 Rcchi in actin Rcchi in actin surce: Fernand Diaz query vectr = riginal query vectr + relevant vectr " nn " relevant vectr Original query Relevant centrid Nn-relevant centrid (+) (-) New query Rcchi in actin surce: Fernand Diaz User feedback: Select what is relevant surce: Fernand Diaz

7 Results after relevance feedback surce: Fernand Diaz Any prblems with this?! q pt = 1! # d j $ 1! # C r C! nr! d j "C r d j "C nr d j C r and C nr are all the relevant and nn-relevant dcuments We get a biased sample! Rcchi 1971 Algrithm (SMART) Relevance Feedback in vectr spaces Used in practice: 1 1 qm = αq D 0 + β d j γ Dr d j Dr D r = set f knwn relevant dc vectrs D nr = set f knwn irrelevant dc vectrs n Different frm C r and C nr q m = mdified query vectr q 0 = riginal query vectr α,β,γ: weights (hand-chsen r set empirically) nr d j d j Dnr Relevance feedback can imprve recall and precisin Hw might it imprve each f these? Which d yu think it s mre likely t imprve? New query mves tward relevant dcuments and away frm irrelevant dcuments

8 Relevance Feedback in vectr spaces Relevance feedback can imprve recall and precisin Relevance feedback is mst useful fr increasing recall in situatins where recall is imprtant n Users can be epected t review results and t take time t iterate Psitive feedback is mre valuable than negative feedback (s, set γ < β; e.g. γ = 0.25, β = 0.75). Many systems nly allw psitive feedback (γ=0) Anther eample Initial query: New space satellite applicatins , 08/13/91, NASA Hasn t Scrapped Imaging Spectrmeter , 07/09/91, NASA Scratches Envirnment Gear Frm Satellite Plan , 04/04/90, Science Panel Backs NASA Satellite Plan, But Urges Launches f Smaller Prbes , 09/09/91, A NASA Satellite Prject Accmplishes Incredible Feat: Staying Within Budget , 07/24/90, Scientist Wh Epsed Glbal Warming Prpses Satellites fr Climate Research , 08/22/90, Reprt Prvides Supprt fr the Critics Of Using Big Satellites t Study Climate , 04/13/87, Arianespace Receives Satellite Launch Pact Frm Telesat Canada , 12/02/87, Telecmmunicatins Tale f Tw Cmpanies + User then marks relevant dcuments with +. Epanded query after relevance feedback new space satellite applicatin nasa es launch aster instrument arianespace bundespst ss rcket scientist bradcast earth il measure Results fr epanded query , 07/09/91, NASA Scratches Envirnment Gear Frm Satellite Plan , 08/13/91, NASA Hasn t Scrapped Imaging Spectrmeter , 08/07/89, When the Pentagn Launches a Secret Satellite, Space Sleuths D Sme Spy Wrk f Their Own , 07/31/89, NASA Uses Warm Supercnductrs Fr Fast Circuit , 12/02/87, Telecmmunicatins Tale f Tw Cmpanies , 07/09/91, Sviets May Adapt Parts f SS-20 Missile Fr Cmmercial Use , 07/12/88, Gaping Gap: Pentagn Lags in Race T Match the Sviets In Rcket Launchers , 06/14/90, Rescue f Satellite By Space Agency T Cst $90 Millin

9 Epanded query after relevance feedback new space satellite applicatin nasa es launch aster instrument arianespace bundespst ss rcket scientist bradcast earth il measure Any prblem with this? Relevance Feedback: Prblems Lng queries are inefficient fr typical IR engine n Lng respnse times fr user n High cst fr retrieval system n Partial slutin: n Only reweight certain prminent terms n Perhaps tp 20 by term frequency Users are ften reluctant t prvide eplicit feedback It s ften harder t understand why a particular dcument was retrieved after applying relevance feedback Will relevance feedback wrk? RF assumes the user has sufficient knwledge fr initial query Brittany Speers hígad Csmnaut Misspellings - Brittany Speers Crss-language infrmatin retrieval hígad Mismatch f searcher s vcabulary vs. cllectin vcabulary: csmnaut/astrnaut

10 Relevance Feedback n the Web Sme search engines ffer a similar/related pages feature (this is a trivial frm f relevance feedback) n Ggle (used t ) n Altavista n Stanfrd WebBase But sme dn t because it s hard t eplain t average user: n Ggle n Alltheweb n msn n Yah n Ecite initially had true relevance feedback, but abandned it due t lack f use Ecite Relevance Feedback Spink et al Only abut 4% f query sessins frm a user used relevance feedback ptin n Epressed as Mre like this link net t each result But abut 70% f users nly lked at the first page f results and didn t pursue things further n S 4% is abut 1/8 f peple etending search Relevance feedback imprved results abut 2/3rds f the time Pseud relevance feedback Pseud-relevance algrithm: n Retrieve a ranked list f hits fr the user s query n Assume that the tp k dcuments are relevant. n D relevance feedback (e.g., Rcchi) Pseud relevance feedback Pseud-relevance algrithm: n Retrieve a ranked list f hits fr the user s query n Assume that the tp k dcuments are relevant. n D relevance feedback (e.g., Rcchi) Wrks very well n average Hw well d yu think it wrks? Any cncerns? But can g hrribly wrng fr sme queries Several iteratins can cause query drift What is query drift? n

11 Epanding the query We wuld like t suggest alternative query frmulatins t the user with the gal f: n increasing precisin n increasing recall Increasing precisin Query assist: n Generally dne by query lg mining n Recmmend frequent recent queries that cntain partial string typed by user What are methds we might try t accmplish this? Increasing precisin Increasing recall: query epansin Autmatically epand the query with related terms and run thrugh inde Spelling crrectin can be thught f a special case f this csmnaut csmnaut astrnaut space pilt Hw might we cme up with these epansins?

12 Hw d we augment the user query? Eample f manual thesaurus Manual thesaurus n E.g. MedLine: physician, syn: dc, dctr, MD, medic n Wrdnet Glbal Analysis: (static; f all dcuments in cllectin) n Autmatically derived thesaurus n (c-ccurrence statistics) n Refinements based n query lg mining n Cmmn n the web Lcal Analysis: (dynamic) n Analysis f dcuments in result set Thesaurus-based query epansin Fr each term, t, in a query, epand the query with synnyms and related wrds f t frm the thesaurus n feline feline cat May weight added terms less than riginal query terms. Autmatic thesaurus generatin Given a large cllectin f dcuments, hw might we determine if tw wrds are synnyms? Tw wrds are synnyms if they c-ccur with similar wrds May significantly decrease precisin, particularly with ambiguus terms n interest rate interest rate fascinate evaluate There is a high cst f manually prducing a thesaurus n And fr updating it fr scientific changes I drive a car I bught new tires fr my car can I hitch a ride with yu in yur car I drive an autmbile I bught new tires fr my autmbile can I hitch a ride with yu in yur autmbile

13 Autmatic thesaurus generatin Autmatic Thesaurus Generatin Eample Given a large cllectin f dcuments, hw might we determine if tw wrds are synnyms? Tw wrds are synnyms if they c-ccur with similar wrds I drive a car I bught new tires fr my car can I hitch a ride with yu in yur car I drive an autmbile I bught new tires fr my autmbile can I hitch a ride with yu in yur autmbile Autmatic Thesaurus Generatin Discussin Quality f assciatins is usually a prblem Term ambiguity may intrduce irrelevant statistically crrelated terms n Apple cmputer Apple red fruit cmputer Discussin Certain query epansin techniques have thrived and many have disappeared (particularly fr web search). Why? Which nes have survived? Since terms are highly crrelated anyway, epansin may nt retrieve many additinal dcuments

14 IR: tuching base