Combining linguistic resources and statistical language modeling for information retrieval

Transcription

1 Combnng lngusc resources and sascal language modelng for nformaon rereval Jan-Yun Ne RALI ep. IRO Unversy of Monreal Canada hp://.ro.umonreal.ca/~ne

2 Bref hsory of IR and NL Sascal IR f*df Aemps o negrae NL no IR Idenfy compound erms Word dsambguaon Mgaed success Sascal NL Trend: negrae sascal NL no IR language modelng 2

3 Overve Language model Ineresng heorecal frameork Effcen probably esmaon and smoohng mehods Good effecveness Lmaons Mos approaches use un-grams and ndependence assumpon Jus a dfferen ay o egh erms Exensons Inegrang more lngusc analyss erm relaonshps Expermens Conclusons 3

4 rncple of language modelng Goal: creae a sascal model so ha one can calculae he probably of a seuence of ords s = 2 n n a language. General approach: Tranng corpus s robables of he observed elemens s 4

5 5 rob. of a seuence of ords... 2 n s Elemens o be esmaed: - If h s oo long one canno observe h n he ranng corpus and h s hard generalze - Soluon: lm he lengh of h h h h n n n h 2...

6 Esmaon Hsory: shor long modelng: coarse refned Esmaon: easy dffcul Maxmum lkelhood esmaon MLE 6

7 7 n-grams Lm h o n- precedng ords Un-gram: B-gram: Tr-gram: Maxmum lkelhood esmaon MLE problem:h =0 n s n s n s 2 # # gram n un C h h C

8 Smoohng Goal: assgn a lo probably o ords or n-grams no observed n he ranng corpus MLE smoohed ord 8

9 Smoohng mehods n-gram: Change he fre. of occurrences Laplace smoohng add-one: add_ one C V nr = no. of n-grams of fre. r Good-Turng change he fre. r o r* r n r n r 9

10 0 Smoohng con d Combne a model h a loer-order model Backoff Kaz Inerpolaon Jelnek-Mercer In IR combne doc. h corpus oherse 0 f Kaz GT Kaz JM ML JM C ML ML

11 Smoohng con d rchle To-sage C f ML r C C f ML ML TS

12 Usng LM n IR rncple : ocumen : Language model M uery = seuence of ords 2 n un-grams Machng: M rncple 2: ocumen : Language model M uery : Language model M Machng: comparson beeen M and M rncple 3: Translae o 2

13 rncple : ocumen LM ocumen : Model M uery : 2 n : un-grams = M = M 2 M n M roblem of smoohng Shor documen Coarse M Unseen ords Smoohng Change ord fre. Smooh h corpus Exemple C GT ML 3

14 eermne 22 h 2 Expecaon maxmzaon EM: Choose ha maxmzes he lkelhood of he ex Inalze E-sep M-sep Loop on E and M C C C 4

15 rncple 2: oc. lkelhood / dvergence beeen M d and M ueson: Is he documen lkelhood ncreased hen a uery s submed? LR Is he uery lkelhood ncreased hen s rereved? - calculaed h M - esmaed as M C Score log M M C 5

16 6 vergence of M and M f C f C f M f M!!!! n C M M f Score *log KL: Kullback-Lebler dvergence measurng he dvergence of o probably dsrbuons Consan *log *log *log C n C n n C M M H M M H M M KL M M M M M M M M M Assume follos a mulnomal dsrbuon :

17 rncple 3: IR as ranslaon Nosy channel: message receved Transm hrough he channel and receve : prob. ha generaes : prob. of ranslang by ossbly o consder relaonshps beeen ords Ho o esmae? Berger&Laffery: seudo-parallel exs algn senence h paragraph 7

18 Summary on LM Can a uery be generaed from a documen model? oes a documen become more lkely hen a uery s submed or reverse? Is a uery a "ranslaon" of a documen? Smoohng s crucal Ofen use un-grams 8

19 Beyond un-grams B-grams MLE 2MLE 3MLE C B-erm o no consder ord order n b-grams analyss daa daa analyss 9

20 Relevance model LM does no capure Relevance Usng pseudo-relevance feedback Consruc a relevance model usng opranked documens ocumen model + relevance model feedback + corpus model 20

21 Expermenal resuls LM vs. Vecor space model h f*df Smar Usually beer LM vs. rob. model Okap Ofen smlar b-gram LM vs. un-gram LM Slgh mprovemens bu h much larger model 2

22 Conrbuons of LM o IR Well founded heorecal frameork Explo he mass of daa avalable Technues of smoohng for probably esmaon Explan some emprcal and heursc mehods by smoohng Ineresng expermenal resuls Exsng ools for IR usng LM Lemur 22

23 roblems Lmaon o un-grams: No dependence beeen ords roblems h b-grams Consder all he adacen ord pars nose Canno consder more dsan dependences Word order no alays mporan for IR Enrely daa-drven no exernal knoledge e.g. programmng compuer Logc ell hdden behnd numbers Key = smoohng Maybe oo much emphass on smoohng and oo lle on he underlyng logc rec comparson beeen and Reures ha and conan dencal ords excep ranslaon model Canno deal h synonymy and polysemy 23

24 Some Exensons Classcal LM: ocumen 2 uery nd. erms. ocumen comp.arch. uery dep. erms 2. ocumen prog. comp. uery erm relaons 24

25 Exensons : lnk erms n documen and uery ependence LM Gao e al. 04: Capure more dsan dependences hn a senence Synacc analyss Sascal analyss Only rean he mos probable dependences n he uery ho has affrmave acon affeced he consrucon ndusry 25

26 Esmae he prob. of lnks EM For a corpus C:. Inalzaon: lnk each par of ords h a ndo of 3 ords 2. For each senence n C: Apply he lnk prob. o selec he sronges lnks ha cover he senence 3. Re-esmae lnk prob. 4. Repea 2 and 3 26

27 27 Calculaon of L C L L R L L arg max arg max L L... L h L L L l l L. eermne he lnks n he reured lnks 2. Calculae he lkelhood of ords and lnks L n L... Reuremen on ords and b-erms lnks

28 Expermens Models WSJ AT FR Avg % change over % change over Avg %change over % change over Avg % change over % change over BM UG BM UG BM UG BM UG ** M ** * * BG BT * BT Table 2. Comparson resuls on WSJ AT and FR collecons. * and ** ndcae ha he dfference s sascally sgnfcan accordng o -es * ndcaes p-value < 0.05 ** ndcaes p-value < Models SJM A ZIFF Avg % change over % change over Avg %change over % change over Avg % change over % change over BM UG BM UG BM UG BM UG M * +9.54** ** * +0.38** BG * +8.96** * BT ** BT Table 3. Comparson resuls on SJM A and ZIFF collecons. * and ** ndcae ha he dfference s sascally sgnfcan accordng o -es * ndcaes p-value < 0.05 ** ndcaes p-value <

29 Exenson 2: Inference n IR Logcal deducon A B B C A C In IR: =Tsunam =naural dsaser rec machng Inference on uery Inference on doc. rec machng 29

30 Is LM capable of nference? Generave model: ~ Smoohng: C : ML change o ML E.g. =Tsunam ML naural dsaser=0 change o naural dsaser>0 No nference compuer>0 0 0 ML 30

31 Effec of smoohng? Tsunam ocean Asa compuer na.dsaser Smoohng nference Redsrbuon unformly/accordng o collecon 3

32 Expeced effec Tsunam ocean Asa compuer na.dsaser Usng Tsunam naural dsaser Knoledge-based smoohng 32

33 33 Exended ranslaon model ' ' ' ' ' ' Translaon model: ' '

34 Usng oher ypes of knoledge? fferen ays o sasfy a uery. erm recly hough ungram model Indrecly by nference hrough Wordne relaons Indrecly rough Co-occurrence relaons f UG or WN or CO 3 WN 2 CO UG C 34

35 Illusraon Cao e al. 05 WN CO 2 n 2 n WN model CO model UG model λ λ 2 λ 3 documen 35

36 Expermens Table 3: fferen combnaons of ungram model lnk model and co-occurrence model WSJ A SJM Model Avg Rec. Avg Rec. Avg Rec. UM / / /2322 CM / / /2322 LM / / /2322 UM+CM / / /2322 UM+LM / / /2332 UM+CM+LM / / /2322 UM=Ungram CM=co-occ. model LM=model h Wordne 36

37 Expermenal resuls Coll. Ungram Model LM h unue WN rel. ependency Model LM h yped WN rel. Avg Rec. Avg %change Rec. Avg %change Rec. WSJ / * 706/ * 79/272 A / ** 3523/ ** 3530/60 SJM / / /2322 Inegrang dfferen ypes of relaonshps n LM may mprove effecveness 37

38 38 oc expanson v.s. uery expanson UG 3 2 UG CO WN UG ocumen expanson 2 UG R uery expanson

39 39 Implemenng E n LM KL dvergence: ne a expanson uery log log log log ; KL Score

40 40 Expandng uery model V R ML V R ML V R ML R ML Score log log log ] [ log model Relaonal : no smoohed M ax.lkelhood ungram model : Classcal LM Relaon model

41 Ho oesmae R? Usng co-occurrence nformaon Usng an exernal knoledge base e.g. Wordne seudo-rel. feedback Oher erm relaonshps 4

42 efnng relaonal model HAL Hyperspace Analogue o Language: a specal co-occurrence marx Bruza&Song he effecs of polluon on he populaon effecs and polluon co-occur n 2 ndos L=3 HALeffecs polluon = 2 = L dsance + 42

43 From HAL o Inference relaon HAL 2 HAL HAL 2 superconducors : <U.S.:0. amercan:0.07 basc:0. bulk:0.3 called:0.5 capacy:0.08 carry:0.5 ceramc:0. commercal:0.5 consorum:0.8 cooled:0.06 curren:0.0 develop:0.2 dover:0.06 > Combnng erms: spaceprogram fferen mporance for space and program 43

44 44 From HAL o Inference relaon nformaon flo spaceprogram - {program:.00 space:.00 nasa:0.97 ne:0.97 U.S.:0.96 agency:0.95 shule:0.95 scence:0.88 scheduled:0.87 reagan:0.87 drecor:0.87 programs:0.87 ar:0.87 pu:0.87 cener:0.87 bllon:0.87 aeronaucs:0.87 saelle:0.87 > degree degree k n k V k IF k n n n degree degree

45 45 To ypes of erm relaonshp arse 2 : Inference relaonshp Inference relaonshps are less ambguous and produce less nose u&fre 93 HAL HAL HAL 2 2 V k IF k n n n degree degree

46 46. uery expanson h parse erm relaonshps log log log log log log Score E R co ML V co ML V R ML Selec a se 85 of sronges HAL relaonshps

47 47 2. uery expanson h IF erm relaonshps log log log log log log Score E R IF ML V IF ML V R ML 85 sronges IF relaonshps

48 Expermens Ba e al. 05 A89 collecon uery -50 Avgr Recall oc. Smooh. Jelnek- Merer LM baselne E h HAL E h IF E h IF & FB % % % rchle % % % Abslue % % % To- Sage Jelnek- Merer /330 rchle 569/330 Abslue 560/330 To- Sage 573/ % % % 588/330 +3% 2240/ % 2366/ % 608/330 +2% 2246/ % 2356/ % 607/330 +3% 25/ % 2289/ % 596/330 +% 222/330 +4% 2356/ % 48

49 Expermens A88-90 opcs 0-50 Avgr Recall oc. Smooh. LM baselne E h HAL E h IF E h IF & FB Jelnek- Mercer % % % rchle % % % Abslue % % % To-Sage % % % Jelnek- Mercer rchle Abslue To-Sage 306/ /330 +3% 3675/ % 356/ /330 +3% 3738/ % 303/ /330 +3% 3572/ % 334/ /330 +2% 373/ % 3895/ % 3930/ % 3842/ % 390/ % 49

50 Observaons ossble o mplemen uery/documen expanson n LM Expanson usng nference relaonshps s more conex-sensve: Beer han conexndependen expanson u&fre Every knd of knoledge alays useful coocc. Wordne IF relaonshps ec. LM h some nferenal poer 50

51 Conclusons LM = suable model for IR Classcal LM = ndependen erms n-grams ossbly o negrae lngusc resources: Term relaonshps: Whn documen and hn uery lnk consran ~ compound erm Beeen documen and uery nference Boh Auomac parameer esmaon = poerful ool for daa-drven IR Expermens shoed encouragng resuls IR orks ell h sascal NL More lngusc analyss for IR? 5