Cross-language Retrieval Experiments at CLEF-2002

Size: px

Start display at page:

Download "Cross-language Retrieval Experiments at CLEF-2002"

Daisy Barton
6 years ago
Views:

1 Cross-language Retrieval Experiments at CLEF-2002 Aitao Chen School of Information Management and Systems University of California at Berkeley CLEF 2002 Workshop: 9-20 September, 2002, Rome, Italy

2 Talk Outline Overview of our CLEF-2002 experiments Evaluation of merging strategies German/Dutch decompounding Query expansion Conclusions

3 Overview of Multilingual Information Retrieval Experiments Query English IR/RF Documents English French/2 IR/RF French Babelfish L&H German/2 Italian/2 IR/RF/ DC IR/RF German Italian Spanish/2 IR/RF Spanish English docs French docs German docs Italian docs Spanish docs merger combined ranked list of documents

4 Multilingual Information Retrieval: Direct Merging English docs French docs Italian docs German docs Spanish docs E e E2 e2 E50 e50 E5 e5 E000 e000 F f F2 f2 F50 f50 F5 f5 F000 f000 I i I2 i2 I50 i50 I5 i5 I000 i000 G g G2 g2 G50 g50 G5 g5 G000 g000 S s S2 s2 S50 s50 S5 s5 S000 s000 () combine ranked lists; (2) sort by raw score; (3) take the top 000 docs Weakness: prone to un-comparable raw relevance scores in individual ranked lists.

5 Multilingual Information Retrieval: Normalized Merging English docs French docs Italian docs German docs Spanish docs E e E2 e2 E50 e50 E000 e000 F f F2 f2 F50 f50 F000 f000 I i I2 i2 I50 i50 I000 i000 G g G2 g2 G50 g50 G000 g000 S s S2 s2 S50 s50 S000 s000 E e/e E2 e2/e E50 e50/e E000 e000/e F f/f F2 f2/f F50 f50/f F000 f000/f I i/i I2 i2/i I50 i50/i I000 i000/i G g/g G2 g2/g G50 g50/g G000 g000/g S s/s S2 s2/s S50 s50/s S000 s000/s () combine ranked lists; (2) sort by normalized score; (3) take the top 000 docs Weakness: prone to skewed distribution of relevant documents over the document subcollections.

6 Optimal Merging (Known Relevance) (red=rel doc; black=irrel doc) Table Rank Run A Run B Run C A B C 2 A2 B2 C2 3 A3 B3 C3 4 A4 B4 C4 Table 2 Table 3 Set Optimal Ranking (0,) {A} Set 2 3 Run A (0,) {A} (,) {A2,A3} (,0) {A3} Run B (2,) {B,B2,B3} (,0) {B4} Run C (,3) {C,C2,C3,C4} Choose the set with the smallest number of irrelevant documents, but the largest number of relevant documents from the set of active sets.

7 Optimal Merging (Known Relevance) (red=rel doc; black=irrel doc) Set Optimal Ranking A Rank Run A A A2 A3 A4 Run B B B2 B3 B4 Run C C C2 C3 C (0,) {A} (,3) {C,C2,C3,C4} (,) {A2,A3} (2,) {B,B2,B3} (,0) {A4} (,0) {B4} C C2 C3 C4 A2 A3 B Set Run A Run B (2,) {B,B2,B3} Run C (,3) {C,C2,C3,C4} 9 0 B2 B3 A4 2 (,) {A2,A3} (,0) {B4} 2 B4 3 (,0) {A3}

8 Performances of MLIR with Different Merging Strategies (Topics: English,TD) (.4705) (.4773) (.4479) (.4008) (.4567) English docs French docs German docs Italian docs Spanish docs Direct Merging (.3762) Normalized Merging (.3570) Optimal Merging (.577) 72.67% 68.96% English French German Italian Spanish No. topics with no rel docs

9 Why decompounding? Topic 09: Computersicherheit (computer security) in title & desc fields, but not in the German document collection. Topic 88: The Dutch compound gekkekoeienziekte (mad cow disease) is not in the Dutch document collection. Topic 3: Fussballeuropameisterschaft in the title, but Europameisterschaft im FuBball in the desc. (B/ss) Topic 5: Scheidungsstatistiken (divorce statistics) in the title, but Statistiken uber die Scheidungsraten in the desc. In Der Spiegel : Literaturnobelpreistrager, Literatur- Nobelpreistrager, Literaturnobelpreis-Tragerin; Literaturnobelpreis v.s. Nobelpreis fur Literatur. The German translation of Latin America by Babelfish was lateinischem Amerika, not Lateinamerika. Bronchialasthma was not translated into English by Babelfish, but Bronchial and Asthma were.

10 German/Dutch Decompounding Procedure Create a German/Dutch base dictionary consisting of single words only (compounds are excluded). Decompose a compound into component words found in the German/Dutch base dictionary. Choose the decomposition with the minimum number of component words. If there are more than one decompositions having the minimum number of component words, choose the decomposition with the highest probability.

11 German Decompounding: Example Compound: fussballeuropameisterschaft (European Football Cup). Base dictionary ball europa fuss fussball meisterschaft s 2. Decompose a compound with respect to the base dictionary.. fuss ball europa meisterschaft 2. fussball europa meisterschaft 3. Choose the decomposition with the smallest number of component words. fussballeuropameisterschaft = fussball europa meisterschaft

12 German Decompounding: Example 2 Compound: wintersports (winter sports). Base dictionary port ports s sport sports winter winters 2. Decompose a compound with respect to the base dictionary. Decompositions log p(d). winter s ports winter sports winters ports Choose the most likely decomposition. wintersports = winter sports

13 Decompounding: Probability of Decomposition C = W W2 W3 W4 p( C) = p( W ) p( W2 ) p( W3 ) p( W4 ) p( w) = n tfc( i= tfc( w) w i ) Relative frequency of w in a collection. tfc(w) is the number of times word w occurs in a corpus. n is the number of unique words (including compounds) in a corpus.

14 Evaluation of Decompounding Test Without With collection Run type decompounding decompounding Change CLEF-2002 German--German % CLEF-2002 Dutch--Dutch % CLEF-200 Dutch--Dutch % CLEF-2002 English--German % (L&H) CLEF-2002 English--German % (Babelfish) CLEF-2002 French German % (Babelfish)

15 Document Ranking query t t2 2 t3 t4 t2 t3 4 t5 3 t8 documents n, { qtf, dtf, ctf }, ql, dl, i i i i =.. n cl x n qtf n dtf n ctf = ql c dl c, cl 4 2 i= i= i= i, x = log i, x = log i x = n logit( p( q, d)) = p p log = b 0 + b x + b 2 x 2 + b 3 x 3 + b 4 x 4

16 Query Expansion Select terms from top-ranked documents after the initial search. Assign weight to selected terms. Combine selected terms with original query terms.

17 Query Expansion (2) Step : term selection. relevant irrelevant w = n n 3 n 2 indexed not indexed n 4 ) Rank terms in the presumed relevant documents by w in descending order; 2) Choose the top-ranked m terms, m = 2 * average-number-of-unique-queryterms. Alternative weighting schemes include a) Maximum Likelihood Ratio; b) Chi-square statistics; c) Mutual information. n n2 n3 n4 Steps 2 & 3: term-weighting and merging. Initial query Selected terms T () T2 (2) T2 (2*0.5) T3 () T3 (*0.5) T4 (0.5) Expanded query T (.0) T2 (3.0) T3 (.5) T4 (0.5)

18 Evaluation of Query Expansion (0 terms/0 docs) Run id Run type No query expansion Query expansion Change bky2monl Dutch-Dutch % bky2mofr French-French % bky2mode German-German % bky2moit Italian-Italian % bky2moes Spanish-Spanish % bky2bienfr English-French % bky2bienfr2 English-French % bky2bidefr German-French % bky2biende English-German % bky2bifrde French-German % bky2bienit English-Italian % bky2bienes English-Spanish % bky2biennl English-Dutch %

19 Evaluation of Decompounding, Stemming and Query Expansion in Monolingual Retrieval (Topics: German, TD) decomp+stem+expan.5234 (5.8%).4393 (26.89%).457 (30.47%).4393 (26.89%) decomp+stem decomp+expan stem+expan.3859 (.47%).3633 (4.94%).445 (9.73%) decomp stem expan baseline.3465

20 English-to-French Dictionary Built from Parallel Texts. fall in sale of cars 2. ski race; car race 3. pop star; galaxy star autome 0.32 tomber 0.08 telever 0.06 race 0.60 courser 0.8 racial 0.05 star 0.62 etoile 0.3 etoiler rock music 5. lead singer rock 0.89 rocher 0.02 pierre 0.0 mener 0.3 conduire 0.08 amener 0.07 principal Run id bky2enfr3 bky2enfr4 bky2enfr5 Resource Babelfish L&H Parallel texts AP

21 Conclusions The simplest direct merging method worked better than the score-normalized method when the intermediate ranked lists were produced under similar conditions (e.g., roughly the same query length, the same number of terms selected from the same number of documents). Decompounding improved the retrieval performance of German/Dutch monolingual and cross-language retrieval to German. The margin of improvement varies from one topic set to another. Query expansion substantially improved the performance of monolingual, cross-language, and multilingual retrieval.

22 THANK YOU

Variable Latent Semantic Indexing

Variable Latent Semantic Indexing Prabhakar Raghavan Yahoo! Research Sunnyvale, CA November 2005 Joint work with A. Dasgupta, R. Kumar, A. Tomkins. Yahoo! Research. Outline 1 Introduction 2 Background