MSU at ImageCLEF: Cross Language and Interactive Image Retrieval

Transcription

1 MSU at ImageCLEF: Cross Language and Interactve Image Retreval Vneet Bansal, Chen Zhang, Joyce Y. Cha, Rong Jn Department of Computer Scence and Engneerng, Mchgan State Unversty East Lansng, MI48824, U.S.A. Abstract. In ths report, we descrbe our studes wth cross language and nteractve mage retreval n ImageCLEF Typcal cross language retreval requres specal lngustc resources, such as blngual dctonares. In ths study, we focus on the ssue of how to acheve good retreval performance gven only an onlne translaton system. We compare two approaches,.e., a translatonbased approach and a model-based approach, and fnd that the later one performs substantally better than the former one. For nteractve mage retreval, we nvestgated the potental use of user relevance feedback (URF), whch was desgned to address the msmatch problem between user queres and system descrptons. Our strategy s to let the system select mportant terms for user feedback before expandng queres. However, our prelmnary results appear to ndcate that the URF approach developed at the current stage s not workng. We report our current nvestgaton and dscuss lessons learned from ths experence. 1 Introducton Emprcal studes have shown that usng mage features to fnd smlar mages s usually nsuffcent [15]. Frst, t s dffcult for users to specfy vsual queres wth low-level vsual features. Second, low level mage features cannot precsely descrbe user nformaton needs. There s a gap between low-level vsual descrptons and user s semantc expectaton [10]. Text queres, on the other hand, are more ntutve and natural for users to specfy ther nformaton needs and expectatons. In ths year s ImageCLEF, we nvestgated two challengng tasks related to textbased mage retreval: 1) Gven mage descrptons n one language and user query n another language, how to effectvely retreve mages usng cross language retreval? In partcular, gven lmted blngual resources (e.g., the onlne blngual translaton system), how to mprove the accuracy of cross lngual nformaton retreval? 2) Gven a target mage n user s mnd, how to nteractvely help users to fnd such an mage. In partcular, we nvestgated the use of user relevance feedback n such a task.

2 In the followng sectons, we devote two sectons to these two tasks respectvely. 2 Cross Language Retreval Usng only an Onlne Translaton System Cross lngual retreval has been one of the major research areas n nformaton retreval durng last few years [1, 2, 5-7, 9]. Most cross lngual retreval algorthms fall nto two categores: the translaton-based approaches, and the approaches based on statstcal models. A smple translaton-based approach wll translate a query nto the language of documents, and relevant documents wll be found by matchng the translated queres wth the documents[1]. Dfferent algorthms can be appled to translate queres, rangng from the smplest one that s based on blngual dctonares to the sophstcated one that s based on a full-scale machne translaton system. Compared to dctonarybased translaton, usng a full-scale machne translaton system has the advantage n that the ambguty of a query s reduced by a full-scale translaton system and only the best translaton of the query s used. However, on the other hand, a cross lngual approach based on the full-scale translaton system can perform poorly f a query s truly ambguous and multple possble translatons need to be consdered. In those cases, dctonary-based translaton approaches for cross lngual retreval wll have advantages because t nclude all possble translatons of query words. Thus, a good cross lngual retreval system should be able to, on one hand, reduce the uncertanty n translatng queres when possble, and on the hand, mantan the uncertanty of query translaton f the query s ambguous. A model-based approach usually utlzes the exstng statstcal machne translaton models that were developed by the IBM group [16]. Gven a translaton model θ, the relevance of a document d to a gven query q s computed as pq ( dθ ; ), whch s the lkelhood of translatng document d nto query q. Compared to the translaton-based approaches, the model-based approaches have advantage n that by usng the translaton probabltes learned from a parallel corpus, we are able to reduce translaton ambguty and yet mantan the uncertanty n translaton at the same tme. Ths s done through the adjustment of translaton probabltes: an unlkely translaton wll be assgned wth a small probablty; meanwhle equally lkely translatons of a query wll be assgned wth smlar translaton probabltes. However, n order to buld a statstcal translaton model, a suffcently large blngual parallel corpus s requred. Acqurng a large parallel corpus s usually expensve and tme consumng, especally for mnor languages. In ths report, we study an approach that frst utlzes the onlne translaton system to create a blngual parallel corpus and then learns a statstcal translaton model based on the created blngual corpus. Unlke the translaton based approaches where only the best translaton s used n nformaton retreval, ths approach mantans the uncertanty n translaton and therefore wll be more robust to the translaton errors. On the other hand, unlke the typcal model-based approaches where a large blngual parallel corpus s requred, ths approach automatcally creates a blngual parallel corpus by applyng the onlne translaton system to translate test documents nto the

3 language of queres. In the followng subsectons, we wll frst overvew the statstcal machne translaton model, and then dscuss the emprcal results wth the proposed approach. 2.1 A Statstcal Translaton Model for Cross Language Informaton Retreval denoted by { j } j For the convenence of dscusson, let s assume that the language of queres s Chnese and the language of documents s Englsh. Let the set of translaton probabltes e c e c θ = tw ( w). Each tw ( w ) s the probablty that translates a Ch- c e nese word w j nto an Englsh word w. The key to statstcal translaton model for cross lngual nformaton retreval s to automatcally learn the set of word translaton probabltes from a parallel corpus. Let the blngual parallel corpus for tranng a { } = 1 N c e c e statstcal translaton model be denoted by Ω= ( s, s ). Each (, ) s s s a transla- c ton par n whch sentence s s the Chnese translaton of sentence s e. N s the total number of translaton pars n the corpus Ω. Accordng to the IBM translaton e c model, p( s s ; θ ),.e., the probablty of translaton an Englsh sentence s e nto a c Chnese sentence s, can be wrtten as: e e ow ( k, s ) V e Vc e c c c e c p( s s ; θ ) o( wk, ) t( wj wk) s j = 1 k = 1 where V e and V c stand for the sze for Chnese vocabulary and Englsh vocabulary, c c respectvely. ow ( k, s ) represents the occurrence of Chnese word w k c n Chnese c e e sentence s. So does ow ( k, s ). Thus, n order to learn translaton probabltes, we can maxmze the log-lkelhood of all translaton pars used for tranng,.e. θ = arg max l( Ω ; θ) = arg max log p( s s ; θ) θ Θ θ Θ = 1 A well-known Expectaton Maxmzaton algorthm can be used to effcently learn the optmal translaton probabltes. More detals can be found n [3]. Fnally, n order to estmate the relevancy of a document d to a query q, probablty p( q d ) s estmated usng the followng expresson: c e e c e e e e c e e log p( q d ; θ ) = log dq p( q q ) p( q d ) p( w q )log p( w d ) c c e c e e ow ( j, q ) pw ( wj) log pw ( d ) j More detals of applyng statstcal translaton model to cross lngual nformaton retreval can be found n [16]. N e c

4 2.2 Our Approach: Tranng a Statstcal Model Usng an Onlne Translaton System Gven the success of the statstcal translaton model for cross lngual nformaton retreval n the TREC evaluatons [13, 14], we would lke to apply t to the cross language mage retreval. However, the bggest problem s to acqure a blngual parallel corpus that shares the smlar content as the text collecton used n the ImageCLEF evaluaton. In order to acqure a blngual corpus, we tred a smple strategy. We frst appled an onlne translaton system to translate the textual descrptons n ImageCLEF nto Chnese sentences. To enhance the dversty of our translaton pars, the Chnese sentences that are generated by the onlne translaton system are further translated back nto Englsh sentences. The fnal blngual corpus s created by aggregatng all the translaton pars together. The onlne translaton system used n our experment s Systran ( Wth the acqured translaton pars, we now can apply the statstcal translaton model to automatcally learn translaton probabltes between Chnese words and Englsh words. Examples of learned translaton probabltes are lsted n Table 1. Note that all Englsh words are stemmed usng the Porter algorthm. Table 1. Examples of translaton probabltes learned from the blngual parallel corpus that s generated by the onlne translaton model. All the Englsh words are stemmed. Chnese Englsh Prob. Chnese Englsh Prob. tower cathedr turret st pnnacl ona buld dunblan clock el squar durham spre andrew church elgn transept dunkeld hous transept The retreval performance usng statstcal translaton model for cross lngual retreval s lsted n Table 2 under the column enttled as Model-based. For the purpose of comparson, we also run the smple translaton-based approach, whch apples the onlne translaton system to translate each Chnese query nto an Englsh query. The results of ths translaton-based approach are also ncluded n Table 2 under the column enttled as Translaton-based. As ndcated n Table 2, the approach based on the statstcal translaton model performs substantally better than the smple translaton-based approach n terms of almost every metrc. In partcular, the major dfference between these two approaches les n the regon when only the top retreved documents are examned. For example, when only the frst fve documents are examned, the precson for the translatonbased approach s only 28.8%, whle the precson for the approach based on statstcal translaton model s 41.6%. Ths fact s further confrmed by the precson results

5 for the low recall ponts. For example, when systems recall 10% of the relevant documents, the precson for the translaton-based approach s only 37.4%, whle the precson for the model-based approach s above 50%. Thus, we conclude that the proposed approach s a better way of utlzng the onlne translaton system for cross lngual nformaton retreval than the smple translaton-based approach. Table 2. Retreval results for both the translaton-based approach and the approach based on the statstcal translaton model. Recall@ Translaton-based Model-based Avg Prec Prec@ 5 doc doc doc Interactve Image Retreval: User Relevance Feedback Compared to example-based mage retreval, text-based mage retreval provdes an ntutve and natural means for users to specfy ther nformaton needs and expectatons. However, text queres also face many challenges [8]. One major problem concerns both the sparsty and nconsstency of textual descrptons [12]. The words used to descrbe an mage or a smlar mage vary from one user to another. Furthermore, the textual descrptons are usually short. Ths vocabulary varaton and the concseness of textual descrptons make t dffcult for the tradtonal text retreval to work effectvely for mage retreval. To address ths problem, we are currently n an on-gong nvestgaton on user relevance feedback (URF) n mage retreval. Here, user relevance feedback s motvated by the success of pseudo relevance feedback (PRF) n nformaton retreval [10]. The dfference between URF and PRF s that, n URF we ntroduce users n the loop to do a santy check on potental expanded terms. Instead of automatcally expandng the query as n PRF, the URF presents a lst of terms to users and ask them to choose relevant terms that can descrbe the target mage. Only those terms chosen by the user wll be used n query expanson.

6 Our hypothess s that ths type of feedback can take advantage of the concseness of textual descrptons and consoldate the nconsstency of user textual queres. On one hand, the concse descrptons make t possble for the system to effcently dentfy potental mportant terms. On the other hand, the system selected terms wll remedy the dfference between query term and mage descrpton. Furthermore, the santy check from the user wll mprove the qualty of query expanson, whch wll ultmately result n the mprovement of fnal retreval results. As a frst step n our nvestgaton, we developed several strateges to select terms and conducted smulatons to evaluate dfferent strateges. We then mplemented the best strategy for the real user study. In the user study, we compared the nterface usng URF wth a standard nterface that only allows users to nteractvely refne or expand ther queres. However, out of our expectaton, the results from user studes were not able to valdate our hypotheses. In fact, the results ndcate the current desgn and mplementaton of URF s not workng. Therefore, n ths secton, rather than presentng a successful story (as much as we wsh), we report our current nvestgaton and dscuss lessons learned from ths experence that are useful for future nvestgaton. 3.1 Term Selecton Term selecton n URF s dfferent from that n PRF. In URF, our goal s to fnd terms from descrptons that are related to the ntal user query terms, however wth large uncertantes as to whether they are relevant. As a frst step, we nvestgated dfferent strateges for term selecton usng smulaton experments. Smulaton studes are mportant snce they can provde some nsghts on whether a strategy can potentally work even before the expensve user studes are conducted. In these smulated experments, the system frst selects a lst of ten terms based on dfferent strateges. To smulate human behavor n dentfyng relevant terms from ths lst, the system pcks terms that occur n the descrpton of the target mages. The pcked terms, together Total Number of Target Retreved Iteraton Number Strategy 1 Strategy 2 Strategy 3 Fg. 1. Performance of three strateges at each teraton pont

7 wth the ntal query terms, wll be sent to the backend retreval engne. Ths process repeats untl ether the target mage s found n the top N (currently, N = 20) retreved mages or the system reaches M teratons (currently, M = 10). To generate the terms, we have expermented wth dfferent strateges. The frst strategy measures the entropy of a term based on the top N retreved results (called Top Set later) and/or the next 100-N retreved results (called Bottom Set later). The dea s that the term wth hgher entropy s more uncertan n terms of whether t descrbes user s nterest. By askng user to confrm those hgher entropy terms, the system can quckly narrow down the search space. The hgher the entropy s, the hgher the weght s gven. We tred dfferent combnatons of retreved results to calculate the entropy for a gven term, specfcally the followng three strateges: Strategy 1: Hgher weghts are gven to terms that have hgher entropy from the Bottom Set and also occur less frequently n the Top Set. Strategy 2: Hgher weghts are gven to terms wth hgher entropy from the Bottom Set. Strategy 3: Hgher weghts are gven to terms wth hgher entropy from the Top Set. In the smulaton experments, we randomly pcked 200 mages from ImageCLEF collecton [4]. For each mage, we provded an ntal query. Then we appled the smulaton process as descrbed above to retreve each mage. Fgure 1 shows the smulaton results from three dfferent strateges as to how many out of 200 mages were successfully retreved as top 20 results at each teraton pont. Results ndcate that there s no sgnfcant dfference between three strateges. All three strateges are more effectve at earler teratons (from 1 to 6) than later ones n the smulaton. At teraton 1, snce no query expanson s used, all three strateges resulted n the same number of successful retrevals only based on the ntal queres. Snce the strategy 2 seems slghtly better, we use the strategy 2 n our user study. We have also expermented wth the nverse correlaton strategy and the synonym strategy. The dea for the nverse correlaton strategy s that f a term s very correlated wth a query term gven by the user, then that term carres less nformaton n dentfyng new mages that mght be of user s nterest. Therefore, we gve a lower weght to the terms that s hghly correlated wth a query term usng a vector space model. The dea for the synonym strategy s that f a term s the synonym of a query term, then t could be very relevant. We want to gve t a hgher weght snce t maybe just a dfferent vocabulary expressng the same meanng. To test ths synonym strategy, we used WordNet. However, our current smulatons have not shown the effectveness of these two strateges n term selecton. Therefore, we dd not nclude these two strateges n the user study. Once one or more prompted terms are selected (ether automatcally by the system n the smulaton or manually by the user n the user studes), those terms wll be used to expand the ntal query n further retreval cycles. The retreval model s based on a statstcal language modelng approach usng textual descrptons of mages [11].

8 3.2 User Studes To valdate our hypothess and evaluate the effectveness of the current URF, we conducted a comparatve study followng the gudelnes provded by the ImageCLEF nteractve track Method Fg. 2. (a) Standard Interface Eght subjects partcpated n the study and each of them was asked to search for 16 mages from the Eurovson St. Andrew collecton provded by ImageCLEF. The subjects were frst asked to complete a screenng questonnare to elct demographc data and data concernng searchng experence. Then the subject was asked to use one nterface to search eght mages (one at a tme). After usng each nterface, the subject was gven a questonnare to ndcate how easy he feels about the search process, and how satsfed he s wth a partcular system. Durng the search, the system also automatcally logged the nformaton such as the orgnal queres from the user, the system retreved results, terms prompted by the system, and the tme spent on searchng, etc. When an mage was found or when fve mnutes were run out, the search stopped. After searchng all mages usng two dfferent nterfaces, each subject was asked to gve an overall rankng of the two nterfaces n terms of ther overall satsfacton and systems effectveness of locatng the target mages. Two nterfaces used n the study are shown n Fgure 2. Fgure 2(a) s a standard nterface to be compared wth, where users could refne or expand ther queres usng ther own terms. Fgure 2(b) s the URF nterface. In the URF nterface, n addton to ten terms prompted by the system for user feedback, the system also shows the query terms that are used so far for the retreval. Users have choces to revse these query terms from prevous teratons by de-select them from the lst. Ths feature was desgned so that the URF nterface s comparable to the standard nterface where users can freely add/remove ther query terms at each teraton Evaluaton Results (b) Interface wth URF

9 Unfortunately, after three users, we found some nconsstency n the system, so we had to dscard the results from those three users. The results shown here are from fve out of eght users. Table 3 shows the effectveness for the two nterfaces. The successful retreval rate s calculated by dvdng the number of target mages that are shown n the top 20 retrevals by the total number of target mages tred for that nterface. Snce the success rate for the nterface B (wth URF) s lower than the nterface A (the standard nterface), we conclude that the nterface B s not effectve based on the current desgn and mplementaton. Table 3. Overall performance of two nterface Standard Interface URF Successful Retreval Average tme 0:48 1:57 Average number of nteractons Dscusson The falure wth the current desgn ponts to several problems that need to be addressed n our future nvestgaton. Users were nvolved n the loop to provde feedback for query expanson. However, one major problem s that many of those terms do not mean much to the user. Certanly, we hope that when a prompted term appears n the descrpton of the target mage, the user would pck that term (as n our smulaton). However, from our studes, we found that even those terms appear n the descrpton, the user stll could not recognze them. Ths caused the bg performance dfference between the smulated experments and the real user study. We feel that there are dfferent classes of terms. Some classes of terms are much easer to dentfy than the others. For example, background and substantal, both terms occurred n the descrpton of a target mage. However, t was very hard for users to recognze them snce they dd not drectly match any salent features conveyed by the mage. On the other, the term brdge would be easer for the user to recognze. It would be deal f the system can only prompt to the users those key terms that could mean somethng to the user. Thus, t would be nterestng to study how users respond to these dfferent terms based on the salent features and semantc content presented n an mage and how to dentfy those sgnfcant terms from the retreved results. Only wth such an understandng, s t possble to buld a potentally effectve URF. In addtonal, as n the tradtonal text retreval, the term msmatchng s another problem for mage retreval. For example, suppose among the ten terms prompted by the system, the user chooses the term road. Even ths term does descrbe some object n the target mage, ths term wll not be effectve f the term street s used n the descrpton, rather than the term road. Therefore, n order to effectvely use URF, the system needs to have a capablty of handlng ths type of msmatchng caused by varatons of terms. Because of the tme lmtaton, here we only brefly descrbe some very prelmnary observatons and problems. We certanly need more n-depth analyss on our

10 collected data. Although the current experment s not successful, what we have learned from ths experence can help us focus on specfc ssues dentfed. We beleve URF stll has a potental n nteractve mage retreval. For example, nstead of only allowng URF as n our current nterface, we can consder addng URF to a standard nterface. However, before that happens, frst of all, we need to reach a better understandng of user cogntve models on descrbng mage content and ts mplcaton n user relevance feedback. 4. Concluson In ths report, we examned two mportant ssues assocated wth cross language mage retreval and nteractve mage retreval: 1) How to mprove the accuracy of nformaton retreval gven that only an onlne translaton system s avalable; 2) How to enhance text-based mage retreval usng the user relevance feedback (URF). Our emprcal results wth cross language retreval have ndcated that an employment of statstcal translaton model s effectve, even when the parallel corpus s created automatcally by an onlne translaton system. Our prelmnary study wth nteractve mage retreval has llustrated that to make user relevance feedback effectve for text-based mage retreval, a carefully desgned procedure of automatc term selecton s crtcal. In partcular, the selected terms should be able to not only dstngush certan mages from others, but also be consstent wth the users percepton of mages. Thus, more n-depth nvestgaton s needed to reach a better understandng of user cogntve models on descrbng mage content and ts mplcaton n user relevance feedback. References 1. Ballesteros, L. and W.B. Croft. Phrasal Translaton and Query Expanson Technques for Cross-Language Informaton Retreval. n Proceedngs of the 20th annual nternatonal ACM SIGIR conference on Research and development n nformaton retreval Ballesteros, L. and W.B. Croft. Resolvng Ambguty for Cross-Language Retreval. n Proceedngs of the 21th Annual Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval Brown, P., et al., The Mathematcs of Statstcal Machne Translaton. Computatonal Lngustcs, (2): p Clough, P., M. Sanderson, and N. Red. The Eurovson St Andrews Photographc Collecton Federco, M. and N. Bertold. Statstcal Cross-Language Informaton Retreval Usng N-Best Query Translatons. n Proceedngs of the 25th Annual Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval

11 6. Gao, J., et al. Improvng Query Translaton for Cross-Language Informaton Retreval Usng Statstcal Models. n Proceedngs of the 24th Annual Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval Hemstra, D. and F.M.G.d. Jong. Dsambguaton Strateges for Cross-Language Informaton Retreval. n Proceedngs of the Thrd European Conference on Research and Advanced Technology for Dgtal Lbrares (ECDL) Kester, L.H., User Types and Queres: Impact on Image Access Systems. ASIS, 1994: p Lavrenko, V., M. Choquette, and W.B. Croft. Cross-Lngual Relevance Model. n Proceedngs of the 25th Annual Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval Mtra, M., A. Snghal, and C. Buckley. Improvng Automatc Query Expanson. n Proceedngs of SIGIR Ponte, J.M. and W.B. Croft. A Language Modelng Approach to Informaton Retreval. n Proceedngs of the 21st annual nternatonal ACM SIGIR conference on Research and development n nformaton retreval Seloff, G.A., Automated Access to Nasa-Jsc Image Archves. Lbrary Trends, (4): p Voorhees, E.M. and D.K. Harman, eds. Proceedngs of the Nnth Text Retreval Conference (Trec-9). 2000: Gathersburgh, MD. 14. Voorhees, E.M. and D.K. Harman, eds. Proceedngs of the Nnth Text Retreval Conference (Trec-10). 2001: Gathersburgh, MD. 15. Westerveld, T. and A.P.d. Vres. Expermental Result Analyss for a Generatve Probablstc Image Retreval Model. n Proceedngs of the 26th ACM SIGIR Xu, J., R. Weschedel, and C. Nguyen. Evaluatng a Probablstc Model for Cross- Lngual Informaton Retreval. n Proceedngs of the 24th Annual Internatonal ACM SIGIR Conference on Research and Development n Informaton Retreval