Cardinal, nominal or ordinal similarity measures in comparative evaluation of information retrieval process

Transcription

1 ardnal, nonal or ordnal slarty easures n coparatve evaluaton of nforaton retreval process Laboratore REODO Unversté laude Bernard Lyon Bd du novebre 98, bat 7 9 VILLEURBANNE edex Tel : + ( 7 9 Fax : + ( 7 9 hrstne MIHEL hrs tne.mchel@ntagne.u-bordeaux.fr Laboratore EM-GRESI MSHA - Esplanade des Antlles, D.U 7 PESSA edex - FRANE Tél : + ( 8 8 / 8 Fax : + ( 8 8 Abstract Slarty easures are used to quantfy the reseblance of two sets. Splest ones are calculated by ratos of the docuent s nuber of the copared sets. These easures are sple and usually eployed n frst steps of evaluaton studes, they are called cardnal easures. Others easures copare sets upon the nuber of con docuents they have. They are usually eployed n quanttatve nforaton retreval evaluatons, soe exaples are Jaccard, osne, Recall or Precson. These easures are called nonal ones. There are re or less adapted n functon of the rchness of the nforaton syste s answer. Indeed, n the past, they were suffcent because answers gven by systes were only coposed by an unordered set of docuents. But usual systes prove the qualty or the vsblty of there answers by usng a relevant rankng or a clusterng presentaton of docuents. In ths case, slarty easures aren t adapted. In ths paper we present soe solutons n the case of totally ordered and partally ordered answer. Introducton The quanttatve evaluaton of a syste s nforaton retreval process s often based upon the coparson of answers. For exaple, n large scale evaluaton, syste s answers are copared each to other (coparatve evaluaton or to a referental set of good answer (dagnostc evaluaton (Hrschann, 9. The calculus of the slarty between two answers and depends on the rchness of the nforaton presentaton forat. Indeed, basc nforaton retreval systes produce lsts of docuents wthout any partcular order. Answer sets are n those cases copared n functon of the nuber of docuents they have (cardnal coparson or the nuber of con docuents they have (nonal coparson. Actual nforaton retreval systes propose ranked lst of docuents as answer, the rank s gven by the relevance degree fro the docuent to the answer. It ay be total or partal. For exaple, web engnes gve a copletely ordered lst of docuents as answer, ths s a total order. But any systes use the clusterng process n order to prove the vsblty of nforaton set. Docuent clusterng algorths attept to group docuents together based on ther slartes / Ths can help users both n locaton nterestng docuent re easly and n gettng an overvew of the retreved docuent set. Inforaton Retreval county has long explored a nuber of post-retreval docuent vsualzaton technques as alternatves to the ranked lst presentaton / docuent networks, sprng ebeddngs, docuents clusterng, and self organzng ap. Of the four aor technques, only docuent clusterng appears to be both fast enough and ntutve enough to requre lttle tranng or adustent te fro the user. (Zar, 99 In these case answer sets are partally ordered. Indeed, only clusters (.e. classes are ranked n order of relevance, docuents are equally ranked n a cluster. Searchers (Tague 99 (Borlung 998 quotes any studes whch hghlght the delay nduced, n the satsfacton of an nforatonal need, by a possble dfcaton of ths order of presentaton. Ordnal easures ust be used n order to take t nto account n the calculus of the slarty, ndeed, cardnal or nonal ones do not do t. But there are none really ordnal easures proposed for evaluaton context. Indeed, easures proposed are st of te cardnal or nonal ones lke Recall, Precson or Jaccard (Losee 9. The a of ths paper s to propose other ones. In the frst part of ths paper we we ll descrbe the usual easures proposed n the case of evaluaton tests: easures based upon nonal or cardnal coparsons. Then, n the second part, we ll present the total ordered forals, the property the slarty easures ust have n ths case and exaples of possble slarty easures. In the thrd and last part we ll present the st general case, the partal order. We ll explan why easures proposed n total order case can t be used there and how we can defne new ordered slarty easures. In the followng secton we wll called : D : a docuents gven as answer. and two sets of docuents defnng two answers to be copared. We can notced that the partal order case s a generalzaton of the total order case; a total order s a partal order wth clusters of one docuent.

2 . No order case.. ardnal coparsons Let suppose that there s not nforaton upon docuents of and. The splest possble coparson between and ay be ade upon the nuber of eleents they have. It s called the cardnal of sets and and t s noted and. orrespondng ndcators are ratos lke:, + or +.. Nonal coparsons Let s consder now that each docuent of and s dentfed as a sngular way.e. lke wth a nae. It s so possble to ake nonal coparson,.e. to dentfy the con docuents of and. The slarty between and wll grow wth the nuber of con docuents. They are atheatcally represented by and graphcally represented as n Fgure. {D, D,, D,, D } and {D,D,, D,, D }. The graphcal representaton s lke Fgure. As before, the slarty ndcators between and wll grow wth the nuber of con docuents. The rank gve re detal about and and pert to defne other crtera. What are they? Docuent s rank n 8 7 Fgure Docuent s rank n.. Possble slarty crtera n a rankng order case Fgure The nuber of con eleents s. Another portant eleent s whch s the nuber of dfferent docuents of and and calculated as : +. Most of usual slarty ndcators are based upon ths nubers. They dffers fro ones to other wth the denonator nuber, calculated n order to noralze the easures fro to. As exaples we can quotes (Boyce 9(Losee 9: The Jaccard s coeffcent [Eq ] The ce s coeffcent The cosne (Salton 8 + The Overlap coeffcent n(, [Eq ] [Eq ] [Eq ] Measures of Recall and Precson used n the case of large scale evaluatons or coparatve test protocols lke TRE (Voorhees 98 are also nonal ones.. The total order case Let suppose now that the docuents of and are personalzed by a nae and presented n a totally ranked way. Let us call D and D the docuents of rank and fro and. If (respectvely s coposed of (respectvely docuents we wll have :... The relatve dfference n order Let suppose that D found as the rank n s the sae docuent as D found at the rank n. The closest and are, the nearest and should be and the hgher the slarty should be. We ll call ths crteron the relatve dfference n order. It represents the dfference n the order of presentaton of D relatve to D. In the followng exaple (Fgure (A, B and ( D have dentcally one con docuent, nevertheless the slarty between A and B s hgher than between and D because the relatve order dfference of D of A and D of B s less than the ones of D of and D of D. A B Fgure... The top-rankng Order of docuent are generally ade upon a relevance crteron, so con docuents presented at the end of the answer are less relevant than the ones presented n the begnnng. Ths crteron ust appear n slarty calculus, we ll call t the top-rankng. The re con docuents are presented early to the users, the re slar the copared sets ust be consdered. As prevous, let suppose that D and D s the sae docuent found at the rank n and at the rank n. The hgher and are, the saller the slarty ust be. In the followng exaple (Fgure the con docuents D of A and D of B are presented both prevous to the user than D of and D of D. (A,B and D

3 (D have both docuent n con but, regards the top-rankng crteron, the slarty between and D s hgher than between E and F. [Eq 8], n + [Eq 9] D Fgure How construct slarty easures quantfyng these crtera? The soluton proposed by Tague (Tague 9 s to cobne n the sae slarty easure an ndcator quantfyng the slarty n ters (.e varyng wth the nuber of con eleents wth an ndcator quantfyng the slarty n order. Ths last one s called the delay ndcator d and s ncreasng wth the orderng dfference of the set to... Delay : ndcator of the order s dfference between sets. There exsts any ethods to construct delay ndcators. Tague s dea s to calculated an adaptaton of the rank correlaton coeffcent eployed n statstc.... Delay derved fro the coeffcent correlaton of rank The correlaton coeffcent of rank quantfes the dfference of two sets n ters of order. It s decreasng wth the nuber of perutaton used to rank the eleents of accordng to the eleents of. The adaptaton presented by Tague (Tague 9 s called R and t s calculated lke: n r( ( + / R( n ( / r( ( + / [Eq ] Where s the total eleent nuber of r(d the rank of the docuent D n f t s present. She calculates the delay d (R as a functon of R. Ths delay ndcator s good because global but s not able to ake appear the two prevous crterons: the relatve order and the freshness of nforaton. Delay n and n presented below do t. E F... Delay calculated varyng the freshness of nforaton By usng the sae reasonng the slarty n ters of top-rankng s decreasng wth the rank and. The delay nduce ay be quantfy by ndcators lke: [Eq ], ( ax( n, [Eq ] ( ax( n, [Eq ], n [Eq ] n [Eq ], n + [Eq ] n +.. onstructon of ordered slarty easures... Type : Measures wth general delay ndcaton Let s consder S ( as any nonal easures lke Jaccard, osne, ce, Recall, Precson prevously descrbe n [Eq,,, ]. Tague construct t s slarty easure by usng the ndcator δ ( as n the further equaton : R S O ( δ ( R S( [Eq ] Ths calculus s possble because the ndcator δ ( R s global to and and so s coherent wth the other general slarty ndcaton S(. Other global delay functon ay be calculated, for exaple wth the eans: [Eq 7] or [Eq 8].. In ths case ordered slarty quantfyng the relatve dfference n order ay be: S ( S( [Eq 9] O And the slarty quantfyng the top-rankng dfference ay be : S ( S( [Eq ] O And t s also possble to cobne delay lke n the followng forula: S ( S( [Eq ] O... Delay calculated varyng the relatve order The crteron defnng the relatve dfference s : the closer and are, the hgher the slarty ust be and so the saller the weght delay ust be. So the slarty s decreasng wth (- or (-. The correspondng weght nduce by the dfference n order ay be quantfy by ndcators lke: [Eq ], [Eq 7] ax(, n... Type : Measures wth precse delay ndcaton Let suppose now that we want to lnk drectly the crtera of relatve dfference or top-rankng to the concerned con docuents. In ths case, correspondng slarty easures can look lke the followng three easures : S S n O D ( [Eq ] n O D ( [Eq ]

4 S n 7 O ( D [Eq ] Type easures are based upon a global ndcator, whch s not very precse. There s an opposte proble wth type easures : freshness of nforaton or relatve order s taken nto account n a precse way but the coparson of the two sets eleents s ust ade upon the ntersecton D. D There s no soluton to ths dlea n the total order forulaton. Nevertheless, there s one f we consder the re general del of partal ordered sets presented below.. The partal order case As we seen n ntroducton, systes tend to use clusterng algorths to prove the vsblty of nforaton. They produce classes of docuents, classes are presented to the user n a rankng way, docuents are usually equally ranked n a class. Ths s a partal order of docuents graphcally represented n Fgure. lass lass lass Fgure Let and be the classes of rank of the sets and, so {,,,,, } and {,,,,, }. Let s D and D are the docuents of sets and. So {D, D,, D, }, and {D, D,, D, }. In ths case, the prevous slarty easures (Eq, 9,,,,, can t be adapted because of the classes forals. Indeed : - Let s reeber that S( s a nonal easure lke for exaple [Eq,,, ]. If we consders the forula, S( can t be calculated wthout breakng the classes herarchy. In ths case we totally loose the clusterng nforaton. - The ndces and haven t the sae sense as before. It s convenent now to speak about r(d, the rank of the docuent D of. The partal order hypothess s that all the docuents of a class have the sae rank but what rank? Takng as r(d s not the only possble soluton. The slarty easure S( and the delay ndcator ust be adapted.... Possble adaptaton of nonal slarty ndcator As we sad before, the calculus of S( has no sense f we keep the classes forals. We advse to consdered [Eq ] as an ndcaton of the sets and nonal slarty. lass lass lass S( S(, [Eq ] Indeed, n ths case, and are nonal sets and the classcal slarty ay be calculated.... Possble delays adaptaton In a partal order case, there s a dlea n the choce of the rank r(d of the docuent D. The splest choce s : r( D [Eq ] The forulas lke [Eq,, 7, 8, 9,,,,,, ] aren t changed except n notaton. For exaple [Eq ] s wrtten : r( D r( D kl k [Eq 7] ax(, ax(, Let s consdered now the case where the rank of the docuents n a class depends on the docuents class nuber. Let s call ( the eleent nuber of class. We can consdered that the docuent D rank are deterned wth : the frst eleent of the class : r ( D ( + [Eq 8] the last eleent of the class : r( D ( [Eq 9] The ean eleent of the class ( r( D ( + [Eq ] The calculus of delay ndcator are exactly ade as n exaple n [Eq 7].Tague (Tague 9 advse to calculate r(d as the eans rank ([Eq ]. It s possble now to adapt type and type slarty easure varyng the chosen adaptatons. But, the probles enuncated at the end of the secton are the sae. Nevertheless, the forals of partal ordered sets let s us thnk that there s another type of possble ordered slarty easures, easures of type.... Type ordered slarty easures : cobnaton of type and. Forals of partal order case ake the constructon of ordered slarty easures as [Eq ] possble. Q (, S (, ϕ(, [Eq ] ϕ (, s defned by fve partculars condtons n order to ake appears the relatve order and the top-rank of nforaton. (, [,]x[, ], ϕ(,> ( [,], ϕ (, (, [,]x[, ], ϕ (, s strctly decreasng n ( fxed (v, [,]x[, ], ϕ (, s strctly decreasng n ( fxed (v [, ] ϕ(, s strctly decreasng n.

5 We can notced that nforaton on slarty n docuent and n order delay are quantfed n the re precse way as possble,.e. for each class. So that s the reason why type easure have both propertes of type and... Exaple of type easure Let s suppose that S(, s the Jaccard ndcator ([Eq ], and ϕ (, the cobnaton of δ ( n defne as ϕ (, δ + δ + [Eq ] wth : and δ ( ( ( ( n ( n ax(, It s possble to construct slarty easure of type lke : Pδ ( J (, δ ( ( + δ ( ( + [Eq ] The easure P δ has been tested n a real evaluaton context : the dagnostc evaluaton of a syste havng a personalzed flterng process upon the user s profle. The a and general ethodology of the study s presented n (Mchel. A coparatve study of the results gven by a classcal Jaccard easure and ths one show that the delay nduce by the relatve order and the top rankng really ustfed the use of an ordered slarty easure (Mchel 99. oncluson Slarty easures are of three types : cardnal, nonal and ordnal ones. ardnal ones are the splest, they ay be used n all the sets descrpton case. Nonal ones are re precse f the sets have ndvdual descrptons of all eleents and ordnals one f, therefore, there s also an ordered classfcaton of the eleents. Sets of docuents proposed as answer of nforaton retreval probles ay be totally or partally ordered. In secton and we propose soe solutons and construct ordered slarty easure by cobnng two crteron : the slarty n ters of docuents (called the nonal slarty and the slarty n ters of order (called the delay. We can notced that ths two crteron can t be appled sultaneously n the total ordered case. Nevertheless, they can n the partal order case. Ths results fro the easures constructon choce, but also fro the fact that, on reverse than n atheatcs, the total order s, n the context of nforaton retreval, a partcular case of the partal order. References (Borlung 98 : BORLUNG P., INGWERSEN P. Measures of relatve relevance and Ranked Half-Lfe : Perforance ndcators for nteractve IR. In Proceedng of the SIGIR 98, -8 august 998, Melbourne, Australa (Boyce 9 : BOYE B.R., MEADOW.T., KRAFT D.H. - Measureent n nforaton scence. Acadec Press 99 8 p. (Ells 9 : ELLIS D. The dlea of easureent n nforaton retreval research. - Journal of the Aercan Socety for Inforaton Scence 7( pp -. (Frcké 98 : FRIKE M Jean Tague-Sutclffe on easurng nforaton In Journal of the Aercan Socety for Inforaton Scence ( 998 pp 8-9. (Harter 9 : HARTER S.P. Varaton n Relevance assessent and easureent of retreval effectveness In Journal of the Aercan Socety for Inforaton Scence 7( pp 7-9. (Harter 97 : HARTER S.P., HERT.A. - Evaluaton of nforaton retreval systes : Approches, Issues, and ethods. In Annual revew of nforaton scence and technology pp -9. (Hrschan 9 : HIRSHMAN L, THOMSON H.S., - Overvew of evaluaton n speech and Natural Langage Processng. In Survey of the state of the art n huan langage technology collectve drecton OLE R./MARIANI J./USZKOREIT H./ZAENEN A./ZUE V. - Rapport NFS/EE. To be publshed abrdge Unversty Press et Gardn Publ. (Losee 9 : LOSEE R. M. The scence of nforaton. Measureent and applcatons Acadec Press, Inc p. (Mchel 99: MIHEL - Evaluaton de systèes de recherche d nforaton, coportant une fonctonnalté de fltrage, par des esures endogènes. Réalsaton et evaluaton d un prototype de systèe de recherche d nforaton avec fltre selon les profls des utlsateurs.- PhD Thess - Unversty Lyon II - January p. (Mchel : MIHEL. agnostc evaluaton of a personalzed flterng nforaton retreval syste. Methodology and experental results. In Proceedngs of RIAO "ontent based ulteda nforaton access", ollège de France, Pars, - aprl (Salton 8 : SALTON G. McGILL M. J. Introducton to dern Inforaton Retreval. New York : McGraw- Hll. (Tague 9 : TAGUE-SUTLIFFE J. - Mesurng nforaton. An nforaton servces perspectves. - Acadec Press p. (Tague 9 : TAGUE-SUTLIFFE J. Soe perspectves on the evaluaton of nforaton retreval systes In Journal of the Aercan Socety for nforaton scence. - 7( pp -. (Voorhees 98 : VOORHEES E.M. HARMAN D. Overvew of the seventh Text Retreval onference TRE 7. In Proceedngs of the seventh Text Retreval onference TRE 7. Gatherburg 9- nveber p. (Zar 99 : ZAMIR O., ETZIONI O. : Grouper : A dynac clusterng nterface to web search results In Proceedng of the Eghth Internatonal World Wde Web onference May Toronto, anada (