Axiometrics: Axioms of Information Retrieval Effectiveness Metrics

Transcription

1 Axiometrics: Axioms of Information Retrieval Effectiveness Metrics ABSTRACT Ey Maalena Department of Maths Computer Science University of Uine Uine, Italy There are literally ozens most likely more than one hunre) information retrieval effectiveness metrics, counting, but a common, general, formal unersting of their properties is still missing. In this paper we aim at improving extening the recently publishe work by Busin Mizzaro [6]. That paper proposes an axiomatic approach to Information Retrieval IR) effectiveness metrics, more in etail: i) it efines a framework base on the notions of measure, measurement, similarity; ii) it provies a general efinition of IR effectiveness metric; iii) it proposes a set of axioms that every effectiveness metric shoul satisfy. Here we buil on their work more specifically: we esign a ifferent improve set of axioms, we provie a efinition of some common metrics, we erive some theorems from the axioms.. INTRODUCTION In the Information Retrieval IR) fiel, accoring to survey in 26 [8], more than 5 effectiveness metrics are ientifie, taking into account only the system oriente ones. This is a rough unerestimate; as iscusse for example in [2], more than one hunre IR metrics exist, let alone useroriente ones or metrics for tasks somehow relate to IR, like filtering, clustering, recommenation, summarization, etc. Figure is a graphical representation of this. This large number is not balance by a complete unersting of the conceptual formal properties that any IR effectiveness metrics shoul satisfy, as iscusse also at the recent SWIRL meeting swirl2/). It is clear that a better unersting of the formal properties of effectiveness metrics woul have several avantages, for example it woul help to avoi wasting time in tuning retrieval systems accoring to the wrong metric. However, this is still lacking. Among some recent attempts to stuy the formal properties of IR metrics see next section), in this paper we focus specifically on the work by Busin Mizzaro [6]. The contribution of that paper is threefol: i) it efines a framework, groune on measurement theory, base on the notions of measure, measurement, similarity; ii) it provies a general efinition of IR effectiveness metric; iii) it proposes a set of axioms that every effectiveness metric shoul satisfy. Although that paper has the merit of proposing to groun on measurement theory to stuy IR metrics, it also has some limits. First, the analysis of the existing metrics in terms of the framework is very brief, concerns only four metrics. Therefore, although the framework Stefano Mizzaro Department of Maths Computer Science University of Uine Uine, Italy mizzaro@uniu.it has been proven aequate to express the axioms, its ability to take into account ifferent metrics has not really been teste. Secon, the axioms are a bit isorganize: they are not clearly categorize, some of them woul probably more justifiable as theorems than basic axioms. Thir, the usefulnes of the axioms, in terms of eriving theorems from them, is almost not teste, as only one theorem is quickly formalize. We try to overcome those limits in the present work This paper is structure as follows. In Section 2 we briefly recall the previous work on formal accounts of IR effectiveness metrics. In Section 3 we briefly summarize the framework notation propose in [6]. In Section 4 some metrics are efine within the framework, to emonstrate its expressiveness power. In Section 5, a set of axioms, ifferent from that propose in [6], is state the axioms are exploite to erive some theorems in Section 6. Conclusions future work are presente in Section RELATED WORK Although formal approaches in the IR fiel have mainly focusse on the retrieval process rather than on effectiveness metrics themselves see, e.g., [9, ]), some research specific to effectiveness metrics oes exist, it is briefly iscusse here. The first early attempts were mae by Swets [3], who liste some properties of IR effectiveness metrics, van Rijsbergen [5, Ch. 7], who followe an axiomatic approach. In [5], Bollmann iscusses the risk of obtaining inconsistent evaluations on a ocument collection on its subcollections. Two axioms on effectiveness metrics, name the Axiom of monotonicity the Archimeean axiom, are propose, their implications are presente as a theorem. These approaches are evelope on the basis of binary relevance either a ocument is relevant or it is not) binary retrieval either a ocument is retrieve or it is not). Here we o not make any assumption on the notions of relevance retrieval binary, ranke, continuous, etc.). Our approach is meant to be more general. Yao [7] focusses on the notion of user preferences to measure the relevance or usefulness) of ocuments. He aopts a framework where user jugements are escribe as a weak orer. On this basis he then proposes a new effectiveness metric that compares the relative orer of ocuments. The propose metric is prove to be appropriate through an axiomatic approach. More recently, Amigó et al. in [] focus their formal analysis on evaluation metrics for text clustering algorithms fin- 7

2 Figure : IR effectiveness metrics from [2]) ing four basic formal constraints. These constraints shoul be intuitive coul point out the limitations of each metric. Moreover, it shoul be possible to prove formally which constraint is satisfie by a metric or a metric family, not just empirically. They foun BCube metrics BCube precision Bcube recall) to be the only ones satisfying the four propose constraints. Amigó et al. in [3] iscuss, with a similar approach, a unifie comparative view of metrics for ocument filtering. They iscovere that no metric for ocument filtering can satisfy all esirable properties unless a smoothing process is performe. Finally, in an even more recent work [4], they start from a set of formal constraints to efine a general metric for ocument organization tasks, that inclue retrieval, clustering, filtering. Another attempt is by Moffat, that in [] lists seven properties of IR metrics: bouneness, monotonicity, convergence, top-weighteness, localization, completeness, realizability. Several metrics are then analyze to unerst whether they satisfy these properties. 3. THE AXIOMETRICS FRAMEWORK In the remainer of this paper, we buil on the framework recently propose in [6], base on the notions of measurement, measurement scale, similarity [6, 2, 4]. Due to space limitations, we can provie only a very short summary; the reaer is referre to [6] for further etails motivations. Given a query q, system user/assessor relevance measurements on a single ocument are enote by q, ) q, ), respectively; the notation is also extene in a straghtforwar way to set of ocuments D) queries Q); the scale of a measurement e.g., a binary relevance assessment ) is enote by scale) = R, N where R means relevant N nonrelevant). a notion of similarity between measurements is efine, enote as sim Q, D), Q, D)) Q,D or more briefly as sim Q,D, ) when Q D are common to both measurements as it is often the case). In this paper we aopt a ifferent approach from [6], we will not use the similarity of sets of queries Q) /or ocuments D): we will nee sim, ) only. In [6] a metric is then efine as a function that takes as arguments two measurements, a set of ocuments D, a set of queries Q, provies as output a numeric value usually in R). In mathematical notation, metric : D Q R. Componentwise, a metric can be ientifie on the basis of five components: scale), scale), a notion of similarity sim, how the values on single ocuments are average over the set D we enote the corresponing averaging function with avgd), how these averages are average over the set Q avgq). We can write: metric scale), scale), sim, avgd, avgq). ) This framework notation shoul be general enough to moel most if not all) effectiveness metrics, as we show next. 4. SOME METRICS In this section we efine the sim, avgd avgq functions for some common metrics to emonstrate that the framework notation shoul be general enough to moel most if not all) effectiveness metrics. Table presents tentative efinitions of some common metrics. The table shoul be unerstable, but we briefly iscuss some metrics. For instance, for both Precision Recall, scale) = scale) = R, N. Let Rel = { D ) = R} Ret = { D ) = R} be the sets of relevant retrieve ocuments, respectively. Similarity is see the table) if a ocument is both retrieve relevant otherwise., the avgd functions for Precision Recall are the arithmetic means over the sets Ret Rel, respectively, both the avgq functions are the arithmetic mean over the set Q. For MAP MAP-like metrics GMAP, logitap, yaap, etc.) we have two ifferent scales: scale) = R, N scale) = Rank. Similarity is more complex on a rank see the formula in the table), since to unerst the similarity of a ocument we nee to analize also other ocuments in the rank. Similarity can then be use to efine AP values then MAP is obtaine using as avgq the arithmetic mean of the AP values. GMAP is similar to MAP, the 8

3 Metric scale) scale) sim, ) avgd avgq Precision { P q = sim Ret, ) if ) = ) Ret R, N Recall otherwise R q = sim, ) P@n q = n MAP if ) = R R, N AP q = if ) = N MAP@n Rank ) = R AP q = ) > )) GMAP otherwise AP q = Rel Rel, ) n R-Prec R-Prec q = log AP) logitap AP q = Rel Rel, ) n Rel Rel ADM [, ] [, ] q, ) q, ) ADM q = D sim, ) Rel, ) sim, ) sim, ) sim, ) sim, ) sim, ) sim, ) i i D Table : Metrics on the basis of their components as per formula ). P q R q q R-Prec q AP q AP q AP q log AP q log AP +ɛ AP +ɛ ADM q only ifference being on avgq since in GMAP the geometric mean is use. GMAP can also be efine in an equivalent way as the average of logarithms, logitap efinition is similar. yaap shoul be similar as well). The table also inclues MAP@n, i.e., MAP compute averaging only the AP values of the relevant ocuments retrieve in the first n rank positions, consiering as the AP of ocuments retrieve after rank n this is the metric use in TREC-like settings). The last row efines ADM [7]. As it is iscusse at length in [6], besies allowing us to efine the metrics, the framework allows us to state the axioms in a way that is inepenent from the scales of the relevance measurements. 5. AXIOMS We now can list some axioms: they efine properties that, ceteris paribus, any effectiveness metric shoul satisfy. Axioms can also be interprete as a set of constraints on a search space. We formalize as axioms the properties of similarity between relevance measurements Subsection 5.), we then present some axioms that efine the relationships between similarity metrics Subsection 5.2), we then present metric-specific axioms Subsection 5.3). 5. Similarity The first axioms represent basic constraints on similarity, metrics are not involve yet. Axiom Similarity of ocuments). Let q be a query, two ocuments, a human relevance measurement a system relevance measurement such that q, ) = q, ) q, ) = q, ). sim, ) = sim, ). Axiom 2 Similarity of queries). Let q q be two queries, a ocument, a human relevance measurement a system relevance measurement such that q, ) = q, ) q, ) = q, ). sim, ) = sim, ). q, Axiom 3 Similarity of two systems). Let q be a query, a ocument, a human relevance measurement two system relevance measurements such that q, ) = q, ). 2) sim, ) = sim, ). 3) Let us remark that 3) oes not entail 2). Diagrams like those in Figure 2 can be helpful to intuitively unerst the situation: Figures 2a) 2b) represent the cases in which respectively overestimate unerestimate or vice-versa) by the same amount; then the similarity represente in the figure by the two arcs on the right) of the two systems is the same, but obviously 2) oes not hol. Conversely, as state by the axiom, when 2) hols then 3) hols as well Figures 2c) 2)). 5.2 From Similarity to Metric 5.2. Different systems The following axiom sets a constraint on the metric in one of the two last cases of Figure 2 Figures 2c) 2)). Axiom 4 Systems with equal effectiveness). Let q be a query, a ocument, a human relevance measurement two system relevance measurements such that q, ) = q, ). 9

4 a) Divergent b) Divergent c) Overestimate ) Unerestimate Figure 2: Two systems having equal similarity to a) Overestimate b) Unerestimate c) Divergent ) Divergent Figure 3: Two systems with ifferent similarity to We now turn to compare a system measurement for two ocuments. Let us assume, without loss of generality, that is more relevant than ) > )). We can consier two cases: metric, ) = metric, ). Remark. Note that by using, in this axiom, a conition like 2) not like 3) the first two cases of Figure 2 are rule out, inee in those cases we cannot state any constraint on the metric: a recall-oriente metric woul give a higher value to a system overestimating all the ocuments retrieving all ocuments means that recall is ), whereas a precision-oriente metric woul o the opposite. sim, ) > sim, ) see Figure 4a)); sim, ) < sim, ) Figure 4b)). In the first case we have a smaller error in the more relevant ocument a larger error in less relevant ocument. In such a case, no constraint can be state on the metric since, as it is often state, earlier rank positions are more important than later ones. Conversely, the secon case allows to state some axioms. We analyze it we start by observing that, since the system coul overestimate or unerestimate the ocuments, the case of Figure 4b) can be subivie into four cases: Axiom 5 Systems with ifferent effectiveness). Let q be a query, a ocument, a human relevance measurement two system relevance measurements such that sim, ) > sim, ) 4) sim, ) > sim, ). 5) overestimates both see Figure 5a)); unerestimates both Figure 5b)); metric, ) > metric, ). Different ocuments overestimates unerestimates Figure 5c)); Remark 2. Conition 4) means that is less wrong than. The combination of 4) 5) means that the two systems are wrong in the same irection: if overestimates unerestimates), then overestimates unerestimates) it even more. Figures 3a) 3b) show these two cases. Conition 4) rules out the other two situations, shown in Figures 3c) 3), in which no constraint on the metric can be state for the same reasons mentione in Remark. unerestimates overestimates Figure 5)). Only the first two cases allow to express some constraints on the metric, again for the same reason of Remark 2). We analyze the first two cases. Let us start by noting that: if overestimates then sim ), )) < sim ), )); 2 6)

5 a) Both overestimate b) Both unerestimate c) Divergent ) Divergent Figure 5: Four possible cases Figure 6: Critical situation The secon axiom concerns the case of Figure 5b). a) Best similarity top b) Best similarity bottom Axiom 7 Unerestimate ocuments). Let q be a query, two ocuments, a human relevance measurement a system relevance measurements such that Figure 4: Two ocuments with ifferent similarity ) > ), ) > ), if unerestimates then sim ), )) > sim ), )); 7) sim, ) < sim, ), if overestimates then sim ), )) > sim ), )); ) 7) 9) hol i.e., both are unerestimate), then 8) if unerestimates then metric, ) > metric, ). sim ), )) < sim ), )). 9) Remark 3. The conition ) rules out the critical case in which both ocuments are unerestimate but there is a swap as shown in Figure 6. In such a case, although the similarity is higher for, no constraint can be Axiom 6 Overestimate ocuments). Let q be a query, impose on the metric, again for the reason of Remark two ocument, a human relevance measurement about top rank positions. In Axiom 6, this aitional coni a system relevance measurements such that tion is not necessary, because if both ocuments are overestimate similarity is higher for the less relevant ocument, ) > ), then no swap is possible. sim, ) < sim, ) In the following we will nee to write that a metric value We can now state the following two axioms. The first concerns the case of Figure 5a). is more affecte by a ocument than by another ocument. Formally, we efine: 6) 8) hol i.e., both are overestimate), then Definition. We write that metric, ) > metric, ). =metric,) 2

6 if only if metric, ) metric, ) > q,d {} q,d metric, ) metric, ) q,d { } q,d to be rea as affects metric value more than ). Analogously, we will write metric,) we will also use,, with similar meanings. A similar notation hols for queries. Axiom 8 System relevance). Let q be a query, two ocuments, a human relevance measurement a system relevance measurement such that sim, ) = sim, ), ) > ), ) ). ) metric,). This means that if system relevance measures on two ocuments are equally correct, system relevance of is higher than system relevance of, then the effectiveness metric shoul be more affecte by than by provie that is not less relevant than ). As alreay mentione, it is usually state that early rank positions affect a metric value more than later rank positions. This can be erive as a corollary of the previous axiom that states a more general principle, inepenent of the scales) simply by taking scale) = Rank. A symmetric axiom can also be state on user relevance measurement: a metric shoul weigh more, be more affecte, by more relevant ocuments. This is perhaps less intuitive than the previous one, but it oes inee seem natural in this framework. Moreover, it is quite easy for an IRS to evaluate a non-relevant ocument as non-relevant, since the vast majority of ocuments in the atabase are non-relevant. Thus, an IRS stating that a non-relevant ocument is non-relevant is somehow oing an easy job, shoul not be reware too much for it. On the other h it shoul be reware when correctly ientifying a relevant ocument. This is generalize formalize as follows. Axiom 9 User relevance). Let q be a query, two ocuments, a human relevance measurement a system relevance measurement such that: sim, ) = sim, ), ) > ), ) ). 2) metric,). Remark 4. Conitions ) in Axiom 8 2) in Axiom 9 are neee to rule out the case in which the two axioms woul result inconsistent. Finally, the following axiom eals with the last case. Axiom Same relevance). Let q be a query, two ocuments, a human relevance measurement a system relevance measurement such that sim, ) = sim, ). If ) = ) ) = ) then metric,). 5.3 Metrics We now turn to the last set of axioms, that are specifically about metrics. The following axiom formalizes Swets s properties see Section 2). To simplify its formulation we enote by the theoretically worst performance, i.e., the relevance measure that gives the worst possible performance accoring to a given assessor relevance measure. Axiom Zero maximum). An effectiveness metric shoul have a true zero in a maximum value M. The theoretically worst best) performances shoul give M) as the metric value. As a normalization convention let M = such that metric, rangemetric) = [, ], metric, ) =, metric, ) =. Axiom 2 Document monotonicity). Let q be a query, D D two sets of ocuments such that D D =, a human relevance measurement two system relevance measurements such that: metric, ) > metric q,d =) >) q,d, ) 3) metric, ) > metric, ). 4) q,d =) q,d =) metric, ) > metric, ). 5) q,d D =) q,d D >) A similar axiom hols for queries, as follows. Axiom 3 Query monotonicity). Let Q Q be two query sets such that Q Q =, D a ocument set, a human relevance measurement two system relevance measurements such that: metric, ) > metric Q,D =) >) Q,D, ) metric, ) > Q,D =) =) metric, ). Q,D metric, ) > metric, ). Q Q,D =) Q Q,D >) These two last axioms can also be interprete as constraints on the avgd avgq functions, respectively. 6. THEOREMS In this section we emonstrate that the axioms can inee be use to erive further properties as theorems. For space limitations, we show only a few theorems in this section, we sketch only one proof; however, the general iea shoul be clear. We also omit several corollaries that, as alreay hinte above, can be erive for specific scales. In this axiom the equal = less than < signs have obviously to be paire in the appropriate way, row by row. We use this notation for the sake of brevity to avoi to state three ifferent very similar axioms. 22

7 Theorem Unbalance ocument). Let q be a query, D a ocument set, / D a ocument, a human relevance measurement two system relevance measurements such that metric, ) > metric, ) 6) q,d q,d metric, ) metric, ) 7) q,d {} q,d {} i.e., is more effective, accoring to the metric, than on D the situation is reverse on D {}). metric, ) < metric, ) 8) i.e., has to be less effective on q as well). Proof. Let us start by noting that 6) correspons to the first or thir row of) 3). Now, by way of contraiction let us assume that the conclusion 8) oes not hol, i.e., metric, ) metric, ). This correspons to 4), with D = {}, with either > or =. In both cases, Axiom 2 entails that metric q,d {}, ) > metric q,d {}, ) 5) with D = {}), which contraicts 7). A similar theorem, with similar proof omitte), hols for queries, as follows. Theorem 2 Unbalance query). Let Q be a query set, q / Q a query, D a ocument set, a human relevance measurement two system relevance measurements such that metric, ) > metric, ) Q,D Q,D metric, ) metric, ). Q {q},d Q {q},d metric q,d, ) < metric q,d, ). Theorem 3 Consistent subocument set). Let Q be a query set, D a ocument set, a human relevance measurement two system relevance measurements such that metric Q,D, ) > metric Q,D, ). S D metric, ) > metric, ) Q,S Q,S i.e., if is more effective than on D, it has to be more effective on a subset of D as well). Remark 5. Recursively applying this theorem we can erive that there is always at least one ocument in D that is consistent with D. A similar theorem hols for query sets as well. Theorem 4. Let Q be a query set, D a ocument set, a human relevance measurement two system relevance measurements such that q Q, D, metric, ) > metric metric Q,D, ) > metric Q,D, )., ). Theorem 5 Monotonicity of ocuments subsets). Let q be a query, D a ocument set, S a subset of D, a human relevance measurement a system relevance measurement such that S, D\S, D\S, metric, ) > ) metric, ) > q,s ) metric, ). metric, ). q,s { } Remark 6. A similar theorem can be state on queries. 7. CONCLUSIONS AND FUTURE WORK Builing on the framework base on measure, measurement, similarity propose in [6], we have efine some common metrics, propose some axioms erive some theorems on IR effectiveness metrics. More generally, when rea together with [6], the contribution of this paper is fivefol: i) the proposal of using measurement to moel in a uniform way both system output human relevance assessment, the analysis of the ifferent measurement scales use in IR; ii) the notions of similarity among ifferent measurement scales the consequent efinition of metric; iii) the efinitions of some metrics within the framework; iv) the axioms; v) the theorems. Several future evelopments can be imagine, as alreay liste in [6]. For example, ifferent measurement scales have been propose in the literature might be use; axioms for iversity, novelty, session metrics coul be ae, taken into account for specific tasks; so on. It is also possible that ifferent sets of axioms can be ientifie. In this paper we have shown a possible set, inee the axioms in [6] are completely ifferent from those propose here; we leave as future work a etaile comparison of the two sets. The question is left open whether there exist other ifferent sets, if they are equivalent or perhaps even contraictory, but we remark that the combination of this paper [6] emonstrates that the framework is expressive allows to formally reason on effectiveness metrics. Acknowlegments We thank Julio Gonzalo Enrique Amigó for long interesting iscussions, Evangelos Kanoulas Enrique Alfonseca for helping to frame the Axiometrics research project, Arjen e Vries for suggesting the name Axiometrics, organizers of participants to) SWIRL 22. This work has been partially supporte by a Google Research Awar. 8. REFERENCES [] Enrique Amigó, Julio Gonzalo, Javier Artiles, Felisa Verejo. A comparison of extrinsic clustering evaluation metrics base on formal constraints. Information Retrieval, 24):46 486, 29. [2] Enrique Amigó, Julio Gonzalo, Stefano Mizzaro. A general account of effectiveness metrics for information tasks: retrieval, filtering, clustering. In Proceeings of the 37th international ACM SIGIR conference on Research & evelopment in information retrieval, pages ACM, 24. [3] Enrique Amigó, Julio Gonzalo, Felisa Verejo. A comparison of evaluation metrics for ocument filtering. In CLEF, volume 694 of LNCS, pages Springer, 2. 23

8 [4] Enrique Amigó, Julio Gonzalo, Felisa Verejo. A general evaluation measure for ocument organization tasks. In Gareth J. F. Jones, Paraic Sherian, Diane Kelly, Maarten e Rijke, Tetsuya Sakai, eitors, SIGIR, pages ACM, 23. [5] P. Bollmann. Two axioms for evaluation measures in information retrieval. In SIGIR 84, pages , Swinton, UK, 984. British Computer Society. [6] Luca Busin Stefano Mizzaro. Axiometrics: An Axiomatic Approach to Information Retrieval Effectiveness Metrics. In ICTIR 23 Proceeings of the 4th International Conference on the Theory of Information Retrieval, 23. [7] V. Della Mea S. Mizzaro. Measuring retrieval effectiveness: A new proposal a first experimental valiation. Journal of the American Society for Information Science Technology, 556):53 543, 24. [8] G. Demartini S. Mizzaro. A Classification of IR Effectiveness Metrics. In ECIR 26, volume 3936 of LNCS, pages , 26. [9] Hui Fang, Tao Tao, ChengXiang Zhai. A formal stuy of information retrieval heuristics. In SIGIR 4, pages 49 56, New York, NY, USA, 24. ACM. [] Hui Fang ChengXiang Zhai. An exploration of axiomatic approaches to information retrieval. In SIGIR 5, pages , 25. [] Alistair Moffat. Seven numeric properties of effectiveness metrics. In AIRS 3, pages 2, 23. [2] Stanley Smith Stevens. On the theory of scales of measurement. Science, ):677 8, 946. [3] J. A. Swets. Information retrieval systems. Science, 4:245 25, 963. [4] Amos Tversky. Features of similarity. Psychological Review, 844), 977. [5] C. J. van Rijsbergen. Information Retrieval. Butterworths, 2n eition, 979. [6] Wikipeia. Measurement Wikipeia, the free encyclopeia [Last visit: August 23]. [7] Y. Y. Yao. Measuring retrieval effectiveness base on user preference of ocuments. Journal of the American Society for Information Science, 462):33 45,