A Probabilistic Fusion Framework

Transcription

1 A Probabilisti Fusion Framework ABSTRACT Yael Anava Faulty of IE&M, Tehnion Oren Kurland Faulty of IE&M, Tehnion There are numerous methods for fusing doument lists retrieved from the same orpus in response to a query. Many of these methods are based on seemingly unrelated tehniques and heuristis. Herein we present a probabilisti framework for the fusion task. The framework provides a formal basis for deriving and explaining many fusion approahes and the onnetions between them. Instantiating the framework using various estimates yields novel fusion methods, some of whih signifiantly outperform state-of-the-art approahes. 1. INTRODUCTION There is a large body of work on fusing doument lists that were retrieved in response to a query from the same orpus (e.g., [6, 17, 22, 23, 31, 39, 12, 13, 3, 9, 38, 15, 29, 44, 7, 4, 24, 36, 40, 11, 16, 5, 26, 35, 43, 21]). The lists that are fused ould be retrieved by using, for example, different query representations, doument representations, or querydoument similarity measures [13]. Many of the existing fusion methods are based on tehniques and heuristis whih might seem, at first glane, quite different. The lak of formal foundation for many of these methods makes it diffiult to formally ompare them and to ontrast the premises on whih they are based. We present a probabilisti framework for the fusion task. We use the framework to provide formal grounds for a priniple employed by most fusion methods: rewarding douments highly ranked in many of the lists to be fused. Speifially, the framework helps to provide formal support for the potential merits of the two aspets on whih this priniple relies, namely, the skimming and horus effets [40]. These effets refer to rewarding douments that are (i) highly ranked in the lists, and (ii) shared by the lists, respetively. We next turn to use the framework to study the lass of linear fusion methods [40]. These methods rank douments using a linear ombination of relevane evidene indued from the lists. The evidene an be based, for example, on doument retrieval sores, or funtions of their ranks, in Permission to make digital or hard opies of all or part of this work for personal or lassroom use is granted without fee provided that opies are not made or distributed for profit or ommerial advantage and that opies bear this notie and the full itation on the first page. Copyrights for omponents of this work owned by others than ACM must be honored. Abstrating with redit is permitted. To opy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speifi permission and/or a fee. Request permissions from permissions@am.org. CIKM 16, Otober 24-28, 2016, Indianapolis, IN, USA 2016 ACM. ISBN /16/10... $15.00 DOI: Anna Shtok Faulty of IE&M, Tehnion annabel@tx.tehnion.a.il Ella Rabinovih IBM Haifa Researh Labs ellak@il.ibm.om the lists. Our framework helps to set ommon probabilisti grounds to many linear fusion methods and to shed light on the onnetions and differenes between them. In addition, we show that CombMNZ [17, 23], whih is a ommonly used non-linear fusion method, an also be explained using the framework. Speifially, CombMNZ is an instantiation of a meta fusion approah that fuses the lists produed by applying two different fusion methods. The existing fusion methods that we analyze an be instantiated from the framework using speifi estimates. We study additional instantiations by varying the estimates used. To that end, we utilize a variety of retrieval-sore normalization shemes, rank-to-sore transformations, and estimates for the effetiveness of a retrieved list. Empirial omparison of the various instantiations, guided by the framework, results in a methodologial analysis of existing and novel fusion methods that emerge from the framework. Several of the novel fusion methods that we derive are shown to signifiantly outperform state-of-the-art fusion approahes. The main ontributions of this paper are: 1. A probabilisti framework for the fusion task that sets formal grounds for deriving and explaining many seemingly different fusion approahes and the onnetions between them. 2. Novel fusion methods that are derived from the framework. Several of these methods signifiantly outperform state-of-the-art fusion approahes. 3. Methodologial empirial analysis, guided by the framework, of existing and novel fusion methods. 2. RELATED WORK As already noted, we show that many fusion methods [17, 3, 4, 24, 36, 25, 11, 16, 26, 35, 33] an be derived or explained using the probabilisti framework we present. Some of these are state-of-the-art methods [17, 40, 24, 36, 25, 11] whih are shown to underperform several new instantiations of the framework presented here. Some fusion methods (e.g., [15, 29]) annot be diretly derived or explained using the probabilisti framework we present. We show that one suh method, CondoretFuse [29], is substantially outperformed by novel methods instantiated from the framework. Some previous haraterization of fusion methods relies on two dimensions [29]: whether the fusion method uses retrieval sores or rank information and whether it utilizes training data. The framework we present provides a more

2 detailed haraterization of fusion methods based on the types of estimates used to instantiate it. For example, training data is used in two apaities: to estimate (i) list effetiveness; and (ii) the relevane of any doument loated in a speifi rank, or blok of ranks, in a retrieved list. There have been empirial explorations of the presumed onditions neessary for effetive fusion [23, 31, 39, 7]. Our framework provides formal support for the effetiveness of fusion methods that employ the priniple of rewarding douments highly ranked in many of the lists that are fused. A geometri probabilisti approah [43], different than ours, was used to explain why fusion is effetive. In other work, statistial priniples were applied to analyze two fusion methods [42]. There is also work on addressing the fusion task using evidential reasoning [20]. In ontrast to our work, these formal approahes were not used to set ommon grounds to, or derive and explain, the various existing fusion methods that we address. Furthermore, in ontrast to this past work, our framework gives rise to novel fusion methods that signifiantly outperform state-of-the-art approahes. We instantiate the framework using ommonly used basi retrieval-sore normalization shemes and rank-to-sore transformations [23, 30, 45, 11, 28]. These instantiations result in existing and novel fusion methods. There are sore normalization approahes based on fitting sore distributions [27, 1, 2], but these are outside the sope of this paper. The linear mixture model that rises in our framework, and is used to explain linear fusion methods, is oneptually similar to that presented in reent work on utilizing relevane feedbak in fusion-based retrieval [32]. However, our framework is more general as it is derived from first probabilisti IR priniples. Furthermore, in ontrast to this work [32] whih foused on exploiting relevane feedbak, we use our framework to explain existing fusion methods, and to devise new ones, whih do not utilize feedbak. 3. FUSION FRAMEWORK Let L be a set of m lists, {L 1,..., L m}, eah of whih ontains k douments that were retrieved in response to query q from a given orpus. The lists an be produed by using various retrieval approahes [13]. We write d L to indiate that doument d is in list L; r L(d) denotes d s rank in L; if d L, then r L(d) def. We use s L(d) to denote d s non-negative sore in L; this ould be the (normalized) retrieval sore of d or a funtion of its rank in L; s L(d) def 0 if d L. The goal of a fusion method is to merge the retrieved lists by assigning a sore F(d; q) to eah doument d in m L i. We next present a probabilisti framework for fusion. We use the framework to set formal grounds for ommonly used fusion priniples, to explain or formally derive many existing fusion methods, and to devise novel fusion approahes. 3.1 A probabilisti framework We rank doument d by its relevane likelihood: p(d q, R); R is the relevane event. Let θ x denote a representation of text x; e.g., a term-based tf.idf vetor in a vetor spae, or a term-based language model in the simplex. Suppose that we use some doument representation, θ d. Then, integrating over all possible query representations, θ q, yields: Z ˆp(d q, R) def p(θ d θ q, R)p(θ q q, R)dθ q; (1) θ q herein, ˆp( ) denotes an estimate for p( ); p(θ q q, R) is the likelihood that θ q effetively represents q for retrieval, as a relevane event happens; p(θ d θ q, R) is the relevane likelihood of θ d given θ q. Thus, an underlying premise is that the relevane likelihood of d s representation is independent of q given θ q. Approximating Equation 1 using a posterior-mode estimate (f., as in the risk minimization framework [19]) results in ranking d using a single estimate, ˆp(θ d θ q, R), ommitted to a single doument and query representations. This is the basis of the BIR and Okapi BM25 models [34] when using boolean and tf.idf term-based vetor representations, respetively. This is also the formal foundation of the doument likelihood model in the language modeling framework [14] 1. Rather than ommitting to a single query representation, we utilize multiple representations. One possible approah is to use multiple pseudo-feedbak-based query models [10, 37]. These term-based models are indued from doument lists retrieved for the query. Here, instead of using termbased representations indued from pseudo-relevant doument lists, we use the lists themselves as query representations. That is, eah list L i ( L) onstitutes a representation of q by the virtue of being retrieved by a method that ranks douments by presumed relevane to q. In other words, L i serves as q s representation in the spae of all ranked lists of douments from the orpus. A doument serves as its own representation as it onstitutes a single-item ranked list 2. Aordingly, we approximate Equation 1 by using only the lists in L as query representations: ˆp(d q, R) p(d L i, R)p(L i q, R). (2) The quality of this approximation depends on the number of retrieved lists and their likelihood to be effetive for q: p(l i q, R). With no relevane judgements for q, an estimate ˆp(L i q, R) is used. The estimate for the relevane likelihood of d given that L i serves as q s representation, ˆp(d L i, R), is heneforth referred to as doument-list relevane assoiation estimate. Using the estimates just desribed we arrive to: F linear (d; q) def ˆp(d L i, R)ˆp(L i q, R), (3) whih is a linear fusion approah as we further disuss below. By Equation 3, d is likely to be relevant if it is strongly assoiated with many retrieved lists that are effetive for q. Given that douments in L i are ranked by their estimated relevane, highly ranked douments should presumably be assigned with higher estimates of doument-list relevane assoiation (ˆp(d L i, R)) than lower ranked douments. Hene, Equation 3 is a probabilisti formal basis for the most fundamental fusion priniple whih integrates the skimming and horus effets mentioned in Setion 1: rewarding douments that are highly ranked in many lists. 1 Equation 1 is a oneptual analog of the basis for the Bayesian extension of the query likelihood method [46], where multiple doument language models and a single query representation are used. Here we do not assume a generative model. 2 More formally, all representations are essentially ranked lists of doument IDs.

3 3.1.1 Linear fusion methods Equation 3 is a formal basis for the lass of linear fusion methods [40]. Originally [40], these were heuristially presented as soring d by a linear ombination of its relevane sores in the lists (speifially, retrieval sores or ranks) with the lists weighed by presumed effetiveness. Yet, Equation 3 provides grounds to a larger lass of linear fusion methods, as it an be instantiated using various estimates ˆp(d L i, R). For example, d ( m L i) need not neessarily be in L i to yield a high estimate as its ontent similarity to douments in L i an serve for estimation. This is the priniple underlying, in spirit, the luster-based fusion approah [18]. Below we show that Equation 3 an be used to explain numerous existing fusion methods. These do not neessarily use proper probability estimates. Rather, measures that ould be viewed as orrelated with these probabilities are used; and, sum-normalizing the measures over douments for ˆp(d L i, R) and over lists for ˆp(L i q, R) yields probability distributions; the normalization does not affet ranking. We note that several of the most effetive linear fusion methods we explore impliitly use a uniform distribution for ˆp(L i q, R). We show in Setion 4 that using in these methods a non-uniform distribution, based on L i estimated effetiveness, an yield retrieval performane that transends state-of-the-art. This finding further attests to the merits of a formal derivation of linear fusion methods. It not only allows to shed light on existing methods, but also enables to methodologially devise new effetive fusion approahes. CombSUM, Borda, and Measure. The methods we disuss next are impliitly based on the assumption that all retrieved lists are effetive to the same extent: ˆp(L i q, R) def 1 m, where m is the number of lists. If the doument sores in L i retrieval sores or funtions of ranks are sum-normalized, i.e., P d L i s Li (d ) 1, then ˆp(d L i, R) def s Li (d) is a probability distribution over the orpus 3. Using these estimates in Equation 3 yields: F CombSUM (d; q) def 1 m s Li (d). (4) This soring funtion is rank equivalent to CombSUM that simply sums the sores of douments in the lists [17]. Setting s Li (d) def k r Li (d) and s Li (d) def 1 ν+r Li (d), respetively, in Equation 4, where ν is a free parameter, k is the number of douments in a list, and r Li (d) is d s rank in L i, results in soring funtions that are rank equivalent to those used by the Borda [3] and reiproal rank ( in short) [11] fusion methods, respetively. The measurebased fusion methods [4] are also instantiations of Equation 4 where doument sores are funtions of their ranks. Speifially, the most effetive rank-to-sore transformation among those onsidered [4], inspired by the AP (average preision) measure (heneforth Measure) is: s Li (d) def 1 + H k H rli (d); H j is the j th Harmoni number. 3 In pratie, various retrieval-sore normalization shemes may be applied as well as rank-to-sore transformations [23, 30, 45]. The resultant doument sores need not neessarily onstitute a probability distribution but an be normalized to that end as noted above. CombMAXandCombMIN. F CombSUM (d; q) in Equation 4 an be bound from above by max Li L s Li (d) and from below by min Li L s Li (d). These are the sores assigned by the CombMAX and CombMIN methods [17], respetively. WeightedCombSUM, Fuse, WeightedBorda. If we drop the assumption used above that all retrieved lists are effetive to the same extent, but still use sum-normalized doument sores for the doument-list relevane assoiation estimates, then Equation 3 beomes F WeightedCombSUM (d; q) def s Li (d)ˆp(l i q, R). (5) We refer to this fusion method as WeightedCombSUM 4. In some previous work on fusion, the estimate for L i s effetiveness likelihood, ˆp(L i q, R), was (impliitly) devised based on either the past retrieval effetiveness of the retrieval method that produed L i [40, 3, 26, 33] or query performane preditors [33]; past retrieval effetiveness was measured over a train set of queries. Setting s Li (d) def 1 r Li and s (d) Li (d) def k r Li (d) in WeightedCombSUM, respetively, and using the (sum normalized) past mean average preision () performane of the retrieval method for ˆp(L i q, R), yields the Fuse [26] and WeightedBorda [3] fusion methods, respetively. PosFuse,, ProbFuse, SegFuse, BayesFuse. The fusion methods disussed above use d s sore in L i, s Li (d), for ˆp(d L i, R) the relevane likelihood of d given L i. There are linear fusion methods where ˆp(d L i, R) is based, in spirit, on an estimate ˆp(r Li (d) M i, R); M i is the retrieval method that produed L i in response to q. This is an estimate for the likelihood that a doument at rank r Li (d) (i.e., d s rank in L i) of a list retrieved by M i for some query q would be relevant to q. The assumption underlying suh fusion methods is that the ranking patterns of retrieval methods, in terms of positioning of relevant and non-relevant douments, are independent of a speifi query. For example, in PosFuse [26], ˆp(r Li (d) M i, R) is the prevalene of queries in a train set for whih the retrieved list ontains a relevant doument at rank r Li (d). In [25], ˆp(r Li (d) M i, R) is the average estimate assigned by PosFuse to the ranks in a window around r Li (d) i.e., applies nearest-neighbors smoothing to the maximum likelihood estimate used by PosFuse. In both PosFuse and, ˆp(r Li (d) M i, R) serves for ˆp(d L i, R). In the ProbFuse [24] and SegFuse [36] methods, ˆp(r Li (d) M i, R) depends only on the blok (segment) of ranks whih r Li (d) is part of. Speifially, the estimate is the average over queries in the train set of the fration of ranks in the blok that is populated with relevant douments. In ProbFuse, ˆp(d L i, R) is obtained by saling the estimate just desribed with the reiproal rank of the blok in L i whih d s rank is part of. In SegFuse, ˆp(d L i, R) is obtained by saling the estimate with (1 + s Li (d)). In BayesFuse [3], ˆp(d L i, R) def log ˆp(r L i (d) M i,r) where ˆp(r Li (d) M i,nr) NR is the non-relevane event. The estimates in the quo- 4 There are fusion methods [35, 21] that onstitute a generalized version of WeightedCombSUM where Equation 3 is applied, and ˆp(d L i, R) and ˆp(L i q, R) are simultaneously learned using training data.

4 tient are devised in a similar way to that ˆp(r Li (d) M i, R) is devised in ProbFuse and SegFuse, i.e., using bloks (segments) of ranks. All methods just desribed (impliitly) use a uniform distribution estimate, 1 m, for ˆp(Li q, R) Instantiating linear fusion methods In Setion we showed that many existing linear fusion methods an be instantiated from Equation 3. These instantiations are based on the hoie of the doument-list relevane assoiation estimate, ˆp(d L i, R), and the list effetiveness estimate, ˆp(L i q, R). We now turn to desribe a suite of estimates that an be used to instantiate Equation 3. Using the estimates results in the methods disussed in Setion or in methods that are novel to this study. Doument-list relevane assoiation estimates. As noted, d s retrieval sore in L i serves in several fusion methods for ˆp(d L i, R). We explore a few retrieval-sore normalization shemes. SumNorm is the sum-normalization sheme where a sore is normalized with respet to the sum of sores in the list. Under the MinMaxNorm sheme [23, 30], the sore is shifted with respet to the minimal sore in the list and then normalized with respet to the differene between the maximal sore and minimal sore in the list. The ZNorm sheme is z-normalization: shifting the mean sore in the list to 0 and saling the variane to 1 [30] 5. In some of the methods disussed in Setion 3.1.1, a doument rank is transformed to a sore that serves for the doument-list relevane assoiation estimate. We onsider the following rank-to-sore transformations. The Borda [3] linear transformation: k r Li (d), where k is the size of the list and r Li (d) is d s rank in L 6 i. In ontrast, and Measure are non-linear transformations: 1 ν+r Li (d) and 1 + H k H rli (d), respetively; ν is a free parameter. These transformations are used in the [11] and Measure [4] methods, respetively, whih were disussed in Setion The next doument-list relevane estimates are those used by the [25], ProbFuse [24] and SegFuse [36] methods disussed in Setion In these methods, the estimate for doument d and list L i is based on the likelihood that any doument at rank r Li (d) in L i would be relevant. The likelihood is estimated using training data for the retrieval method, M i, that produed L i in response to q. List effetiveness estimates. For the likelihood of list effetiveness, ˆp(L i q, R), we onsider either a uniform distribution ( 1 ) as is the ase in many fusion methods, or measures based on the past performane of M i. As noted above, m the of M i over a train set was used in some work [3, 26, 33]. Here, we also onsider the average preision at 10 (p@10) of M i over the train set, denoted P, and the J measure [6]: the orrelation between the ranking of a list and its most effetive re-ranking attained by positioning the rel- 5 Variants of MinMaxNorm (Max and MM-Stdv) and ZNorm (UV) were proposed for federated searh over nonoverlapping orpora [28]. We found that these variants underperform the MinMaxNorm and ZNorm shemes applied here for fusion of lists from the same orpus. Atual numbers are omitted as they onvey no additional insight. 6 Lee [23] used a transformation attained from Borda s transformation by applying shift and sale: 1 r L i (d) 1 k. evant douments before the non-relevant ones; J was used in work on linear fusion [6, 40]. Query-performane predition methods [8], whih do not utilize training data, ould also be used to estimate list effetiveness. However, a reent study showed that there is little merit in doing so [33]. Methods. We use X-Y to denote a method instantiated from Equation 3 using X as the doument-list relevane assoiation estimate and Y as the list effetiveness estimate. If Y is the uniform distribution estimate, we simply use X to name the method. For example, the method [11] is essentially -uniform. The CombSUM method [17] (Equation 4) has several variants named X, instantiated by varying the retrieval-sore normalization sheme or the rankto-sore transformation: SumNorm, MinMaxNorm, ZNorm, Borda,, and Measure. The methods -Y, -Y, ProbFuse-Y and SegFuse-Y with Y {, P, J}, and all X-P methods are novel to this study CombMNZ We now turn to examine the CombMNZ method [17, 22] whih often serves as a referene omparison in work on fusion. CombMNZ sores d by F CombMNZ (d; q) def {L i : d L i} s Li (d), (6) whih P is rank equivalent to saling the CombSUM sore ( 1 m m sl i (d)) with the number of lists d is in. CombMNZ annot be diretly derived from Equation 3, as it is not a linear fusion method. To derive CombMNZ, we interpolate, using a free parameter λ, two estimates for p(d q, R). This results in a third estimate: We set ˆp 3(d q, R) def λˆp 1(d q, R) + (1 λ)ˆp 2(d q, R). (7) ˆp 1(d q, R) def 1 m s Li (d), (8) as in Equation 4 whih was derived from Equation 2 via Equation 3. We then define ˆp 2(d q, R) using the linear estimate from Equation 3 as follows. For d in L i, we set ˆp(d L i, R) def 1, where k is the number of douments in k the list; if d L i, ˆp(d L i, R) def 0. That is, we assume that all douments in a list are assoiated with it to the same extent; ˆp(d L i, R) is a probability distribution over the orpus. We also assume that all lists are effetive to the same extent: ˆp(L i q, R) def 1, as was the ase in deriving Equation m 4. We thus arrive to ˆp 2(d q, R) def 1 {Li : d Li}, (9) km whih gives rise to the NumLists fusion method: a doument is sored by the number of lists it is in. Plugging the estimates from Equations 8 and 9 in Equation 7 yields: ˆp 3(d q, R) λ m s Li (d) + (1 λ) {Li : d Li}, km

5 whih is rank equivalent to: F ArithCMNZ (d; q) def α s Li (d) + (1 α) {L i : d L i} ; (10) α is a free parameter. We term the soring method in Equation 10 ArithCMNZ as d is sored by the weighted arithmeti mean of the two fusion-based sores (estimates) used by CombMNZ; namely, the CombSUM sore and the NumLists sore 7. The sore assigned by ArithCMNZ in Equation 10 is bound from below by that assigned by GeoCMNZ, whih is the geometri mean of the two fusion-based sores 8 : F GeoCMNZ(d; q) def ` s Li (d) α` {Li : d L i} (1 α). (11) Thus, both ArithCMNZ and GeoCMNZ fuse two fusionbased doument sores. Eah of these sores is an estimate for p(d q, R) that results from applying linear fusion upon the retrieved lists 9. Hene, we refer to ArithCMNZ and GeoCMNZ as meta fusion methods. Now, setting α 0.5 in Equation 11 results in a soring funtion that is rank equivalent to that of CombMNZ in Equation 6. Thus, we get that CombMNZ is a speifi instantiation of a meta fusion method, namely GeoCMNZ. CombMNZ variants. Equation 8, whih is one of the fusion methods applied in CombMNZ, was derived as Equation 4 from Equation 2. Speifially, the doument sore used in CombMNZ, s Li (d), serves as one of two estimates for doument-list relevane assoiation. (The other is 1 used in k Equation 9.) As in Setion 3.1.2, to indue s Li (d), we an use the retrieval-sore normalization shemes: SumNorm, MinMaxNorm, ZNorm, and the rank-to-sore transformations: Borda, and Measure. The sores assigned by, ProbFuse and SegFuse an also serve for the assoiation estimates. We name the resultant methods CombMNZ- X, where X is a doument-list relevane assoiation estimate. The CombMNZ-X variants, exept for X {SumNorm, Min- MaxNorm, ZNorm, Borda}, are novel to this study EVALUATION In what follows we present an empirial evaluation of (i) the linear fusion methods instantiated from the probabilisti 7 The ArithCMNZ soring method is rank equivalent to a previously proposed ranking funtion [16]. This funtion was devised by assuming that all douments in the retrieved lists are generated by a multinomial mixture model with Dirihlet priors. Thus, the derivation of ArithCMNZ provides an alternative explanation for this approah. 8 The GeoCMNZ method is rank equivalent to a previously proposed method named CombGMNZ [23]. Hene, the derivation of GeoCMNZ using the probabilisti framework helps to set formal grounds for CombGMNZ [23] whih was presented as a heuristi alternative to CombMNZ. 9 ArithCMNZ and GeoCMNZ ould be viewed as fusing two ranked lists of douments. Eah of the two ontains all douments in m L i. The first list is ranked by CombSUM and the seond is ranked by NumLists. 10 While Borda s transformation was not used in CombMNZ, Lee s shift-and-sale variant of Borda s transformation [23], mentioned above, was. We found that the performane of the resulting two CombMNZ variants is the same. framework as desribed in Setion 3.1.2; some are existing methods while others are novel; and, (ii) the variants of the CombMNZ method and the MNZ meta fusion methods from Setion We start by desribing the experimental setup details in Setion Experimental setup Data and evaluation measures. We use for experiments runs submitted to traks of TREC. Eah run is produed by some retrieval method and ontains doument lists that were retrieved in response to the queries in a trak. The traks are detailed in Table 1. These are part of old and new TRECs and use newswire and Web doument olletions. The main experimental setting used for evaluation, unless otherwise stated, is as follows. We follow ommon pratie in work on fusion and fuse a relatively small number of randomly seleted runs (e.g., [30, 29, 24, 36]). Speifially, we randomly sample 10 runs from all those submitted to a trak. The fusion methods are applied for eah query in the trak on the m 10 orresponding doument lists in these runs. We use the k 1000 most highly ranked douments in eah list in a run as the retrieved list to be fused. If there are less than 1000 douments in a list, we use them all 11. We use 30 suh random samples of runs, and report the average retrieval performane over the samples. In Setion we study the effet of varying the number of lists that are fused, m, on performane. We show that the performane patterns are in line with those for m 10 whih onstitutes the main experimental setting. Runs that ontain douments with NaN sores and queries for whih there are no relevant douments were exluded from the evaluation 12. Some runs ontain negative retrieval sores. This has undesirable effet on the SumNorm retrievalsore normalization sheme. Thus, only for this sheme, as a pre-proessing step, all retrieval sores in all runs are shifted with respet to the minimal sore in the list. The ZNorm normalization sheme results in negative normalized sores. Thus, the integration with list effetiveness estimates in the linear fusion methods, and with NumLists in CombMNZ, is problemati. Therefore, only for the ZNorm sheme, as a post-proessing step, all the normalized sores in all runs are shifted with respet to the minimal sore in the list [30]. For evaluation measures we use mean average preision () at utoff 1000, and preision of the top 10 douments (p@10). Statistially signifiant differenes of performane are determined using the two-tailed paired t-test omputed at a 95% onfidene level based on the average performane per query over the 30 samples of runs. Free-parameter values. Unless stated otherwise, the freeparameter values of all the methods we evaluate are set using leave-one-out ross validation performed over queries; is optimized in the training phase using grid searh. A few of the fusion methods that we explore utilize a train set of queries to (i) determine the past performane of a retrieval method used to produe a doument list; and (ii) set 11 In these ases, the lowest rank in the list, k, used for omputing the Borda and Measure rank-to-sore transformations is set to 1000 to avoid bias. 12 All in all, there are two runs in TREC10 that ontain NaN sores. TREC18 and TREC19 have one and two queries, respetively, with no relevant douments.

6 Table 1: TREC data used for evaluation. # runs is the number of runs submitted to a trak. TREC Trak # dos Data Queries # runs TREC3 Ad ho 741,856 Disks 1& TREC7 Ad ho 528,155 Disks 4&5-CR TREC8 Ad ho 528,155 Disks 4&5-CR TREC9 Web 1,692,096 WT10G TREC10 Web 1,692,096 WT10G TREC12 Robust 528,155 Disks 4&5-CR TREC18 Web 50,220,423 ClueWeb (Category B) TREC19 Web 50,220,423 ClueWeb (Category B) the estimate for the probability that a doument positioned at a speifi rank of a list is relevant. The train query set, whih is omposed of all queries in a trak exluding the query held out for testing, is used to devise these estimates. We note that the past performane of a retrieval method over all queries exept for the held-out query does not neessarily onstitute a highly aurate estimate of the method effetiveness for this query. Indeed, the performane of retrieval methods substantially varies aross queries [41]. For example, we found that the performane of the - linear fusion method (see Setion 3.1.2), whih uses the rank-to-sore transformation and estimates list effetiveness based on past performane, is substantially and statistially signifiantly lower than that of -orale. In -orale, is used as a rank-to-sore transformation, but the list effetiveness estimate is the atual AP (average preision) attained for the query at hand. A ase in point, - posts.341,.269 and.278 performane for TREC7, TREC10 and TREC19, respetively; -orale posts.405,.332 and.342 for these three traks, respetively. The value ranges of free parameters are the following. In the rank-to-sore transformation [11] (see Setion 3.1.2), the ν parameter is set to values in {0, 10,..., 100, 500}. For [25], we set the window size to values in {1, 2, 5, 10, 20}. For ProbFuse [24], we use the variant whih treats unjudged douments in the train query set as non-relevant; the segment size is set to values in {2, 5, 10, 25, 50, 100, 500}. For SegFuse [36], we follow the original implementation [36] and set the segment size to (10 2 (i 1) ) 5, where i is the segment number in the list. The α parameter used in the ArithCMNZ and GeoCMNZ methods (see Setion 3.1.3) is seleted from {0, 0.1,..., 0.9, 0.95, 0.975, , 0.99}. 4.2 Experimental results In Setion we present an in-depth empirial evaluation of various linear fusion methods existing and novel instantiated from the framework using the estimates from Setion The naming onventions for the linear fusion methods, as stated in Setion 3.1.2, are: (i) X-Y for a method that uses both an X doument-list relevane assoiation estimate and a Y list effetiveness estimate; and (ii) X for a method that uses the X doument-list relevane assoiation estimate and a uniform list effetiveness estimate. Before diving into the omprehensive analysis in Setion 4.2.2, we present one of the main findings it gave rise to the empirial merits of highly effetive novel linear fusion methods instantiated from the framework with respet to existing state-of-the-art fusion methods Main result In Setion we show that the best performing linear fusion methods instantiated from the framework are Slide- Fuse-Y SegFuse-Y and -Y with Y {, P}. These methods are novel to this study. We also show that, in general, is a better performing list effetiveness estimate than P. Thus, in Table 2 we present the performane of -, SegFuse- and -. These three novel methods use the doument-list relevane assoiation estimate applied by [25], SegFuse [36] and [11], respetively, and also utilize the list effetiveness estimate;, SegFuse and are existing state-ofthe-art fusion methods used for referene. We also present for referene the performane of the best performing run among those fused. As additional baselines we use non-linear fusion methods: (i) CombMNZ- MinMaxNorm, a ommonly used CombMNZ variant [23, 30, 45] (see Setion for details); (ii) CombMNZ-SegFuse whih is novel to this study and is shown in Setion to outperform other CombMNZ variants we onsider; and, (iii) CondoretFuse [29]. Table 2 shows that most fusion methods outperform the best performing run among those fused. We also see that CondoretFuse is the least effetive fusion method in the table. The novel linear methods instantiated from the framework, -, SegFuse- and -, outperform their existing (state-of-the-art) ounterparts,, SegFuse and, respetively, in a vast majority of the relevant omparisons (8 traks 2 evaluation measures); most improvements are statistially signifiant. The few ases where the novel methods are outperformed by their existing ounterparts, albeit rarely to a statistially signifiant degree, are for the TREC18 and TREC19 traks. Note that and SegFuse use training data to estimate doument-list relevane assoiation. Our novel - and SegFuse- methods were instantiated by using the same training data to also estimate past performane of the retrieval method. Table 2 also shows that the ommonly used CombMNZ- MinMaxNorm is statistially signifiantly outperformed by CombMNZ-SegFuse whih is a novel variant instantiated based on our framework. Yet, CombMNZ-SegFuse, whih is a non-linear method that does not utilize a list effetiveness estimate, is outperformed in most relevant omparisons by the novel linear methods (-, SegFuse- and -) that utilize suh an estimate Comparative analysis of linear fusion methods We turn to present a omprehensive performane omparison of all linear fusion methods instantiated from our framework. Refer bak to Setion for details regarding the instantiations. Table 3 presents the performane numbers. In what follows, we ontrast the performane of two methods with respet to 16 relevant omparisons (8 traks 2 evaluation measures), and refer to the orresponding statistially signifiant differenes. We do not mark statistially signifiant differenes in Table 3 to avoid luttering. Doument-list relevane assoiation estimates. We ompare doument-list relevane assoiation estimates, referred to below also as doument-list relevane estimates, by assuming a uniform list effetiveness estimate. Aordingly, the estimates are ompared by the resultant performane of

7 Table 2: Main result table. Comparing the most effetive run among those fused (Best Run), existing state-of-the-art linear fusion methods instantiated from the framework ( [25], SegFuse [36] and [11]), highly effetive novel linear fusion methods instantiated from the framework (-, SegFuse- and -) and non-linear fusion methods (CombMNZ-MinMaxNorm [23, 30, 45], the novel CombMNZ-SegFuse variant, and CondoretFuse [29]). Bold: the best result in a olumn. a, b, and d mark statistially signifiant differenes with -, SegFuse-, - and CombMNZ- MinMaxNorm, respetively. TREC3 TREC7 TREC8 TREC9 TREC10 TREC12 TREC18 TREC19 Method p@10 p@10 p@10 p@10 p@10 p@10 p@10 p@10 Best Run SegFuse.373 a,b,d.689a,b,d.317a,b d.607,d.240 a,b.407 a,b.736 a.336 a,b,d.596a,b,d.343a,b,d.568a,b,d.257a,b,d.351a d.405 a,b.733 a,b.337 a,b,d.600a,b d.346 a,b d.572 a,b d.260 a,b,d.355a d.335 a,b.237 a,b.264a,b d.266a,b d a,b.416 a,b d.413 a,b.440 a,b.200 a,b,d.378a,b,d.221a,b,d.376a,b,d.300 a,b,d.471 d.246b d.463a,b d d.301 b,d.468 d.243b,d.448b.277 b a,b.729 a,d.312a,b.565 a,b,d.327a,b,d.553a,b,d.247a,b,d.337a,b,d.260a,b,d.409a,b d.297 a,b,d.468 d.226a,b,d.454 d d b,d.745b,d.358,d.620,d.357,d.585,d.270b d.364,d.275,d.429,d.302 d.471 d.243b,d.448b.275 b.435 SegFuse-.411 a d.737 a.356,d.619,d.357,d.590,d.266 a d.359 d.273 d.426,d.303 d.470 d.239 a,d.435a.272 a a d.738 a.341 a,b d.603 a,b d.348 a,b d.573 a,b d.266 d.356 a d.269 a d.413a,b d.302 d.470 d.251 a,b d.458 b d.278 b.439 CombMNZ-MinMaxNorm.404 a,b.735 a.307 a,b.574 a,b.316 a,b.543 a,b.229 a,b.320 a,b CombMNZ-SegFuse.409 a,d.738a.331 a,b,d.594a,b,d.339a,b,d.566a,b,d.258a,b,d.355 d CondoretFuse.372 a,b,d.706a,b,d.283a,b,d.551a,b,d.312a,b.540 a,b.233 a,b.322 a,b.248 a,b.266a,b d.234 a,b.402 a,b.414 a,b d,d.393a,b.290 a,b.299 a,b.457 a,b.217 a,b ,d.470 d.251a,b d.467a,b,d.289a,b,d.458a,b.457 a,b.161 a,b,d.360a,b,d.202a,b,d.397,d.285 a,b linearly fusing them. This amounts to applying the Comb- SUM method with varying doument-list relevane estimates. (See Setion for details.) We first ompare the retrieval-sore normalization shemes used to indue doument-list relevane estimates. Table 3 shows that MinMaxNorm and ZNorm outperform SumNorm in a vast majority of the relevant omparisons; most improvements are statistially signifiant. ZNorm outperforms MinMaxNorm in most relevant omparisons, but only a few improvements are statistially signifiant. In omparing the rank-to-sore transformations used to indue doument-list relevane estimates we find the following in Table 3. outperforms any other rank-to-sore transformation in a majority of the relevant omparisons. Most of the improvements over Borda are statistially signifiant while only a few of the improvements over Measure are statistially signifiant. Indeed, Measure is the seond best rank-to-sore transformation. In ontrast to Borda, and Measure are non-linear transformations. Furthermore, is the only transformation among the three that inorporates a free parameter. This an explain to some extent s relative effetiveness. (Reall that all parameters of all methods were set using ross validation.) Table 3 shows that the and Measure rank-to-sore transformations outperform in most relevant omparisons (often statistially signifiantly) the retrieval sore normalization shemes (MinMaxNorm, SumNorm and ZNorm). We note that in some past work [23] it was shown that using retrieval sores for doument-list relevane estimates yields better performane than using rank-to-sore transformations. However, the study was performed for CombMNZ, and took plae before [11] and Measure [4] were introdued. We next ompare [25], ProbFuse [24], and Seg- Fuse [36]. In these methods, the doument-list relevane estimate is based on the past performane of the retrieval method for the rank, or blok thereof, in whih the doument appears. Both and SegFuse outperform Prob- Fuse in all relevant omparisons with almost all improvements being statistially signifiant. While neither nor SegFuse dominates the other in a pairwise omparison, posts more (statistially signifiant) improvements over the other X fusion methods that use only doument-list relevane estimates without list-effetiveness estimates. Using a pairwise omparison of all doument-list relevane estimates, in terms of the number of relevant omparisons in whih one method outperforms the other, we find that is the best performing, SegFuse is the seond best and is the third best. is outperformed in a vast majority of the relevant omparisons by and Seg- Fuse. We note that and SegFuse use training data to estimate the relevane of a doument in a rank or blok of ranks, while is a rank-to-sore transformation whih relies on a free parameter tuned using training data. It is also interesting to note that outperforms ProbFuse in a majority of relevant omparisons, often to a statistially signifiant degree. ProbFuse as SegFuse and uses training data to estimate rank relevane. List effetiveness estimates. We next ompare the list effetiveness estimates, P and J when used together with the doument-list relevane estimates (i.e., the X-Y methods). In a vast majority of the relevant omparisons, X-Y is statistially signifiantly superior to X. Thus, we see that using list effetiveness estimates that are based on the past performane of the retrieval method is of muh merit. Overall, both the and P measures are very effetive and outperform the J measure to a statistially signifiant degree in a vast majority of the relevant omparisons. The measure outperforms the P measure in a majority of the relevant omparisons; the vast majority of these improvements are statistially signifiant. Further analysis of Table 3 reveals that the best performing methods in desending order of effetiveness are: -Y, SegFuse-Y and -Y with Y {, P}. As already noted, all six methods are novel to this study. Now, [25], SegFuse [11] and [11] are existing stateof-the-art methods whih were first introdued without integrating a list effetiveness estimate. Integrating eah of these three methods with the or P list effetiveness estimates yields statistially signifiant performane improvements in a vast majority of the relevant omparisons.

8 Table 3: All linear fusion methods instantiated from the framework. Bold: best result in a olumn. Method TREC3 TREC7 TREC8 TREC9 TREC10 TREC12 TREC18 TREC19 p@10 p@10 p@10 p@10 p@10 p@10 p@10 p@10 SumNorm MinMaxNorm ZNorm Borda Measure ProbFuse SegFuse SumNorm SumNorm -P SumNorm-J MinMaxNorm MinMaxNorm -P MinMaxNorm-J ZNorm ZNorm -P ZNorm-J Borda Borda -P Borda-J P J Measure Measure -P Measure-J P J ProbFuse ProbFuse -P ProbFuse-J SegFuse SegFuse -P SegFuse-J Summary of findings for linear fusion methods. is the most effetive rank-to-sore transformation; it also outperforms all retrieval-sore normalization shemes. The doument-list relevane estimates used by and Seg- Fuse yield the best fusion performane among all onsidered estimates; is somewhat more effetive than Seg- Fuse. Using list effetiveness estimates based on the past performane of the retrieval method is of muh merit; is the best performing list effetiveness estimate of the three non-uniform estimates studied. Finally, the most effetive methods, in desending order, are -Y, SegFuse-Y and -Y with Y {, P}; these are novel to this study. Varying the number of lists fused. Thusfar, we studied the performane of fusing m 10 runs whih are randomly seleted from all those submitted to a trak. In Figure 1 we study the performane when varying the number of randomly seleted runs that are fused: m {3, 5, 10, 20}. We present the performane of - and - whih were found above to be among the best performing linear methods; reall that these are novel to this study. (The performane of SegFuse- is very lose to that of - and is thus not depited to avoid luttering the figure.) To demonstrate the effet of using nonuniform list effetiveness estimates, we also present the performane of and whih do not use suh estimates. was found above to be the best performing linear method using a uniform list effetiveness estimate. We see in Figure 1 that the performane of all fusion methods monotonially inreases with inreased number of runs. Figure 1 also shows that in almost all ases - outperforms and - outperforms. These findings provide further support to the merits of using list effetiveness estimates. We also see that in most ases, - outperforms - and outperforms. Thus, we arrive to the onlusion that the relative performane patterns of the fusion methods onsidered here, when varying the number of fused runs, are in line with those presented above for fusing 10 runs CombMNZ In Table 4 we ompare the performane of the CombMNZ variants that were instantiated based on our framework in Setion 3.1.3; CombMNZ-X, exept for X {SumNorm, MinMaxNorm, ZNorm, Borda}, are novel to this study. The novel CombMNZ- variant, whih uses the rank-to-sore transformation, outperforms in most relevant omparisons all other variants that use rank-to-sore transformation or retrieval-sore normalization sheme (SumNorm, MinMaxNorm, ZNorm, Borda, Measure). Table 4 also shows that CombMNZ-SegFuse and CombMNZ-, variants novel to this study, post the best performane; CombMNZ- SegFuse posts more (statistially signifiant) improvements over the other variants than CombMNZ-. Next, we analyze the performane of the meta fusion methods from Setion 3.1.3: ArithCMNZ and GeoCMNZ. Reall that CombMNZ is an instane of GeoCMNZ with α 0.5. We use the MinMaxNorm retrieval-sore normalization to indue a doument-list relevane assoiation estimate. CombMNZ-MinMaxNorm is among the best per-

9 TREC3 TREC7 TREC8 TREC TREC10 TREC12 TREC18 TREC Figure 1: Fusing m randomly seleted runs. Note: figures are not to the same sale. Table 4: Comparison of the CombMNZ variants. Bold: best result in a olumn. a and b mark statistially signifiant differenes with CombMNZ- and CombMNZ-SegFuse, respetively. TREC3 TREC7 TREC8 TREC9 TREC10 TREC12 TREC18 TREC19 Method p@10 p@10 p@10 p@10 p@10 p@10 p@10 p@10 CombMNZ-SumNorm a b.327 a a b a b.310 a b.241 a b.380 a b.283 a b.433 a b.225 a b.433 a b.254 a b.396 a b CombMNZ-MinMaxNorm.404 a b a b.574 a b.316 a b.543 a b.229 a b.320 a b.248 a b.402 a b.290 a b.457 a b.217 a b.437 a b.270 b.457 CombMNZ-ZNorm.403 a b a b.567 a b.322 a b.549 a b.237 a b.328 a b.249 a b.404 a b.293 a b.461 a b.224 a b.447 b.269 b.447 CombMNZ-Borda.393 a b.709 a b.283 a b.516 a b.276 a b.473 a b.192 a b.274 a b.235 a b.383 a b.268 a b.428 a b.220 a b a b.436 CombMNZ-.404 a b.730 a b.313 a b.568 a b.327 a b.550 a b.246 a b.337 a b.258 b.406 a b.297 a b a b b.447 CombMNZ-Measure.400 a b.728 a b.300 a b.557 a b.308 a b.533 a b.228 a b.323 a b.251 a b.404 a b.287 a b.459 a b.226 a b b.452 CombMNZ b b b.437 b CombMNZ-ProbFuse.401 a b a a a b.347 b.257 b.407 a b.296 a b a b.451 b.282 a b.448 b CombMNZ-SegFuse a a a.458 a forming CombMNZ variants in past work [23, 30, 45]. We also use the SegFuse doument-list relevane estimate sine CombMNZ-SegFuse, whih is novel to this study, was the best performing in Table 4. In ontrast to CombMNZ, the ArithCMNZ and GeoCMNZ meta fusion methods rely on a free parameter, α. We set α either using leave-one-out ross validation as was the ase for all free parameters of all methods thusfar, or by average preision (AP) optimization on a per-query basis. The latter orale experiment is intended to demonstrate the potential of the meta fusion methods when using a highly effetive value of α. Performane numbers are presented in Table 5. Table 5 shows that in the orale setting, CombMNZ, whih is a speial ase of GeoCMNZ, is outperformed by ArithCMNZ and GeoCMNZ to a statistially signifiant degree in all relevant omparisons. In general, GeoCMNZ outperforms ArithCMNZ, sometimes to a statistially signifiant degree. When setting α using ross validation, and using Min- MaxNorm, ArithCMNZ and GeoCMNZ still outperform often to a statistially signifiant degree CombMNZ in most relevant omparisons; yet, the performane differenes are naturally smaller than those attained for the orale setting. When using SegFuse, GeoCMNZ (but not ArithCMNZ) performs as well as CombMNZ in most relevant omparisons. In ontrast to the ase here, Lee [23] did not find that a rank-equivalent version of GeoCMNZ an outperform CombMNZ. This is due to a different value range used for the α parameter, and the fat that SegFuse was not used. We onlude that CombMNZ is a speial ase of, and an potentially be signifiantly outperformed by, a meta fusion method, GeoCMNZ. Yet, the value of the free parameter α on whih GeoCMNZ relies has signifiant impat on its improvements over CombMNZ; these also vary with respet to the doument-list relevane assoiation estimate used. 5. CONCLUSIONS We presented a probabilisti framework for the fusion task. The framework provides formal ommon grounds to many seemingly different approahes. Instantiating the framework using various estimates gave rise to novel fusion methods that signifiantly outperform the state-of-the-art. Aknowledgments. We thank the reviewers for their omments. This work was supported in part by the Tehnion- Mirosoft Eletroni Commere Researh Center. 6. REFERENCES [1] A. Arampatzis and J. Kamps. A signal-to-noise approah to sore normalization. In Pro. of CIKM, pages , [2] A. Arampatzis and S. Robertson. Modeling sore distributions in information retrieval. Information Retrieval, 14(1):26 46, [3] J. A. Aslam and M. Montague. Models for metasearh. In Pro. of SIGIR, pages , [4] J. A. Aslam, V. Pavlu, and E. Yilmaz. Measure-based metasearh. In Pro. of SIGIR, pages , [5] N. Balasubramanian and J. Allan. Learning to selet rankers. In Pro. of SIGIR, pages , 2010.