B. Piwowarski P. Gallinari G. Dupret

Size: px

Start display at page:

Download "B. Piwowarski P. Gallinari G. Dupret"

Ethel Little
5 years ago
Views:

1 PRUM Precision Recall with User Modelling Application to XML B. Piwowarski P. Gallinari G. Dupret

2 Outline Why measuring? XML Information Retrieval Current metrics GPR, PRng, XCG PRUM A new user model A new metric Comparison

3 Why measuring? The evaluation methodology started early (Cleverdon, 1967) Justification of pragmatic/theoretic developments In new IR paradigms (XML, video, etc.), measures have the same role to play but none of the proposed metrics are satisfying.

4 XML Retrieval (and INEX) more precise access by giving more specific answers (ie. XML nodes) Assessments: a new scale Specificity (S): The extent to which a document component is focused on the information (4 values: 0 to 3) Exhaustivity (E): The extent to which the information contained in a document component satisfies the user's query need (4 values: 0 to 3)

5 Precision Recall E0S0 E3S2 E3S3 E0S0 Only exact answers are rewarded Too strict! Overlap problem in recall base E2S3 E1S3

6 Generalised Precision Recall E0S E2S E3S2 E3S3 E0S E1S3 q ExSy = Assign a relevance value to near misses (quantisation) Recall base is expanded Overlap is a problem

7 PRng (N. Gövert) Exhaustivity and specificity based on the notion of an ideal concept space upon which precision and recall are defined. Considers overlap in retrieval results Very sensitive to list order Does not consider overlap in recall-base (not completely true)

8 PRng (N. Gövert) 0.75 a (30) Recall(a,b,c) = x x 0.5 = b(20) c (10) 0.25 Recall(c,b,a) = x 1 + 1/3 x 0.75 = 1.25 Recall(c,b,a) = x 1 + 1/3 x 0.75 = 1.25

9 XCG (G. Kazai) Based on cumulated gain measure for IR Accumulates gain obtained by retrieving elements at fixed ranks Based on the construction of an ideal recallbase Consider overlap in both retrieval results and recall-base

10 XCG (G. Kazai) The most consistent metric Normalisation maximises the gain of near misses Normalisation to ensure that performance is bounded by an ideal system but No precise user model No precision dimension

11 Towards user models Web and others Quintana Dunlop leads to: Tolerance To Irrelevance (A. de Vries) The first real user model in XML IR Some theoretic problems

12 PRUM: main ideas Construction of an ideal recall base without overlap (like XCG) Definition of a precise user model (which includes T2I as a special case) Sound formal grounds (based on Raghavan's probabilistic precision-recall)

13 The PRUM user model Retrieved list a b c d e f g h j b a c k e d g h i f The user wants to see 4 relevant elements

14 The PRUM user model Consulted elements a Seen elements a b c k e d P=0 f g h i f The user has seen 0 relevant element

15 The PRUM user model Consulted elements a Seen elements a b b c k leads to an unseen relevant e d P=1/2 j g h i f The user has seen 1 relevant element

16 The PRUM user model Consulted elements a Seen elements a b c b c k e d j g h i f P=2/3 The user has seen 2 relevant elements

17 The PRUM user model Consulted elements a Seen elements a b c d b c k e d j g h i f P=2/4 The user has seen 2 relevant elements

18 The PRUM user model Consulted elements a Seen elements a b c d e e b c k d j g h i f P=3/5 The user has seen 4 relevant elements

19 PRUM stochastic user model x 1 2 x 1 2 x x 1 x 2/ x 1 2

20 PRUM stochastic user model x P(x->)

21 PRUM definition Extends Raghavan's definition P Lur Cs, L=l,Q=q The element Leads to an Unseen Relevant The element is Consulted by the user The user wants to see l % of the relevant elements

22 Computing PRUM Two probabilities have to be computed at each rank in the consulted list: The probability that the user has seen s distinct relevant elements P F i =s The probability that the user sees a new relevant element (knowing that ) P F i F i 1 F i 1 =s

23 Experiments (INEX 2004) 5 hypothetic systems 1) ideal (I) = ancestors (A) 2) parent (P) ~ biggest child (B) 3) document (D) Orders GRP: A >> I > P > D >> B RPng: A > I ~ P > D >> B PRUM: I = A >> B >> P >> D

24 Conclusion PRUM Extension of Precision-Recall (probabilistic) Explicit parameters ( GPR, PRng, XCG) Good behaviour of the metric Adapted to several IR formalisms Work to be done More realistic user model parameters Shortcomings Graded scale Complexity

Applying the Divergence From Randomness Approach for Content-Only Search in XML Documents

Applying the Divergence From Randomness Approach for Content-Only Search in XML Documents Mohammad Abolhassani and Norbert Fuhr Institute of Informatics and Interactive Systems, University of Duisburg-Essen,