Digital Library Evaluation by Analysis of User Retrieval Patterns

Transcription

1 Digital Library Evaluation by Analysis of User Retrieval Patterns Johan Bollen, Somasekhar Vemulappali and Weining Xu Old Dominion University Department of Computer Science jbollen September 15, 2002 Page 1

2 Problem Statement 1. Evaluation of DLs (a) Collection evaluation: impact, relations among records in collection (b) User community: characteristics and structure. 2. General Approach (a) User retrieval patterns recorded in DL server logs determine network of document relations (b) Document impact and user community studied by proxy of graph-theoretical characteristics of document networks 3. Structure of presentation: (a) Methodology for adaptive linking from user retrieval patterns (b) Determination of document impact measures (c) Characteristics of user community September 15, 2002 Page 2

3 Journal Impact 1. De facto standard is ISI JCR citation database (a) Number of citations to article in journals (b) Normalized by number of published articles 2. Impact for year x, journal i at year x IF x i A B (a) A = number of citations 2 years preceeding x (b) B = number of articles published 2 years preceeding x 3. Integral part of the evaluation of: (a) Journal impact (b) Quality of research (individuals, departments, countries, DLs) (c) DL impact: IFs for collection. September 15, 2002 Page 3

4 Citation Frequency and the ISI IF 1. Validity: (a) Author motivations (b) Biases (c) Do citations indicate agreement, opposition, impact, recommendations? (d) Few authors, many readers. 2. Technical reports, digital publications? (a) Citations of gray publications are not counted (b) Web page can be frequently read, but not often cited: is it not influential? 3. Citations: static and universal (a) do not change over time (b) do not adapted to specific user community September 15, 2002 Page 4

5 Our approach 1. Focus is user community, not author citations 2. Usage defines document relations (a) User community retrieval patterns determine doc relations (b) Co-retrieval: evidence strengthens document relation weight (c) Goal: capture document relations from collective retrieval patterns (d) Document networks analysis (graph-theory) i. Document centrality: impact ii. Structure indicates characteristics of user community September 15, 2002 Page 5

6 Basic principles of document network generation 1. User retrieval behavior exhibits certain regularities (a) User interests result in propensity for retrieval of related documents (b) Documents frequently retrieved in close temporal proximity are associatively related to a degree corresponding to the retrieval frequency. 2. DL server logs: definition of co-retrieval event (a) A pair of any two retrieval requests by the same user within a given period of time labeled t. (b) Reconstructed from sequences of retrievals found in DL server logs (c) Retrieval: broad category (any token of user interest) 3. Pitfalls (a) DL needs to maintain retrieval logs (b) IP often used as user ID (c) Operates on statistical regularities, i.e. pertains to user community, NOT individual users September 15, 2002 Page 6

7 Generation of co-retrieval events userid data/time Document ID Service /Aug/2001:17:12: ;14;4;69 ddacfrtip SciServer /Aug/2001:17:14: ;25;4;499 prtuauma SciServer /Aug/2001:17:15: ;19;2;281 apiaaaortars SciServer /Jun/2001:12:03: ;38;3;347 otfrim SciServer /Jun/2001:12:04: ;19;3;211 aotdfmfct SciServer /Jun/2001:13:13: ;25;2;113 asrgt3tr SciServer Table 1: An example of the transformed Science Server log used to reconstruct journal and document co-retrieval events. September 15, 2002 Page 7

8 100 ( , ) ( , ) 101 ( , ) ( , ) Table 2: Co-retrieval events derived from set of user retrieval requests in table 1. September 15, 2002 Page 8

9 Generation of document networks Document network: G V E W V v 0 v 1 v n E V 2 i jm i j 0 for (i=1; i n+1; i++) e i v i v j t v i v j : m i j f e i W : E 0. n n matrix M set of co-retrieval events C c 1 c 2 c k c i v i v j t v i v j each co-retrieval c i m i j r r f c i f c k is referred to as reinforcement function for this presentation f c k 1 September 15, 2002 Page 9

10 Early applications to hypertext Systems 1. Principia Cybernetica Web site (U. Brussels) nodes, 300,000 log lines 3. Symmetric and Transitive closure (a) To increase matrix density (b) Same linking methodology as discussed 4. Used mainly for analysis purposes and recommendation systems September 15, 2002 Page 10

11 action movement change influence power authority knowledge control system method experience reason development mind education thought body research PCP server log graph person life September 15, 2002 Page 11

12 Network Matrix number label (1) act (2) action (3) age (4) amount (5) area (6) art (7) attention (8) authority (9) bed (10) blood Table 3: Sample of matrix representing final structure of adaptive hypertext network trained in experiment 1 September 15, 2002 Page 12

13 Digital Library Applications 1. Los Alamos National Laboratory Research Library (LANL RL): (a) Large digital library with focus on management of own material (b) User community: focus on physics and nuclear science (c) SciSearch database: +20,000,000 articles (d) Downloadable: PDF format after search and metadata retrieval 2. Availability of server logs from 1998 to present (a) Download rates: 10,000/mo. (b) Server logs used: 2001 (c) Log file used: 5 months, 40,847 requests, 1829 journals and 20,720 articles September 15, 2002 Page 13

14 Methodology 1. Reconstruction of co-retrieval events from LANL RL Server logs. (a) Logs contain IP number, date and time of request and document ID (b) Reconstruct sequences of retrieval requests by users i. Sort server log by user and date-time of request. ii. Each pair of documents retrieved by same user within certain period of time: co-retrieval 2. Generate Associative Network of Journals (a) Co-retrieval: link weight between co-retrieved document increases (b) Scan all co-retrieval events and apply learning rules (c) Result: document network 3. For LANL RL (a) First prototype: journal networks for scale (b) Every article appears in journal (c) Ability to validate network structure to existing data (Journal Citation Records) September 15, 2002 Page 14

15 LANL RL full-text retrieval September 15, 2002 Page 15

16 September 15, 2002 Page 16

17 Analysis application September 15, 2002 Page 17

18 Generated Network journals for 20,720 documents 2. density 0.176% 3. min(m i j ) = 1, max(m i j ) = mean link weight = 1.196, standard deviation=0.821 for all m i j : m i j 0 September 15, 2002 Page 18

19 Matrix Journal ADV COLLOID INTERFAC ADVAN MATH ANALYT CHIM ANALYT BIOC ANIMAL BEHAVIOUR ANN PHYS-NEW YORK APPLIED ACOUSTICS APPL PHYS LETT ARCH RATION MECH AN ARCH BIOCHEM BIOPHYS ARTIF INTELL ASTRONOM J PUBL ASTRON SOC JPN ASTRON ASTROPHYS ASTROPHYSICAL JOURNAL September 15, 2002 Page 19

20 Five strongest connections Frequency Start node End Node 22.0 PHYSICA C PHYSICA B 18.0 PHYSICA B PHYSICA C 14.0 NUCL INSTRUM METH A IEEE T NUCL SCI 12.0 PHYSICA B J MAGN MAGN MATER 11.0 J MAGN MAGN MATER PHYSICA B Table 4: Five pairs of journals for which highest co-retrieval frequencies have been found. September 15, 2002 Page 20

21 Relations Links from NUCL INST METH A ISSN Weight Title IEEE T NUCL SCI FUSION ENG DES APPL RADIAT ISOTOPES NUCL PHYS B-PROC SUP RADIAT MEAS PROG NUCL ENERG J RADIOANAL NUCL CH J NUCL MATER ANN NUCL ENERGY x 2.0 NUCL INSTRUM METH B September 15, 2002 Page 21

22 Analysis 1. Which journals have highest impact according to network structure? (a) Network structure corresponds to user community (b) Relations defined by users instead of those defined by authors 2. How do user-based ranking differ from ISI IF, and where? (a) Differences reveal community-specific characteristics (b) Contrast general level impact vs. community-specific September 15, 2002 Page 22

23 Journal Consultation Frequency as an expression of journal impact F v i n m i j j 1 n m ji (1) j 1 1. Corresponds to journals that have highest in- and out-degree 2. Most central to user retrieval patterns 3. Can therefore be considered highest impact September 15, 2002 Page 23

24 Impact Factor Discrepancy Ratio Journal usage frequency confounds two distinct factors: 1. Journal relevance or impact to a specific user community. 2. General impact or relevance of a journal to the larger scientific community. IFDR=ratio of user impact over ISI IF. High IFDR = high local impact, low general IF Low IFDR = low local impact regardless of high IF r f i JCF i IF i (2) September 15, 2002 Page 24

25 JCF values for LANL RL: 15 top JCF journals Frequency IF Journal Title PHYSICA B NUCL INSTRUM METH A APPL CATAL A-GEN MAT SCI ENG A-STRUCT J ALLOY COMPD IEEE T NUCL SCI CATAL TODAY J CATAL SURF SCI J NUCL MATER Table 5: List of 10 most frequent journals in LANL RL Science Server network and 2000 Impact Factors. September 15, 2002 Page 25

26 JCF values compared to ISI IF JCF vs. IF 1. IF(2000) retrieved. 2. Spearman rank order correlation caculated over non-zero entries IF(2000) JCF - IF: 0.13 (p 0 05) JCF September 15, 2002 Page 26

27 JCF and IF ratio r f JCF IF Journal Title ASTROPHYS SPACE SCI NUCL ENG DES PHYSICA B J MATER PROCESS TECH ADV SPACE RES ACTA ASTRONAUT J RADIOANAL NUCL CH PROG NUCL ENERG NUCL INSTRUM METH A MATH COMPUT SIMULAT Table 6: List of 10 journals for which highest ratio of JCF and ISI IF has been found. September 15, 2002 Page 27

28 Conclusion 1. Document linking (a) General methodlogy to link documents (not just journals) (b) User community specific links (c) Low cost generation (d) Independent of keyterms and metadata (e) Can be applied to electronic publications, tech reports etc 2. DL evaluation (a) Generated networks correspond to characteristics of user community (b) Impact measures derived from DL log files (c) Large discrepancy with ISI IF (authors vs. readers?) (d) Discrepancy reveals specifics of community 3. Future: (a) Track sub-communities by cluster analysis and MDS (b) Study evolution of user communities (c) Generation of set of freely available journal and document impact norms September 15, 2002 Page 28