Exploring Large Digital Library Collections using a Map-based Visualisation

Size: px

Start display at page:

Download "Exploring Large Digital Library Collections using a Map-based Visualisation"

Elfreda Fay West
6 years ago
Views:

1 Exploring Large Digital Library Collections using a Map-based Visualisation Dr Mark Hall Research Seminar, Department of Computing, Edge Hill University

2 The information access problem

3 The information access problem

4 The information access problem Search works If you know what you are looking for If you know what the right keywords are for the collection If you are looking for a specific thing Search does not work If you don t know what you are looking for If you don t know what the right keywords are If you are looking for an overview over a topic If you want to find out what kind of things a collection contains

5 The information access problem Mass digitisation has created a scaling problem Europeana The European Digital Library > 24 million records The UK National Archives > 11 million records The British Library > 56 million records

Alternative access methodologies Recommendation Facetted search Visualisations

6 Alternative access methodologies Recommendation Facetted search Visualisations

7 Spatialisation Turn a higher-dimensional semantic space into a two-dimensional representation Map similarity in the higher-dimensional space into distance in the two-dimensional space Provides a visual overview over the topics in a collection People readily understand the distance similarity metaphor

8 Spatialisation A number of algorithms exist Multi-Dimensional Scaling Self-Organising Maps Issues Computationally complex Semantic overload Interpretation problems

9 Potential solution Use hierarchical structures to overcome the issues Everything Agriculture Arts Culture Technology Art Craft Design Visual arts Artisans Crochet Watchmaker Each topic can be processed independently Structure can be used to provide visual summaries

10 Hierarchical spatialisation algorithm 1. Pre-processing 1. Tree pruning 2. Item pruning 3. Vectorisation 2. Spatialisation 1. Initial spatialisation 2. Final positioning 3. Post-processing

11 Pre-processing Ensures that the hierarchy is compatible with the core algorithm Hierarchy must be a full tree Items must only be assigned to leaf topics in the tree Ensures that all items & topics have the necessary pre-calculated data for the spatialisation

12 Tree pruning Transforms the hierarchy from a Directed-Acyclical Graph to a tree

13 Item pruning Ensures that items are only assigned to leaf topics

14 Vectorisation Each object to spatialise with MDS must be defined via a vector Extract keywords from titles and descriptions of items Filter keywords that appear less than 5 times in the collection or in more than half the documents From the keywords use TFIDF (term frequency inverse document frequency) to create the vectors Items Use item s keywords Topics Use the keywords of all items that belong to the topic or to one of its child topics tf t, d = f(t) d D idf t, D = log d D: t d tfidf t, d, D = tf idf

15 Core spatialisation Hierarchy is spatialised bottom-up Parent topic is spatialised after all its children have been spatialised

16 Core spatialisation Degenerate MDS Initial spatialisation Neighbourhood graph Final, compact spatialisation

17 Parallelisation Use the inverse tree as an activation graph Everything Agriculture Arts Culture Technology Art Craft Design Visual arts Artisans Crochet Watchmaker

18 Parallelisation Enables the algorithm to scale to large data-sets items processed in ~16 hours on a multi-core desktop processor Limited by the shared map storage backend

19 Placement Due to the parallel nature of the algorithm topic areas will overlap

20 Post-processing Re-calculate boundaries to achieve visual attractivity

21 Semantic map

22 Semantic map Generally provides overviewing and exploration support Hierarchy provides overview labels at higher zoom levels Interaction follows the widely adopted Googlemaps pattern (zoom / pan) At lower zoom levels allows interaction with individual items Provides a natural interface for touch-based devices

23 Semantic map Algorithm written in Python Data stored in PostgreSQL + PostGIS database Individual tiles rendered using Mapnik for the actual rendering TileLite for caching and serving Web-based user interface provided via Leaflet

24 Where next? Evaluation Support continuous updates to the map Create more natural boundaries

25 Thank you Questions? See a demo at

Fall 2018: Introduction to Data Science GIRI NARASIMHAN, SCIS, FIU

Fall 2018: Introduction to Data Science GIRI NARASIMHAN, SCIS, FIU !2 Data Wrangling !3 Database Join (Python merge) unames = ['user_id', 'gender', 'age', 'occupation', 'zip'] users = pd.read_table('data/ml-1m/users.dat',