Study on LOD requirement main findings

Transcription

1 Study on LOD requirement main findings ESS LOD 2017 Malta, 18 Dr. Nikolaos Loutas Data & Analytics A study delivered by EU Services for Eurostat

2 Agenda Scope and objectives of the study What is linked data Our approach Our findings 2

3 Scope and objectives of the study The identification and study of LOD initiatives in the ESS The identification and assessment of implementations, supported use cases and public learning resources for LOD in statistics The identification of current initiatives and projects on LOD in statistics The proposition of proof of concepts that demonstrate the benefits of LOD for official statistics The identification of a high level architecture for LOD in statistics The definition of a joint LOD strategy at ESS level 3

4 What is linked data Linked data is a set of design principles for connecting and sharing machinereadable data within and across organisations, and on the Web. The four design principles of Linked Data (by Tim Berners Lee): 1. Use Uniform Resource Identifiers (URIs) as names for things. 2. Use HTTP URIs so that people can look up those names. 3. When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL). 4. Include links to other URIs so that they can discover more things. 4

5 Our approach Desk research On site visits Audioconference Use cases of LOD in statistics LOD benefits, challenges and good practices High level architecture for LOD in statistics Proposal for a joint LOD strategy for the ESS 5

6 Your contributions Alain Nadeau (FSO, Switzerland) Darren Barnes (ONS, United Kingdom) Eoin McCuirc (CSO, Ireland) Monica Scannapieco (ISTAT, Italy) Franck Cotton (INSEE, France) Phil Archer (W3C) Bill Roberts (Swirrl, United Kingdom) Evangelos Kalampokis (Centre for Research and Technology Hellas) Joanne Hillermann (Australian BoS, Australia) 6

7 Our analysis framework Key Partnerships Key Activities Value Propositions Channels Customer Segments Key Resources Cost Structure Revenue Systems All credits to businessmodelgeneration.com 7

8 Our findings Why are NSIs using Linked Open Data? Interconnect several official statistics datasets (covering both data and metadata) housed in different databases, data stores and data warehouses within a NSI. 1 Interconnect several official statistics datasets (covering both data and metadata) housed in different databases, data stores and data warehouses of different NSIs and/or Eurostat. 2 Publish official statistics in a linkable, machine-readable format, which can easily be reused and integrated with other types of data, e.g. geospatial, weather, etc. 3 8

9 Our findings Value propositions Flexible data integration within NSIs and with others LOD facilitates statistical data integration and enables the interconnection of previously disparate datasets of different formats and type, housed in different data warehouses, data bases, data stores and files within and across NSIs. Promoting data standardisation LOD allows and encourages NSIs to enhance the statistical data and metadata standardisation process. LOD in statistics heavily capitalises on and promotes standards, such as HTTP URIs, RDF, SDMX and StatDCAT-AP. Creation of partnerships Integration of data requires collaboration on LOD initiatives and creation of partnerships between NSIs as well as with research and industry. Innovation and development of new services Availability of LOD and its easy integration with other internal or external data (both statistical and from other data domains) can give rise to new services offered by the NSIs or other organisations. Increase in data quality The increased use of LOD triggers demands to improve data quality. Through crowdsourcing, feedback and self-service mechanisms, errors can be progressively corrected. 9

10 Our findings Overview of use cases from the ESS Interconnect datasets within a NSI LOD for territorial bases (ISTAT, Italy) Selecting the best place to live or to invest (Maynooth University) LOD for fact-checking Finding data for a postcode (ONS Geography) Accessing and querying census data (CSO, Ireland) Evolution of Swiss communes (FSO, Switzerland) Interconnect official statistics datasets from different NSIs and/or Eurostat Scottish Index of Multiple Deprivation (Scottish Government) Relate/correlate different sources which provide information about a specific domain (Evangelos Kalampokis) Publish official statistics in machine-readable, linkable formats Providing catalogues of linked metadata of open datasets (EU Open Data Portal and European Data Portal) ModernStats - Linked Open Metadata (UNECE) Integrated access to EU and BEA data (Eurostat and BEA) Digital Agenda Scoreboard (DG CONNECT)

11 Our findings Examples of use cases Metadata catalogue of the EU Open Data Portal and the European Data Portal In production Gives access to data and metadata of institutions and other bodies of the EU. Based on common encoding rules and standardized vocabularies. Available as Linked Data. Accessible through API (REST and SPARQL).. Target audience Data Publishers: EU institutions and agencies Data Users: academia, data journalists, developers, NGOs, businesses, public administrations, etc. Integrated access to EU and BEA data Prototype Allows to access and compare data series (e.g. GDP) published by the US and the EU. Consist in a R library that allows to: List available integrated datasets; Describe the structure of the output integrated datasets; Fetch information on original datasets; and Fetch the integration logic (e.g. structural transformations, transcodings). Data Publishers: Eurostat, BEA Data Users: academia, researchers, data journalists, developers, businesses, public administrations, standard organisations, citizens, etc. 11

12 Our findings Examples of use cases Scottish Index of Multiple Deprivation (SIMD) In production Deprivation is a challenge with multiple dimensions (e.g. education, employment, crime, health) The service provides a consistent relative ranking of the data zones based on multiple datasets. Each of the dimensions can be ranked individually. Results are displayed on an interactive map. LOD for fact-checking Study The increased publication of data required a way to automatize fact checking The proposed solution follows the following steps: Retrieving the structure of the relevant dataset; Retrieving code lists that provide values; Retrieving the value of the required observation; and Validate the information. Target audience Data Publishers: Scottish Government Data Users: academia, researchers, data journalists, businesses, local public administrations, citizens, NGOs, etc. Data Publishers: Statistical institutions (NSOs, Eurostat, BEA, etc.) Data Users: academia, researchers, data journalists, citizens, NGOs, etc. 12

13 Our findings Examples of use cases Selecting the best place to live or to invest In production Provides a website that allows to visualise the census 2011 data of Ireland. Allows to explore different dimensions (e.g. religion, housing type, labour forces, general health.). Variables are mapped at Electoral Division (ED) and Small Area (SA) spatial scales Digital Agenda Scoreboard In production Provides a visualisation tool which allows the user to explore open datasets through the interactive creation of charts and tables. Data can be downloaded, including the metadata providing labels, definitions, methodological notes, references and sources. Datasets are stored in a semantic data repository to facilitate their reuse and the linking to other data published on the web. Target audience Data Publishers: Central Statistics Office, Ireland. Data Users: academia, researchers, data journalists, developers, businesses, public administrations, citizens, NGOs, etc. Data Publishers: DG CONNECT Data Users: academia, researchers, data journalists, developers, businesses, public administrations, standard organisations, citizens, NGOs, etc. 13

14 Our findings Cost structure Development Maintenance Promotion Subcontracting creation of tools In-house development Technical maintenance, only limited as in most cases LOD is in pilot phase Communication and publicity, contests, prizes Licensing Licensing for LOD tools in case open source solutions are not used 14

15 Our findings Customer segments NSIs and Eurostat Public administrations Businesses NGOs Data journalists Academia and researchers Citizens 15

16 Our findings Channels NSIs portals Browser-based access to LOD, e.g. intuitive link navigation Endpoint/API SPARQL, REST, URI dereferencing Mobile Apps Customer relationships Contests Hackathons, app contests, prizes Feedback Customer support, CRM 16

17 Our findings Key resources Data & Metadata Web standards, such as HTTP URIs and RDF Data standards, such as SDMX, RDF Data Cube and StatDCAT-AP. Technology & Infrastructure Building blocks frequently used: Data preparation SPARQL endpoint Website JSON-stat API REST API Data browsers Linked Data Governance People & Capabilities Define overall priorities with respect to the main value proposition Performing common analysis and on-going evaluation Data licensing: most common licence is Creative Commons Attribution 4.0. URI policy: to guarantee persistence, resolvability, and uniformity of Web identifiers. Skill may be available in house, outsourced or a combination of both. Skills typically include: Technical skills (e.g. PHP, JAVA, data management, data quality) LOD knowledge (e.g. LOD principles, standards and technologies) Communication and team skills (e.g. enhanced communication) Statistical knowledge 17

18 Our findings Key partners Industry Academia Outsourcing technical development, consultancy Knowledge and expertise sharing, common projects, tools development Key activities Requirements Development Maintenance Pre-implementation analysis, on-going evaluation Selection of data, creation of tools to transform, link, publish and visualise data Governance, management, user support Promotion Communication and publicity 18

19 Our findings Enablers and good practices for setting up successfully a LOD service Enablers Flexible way of integrating data, with minimum impact on current infrastructure Allows to access data at different levels of granularity from data points to datasets Promotes the need for data standardisation Opportunities for new data-enabled services Ease of data navigation via browsing the links (URIs) Ease of model updates because of the flexibility of RDF Emerging best practice guidance Creates partnerships between public administration, academia, standards organisations and industry Good practices Identify clear use cases, target users and benefits Start small, think big Use a trial and error approach with welldefined iterations Look for support and knowledge from the community and collaborate Rely on standards for data and metadata contribute to standardisation discussions Provide different ways of accessing the data, from APIs to visual interfaces, to cater for different types of users Provide persistent URIs and open licensing Measure the use and the expected benefits 19

20 Our findings Roadblocks for LOD in statistics Roadblocks for LOD in statistics Low awareness within the ESS of the technology and its benefits Insufficient promotion of success stories and implemented use cases Perceived lack of users demand for LOD Lack of management buy-in and support Organisational resistance because of changes in dissemination of official statistics (new technology, new data formats, new data standards ) Scarcity of necessary skills and competencies in the ESS Proliferation of so-called standards. The ESS is not partaking in the development of standards for LOD in statistics Uncertainty of persistence of the URIs and the data behind them; lack of service level guarantees An initial investment is required 20

21 Annex

22 Overview Central Statistics Office Ireland (L)OD Current initiatives (L)OD Future plans (L)OD Main users OD: publishing around 4646 datasets which reach at least 3*. More than 90 publishers and 14 domains. LOD: POC on Census data, published as LOD. Project to classify the geography levels in Ireland. Two competitions to encourage and promote the reuse of OD and LOD. App competition: participants had to use data from Census BT Young scientist: it targets students. (L)OD: there is a clear intention to expand the supply of OD/LOD. The future plans include, for example, the correct implementation of persistent URIs. CSO hopes to reach 4*for all their data by the end of OD main users: Tourist: person interested in high level information; Farmer: person interested in historical data/deeper data; and Miner: person who wants all data. LOD main users: no tracking available Key partners: Insight: technical development; and DEPR: OD aspects. LOD Enablers/Inhibitors Data skills and capabilities; Easy access and use; and Publishing data in an open format and licence. Fear about the effort needed to deploy LOD Lack of LOD understanding Limited reuse LOD Benefits Expected: Data integration; Data quality; New services; Creation of partnerships; and Data standardisation processes. Observed: N/A Aware of the EDP: Planned integration with the EDP: 22

23 Overview Central Statistics Office Ireland LOD Infrastructure LOD Skills/competencies LOD Risks/Issues Based on the principle single source multiple output where the input data format is PC-AXIS and multiple outputs are generated through a transformation process. The generation of LOD is done through the following transformation PC- AXIS -JSON RDF. There is a simple server for LOD. URI policy: YES The understanding of the data available and the infrastructure used to publish them (in-house); and The understanding of the semantic web for the implementation (external). Lack of skills and capabilities; Too technical ways of communication; Performance; Persistence of URIs; and Organisational resistance LOD Investment/FTEs Investment: an increase of cost is expected to switch from OD to LOD (training, servers, etc.) but operating costs should stay stable. Development costs; Maintenance of the server ; Working time; and Promotion FTE: No resources working full time. On average it represents 1/5 of an FTE. LOD Distribution Channels: Web API Website; and Mobile app. Formats: PX; JSON; RDF; TSV; and CSV Licences: Creative Commons Attribution (version 4.0 cc-by) LOD Learning resources Used: Books; Web; and Conferences Provided: Aware of StatDCAT-AP: Planned implementation of StatDCAT-AP: 23

24 Overview Office for National Statistics UK (L)OD Current initiatives (L)OD Future plans (L)OD Main users OD: publishing OD for more than 15 years. More than 1000 OD spreadsheets available. LOD: statistics.data.gov.uk allows the access to geography products in a machinereadable (LD) form. An API is provided. ONS is currently carrying out a small LD experiment related to geography data. OD: more reusable data and open standards. LOD: The next phase of the LD experiment will look at interlinking and disseminating a bigger number of datasets: Internal (ONS data): population, housing; and External (wider governmental experiment): education, health, economic, labour-market. High-fidelity prototype tool to publish data as LOD. OD main users: local authorities, Academia, journalists, Parliament, Central Bank, treasury departments, charity organisations. LOD main users: no tracking available. Key partners: SWIRRL LOD Enablers/Inhibitors Interoperability: standards and well connected systems, URIs; More LOD users; LOD trainings and guidance; and Support from a political perspective Not enough demand; and Technological aspects LOD Benefits Expected: Data integration; New services; and Data standardisation processes. Observed: N/A Aware of the EDP: Planned integration with the EDP: 24

25 Our findings Office for National Statistics UK LOD Infrastructure LOD Skills/competencies LOD Risks/Issues Description: the current infrastructure at ONS does not support LOD. ONS has a relational database and does not have any LD capability in-house. The geography linked data pilot is hosted by the contractor. LOD Investment/FTEs Investment: there is not enough demand for linked data and this makes large-scale investment in LOD difficult. LOD will lead to cost reductions by creating more efficiency and avoiding the duplication of data. Technical capabilities: resources who can deal with data and translate CSV or Excel files into RDF; and Good understanding of linked statistical data standards, e.g. Data Cube, StatDCAT-AP, and principles, such as persistent URIs. Channels: Web API LOD Distribution Formats: SDMX; XML; JSON; RDF; and CSV Licences: Open government licence Lack of skills and capabilities; Lack of awareness; High investment; Lack of standards, e.g. code lists, vocabularies, ontologies in order to have interoperable and harmonised datasets; Lack of a leader institution centralising and managing all the aspects related to LOD in order to push LOD to all governmental departments and public sector bodies. Used: N/A Provided: N/A LOD Learning resources Aware of StatDCAT-AP: Planned implementation of StatDCAT-AP: 25

26 Overview Federal Statistical Office Switzerland (L)OD Current initiatives (L)OD Future plans (L)OD Main users OD: publishing OD for more than 160 years. LOD: Census data: there are around two million triples at a low level of aggregation. SDMX classification to reflect all changes on Swiss communes (since 1960): LOD dataset created on that basis. SPARQL to aggregate on-the-fly dataset within the datastore. OD: more reusable data and open standards. LOD: FSO plans to move to a single source of data for all their apps that allow to access data (SDMX format). FSO is working on the specification of ETL processes aimed at the production of LOD out of their central datawarehouses. By the end of 2017 they would like to offer all census datasets since 2000 (or 1990) as LOD. OD main users: around 100,000 requests on the Open Data portal daily. LOD main users: universities, researchers and journalists. Key partners: groups in Switzerland with SDMX knowledge, Swiss universities, Government agencies that publish data on the Swiss Open Data Portal. LOD Enablers/Inhibitors Good IT skills; Clear understanding of statistical data; management processes, including creation, dissemination, archiving, etc.; Uniform semantic and URI policies. The use of LD (including SPARQL and visualisation) is complex. LOD Benefits Expected: Data integration; The implementation of SDMX metadata may lead to efficient LOD data production; New services; and Data standardisation processes. Observed: N/A Aware of the EDP: Planned integration with the EDP: and proofs of concept 26

27 Overview Federal Statistical Office Switzerland LOD Infrastructure LOD Skills/competencies LOD Risks/Issues Description: FSO has currently a small prototype (a webpage with two datasets). The process to transform and publish data is made through XML and command lines. FSO is currently working on the optimisation of this process. URI policy: it has been an issue to implement specific semantic and URI rules among statistical organizations. SDMX knowledge; Good understanding of linked statistical data standards and principles, such as persistent URIs. No major risks LOD Investment/FTEs Investment: Expected increase in costs in the deployment phase; Non-expected cost reduction in the long term. Channels: SPARQL endpoint Beta app LOD Distribution Formats: SDMX; RDF; Expected: JSON; TSV and CSV. Licences: N/A Used: N/A Provided: N/A LOD Learning resources Aware of StatDCAT-AP: Planned implementation of StatDCAT-AP: 27

28 Overview Italian Statistical Office Italy (L)OD Current initiatives LOD Future plans (L)OD Main users OD: Data related to commuting aspects available LOD: 2011 Census data available Data related to territory aspects available ISTAT started with providing a SPARQL endpoint, then an RDF browser was put in place and last the REST API which shows that interest has increased. LOD: ISTAT wants to provide more LOD 2016: Data related to labour to be available in English 2017: ANNCSU (Archivio Nazionale dei Numeri Civici delle Strade Urbane (national archive for street addresses) There is a request for more LOD especially from the open data community (in particular for street addresses). LOD main users: Citizens Municipalities Journalists Transport companies developing apps There is a need to use also for back-office by integrating current registries together such as families, territory, activities. Key partners: AGID main driver for open data recommendations LOD Enablers/Inhibitors App contests such as APP4Italy are facilitators in the use of LOD and provide value to LOD data/users Detailed analysis before developing N/A LOD Benefits Observed: Number of LOD users is significantly increasing (e.g. municipalities such as Bologna reusing the ontology, transport companies using the apps, etc. Flexible data integration: bidirectional links with ISPRA (Environment Protection Agency) at level of municipality (owl:sameas) Aware of the EDP: Planned integration with the EDP: 28

29 Overview Italian Statistical Office Italy LOD Infrastructure LOD Skills/competencies LOD Risks/Issues Description: Triple store/sparql endpoint based on Oracle Exadata. Website and REST API 2 Data browsers: GIS system linked to the triple store Data warehouse (I.STAT): portal that maps uploaded data in SDMX and publish data in different formats (CSV, etc.) URI policy: no official URI policy Managing database; Knowledge of RDF; Knowledge of programming languages such as PHP and Java; Knowledge of web server and application server technologies; (Everything is developed in house). Performance; Structural transformation (CSV to RDF) is a delicate process; Difficult to show the added value for LOD. LOD Investment/FTEs Investment: Developing in-house tools (website, web services, data warehouse), reuse existing software (Pubby), reuse existing standards (DCAT-AP, ADMS, DCMI, SKOS, SDMX for sex classification, W3C PROV ontology, GeoNames). FTE: Difficult to measure it: 5 developers (but not full time). LOD Distribution Channels: Web services based on REST technology SPARQL endpoint with guided queries Website driving the user in choosing the best way to get data. Formats: Web: CSV, TSV, text, JSON, XML, HTML SPARQL ep: CSV, TSV, text, HTML Dataset are provided as ZIP files Licences: CC-BY 3.0 (Creative Common) Used: N/A LOD Learning resources Provided: Video explaining how to use the website: Aware of StatDCAT-AP: Planned implementation of StatDCAT-AP: 29

30 Overview National Institute of Statistics and Economic Studies France (L)OD Current initiatives LOD Future plans (L)OD Main users OD: All the aggregated statistical data collected by INSEE is published as OD. LOD: Metadata: nomenclature of French activity, professions and socio-professional categories and jurisdictional categories as well as geographical entities. Data: Census, list of communal equipment (info about sport and education), BDM database. Internal LD: thesaurus of statistical concepts. LOD: Consolidate the infrastructure of a new data centre. Extend the publication of metadata in RDF including quality metadata by using SIMS and SDMX. Publishing indicators of constructions, price and service as LOD. Build a new tool to manage LOD Promote the reuse of metadata at European/world level. (Long term) extend LOD to other administrations that produce statistical data (mainly ministries). LOD main users: There is no monitoring of the usage of LOD. INSEE is aware that some entities (including the French water agency) are using the data Key partners: IGN, Atos, GraphDB, Mondeca UNECE, Eurostat, ISTAT, ABS, Statistics Canada, ONS (ESSNET), DDI LOD Enablers/Inhibitors Reusing LOD internally and externally Internal promotion and training can help to demystify LOD The key to success is communication N/A LOD Benefits Observed: Quality improvements such as the correction of incoherence in nomenclatures and the normalisation of data Expected: LOD will probably help to propagate its nomenclatures to the INSEE s partners. Positive impact on the image of INSEE in the NSI community Aware of the EDP: Planned integration with the EDP: 30

31 Overview National Institute of Statistics and Economic Studies France LOD Infrastructure LOD Skills/competencies LOD Risks/Issues Description: GraphDB is used for the triple store which implements the Sesame API Datalift app is the front end of GraphDB Datalift exposes the SPARQL endpoint (manually updated), webpages (automatically updated) and the management module of the URIs. 2 servers managed by the production service URI policy: A URI policy has been developed in the context of the datalift project LOD Investment/FTEs Investment: External publication: very low cost (license of graphdb + server). Internal publication: in overall, the cost is the same for internal and external publication INSEE is currently developing an interface of management for metadata ( 110,000) FTE: Less than ½ day per week INSEE used to work with a contractor at the beginning Currently, everything is done internally. LOD Distribution Channels: The SPARQL endpoints The datasets can be directly downloaded (as zip turtle) REST API Formats: The nomenclature are published in SKOS Specific ontology for the publication of the population and the geography Licences: No official standard license Usually managers do not understand LD technologies There is a need to offer specific use cases Technology inhibition: there is a need to demystify LOD, explain the technology in simpler terms, easy to be understood both by management and technicians It is more difficult to understand the data model than SPARQL Used: N/A LOD Learning resources Provided: All the projects are open source. The RDF website of INSEE contains links to LOD knowledge Aware of StatDCAT-AP: Planned implementation of StatDCAT-AP: 31

32 Overview Phil Archer (W3C), Bill Roberts (Swirrl) and Evangelos Kalampokis (CERTH) LOD Current initiatives LOD Future plans LOD Risks/Issues Horizon Open government intelligence : improve public services through LOD in the field of statistics; Geographical : standardise codelists for LOD (Swirll); W3C driving the standardisation of linked data vocabularies Increase/encourage the use of statistical linked data, in combination with other types of data, e.g. geospatial data; Support the linking of statistical data within and across countries; Continue the development of production ready linked data platforms and tools that support different data formats; Align linked data vocabularies with de facto market standards, such as schema.org; Work on new graph query languages. Lack of knowledge regarding: data analysis, LOD technologies/tools and semantic formats (such as RDF); Lack of validation tools; Lack of learning/training resources; Performance issues; Data quality issues; Lack of standardisation (URIs, vocabularies, ontologies); Different ways to interpret/implement a standard. LOD Enablers/Inhibitors Data quality and integration; Visualisation tools to extract and explore data; Communicate the goal and benefits of LOD; Share training sessions, courses, best practices and recommendations. Complexity Organisational resistance LOD Benefits Expected: Data integration; and Data standardisation processes. Observed: Increase internal efficiency (e.g. exchange of data, quality improvement); Improve particular processes/tasks through services to link information from multiple sources. LOD Skills/competencies Programming/querying skills; Data analysis skills; Knowledge of (statistical) linked data standards, such as RDF Data Cube, and experience with their implementation. Aware of StatDCAT-AP: 32

33 Get in touch with us to know more Nikolaos Loutas Daniel Brulé Makx Dekkers This publication has been prepared by EU Services for Eurostat under DI07171 specific contract 353. refers to Enterprise Advisory bvba which is a member firm of PricewaterhouseCoopers International Limited, each member firm of which is a separate legal entity. 33