APPLYING BIG DATA TOOLS TO ACQUIRE AND PROCESS DATA ON CITIES

Size: px

Start display at page:

Download "APPLYING BIG DATA TOOLS TO ACQUIRE AND PROCESS DATA ON CITIES"

Kathlyn Butler
5 years ago
Views:

1 APPLYING BIG DATA TOOLS TO ACQUIRE AND PROCESS DATA ON CITIES JACEK MAŚLANKOWSKI, Ph.D. DEPARTMENT OF BUSINESS INFORMATICS FACULTY OF MANAGEMENT UNIVERSITY OF GDAŃSK, POLAND 1

2 AGENDA 2 Prerequisites Possible use cases and pilots conducted Conclusions

3 3 PREREQUISITES CHALLENGES AND OVERVIEW THE GOAL OF THE STUDY AND GENERAL CHARACTERISTICS

Based on UNECE taxonomy Big Data sources can be divided into three categories: Human sourced information Process

4 CHALLENGES AND OVERVIEW 4 Lacks of statistical data at regional level (sampling frame, cost). We have to identify new data sources to acquire the data with minimum cost (no cost) at regional or city level. Based on UNECE taxonomy Big Data sources can be divided into three categories: Human sourced information Process mediated data Machine generated data Lots of Big Data sources exist but it is not easy to access them (e.g., MCR, CDR, road sensors, )

Smartcitywillbecomesmarterwiththecurrentandreliable data help in decision-making

5 OVERVIEW AND THE GOAL OF THE STUDY 5 The goal of the paper was to study and analyze the possible alternative Big Data sources for analysis at regional (city) level Smartcitywillbecomesmarterwiththecurrentandreliable data help in decision-making process The pilot surveys conducted in various statistical areas will be showninthispaper

6 6 POSSIBLE USE CASES AND PILOTS CONDUCTED METHODS, TECHNOLOGIES, DATA SOURCES

7 WHAT DO WE NEED TO ACQUIRE AND PROCESS BIG DATA? 7 Methods Web scraping Text Mining/NLP Machine Learning Data quality framework Input Throughput Output

8 LIST OF ALTERNATIVE DATA SOURCES 8 Business data Job vacancies Data on enterprises Accomodation establishments (hotel chains) Road traffic Social statistics Life satisfaction People s opinion

9 CASE 1: JOB VACANCIES 9 Data we have Variety of European Labour Force surveys demand, supply, earnings etc. Sampling does not allow to process the data at regional level. Several Big Data sources exist. Data to acquire with Big Data tools Job supply at regional and city level Job demand at regional and city level Number of job offers scraped (thous.): 140; 124; 46; 39; 9.9 Duplicates in the data Classification is needed Location is not always shown

10 CASE 1: RESULTS 10 Number of enterprises: 10.3 thous. Number of locations: 9.5 thous Job vacancies by the city JOB VACANCIES Warszawa Wrocław Kraków Poznań Katowice Łódź Gdańsk Niemcy Szczecin LOCATION Liczba ofert przed klasyfikacją Liczba ofert po klasyfikacji

11 CASE 2: ENTERPRISES 11 What we know? Business registers Not frequently updated Don t have full information on enterprise What can be observed? Current NACE (type of economic activity) Communication channels (what are the most important channels) Methods that enterprise uses for its activities (e.g., e-commerce)

and less 20.01-40PLN 40.01-60PLN 60.01-80PLN 80.01-100PLN 73% 100.01PLN and more What can be added?

12 CASE 3: ACCOMODATION ESTABLISHMENTS (INCLUDING HOTEL CHAINS) 12 What we know? List of registered hotels and places Price category of agritourism places 2% 2% 1% 14% 8% 20PLN and less PLN PLN PLN PLN 73% PLN and more What can be added? Full, updated list of accomodation base The categories of objects even not registered The object misplaced their category

13 CASE 4: ROAD TRAFFIC 13 Current data sources Traffic loops Road traffic sensors Possible data sources Google Maps Traffic density and anomalies (access to historical data)

14 CASE 5: PEOPLE S OPINION 14 Current work We have sampling data that are gathered and processed not in the real time Process of typical opinion pool is expensive Data possible to gather

71 0.65 34 calm 0.43 0.30 0.35 10 upset 0.67 0.15 0.25 13 depressed 0.59 0.62 0.

15 CASE 6: SOCIAL STATISTICS LIFE SATISFACTION 15 Twitter popularity in Europe Training and testing dataset precision recall f1-score support happy neutral calm upset depressed discouraged indeterminate avg / total

16 THE FRAMEWORK 16 (1) Web scraping HTML/PHP/ASP etc. (2) Twitter API Text (3) Machine Learning Classification

17 17 CONCLUSIONS SUMMARY FUTURE WORK

18 SUMMARY 18 There is a variety of alternative data sources The quality is an issue Bad statistical data drives out good

19 FUTURE WORK 19 Combining the data Increase the data quality Publish a dashboard

20 Dashboard for labour market mismatch 20

21 21 THANK YOU! JACEK MAŚLANKOWSKI UNIVERSITY OF GDAŃSK POLAND

WP7 Multi Domains Setting the scene

WP7 Multi Domains Setting the scene WP7 Multi Domains General summary of WP7 work done, Review of the future deliverables and milestones. WP7 Multi Domains TEAM WP0: CO-ORDINATION WP 1 : Webscraping /