SGA-2: WP6- Early estimates. SURS: Boro Nikić, Tomaž Špeh ESSnet Big Data Project Brussels 26.,

Transcription

1 SGA-2: WP6- Early estimates SURS: Boro Nikić, Tomaž Špeh ESSnet Big Data Project Brussels 26.,

2 Outline WP6 state of affairs WP6 pilot: Early estimates of economic indicators Methods Data sources Quality measures Tools WP6 coordination meeting, Ljubljana Planning for remaining period of SGA-2 2

3 3

4 WP6 Deliverables (SGA-2 only): 6.6 Report about the impact of one (or more) big data (and other) sources on economic indicators (correlation, time lag, selectivity, etc.) 6.7 Recommendation about the methodology and process of calculating estimates for at last one early economic indicator 6.8 At least one example of calculated concrete estimates for one of the economic indicators with quality assessment of the input, throughput and output phase of the process month 6.9 Report and recommendations about the IT infrastructure needed for the storage, analysing 4

5 Work done in period of SGA-1 Investigate multiple big data, administrative and other existing sources in order to produce early estimates for statistical purposes ( data access and data handling ) Collaboration with WP7 team -Joint meeting in Warsaw Nowcasting the turnover indicators (April 2016) One of the pilots that was started in WP6 SGA-1 Statistics Finland (basic proposal), Statistical Office Slovenia Interesting methodological suggestions for estimating early economic indicators SURS decided to start with this idea Finalising the proposal for SGA2 (November 2016) 5

6 Proposed pilot for WP6 SGA-2 Title of the pilot: Early estimates of economic indicators Main economic indicators: Gross domestic product (GDP) Consumer price index (CPI) Retail sale Balance of payments Economic sentiment indictors New leading economic indicators 6

7 1.METHOD Create and test the methodology of creating early estimates for at least one of the main economic indicators. Aim of the pilot 3.QUALITY Define and test the quality measures which assess quality of the sources, statistical production and statistical results 2.DATA SOURCE Investigate multiple Big data and other existing sources for purposes of early estimates of at least one of the main economic indicators 4.REUSE/SHARE Many of the sources are available in most of the countries so it is possible to test them and create the results for more than one country. 7

8 Method Nowcasting model The idea and a short example of the model came from partners from Statistics Finland (share/reuse). The nowcasting model consists of 2 stages: Principal Component Analysis (PCA) is used to extract principal components from enterprise data. For each enterprise included in the model, time series of data without any missing values is needed. Then, first few principal components are chosen. Linear regression is used: the time series of interest (e.g. GDP) is the dependent variable (Y) and the chosen principal components are the predictors (X 1,, X n ). Seasonal component and other predictors can be added 8

9 Early estimates of GDP at SURS Time series of interest: GDP in constant prices (chain-linked volumes, reference year 2010) from 2008Q1 to 2016Q4. 8 different testing spans: the first period is always 2008Q1; the last period is 2014Q1 or 2014Q2 or or 2016Q4. 3 different sets of enterprise data: D1: turnover in industry; D2: turnover in retail trade; D3: turnover in industry and retail trade (i.e. D1 and D2 together). 9

10 Data sources Big Data Availability (SURS) Job Vacancies Ads from job portals Traffic loops Yes Yes Social media data (Twitter, Facebook, ) Data from supermarket chains? Yes Mobile phone data? Transaction data from banks? 10

11 Traffic loops in Slovenia Around 660 traffic loops 10 categories of vehicles Frequencies of certain type of vehicle are available in 15 min interval Data since 2005 (2011) Sample data already at SURS 11

12 Traffic sensors Traffic sensors count different vehicles: TRAFFIC SENSOR QLD3 QLD5 QLD6 QLTC8 QLTC10 QLD COUNTS: all vehicles without separation vehicles in 5 categories: motorcycles, buses and buses with trailers (krpan), passenger cars and light lorries, medium and heavy lorries, trailing vehicles and road tractors vehicles in 10 categories: motorcycles, busses, busses with trailers (krpan), passenger cars, light lorries, medium lorries, heavy lorries, trailing vehicles, road tractors, unknown vehicles in 10 categories: like above vehicles in 10 categories: like above different sensor types throughout measurements Multiple sensors can be set on the same road section! 12

13 Sources of data Three different sources with different data types according to position set-up: 1. Administrative data traffic sensors on the borders (from the Ministry of Infrastructure) 2. Ministry of Infrastructure national roads sensors 3. City Municipalities city roads sensors options according to rawness /editing of data: 1. raw data 11 quantities per channel, 2 channels, 15- minute intervals, 2. Edited data or a count spot 8 variables per channel with 2 channels, aggregated data of traffic sensors on the same road section, 15-minute, hourly, daily intervals 13

14 Traffic data and calculation of rapid GDP estimates Searched for big data sources with a timely dimension and easy to acquire. Preliminary test with current estimates of GDP fitted on traffic data. The intention: the analysis of improvement of rapid short term estimates using traffic data. 14

15 Chosen source: MoI data The chosen data: 15-minute edited data on national roads from the Ministry of Infrastructure. Why edited data: costs of unification of same-section traffic sensors greater than loss of information! 15

16 Data files: Data acquisition More than 3500 files, Files represent yearly data for one count spot for the period between 2011 and 2016, More than 24 GB of data, Unstructured head in every file, 15-minute interval microdata, Missing data present in the form of missing observations or zeros in variables, Errors in data structure (multiple separator signs, separator signs in variable names, unequal number of variable names and columns, metadata in the middle of microdata ) 16

17 Data preparation Programming work in Python: 1. reading text files, 2. recognizing un-/structured parts, number of variables, time and date components 3. recognizing format errors, 4. new object stevci (sensors) with defined functions: 1. transformations, aggregations, missing data storage, metadata storage (position, time period, neighbours, text file properties ) 5. imputations and imputation testing Why Python? Use of multi-thread capacities Built-in algorithms for sparsely structured data Open code 17

18 Use of combined data for rapid estimation of GDP Prepared data used as regressors in GDP flash estimates: Constant prices of GDP, quarterly, regressand Industry Enterprises' productions as primary Different types of traffic data as secondary All count spots on all roads All count spots on regional roads Goods motor vehicles on all roads Goods motor vehicles on regional roads Different PCA criterion: Percent of explained variability Number of cases/observations per variable Scale of eigenvalues 18

19 Result interpretations PCA criterion 80 is optimal for our time series The best traffic data is represented by goods vehicles on regional roads data Open issue: Number of observations is too small for conclusions 19

20 Result interpretations Our assumption about traffic data were right In almost every instance traffic data corrects the estimates The errors between official and estimated values are reduced 20

21 Future work Extending the GDP series Improve traffic data processing Determine accuracy Other regressors and use in other regressor positions (main/secondary) Idea: estimating Industry Production Index (IPI) Nowcasting components of GDP 21

22 Software & Infrastructure STATISTICAL PRODUCTION Data preparation Modeling Results Sandbox for insensitive big data (e.g. road sensor data) Internal IT environments for sensitive data 22

23 SGA-2 WP6 Activities done SURS successes to have access to one big data source Big data source was statistically processed First estimates of early economic indicator using also big data source were produced WP6 coordination meeting, Ljubjana Oct. One of nowcasting methods (together with IT application) to be shared among members of WP6 Initial inventory of data sources prepared Some quality indicators were investigated 23

24 SGA-2 WP6 plans & issues Big data and other sources reliable for estimation of early economic indicators > final decision about sources to use (availability of data in each country) Economic indicators which could be tested during SGA-2. > decision (country by country if possible) about testing set of about early economic indicators Work on early estimates pilot (IT application, method, data processing and estimates improvements) The code will be tested by other countries on their data by our methods; their methods; try to evaluate other indicators >Ideas for improvement Methods for nowcasting >determination of methods which could be tested (implemented) Statistics Finland continued with very useful contribution regarding nowcasting methods Quality evaluation with focus on precision criteria and validation of process and results. 24