M4D Data Quality Check

Transcription

1 M4D Data Quality Check Detail Dataset: _SeGi_I-6ESPONindicatorsSeGI_syntaxChecked.xls Check date: 2013/03/21 Checked by: AC/MC NUTS level 2: NUTS date 2006: General Data for NUT0 and NUTS2 regions are both present in the dataset. This quality check is concerned solely with the NUTS2 data. The spreadsheet contains the four usual worksheets Dataset, Indicator, Source and Data. A fifth object, named SPSS, cannot be seen in Excel but is visible when the spreadsheet is loaded into ArcGIS 10.1 and R (using the RODBC package). Variables in the dataset (from Indicator worksheet) Indicator Name Data Type Data Values ene-prod_abs_rtd float TOE/inhabitants ene-ren_sh_rtp float renewable energy production/energy production ene-supply_abs_rtd float Units/inhabitants ene-supply_abs2_rtd float Employed persons/inhabitants ene-price_abs_rtd float /kwh env-wat_abs_rtd float Units/inhabitants env-sew_abs_rtd float Units/inhabitants env-was_abs_rtd float Units/inhabitants tra-air_abs_rtd float Units/inhabitants tra-wat_abs_rtd float Units/inhabitants tra-lan_abs_rtd float Units/inhabitants tra-mot_abs_rtd float length/area tra-rod_abs_rtd float length/area tra-ral_abs_rtd float length/area frg-rod_abs_rtd float tonne-km/inahbitants frg-ral_abs_rtd float tonne-km/inahbitants tra-bus_abs_rtd float Number/inhabitants tra-pos_abs_rtd float Units/inhabitants tra-pri_abs_rtc float brc-rtv_abs_rtd float Units/inhabitants inf-tel_abs_rtd float Units/inhabitants inf-tel_abs2_rtd float Employed persons/inhabitants inf-pri_abs_rtc float bro-hsd_sh_rtp float Households with access to broadband/households ser-cos_abs_rtd float Employed persons/inhabitants ser-vet_abs_rtd float Units/inhabitants ser-vet_abs2_rtd float Employed persons/inhabitants ser-emp_abs_rtd float Units/inhabitants edu-enr_abs_rtd float Students/population group edu-enr_abs2_rtd float Students/population group edu-enr_abs3_rtd float Students/population group edu-str_abs_rtd float Students/teachers edu-str2_abs_rtd float Students/teachers edu-scl_abs_rtd float Students/classes 1

2 edu-scl2_abs_rtd float Students/classes edu-pri_abs_rtd float EU27 = 100 edu-exp_abs_rtd float PPS/inhabitants hst-bed_abs_rtd float Beds/inhabitants hst-bed2_abs_rtd float Beds/inhabitants drs-pop_abs_rtd float physician and doctors/inhabitants nrs-pop_abs_rtd float nurses and midwives/inhabitants hlt-exp_abs_rtd float expenditures/inhabitants cre-chi_sh_rtp float children in care/population age group cre-chi2_sh_rtp float children in care/population age group cre-chi3_sh_rtp float children in care/population age group cre-chi4_sh_rtp float children in care/population age group clt-fcl_abs_rtd float Number of facilities/inhabitants spt-sta_abs_rtd float Number of stadiums/inhabitants spt-sta_abs2_rtd float Seats/inhabitants hou-exp_abs_rtd float expenditures/inhabitants hou-ovb_sh_rtp float persons with cost overburden/persons of permanent risk of pverty The metadata for frg-rod_abs_rtd, frg-ral_abs_rtd and hou-ovb_sh_rtp contain spelling errors. The spelling of inhabitants and poverty needs to be corrected. These are highlighted in the list above. Missing values Values are coded as not present for some areas. Missing Values Analysis Index All NUTS2 values are missing for (shown as a red column in the display above) Indicator ene-prod_abs_rtd2009: 271 (100%) Indicator ene-ren_sh_rtp: 271 (100%) Indicator ene-price_abs_rtd: 271 (100%) Indicator tra-ral_abs_rtd: 271 (100%) Indicator frg-rod_abs_rtd: 271 (100%) Indicator frg-ral_abs_rtd: 271 (100%) 2

3 Indicator tra-pri_abs_rtc: 271 (100%) Indicator inf-pri_abs_rtc: 271 (100%) Indicator edu-str_abs_rtd: 271 (100%) Indicator edu-str2_abs_rtd: 271 (100%) Indicator edu-scl_abs_rtd: 271 (100%) Indicator edu-scl2_abs_rtd: 271 (100%) Indicator edu-pri_abs_rtd: 271 (100%) Indicator edu-exp_abs_rtd: 271 (100%) Indicator hlt-exp_abs_rtd: 271 (100%) Indicator cre-chi2_sh_rtp: 271 (100%) Indicator cre-chi_sh_rtp: 271 (100%) Indicator cre-chi3_sh_rtp: 271 (100%) Indicator cre-chi4_sh_rtp: 271 (100%) Indicator hou-exp_abs_rtd: 271 (100%) Indicator hou-ovb_sh_rtp: 271 (100%) Portions of the NUTS2 values are missing for Indicator ene-supply_abs_rtd2009: 14 (5.17%) Greece, Malta Indicator ene-supply_abs2_rtd2009: 40 (14.76%) France, Greece, Malta Indicator env-wat_abs_rtd2009: 1 (0.37%) Malta Indicator env-sew_abs_rtd2009: 14 (5.17%) Greece, Malta Indicator tra-air_abs_rtd2009: 1 (0.37%) Malta Indicator env-was_abs_rtd2009: 14 (5.17%) Greece, Malta Indicator tra-wat_abs_rtd2009: 2 (0.74%) Luxembourg, Malta Indicator tra-lan_abs_rtd2009: 2 (0.74%) Zahodna Slovenija, Malta Indicator tra-rod_abs_rtd2009: 2 (0.74%) Portugal: Acores, Madeira Indicator tra-pos_abs_rtd2009: 1 (0.37%) Malta Indicator tra-bus_abs_rtd2009: 4 (1.48%) France Departements d'outre Mer Indicator brc-rtv_abs_rtd2009: 1 (0.37% Malta Indicator inf-tel_abs_rtd2009: 1 (0.37%) Malta Indicator inf-tel_abs2_rtd2009: 28 (10.33%) France, Luxembourg, Malta Indicator ser-vet_abs_rtd2009: 17 (6.27%) Greece, Luxembourg, Malta Indicator ser-vet_abs2_rtd2009: 43 (15.87%) Spain: Ceuta, Melilla, France, Greece Luxembourg, Malta Indicator ser-emp_abs_rtd2009: 1 (0.37%) Malta Indicator clt-fcl_abs_rtd2006: 4 (1.48%) France Departements d'outre Mer Indicator spt-sta_abs_rtd2009: 4 (1.48%) France Departements d'outre Mer Indicator spt-sta_abs2_rtd2009: 4 (1.48%) France Departements d'outre Mer All NUTS2 data are present for the following variables: Indicator tra-mot_abs_rtd2009: 0 (0%) Indicator bro-hsd_sh_rtp2010: 0 (0%) Indicator ser-cos_abs_rtd2009: 0 (0%) Indicator edu-enr_abs_rtd2009: 0 (0%) Indicator edu-enr_abs2_rtd2009: 0 (0%) Indicator edu-enr_abs3_rtd2009: 0 (0%) Indicator hst-bed_abs_rtd2008: 0 (0%) Indicator hst-bed2_abs_rtd2008: 0 (0%) Indicator drs-pop_abs_rtd2008: 0 (0%) Indicator nrs-pop_abs_rtd2008: 0 (0%) Duplicated Rows The following regions appear to have duplicated data for ALL variables: NUTSCode DK01 DK02 DK03 NUTS Region Name Hovedstaden Sjælland Syddanmark 3

4 DK04 DK05 Midtjylland Nordjylland The following regions appear to have duplicated data for a subset of variables: CH01 Région lémanique CH02 Espace Mittelland CH03 Nordwestschweiz CH04 Zürich CH05 Ostschweiz CH06 Zentralschweiz CH07 Ticino Potentially Anomalous values A number of regions appear to have potentially anomalous values either unusually high or unusually low. These values may well be correct, even though the data check has identified them as unusual. Data check for ene-prod_abs_rtd Data check for ene-ren_sh_rtp Data check for ene-supply_abs_rtd AT Burgenland (A) AT Kärnten AT Steiermark AT Oberösterreich AT Salzburg AT Tirol DK Syddanmark DK Midtjylland DK Nordjylland ES Comunidad Foral de Navarra ES La Rioja ES Aragón ES Comunidad de Madrid ES Castilla y León ES Castilla-La Mancha ES Extremadura FI Åland FR Île de France FR Martinique IS Ísland ITC Valle d'aosta/vallée d'aoste ITD Provincia Autonoma Bolzano/Bozen NO Vestlandet NO Nord-Norge SE Småland med öarna SE Västsverige SE Norra Mellansverige 4

5 SE Mellersta Norrland SE Övre Norrland Data check for ene-supply_abs2_rtd BG Yugozapaden DE Mittelfranken RO Bucuresti - Ilfov SK Bratislavský kraj Data check for ene-price_abs_rtd Data check for env-wat_abs_rtd AT Oberösterreich AT Salzburg CZ Severozápad CZ Severovýchod DK Sjælland DK Syddanmark DK Midtjylland DK Nordjylland ES Illes Balears ES Canarias FI Itä-Suomi FI Länsi-Suomi FI1A Pohjois-Suomi FI Åland HU Nyugat-Dunántúl HU Dél-Dunántúl HU Észak-Magyarország HU Észak-Alföld HU Dél-Alföld IS Ísland NO Hedmark og Oppland NO Vestlandet NO Trøndelag NO Nord-Norge PL Zachodniopomorskie TR Manisa TR Konya TR Kirikkale UKD Cumbria UKM Highlands and Islands Data check for env-sew_abs_rtd

6 AT Burgenland (A) AT Niederösterreich AT Kärnten AT Steiermark AT Oberösterreich AT Salzburg AT Tirol AT Vorarlberg CY ??p??? / Kibris DE Niederbayern DE Oberpfalz DE Mittelfranken DEB Trier DED Chemnitz FI Åland HU Dél-Dunántúl PL Wielkopolskie PL Zachodniopomorskie PL Lubuskie Data check for env-was_abs_rtd CZ Praha CZ Strední Cechy CZ Jihozápad CZ Severozápad CZ Severovýchod CZ Jihovýchod CZ Strední Morava CZ Moravskoslezsko DE Niederbayern DE Oberpfalz DE Oberfranken DE Unterfranken DE Schwaben FR Lorraine FR Franche-Comté ITF Basilicata Data check for tra-air_abs_rtd BE Prov. Limburg (B) FI Åland FR Île de France FR Corse FR Guadeloupe FR Martinique FR Guyane FR Réunion IS Ísland 6

7 ITD Provincia Autonoma Bolzano/Bozen NO Oslo og Akershus NO Nord-Norge SE Stockholm SE Mellersta Norrland SE Övre Norrland UKJ Surrey, East and West Sussex UKM Highlands and Islands Data check for tra-wat_abs_rtd DE Bremen DE Hamburg DE Lüneburg DE Weser-Ems FI Åland FR Corse FR Guadeloupe FR Martinique FR Guyane GR Anatoliki Makedonia, Thraki GR Kentriki Makedonia GR Dytiki Makedonia GR Thessalia GR Ipeiros GR Ionia Nisia GR Dytiki Ellada GR Sterea Ellada GR Peloponnisos GR Attiki GR Voreio Aigaio GR Notio Aigaio GR Kriti HR Sjeverozapadna Hrvatska HR Središnja i Istocna (Panonska) Hrvatska HR Jadranska Hrvatska ITD Veneto NL Groningen NL Friesland (NL) NL Overijssel NL Gelderland NL Flevoland NL Noord-Holland NL Zuid-Holland NL Zeeland NL Limburg (NL) NO Oslo og Akershus NO Agder og Rogaland NO Vestlandet NO Trøndelag NO Nord-Norge PT Algarve SE Stockholm SE Västsverige TR Aydin UKM Highlands and Islands 7

8 Data check for tra-lan_abs_rtd TR Kastamonu TR Trabzon Data check for tra-mot_abs_rtd AT Wien BE Prov. Vlaams-Brabant DE Berlin DE Bremen DE Hamburg DEA Düsseldorf ES Comunidad de Madrid NL Utrecht NL Zuid-Holland NL Noord-Brabant NL Limburg (NL) PT Lisboa UKD Greater Manchester UKD Merseyside UKG West Midlands DEC NA Data check for tra-rod_abs_rtd AT Wien BE Région de Bruxelles-Capitale / Brussels Hoofdstedelijk Gewest BE Prov. Antwerpen BE Prov. Limburg (B) BE Prov. Oost-Vlaanderen BE Prov. Vlaams-Brabant BE Prov. Hainaut CH Nordwestschweiz CH Zürich MT Malta NL Utrecht NL Zuid-Holland NL Noord-Brabant NL Limburg (NL) UKD Greater Manchester UKD Merseyside UKE West Yorkshire UKG West Midlands UKI Inner London 8

9 UKI Outer London Data check for tra-ral_abs_rtd Data check for frg-rod_abs_rtd Data check for frg-ral_abs_rtd Data check for tra-bus_abs_rtd IS Ísland NO Vestlandet TR Istanbul TR Tekirdag TR Balikesir TR Izmir TR Aydin TR Manisa TR Bursa TR Kocaeli TR Ankara TR Konya TR Antalya TR Adana TR Hatay TR Kirikkale TR Kayseri TR Zonguldak TR Kastamonu TR Samsun TR Trabzon TRA Erzurum TRA Agri TRB Malatya TRC Gaziantep Data check for tra-pos_abs_rtd BE Prov. Oost-Vlaanderen DE Hannover FR Île de France 9

10 Data check for tra-pri_abs_rtc Data check for brc-rtv_abs_rtd CY ??p??? / Kibris CZ Praha ES Región de Murcia ES Canarias FI Åland GR Anatoliki Makedonia, Thraki GR Kentriki Makedonia GR Dytiki Makedonia GR Thessalia GR Ipeiros GR Ionia Nisia GR Dytiki Ellada GR Sterea Ellada GR Peloponnisos GR Attiki GR Voreio Aigaio GR Notio Aigaio GR Kriti HU Közép-Magyarország HU Közép-Dunántúl HU Nyugat-Dunántúl HU Dél-Dunántúl HU Észak-Magyarország HU Észak-Alföld HU Dél-Alföld NO Nord-Norge SI Vzhodna Slovenija SI Zahodna Slovenija UKI Inner London Data check for inf-tel_abs_rtd DEA Köln FI Åland FR Île de France FR Martinique ITC Liguria LU Luxembourg (Grand-Duché) RO Bucuresti - Ilfov TR Tekirdag TR Balikesir TR Aydin TR Bursa TR Konya TR Antalya TR Adana TR Hatay 10

11 TR Kirikkale TR Samsun TR Trabzon UKI Inner London Data check for inf-tel_abs2_rtd AT Wien BE Région de Bruxelles-Capitale / Brussels Hoofdstedelijk Gewest BG Yugozapaden CZ Praha DE Darmstadt DEA Köln DK Hovedstaden ITE Lazio NL Groningen NO Oslo og Akershus PL Mazowieckie RO Bucuresti - Ilfov SE Stockholm SE Mellersta Norrland SK Bratislavský kraj UKC Northumberland and Tyne and Wear UKD Cheshire UKH Bedfordshire and Hertfordshire UKI Inner London UKJ Berkshire, Buckinghamshire and Oxfordshire UKK Gloucestershire, Wiltshire and Bristol/Bath area Data check for inf-pri_abs_rtc Data check for bro-hsd_sh_rtp RO Sud-Vest Oltenia Data check for ser-cos_abs_rtd NL Utrecht NL Noord-Holland SE Stockholm SK Bratislavský kraj UKD Cheshire UKF Leicestershire, Rutland and Northamptonshire UKH Bedfordshire and Hertfordshire 11

12 UKI Inner London UKJ Berkshire, Buckinghamshire and Oxfordshire UKJ Surrey, East and West Sussex Data check for ser-vet_abs_rtd BE Prov. West-Vlaanderen BE Prov. Brabant Wallon BE Prov. Luxembourg (B) BE Prov. Namur ITD Emilia-Romagna ITE Toscana ITE Marche NO Hedmark og Oppland NO Trøndelag Data check for ser-vet_abs2_rtd DK Syddanmark DK Nordjylland UKD Cumbria UKD Cheshire UKD Lancashire UKE North Yorkshire UKG Herefordshire, Worcestershire and Warwickshire UKG Shropshire and Staffordshire UKK Gloucestershire, Wiltshire and Bristol/Bath area UKK Dorset and Somerset UKK Cornwall and Isles of Scilly UKK Devon UKM North Eastern Scotland UKM Highlands and Islands Data check for ser-emp_abs_rtd NL Groningen NL Drenthe NL Overijssel NL Gelderland NL Flevoland NL Utrecht NL Noord-Holland NL Zuid-Holland NL Zeeland NL Noord-Brabant NL Limburg (NL) SE Stockholm 12

13 UKH Bedfordshire and Hertfordshire UKI Inner London UKJ Berkshire, Buckinghamshire and Oxfordshire UKJ Surrey, East and West Sussex Data check for edu-enr_abs_rtd DK Sjælland DK Nordjylland IE Border, Midland and Western IE Southern and Eastern Data check for edu-enr_abs2_rtd BE Région de Bruxelles-Capitale / Brussels Hoofdstedelijk Gewest BE Prov. Antwerpen BE Prov. Limburg (B) BE Prov. Oost-Vlaanderen BE Prov. West-Vlaanderen CZ Praha SK Bratislavský kraj TRA Agri TRB Van TRC Sanliurfa TRC Mardin Data check for edu-enr_abs3_rtd AT Wien BE Région de Bruxelles-Capitale / Brussels Hoofdstedelijk Gewest CH Zürich CZ Praha DK Hovedstaden GR Kentriki Makedonia GR Dytiki Makedonia GR Ipeiros HU Közép-Magyarország NO Oslo og Akershus NO Trøndelag PL Mazowieckie RO Bucuresti - Ilfov SI Zahodna Slovenija SK Bratislavský kraj TR Bursa 13

14 Data check for edu-str_abs_rtd Data check for edu-str2_abs_rtd Data check for edu-scl_abs_rtd Data check for edu-scl2_abs_rtd Data check for edu-pri_abs_rtd Data check for edu-exp_abs_rtd Data check for hst-bed_abs_rtd DE Mecklenburg-Vorpommern Data check for hst-bed2_abs_rtd BE Prov. Antwerpen BE Prov. Oost-Vlaanderen BE Prov. West-Vlaanderen BE Prov. Namur NL Drenthe NL Utrecht PT Região Autónoma dos Açores PT Região Autónoma da Madeira Data check for drs-pop_abs_rtd AT Wien CZ Praha 14

15 ES Aragón ES Ciudad Autónoma de Ceuta ES Ciudad Autónoma de Melilla GR Kentriki Makedonia GR Ipeiros GR Attiki GR Kriti ITC Liguria ITE Lazio SK Bratislavský kraj Data check for nrs-pop_abs_rtd SE Mellersta Norrland Data check for hlt-exp_abs_rtd Data check for cre-chi_sh_rtp Data check for cre-chi2_sh_rtp Data check for cre-chi3_sh_rtp Data check for cre-chi4_sh_rtp Data check for clt-fcl_abs_rtd AT Wien AT Salzburg AT Tirol AT Vorarlberg BE Région de Bruxelles-Capitale / Brussels Hoofdstedelijk Gewest CH Région lémanique CH Nordwestschweiz CH Ticino CZ Praha DE Oberbayern 15

16 DE Berlin DE Hamburg DE Darmstadt DEA Köln DK Hovedstaden ES Illes Balears FI Åland FR Île de France IS Ísland LI Liechtenstein LU Luxembourg (Grand-Duché) NL Friesland (NL) NL Noord-Holland NO Oslo og Akershus SI Vzhodna Slovenija SK Bratislavský kraj UKI Inner London Data check for spt-sta_abs_rtd AT Burgenland (A) AT Vorarlberg CH Ticino CY ??p??? / Kibris DK Midtjylland IE Border, Midland and Western NO Hedmark og Oppland NO Agder og Rogaland PT Algarve AL NA BA NA Data check for spt-sta_abs2_rtd IE Border, Midland and Western IE Southern and Eastern UKD Greater Manchester UKE South Yorkshire UKI Inner London UKL East Wales AL NA Data check for hou-exp_abs_rtd Data check for hou-ovb_sh_rtp

17 Suggestions for checking the data This report seems to be somewhat lengthy. We have checked variables for the NUTS2 or equivalent regions only. This means that data for some variables is not present this is noted in the listing above. We are of the opinion that most of these identified anomalous values may be false positives the data distributions exhibit a degree of positive skewness. The, sometimes lengthy, right tails on the distributions are typical of socio-economic data (for example, most people earn a moderate salary, but a few have extremely high salaries). Please confirm that the values are correct. The duplicated rows should be checked please confirm that the duplicated values correct? Missing data please confirm that these values are not present. If these values are as intended then there is no further need for a data check. 17