GMES Initial Operations / Copernicus Land monitoring services Validation of products

Transcription

1 GMES Initial Operations / Copernicus Land monitoring services Validation of products Validation Services for the geospatial products of the Copernicus land Continental and local components including in-situ data (lot 1) Open Call for Tenders - EEA/MDI/14/010 European Settlement Map 2016 VALIDATION REPORT Document: Date: 09/06/2017

2 Consortium Partners Prime Contractor SIRS SAS, France Main Partners GAF AG, Germany Joanneum Research, Austria e-geos, Italy Experts Specto Natura, UK ISPRA, Italy IGN France International, France Terranea, Germany NLSI, Iceland Document Preparation and Release Affiliation Name(s) Date Signature Author Specto Natura Geoff Smith 13/04/2017 Contributions SIRS Christophe Sannier 9/06/2017 Review Approval EEA Hans Dufourmont Document Issue Record Issue Date Author(s) Description of Change /04/2017 Geoff Smith Initial draft /05/2017 Geoff Smith General update /05/2017 Geoff Smith General update /06/2017 Geoff Smith General update /06/2017 Geoff Smith General update, first release /06/2017 Christophe Sannier Formatting issues GIO Land Product Validation Page 1 / 27

3 Executive Summary This report covers the validation of the European Settlement Map 2016 (also referred to as 'EUGHSL2016') alongside a comparison with the High Resolution Layer (HRL) on Imperviousness for 2012 (IMD2012). The EUGHSL2016 is a geospatial dataset that maps human settlements in Europe based on satellite imagery and ancillary data with a similar approach to the Global Human Settlement Layer (GHSL). The specification for EUGHSL2016 suggests it represents the percentage of built-up area coverage per spatial unit, with the definition of built-up being limited to structures which have a vertical extent and excludes sealed surfaces. The GHSL records human settlements and relies on the definition of BU structure (building) and BU areas (area where buildings are found). The GHSL method uses machine learning techniques to understand systematic relations between morphological and textural features extracted from the multispectral and, where available, panchromatic satellite imagery to identify human settlement % of the input satellite imagery comes from Satellite Pour l Observation de la Terre (SPOT) 5 and 6 missions with a spatial resolution of 2.5 m and covering the period from 2010 to The IMD2012 layer addresses some of the same issues as the EUGHSL2016, but there are also some fundamental differences which will impact on any comparisons and the onward use of the two products in downstream applications. IMD2012 captures the spatial distribution of artificially sealed areas, including the level of sealing of the soil per area unit, for a particular reference year. The level of sealed soil (imperviousness degree 1-100%) is produced using an automatic algorithm based on calibrated normalised difference vegetation index (NDVI). Further details of the HRL IMD2012 validation can be found in the report HRL IMPERVIOUSNESS DEGREE 2012 VALIDATION REPORT. The level of detail within the available specifications of the EUGHSL2016 layer was lower than that defined for the IMD2012 and other products. However, from what was available there was a close alignment to the actual product, but there were a few short comings with the definition of required thematic accuracy, the details of the documentation and compliance of the metadata. The scatterplots of the density values for the EUGHSL2016 against reference data showed that at the European level the relationship was linear and positive, although there was a large amount of scatter about the 1:1 line, with a suggested that there was a constant offset of around %. For the individual biogeographical zones there were a wide range of distribution types due to the different areal extent and landscape characteristics of the regions. The regions tended to support the finding from the whole sample set that there was an underestimation of building density in the EUGHSL2016 and significant amounts of omission and commission. The linear regression analysis which estimated the relationships between the product and reference results produced R 2 for both the Pan-European and biogeographical zone regionalisations of around 0.7, but with some variability between 0.56 and These R 2 were lower than those achieved by the IMD2012 which were around 0.8. For the slopes of regression lines, the majority are below 1 suggesting the EUGHSL2016 is generally under estimating building density with no obvious biogeographical or political pattern to the results. The overall thematic accuracies exceeded the established result of 96 % from the LUCAS exercise as expected due to the urban area making up a relatively small fraction of the European landscape even in the most densely built-up areas. However, when using the user s and producer s accuracy requirements set by the IMD layer and a 15 % threshold for built-up, only 13 % of the country and country group regionalisations reach the minimum of 85 % and only reached the target of 90 %. For the whole European area, the results were around 70 % for user s and producer s accuracies and the poorest results were down to just over 30 %. The 95 % confidence intervals results for both the user s and producer s accuracy produced a broad range of values with some below 0.1%, but also some extremely high values which require further investigation. Based on the results available in this report, the EUGHSL2016 layer is meeting, or nearly meeting, its required technical specifications in all areas except for thematic accuracy. However, as the thematic accuracy is not clearly defined, this must be judged in the context of other similar products such as the IMD2012. The thematic accuracies of the EUGHSL2016 are below those of the IMD2012 being around 70 % and 82 % respectively. Further work should be done on establishing thresholds and clear user s and producer s accuracy requirements for the EUGHSL2016. GIO Land Product Validation Page 2 / 27

4 Table of Contents Validation Framework Products to be validated Validation Criteria EUGHSL Validation approach Completeness... Error! Bookmark not defined Logical consistency... Error! Bookmark not defined Conceptual consistency... Error! Bookmark not defined Domain consistency... Error! Bookmark not defined Format consistency... Error! Bookmark not defined Additional logical consistency checks... Error! Bookmark not defined Positional Accuracy... Error! Bookmark not defined Thematic Accuracy... Error! Bookmark not defined Level of reporting... Error! Bookmark not defined Stratification and sample design Response Design Estimation and analyses procedures Temporal Quality... Error! Bookmark not defined Usability... Error! Bookmark not defined INSPIRE compliant metadata... Error! Bookmark not defined. 3. Validation check list... Error! Bookmark not defined. 4. Thematic accuracy EUGHSL2016 scatterplots & regression analysis EUGHSL2016 correspondence analysis EUGHSL2016 / IMD2012 comparison Conclusions and recommendations GIO Land Product Validation Page 3 / 27

5 List of Figures Figure 2-1: Example of SSUs organised in a 5x5 20m grid Figure 2-2: An example of the PSU (yellow square) and the SSU (green and red dots) for one sample location based on a LUCAS point Figure 4-1: Scatterplot for all replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line Figure 4-2: Scatterplots for the Alpine and Anatolian biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line Figure 4-3: Scatterplots for the Arctic and Atlantic biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line Figure 4-4: Scatterplots for the Black Sea and Boreal biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line Figure 4-5: Scatterplots for the Continental and Macaronesia biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line.. 18 Figure 4-6: Scatterplots for the Mediterranean and Pannonian biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line.. 18 Figure 4-7: Scatterplot for the Steppic biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line Figure 4-8: A plot of the EUGHSL2016 (ESM100m) against the IMD reference data (CODE12) Figure 4-9: Histograms of the EUGHSL2016 value (ESM100m) for different values of IMD2012 at the sample points Figure 4-10: Scatterplot for all the replicate 1 data for the EHSL (ESM100m) against the IMD reference data (CODE2012) with 1:1 line Figure 4-11: Examples of the EHLS only (blue), IMD only (red) and combined results (green) for 2012 compared to Open Street Map for the area to the west of London Heathrow Airport. 24 GIO Land Product Validation Page 4 / 27

6 List of Tables Table 2-1: Map projection details for LAEA-ERTS Error! Bookmark not defined. Table 2-2: Distribution of sample units per main strata and substrata Table 2-3: Weight factor to be applied to each stratum and substratum for constructing confusion matrices Table 4-1: A summary of regression line parameters for the scatterplots derived from the first replicates Table 4-2: A summary of the correspondence results for the 2012 blind interpretation of EUGHSL2016 using a 15 % threshold, country and country groups only. Green cells exceed the 85 % minimum accuracy requirement, orange are within 5% and red cells fall more than 5% short Table 4-3: A summary of the proportions of EHLS and IMD GIO Land Product Validation Page 5 / 27

7 List of Abbreviations BRME CLC CRS DWH EEA EO ETC ULS ETRS EUGHSL2016 EUREF FP7 FTS LM GDAL GHSL GIO GMES GRS HRL IMD IMC INSPIRE LAEA LUCAS MMU MMW NDVI PSU SSU TIFF Biogeographical Regions Map of Europe CORINE Land Cover Coordinate reference system Data warehouse European Environment Agency Earth Observation European Topic Centre on Urban, Land and Soil systems European Terrestrial Reference System European Settlement Map 2016 Regional Reference Frame Sub-Commission for Europe 7 th Framework Programme Fast Track Service Land Monitoring Geospatial Data Abstraction Library Global Human Settlement Layer GMES Initial Operations Global Monitoring for Environment and Security Geodetic Reference System High Resolution Layer Imperviousness density Imperviousness density change Infrastructure for Spatial Information in the European Community Lambert Azimuthal Equal-Area Land Use/Cover Area frame Survey Minimum Mapping Unit Minimum Mapping Width Normalised Difference Vegetation Index Primary Sample Unit Secondary Sample Unit Tagged Image File Format GIO Land Product Validation Page 6 / 27

8 1. Validation Framework The validation framework is defined by a comprehensive analysis of the product specifications to determine the criteria to be used for the validation exercise Products to be validated The European Settlement Map 2016 (also referred as 'EUGHSL2016') is a geospatial dataset that that maps human settlements in Europe based on satellite imagery and ancillary data. It has been produced with the same technology used for the Global Human Settlement Layer (GHSL) by the European Commission, Joint Research Centre, Institute for the Protection and Security of the Citizen, Global Security and Crisis Management Unit. Its production has been partly financed by the European Commission, Directorate General of Regional and Urban Policy. The specification for EUGHSL2016 suggests it represents the percentage of built-up area coverage per spatial unit. In this case the definition of built-up is limited to structures which have a vertical extent and excludes sealed surfaces. The GHSL records human settlements and relies on the definition of BU structure (building) and BU areas (area where buildings are found). The working definition of BU structure (building) used in the GHSL is as follows: buildings are enclosed constructions above ground which are intended or used for the shelter of humans, animals, things or for the production of economic goods and that refer to any structure constructed or erected on its site. The GHSL definition does not impose any permanency on the BU structure and is thus more inclusive, encompassing temporary human settlements such as refugee or internal displaced people (IDP) camps and slums or informal settlement concept. The buildings identified in the GHSL do not need to be part of any wider urban context. The GHSL method uses machine learning techniques to understand systematic relations between morphological and textural features, extracted from the multispectral and panchromatic (if available) satellite imagery to identify human settlement. The input satellite imagery comes from Satellite Pour l Observation de la Terre (SPOT) 5 and 6 missions and has a spatial resolution of 2.5 m % of the EUGHSL2016 is derived from SPOT 5 and 6 data with the remainder from other image sources and ancillary data such as pen Street Map. The image data covers the period from 2010 to The imagery is analysed by an automatic information extraction process based on machine learning which exploits the following reference data as training sets: CORINE Land Cover 2000 (CLC2000) 100 m raster version High-Resolution layers (HRL) on Imperviousness 2006 (IMD2006) 20 m raster The development of the methodology relies on the following assumptions regarding the built-up structures: they manifest themselves as image structures, often rectangular, and of specific size (relatively small). they are brighter than the surrounding image structures in the optical spectrum. they are characterized by high contrast relative to their neighbourhood and often cast shadows, which increases the local spectral heterogeneity in built-up areas. built-up area contains usually a group of buildings with some open area among them, e.g., with a road in the middle. built-up areas often consist of heterogeneous construction materials, which produces heterogeneous radiometric response in the same spatial domain. The accuracy of EUGHSL2016 has been improved through a process of benchmarking with population data, ensuring as much as possible consistency between residential population and built-up. Where statistically insignificant omission errors occurred, detection of built-up was imposed over populated cells. Where statistically insignificant commission errors occurred, these have been treated by adding masks using Urban Atlas GIO Land Product Validation Page 7 / 27

9 (code and 12210). External vector building footprint datasets have also been integrated in the layer (where available). The final EUGHSL2016 product is released at 10 m spatial resolution (i.e. the spatial unit is a pixel of 10 x 10 m). The product represents the density of buildings as an integer percentage per pixel. A systematic validation against the LUCAS 2012 data resulted in more than 96% accuracy for the built-up class. An updated version of the European Settlement Map at 100m resolution is also provided which is an aggregated version of ESM2016_10m. The EUGHSL2016 will be assessed alongside the HRL on Imperviousness for 2012 (IMD2012). Pan-European HRLs provide information on specific land cover characteristics, and are complementary to land cover / land use mapping such as in the CLC datasets. The HRLs are produced from 20 m spatial resolution satellite imagery through a combination of automatic processing and interactive rule-based classification, but their pixels are aggregated into 100 by 100 m grid cells for final products which are validated here. Pan-European wall to wall products will cover the EEA39 countries. Built-up areas are characterized by the substitution of the original (semi- ) natural land cover or water surface with an artificial, often impervious cover. These artificial surfaces are usually maintained over long periods of time. The HRL for Imperviousness captures the spatial distribution of artificially sealed areas, including the level of sealing of the soil per area unit. The level of sealed soil (imperviousness degree 1-100%) is produced using an automatic algorithm based on calibrated normalised difference vegetation index (NDVI). Further details of the HRL IMD2012 validation can be found in the report HRL IMPERVIOUSNESS DEGREE 2012 VALIDATION REPORT. A density threshold of 30% was used to derive a built-up layer from the EUGHSL2016 to be comparable with the imperviousness layer. There was a certain amount of debate over the actual threshold to be used for EUGHSL2016, but a number of tests suggested the 30% level would be reasonable. This was not intended to be a separate product, but instead was calculated for the verification process only, because density products cannot be verified. The IMD2012 layer addresses some of the same issues as the EUGHSL2016, but there are some fundamental differences which will impact on any comparisons and the onward use of the two products in downstream applications Validation Criteria For the EUGHSL2016 layer the main criteria selected for the validation will be: Completeness, the amount of omission and commission. Logical consistency, the adherence to formats, conventions and conceptual aspects. Thematic accuracy, the correspondence with reference data. Temporal quality, the alignment of the results with the reference year. Usability Metadata, the presence of sufficient metadata to describe the product. Other validations are either not applicable (topological consistency is not relevant for a raster dataset) or being dealt with by other aspects of the project (positional accuracy is an assessment of the image data) EUGHSL2016 The validation of the thematic accuracy of the EUGHSL2016 was done at three levels: A scatterplot of the density values extracted from the sample units for both the reference and map data should be made with a view to assess the correlation between reference and map values and identify any systematic bias (slope and intercept of the regression line significantly different for 1 and 0 respectively). A threshold should be applied to the density values for reference and map data to produce binary urban attributes for both the reference and map data layers. For EUGHSL2016, the threshold should be set to 30 % with density values lower than 30 % classified as 0 (non-built-up) and density values greater than GIO Land Product Validation Page 8 / 27

10 or equal to 30 % classified as 1 (built-up). There was no minimum acceptable thematic accuracy set in the specification, however an existing verification exercise against LUCAS data gave an overall accuracy of 96 %. A comparison was made with the IMD2012 results given the known differences in specifications and definitions. 2. Validation approach 2.1. Introduction The validation approach will provide guidance on how the products will be validated by defining suitable indicators or metrics. Detailed completeness and logical consistency checks had already been performed as part of the semantic checks undertaken by ETC ULS for the HRL products and a summary of the findings was provided as part of the validation report for the HRL IMD layers. Therefore, the aim of this validation exercise is not to repeat these, but to focus specifically on thematic accuracy Thematic Accuracy Level of reporting The level of reporting for the validation results is at pan-european level. However, results are also provided at a disaggregated level of countries or groups of countries with an area greater than 90,000km². Analysis of the results at a more disaggregated scale will contribute to assess regional differences if any and the causes of these differences Stratification and sample design Overview A stratified systematic sampling approach based on LUCAS is used for all thematic layers adapting the number of replicates to each stratum. The LUCAS sampling is densified for small strata. Using LUCAS sampling ensures coherence between the different layers and traceability. The EUGHSL2016 uses the same approach as the IMD / IMC products, with a stratification applied at two levels: 1. Stratification according to countries or group of countries with an area greater than 90,000 km². 2. Stratification based on a series of omission / commission strata. The number of primary sample units (PSUs) per stratum should be such to ensure a sufficient level of precision at reporting level. The minimum number of PSUs per stratum should be set at 50 if possible. Priority is given to strata which are known to be difficult to map: i.e. changes and difficult classes. The second level stratification was defined as follows: Commission Low Probability: Imperviousness Degree % & CLC impervious classes (minimum of 50 PSUs per country / group of countries) Commission High Probability: Imperviousness Degree % & CLC non impervious classes (minimum of 50 PSUs per country / group of countries) Omission High Probability: Imperviousness Degree 0-29% & CLC impervious classes (minimum of 100 sample units per country / group of countries) Omission Low Probability: Rest of the area (minimum of 50 PSUs per country / group of countries) Commission Change : all changes [increased and decreased] (minimum of 50 PSUs per country / group of countries) Commission Change : all changes [increased and decreased] (minimum of 50 PSUs per country / group of countries) GIO Land Product Validation Page 9 / 27

11 CLC impervious classes were defined as follows based on CLC2006: = continuous urban fabric = discontinuous urban fabric = industrial, commercial areas = ports = airports The validation exercise covers the whole study area to be valid (e.g. use of low and high probability omission strata for EUGHSL2016 with low sampling intensity in low probability stratum). Each PSU corresponds to one EUGHSL2016 pixel. Each PSU is then associated to secondary sampling units (SSUs) corresponding to a 5x5 grid with 20m between each SSU (Figure 2-1). The original assumption was that each SSU can then be associated with the corresponding HRL 20m layer pixel, but in this case each SSU is associated with 4 EUGHSL m pixels. Figure 2-1: Example of SSUs organised in a 5x5 20m grid Sampling units will be different for each layer due to different stratification approaches, but some sample locations will be shared thanks to using LUCAS as the basis for selecting sample units IMD2012/IMC sample sizes There was a total of 22,910 sample units selected for this specific contract. However, for the analysis of EUGHSL2016 the French Dominions were excluded leaving sample units (Table 2-1): Table 2-1: Distribution of sample units per main strata and substrata Omission Commission LABEL High Low High Low Change0609 Change0912 Total AL+ME+MK+RS+XK AT + CH + LI BA + HR + SI BE + LU+ NL + DK BG CZ + SK DE EE + LT + LV EL + CY ES FI FR HU GIO Land Product Validation Page 10 / 27

12 IE + UK IS IT + MT NO PL PT RO SE TR TOTAL To ensure that unequal inclusion probabilities are accounted for in the construction of the error matrix, weights are applied to each stratum as shown in (Table 2-2) following the formula presented at the beginning of section Table 2-2: Weight factor to be applied to each stratum and substratum for constructing confusion matrices Omission Commission LABEL High Low High Low Change0609 Change0912 AL+ME+MK+RS+XK 0, , , , , , AT + CH + LI 0, , , , , , BA + HR + SI 0, , , , , , BE + LU+ NL + DK 0, , , , , , BG 0, , , , , , CZ + SK 0, , , , , , DE 0, , , , , , EE + LT + LV 0, , , , , , EL + CY 0, , , , , , ES 0, , , , , , FI 0, , , , , , FR 0, , , , , , HU 0, , , , , , IE + UK 0, , , , , , IS 0, , , , , , IT + MT 0, , , , , , NO 0, , , , , , PL 0, , , , , , PT 0, , , , , , RO 0, , , , , , SE 0, , , , , , TR 0, , , , , , The sample units were provided to the bulk interpretation team as separate shapefiles in which all the information on strata was removed to ensure the independence of the interpretation. GIO Land Product Validation Page 11 / 27

13 Response Design Overview The LUCAS points were re-interpreted based on available in situ and reference data, but the LUCAS thematic information is not used directly during the process. The response design for most data set being validated during this project was based on the interpretation of thematic classes or surface characteristics (e.g. soil sealing or presence or absence of built features) at the SSU point level taking into account the details of the product specifications (MMU, MMW, class definitions, etc. ). The interpretation was based on a combination of available in situ data, for example, the validation could use the VHR mosaic alongside the imagery used in production and virtual globes datasets. A double blind approach was adopted as the initial process to guarantee the complete independence of the validation data from the map products. This may underestimate their accuracy where SSU points are uncertain. This was resolved by the plausibility approach for which the interpreter checks the map value to assess whether it can be considered correct or not, within the frame of accepted product specifications. Also, density values may be adjusted based on experience of known uncertainties to allow a more realistic comparison EUGHSL2016 The interpretation of each PSU at a particular point in time is based on the assessment (building or non-building) of 5 x 5 grid of SSU points to derive a density value of building density at the PSU level (Figure 2-1). The values are thus represented by percentage values in 4% steps from 0 100%. The process is then repeated for each time point. The datasets against which the interpretation is performed are divided in two main groups, guiding data and reference data. The guiding data were used in the production of the IMD and IMC layers and are hosted by the EEA as web services. The available guiding data were: 2012 HRIMC1 25 m dataset (not used for original production) 2009 HRIMC1 20 m dataset (used during production) 2006 HRIMC1 20 m dataset (known issue: Shift in relation to 2012 & 2009 datasets) The reference data were to provide more spatial detail and strong landscape context to the assessment. The available reference data were: CORE003 VHR data Bing maps image / cartography layer Open Street map data GoogleEarth image / cartography data Further in-situ data The interpretation process was controlled and the results recorded by the selected sample locations. For each location there were two vector layers. The sample points, or PSU, were represented by a 100 m by 100 m rectangle with a centre point at the selected location and the grid points, or SSU, were represented by a set of points on a 5 x 5 grid with 20 m spacing within each PSU (Figure 2-2). Each dataset contained a set of attributes to both define the characteristics of the sample point, PSU or SSU, and the results of the assessment. GIO Land Product Validation Page 12 / 27

14 Figure 2-2: An example of the PSU (yellow square) and the SSU (green and red dots) for one sample location based on a LUCAS point. For the IMD / IMC the assessment process was applied to approximately 22,502 PSUs each of which contained 25 SSU assessed for sealed and non-sealed surface, making in excess of 500,000 total interpretations per time point. For the EUGHSL2016 it was not felt necessary to re-interpret all of the PSUs due to the similarity of the specifications. A subset of PSUs were therefore selected where the difference between IMD2012 and EUGHSL2016 was greater than 20%. This resulted in 7,412 PSUs which were re-interpreted and the SSUs were assigned to buildings and non-buildings. To perform an internal quality check 10% of the assessments undertaken by the bulk interpretation team were repeated by an independent team. For each PSU and its constituent SSUs the following process was followed to complete the assessment. At the PSU level the guiding data was screened for clouds and cloud shadows and a note was made in the comment attributes when no useable data was available for a reference year. The presence of any spatial shifts either between the guiding datasets or against reference datasets were also noted. At the SSU level within each PSU the first step was to assess the guiding data in the context of the available reference data to understand the relationship between the guiding data and the reference data. Then, using the guiding data for 2012 in the context of the reference data, an assessment was made whether at the location of each SSU there was a building or not. If there was a building present a value of 1 was assigned to the VAL_2012 attribute on the SSU point feature. If no building was present a check was made to ensure that the VAL_2012 attribute for the SSU point feature was set to 0. If necessary, the UNCERTAIN and COMMENT attributes were used to record SSUs where the coding of sealing was unclear or certain issues occurred. The assessment was made for the actual location were each SSU point feature lay in the guiding and reference data and visible in the imagery. For instance, buildings under tree canopies and over lawns are recorded as buildings Estimation and analyses procedures Thematic accuracy should be presented in the form of an error matrix. Unequal sampling intensity resulting from the stratified systematic sampling approach should be accounted for by applying a weight factor (p) to each sample unit based on the ration between the number of samples and the size of the stratum considered: GIO Land Product Validation Page 13 / 27

15 Where i and j are the columns and rows in the matrix, N is the total number of possible units (population) and π is the sampling intensity for a given stratum. Overall accuracy and User and producer accuracy should be computed for all thematic classes and 95% confidence intervals should be calculated for each accuracy. The standard error of the error rate can be calculated as follows: σ h = p h(1 p h ) where nh is the sample size for stratum h and ph is the expected error rate. The standard error is calculated for each stratum and an overall standard error is calculated based on the following formula: n h σ = w h 2. σ h 2 In which is the proportion of the total area covered by each stratum. The 95% confidence interval is +/ GIO Land Product Validation Page 14 / 27

16 3. Thematic accuracy The thematic accuracy is assessed using a number of approaches and within a number of regionalisations to better understand the thematic characteristics across biogeographic regions and countries. The assessment here was based on a sample of reference points EUGHSL2016 scatterplots & regression analysis A scatterplot is a way of displaying data against Cartesian coordinates to show and compare values for two variables within a dataset. The data is displayed as a series of points, where the x and y locations relate two variables assigned to a particular recording instance, in this case a PSU. For this validation exercise the position / value on the horizontal axis represented the reference (VAL_2012) information and the positon / value on the vertical axis represents product (ESM100m) information. In this way, the relation of the reference and product information for a point can be compared to a 1:1 line which runs diagonally across the scatter plot. In such a rich dataset the points are likely to overlap, so a level of transparency is used for each point so if more than one point occupies the same location the strength of the colour increases. The scatterplots in this section are based on the first replicates of the sampling scheme only. This sub sampling minimises the impact of the large numbers of values which tend to lie on the x and y axis and distort the visual appears of the scatterplots and the regression results. The points that lie exactly on the x and y axes are related to omission and commission rather than the calibration of the EUGHSL2016 values themselves. From Figure 3-1, it can be seen that at the European level the relationship is linear and positive, although there is a large amount of scatter about the 1:1 line. There also seems t be a tendency for the EUGHSL2016 to underestimate the density of buildings and it might be suggested that there is a constant offset of around %. Figure 3-1: Scatterplot for all replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. GIO Land Product Validation Page 15 / 27

17 Figure 3-2 through Figure 3-7 show scatterplots for the biogeographical regions individually. There are a wide range of distribution types due to the different sizes and landscape characteristics of the biogeographic regions. The Arctic region (Figure 3-3) has no urban areas represented in the replicate 1 samples so all the points are zero for both the reference data and the product. The Continental region (Figure 3-5) would appear visually to give the best relationship between the reference data and the product, with less bias and similar amounts of omission and commission. For the regions with a small number of points it is difficult to come to any clear conclusions. Those with enough points to show a distribution to support the finding from Figure 3-1 that there is an underestimation of building density and significant amounts of omission and commission. To quantitatively summarise the results displayed in the scatterplots, a linear regression analysis is performed to estimate the relationships between the reference and product information. The analysis produces a coefficient of determination (R 2 ) which is a gives information about the goodness of fit of the estimated regression model. Coefficients of determination closer to 1 represent a better fit. In this case as the reference and map information are meant to represent the same information then it is useful to also consider the slope and intercept of the estimated regression model. The slope should therefore approach 1 and the intercept should be close to 0 for the required relations. Deviations from the expected values give an indication of the correspondence of the reference and map data. As with the scatterplots above, the regression analysis was produced using the replicate 1 samples only to give a more representative indication of the calibration of the map densities to the reference data (Table 3-1). There are some missing items in the table due to the limitations of the data available for Iceland and the Arctic biogeographic regions not allowing the successful completion of the regression analysis. As noted earlier, none of the replicate 1 Arctic region samples contain any buildings thus making the assessment impossible. The R 2 for the Pan-European level and the biogeographical regions tend to be around 0.7, but with some variability which ranges from 0.56 to The biogeographic regions R 2 values are not only effected by the changing character and make-up of the landscape, but also the numbers of samples used in each region. The exceptions to the R 2 results are the Artic region with no results and the Macaronesia region, four archipelagos in the North Atlantic Ocean off the coasts of the Europe and Africa. The later represents a very small sample and possible unique urban situations. These R 2 are lower than those achieved by the HRL Imperviousness which were around 0.8. In the case the slopes of regression lines they seem to be distributed around 1 showing examples of both underand over-estimation of building density in the map product depending on the regionalisation. The majority are below 1 suggesting the EUGHSL2016 is generally under estimating building density with no obvious biogeographical or political pattern to the results. This is supported later by the general dominance of omission errors in the correspondence analysis. GIO Land Product Validation Page 16 / 27

18 Figure 3-2: Scatterplots for the Alpine and Anatolian biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. Figure 3-3: Scatterplots for the Arctic and Atlantic biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. Figure 3-4: Scatterplots for the Black Sea and Boreal biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. GIO Land Product Validation Page 17 / 27

19 Figure 3-5: Scatterplots for the Continental and Macaronesia biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. Figure 3-6: Scatterplots for the Mediterranean and Pannonian biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. Figure 3-7: Scatterplot for the Steppic biogeographic regions using replicate 1 data for the EHSL (ESM100m) against reference data (VAL_2012) with 1:1 line. GIO Land Product Validation Page 18 / 27

20 Table 3-1: A summary of regression line parameters for the scatterplots derived from the first replicates. ESL2016 n R² Slope Intercept European All ALP ANA ARC 90 na na na ATL BLS BOR CON MAC MED PAN STE AL+ME+MK+RS+XK AT + CH + LI BA + HR + SI BE + LU+ NL + DK BG CZ + SK DE EE + LT + LV EL ES FI FR HU IE + UK IS 89 na na na IT NO PL PT RO SE TR GIO Land Product Validation Page 19 / 27

21 3.2. EUGHSL2016 correspondence analysis To undertake the correspondence analysis a density threshold had to be set for the EUGHSL2016. There was no guidance on how to set this threshold, particularly as the EUGHSL2016 was only mapping buildings and not all urban features. When the EUGHSL2016 was plotted against the reference data for the imperviousness layer (Figure 2-1) the best fit line had a slope close to 0.5 suggesting that the equivalent threshold to the 30 % used for IMD would be 15 %. Figure 3-8: A plot of the EUGHSL2016 (ESM100m) against the IMD reference data (CODE12). As a further assessment of the most suitable threshold for built-up, the EUGHSL2016 results were compared to IMD2012 for the sample points where the IMD2012 has values around the 30 % threshold (i.e. 24, 28, 32 & and 36 %). Figure 3-9 shows the distribution of EUGHSL2016 values for the four IMD2012 values and the mean of these results. It also appears to suggest a threshold of 15 % would be more appropriate for the EUGHSL2016, with a generally constant value below 15 % and then a rapid decline in the histogram above it. GIO Land Product Validation Page 20 / 27

22 Figure 3-9: Histograms of the EUGHSL2016 value (ESM100m) for different values of IMD2012 at the sample points. For each of the country / country group regionalisations, the correspondence analysis (user s and producer s accuracies) and the 95 % confidence intervals were calculated (Table 3-2). The selection of the 15 % threshold has produced a slight improvement in the correspondence results compared to earlier tests with the 30 % threshold with some country and country groups achieving the minimum accuracy requirement for IMD and some others falling with 5 % of the requirement. However, these represent less than half of the metrics, therefore the majority are still further than 5 % away from the accuracy requirement. There is no clear relationship between the country or country groups for user s and producer s accuracy. The producer s accuracy results contain the highest single accuracy value, but more of the user s accuracy results exceed the various requirement thresholds. The accuracy results range from a little over 90 % down to almost 30 % with mean for the user s accuracies is 73.8 % while the mean for the producer s accuracies is of 68.6 %. The 95 % confidence interval result show a broad range of values between country and country groups. The majority of the 95 % confidence intervals for the producer s accuracy are all less than 1 %, with a few below 0.1 % giving support to the representativeness of those results. However, the user s accuracy produced a broad range of 95 % confidence interval results with some below 0.1%, but also some extremely high values which require further investigation. GIO Land Product Validation Page 21 / 27

23 Table 3-2: A summary of the correspondence results for the 2012 blind interpretation of EUGHSL2016 using a 15 % threshold, country and country groups only. Green cells exceed the 85 % minimum accuracy requirement, orange are within 5% and red cells fall more than 5% short Blind (15 %) User 95% CI Prod 95% CI European All AL+ME+MK+RS+XK 81.9 < AT + CH + LI 83.0 < BA + HR + SI BE + LU+ NL + DK BG 82.2 < CZ + SK 87.9 < DE EE + LT + LV EL ES FI 31.3 < FR HU 82.6 < IE + UK 68.6 < IS 73.5 < <0.1 IT NO 80.0 < PL 72.0 < PT 88.9 < RO 75.4 < SE 86.1 < TR GIO Land Product Validation Page 22 / 27

24 3.3. EUGHSL2016 / IMD2012 comparison From the definitions of EUGHSL2016 and IMD2012 there are obvious differences in what the two products aim to map. However, comparisons between the products could be enlightening to show how they relate and to pull out different aspects of the urban environment. Figure 3-10 shows the EUGHSL2016 product data (ESM100m) plotted against the reference data for the imperviousness data (CODE2012). As anticipated from the specification, the building densities are lower than the imperviousness which includes all sealed surfaces. Figure 3-10: Scatterplot for all the replicate 1 data for the EHSL (ESM100m) against the IMD reference data (CODE2012) with 1:1 line. The EHSL and IMD layers can be visualised together to get a feeling for which features are mapped by the different approaches. In Figure 3-11 both the products represent the landscape in a reasonable fashion given their specifications. There is a certain amount of noise and the combined impacts of the 100 m aggregation and 30 % threshold are visible in places. GIO Land Product Validation Page 23 / 27

25 Figure 3-11: Examples of the EHLS only (blue), IMD only (red) and combined results (green) for 2012 compared to Open Street Map for the area to the west of London Heathrow Airport. The combined dataset is summarised by area and percentage for Europe in Table 3-3. As expected the IMD maps considerably more area as urban than EHLS. The total figure is somewhat less than that estimated by CLC2012 at around 4.1% which also contains non-sealed surfaces due to the minimum mapping unit of 25 ha. Table 3-3: A summary of the proportions of EHLS and IMD. Area Percentage IMD only % EHLS only % Combined % 2.81% GIO Land Product Validation Page 24 / 27

26 4. Conclusions and recommendations This report represents a thorough examination of the EUGHSL2016 layer and a comparison with Copernicus Imperviousness layer. It provides an independent validation assessment of the performance of the product against its design specifications with blind interpretations putting clear boundaries on the thematic quality of the product being made available to the wider community. The review of the specifications of the products in chapter Error! Reference source not found. shows that the level of detail for the EUGHSL2016 was lower than that defined for the imperviousness layer. There was a close alignment of the specification and the actual product, but there were a few short comings with the definition of thematic accuracy, documentation and metadata. Also, the product does not include details on the image data used for the production (e.g. time stamps, sensor, ancillary information etc.) which would be extremely useful when trying to interpret the results of studies such as this. The scatterplots for the EUGHSL2016 against reference data shown in section 3.1 give the overall situation for the layer across Europe as well as for regionalisations by biogeographic zone. At the European level the relationship was linear and positive, although there is a large amount of scatter about the 1:1 line, with a suggested that there is a constant offset of around %. For the individual biogeographical zones there were a wide range of distribution types due to the different areal extent and landscape characteristics of the regions. These ranged from the Arctic region which had no urban areas present in the samples to the Continental region which visually gave a strong relationship between the reference data and the product. The remaining regions tended to support the finding from the whole sample set that there was an underestimation of building density in the EUGHSL2016 and significant amounts of omission and commission. The linear regression analysis which estimated the relationships between the reference and product results produced R 2 for the Pan-European level and the biogeographical zones of around 0.7, but with some variability which ranges from 0.56 to The biogeographic regions R 2 values were not only effected by the changing character and make-up of the landscape, but also the numbers of samples used in each region. These R 2 were lower than those achieved by the HRL Imperviousness which were around 0.8. In the case the slopes of regression lines they seem to be distributed around 1 showing with examples of both under- and over-estimation of building density in the map product depending on the regionalisation. The majority are below 1 suggesting the EUGHSL2016 is generally under estimating building density with no obvious biogeographical or political pattern to the results. For the correspondence analysis, the overall accuracies exceeded the established thematic accuracy of 96 % from the LUCAS exercise as expected as the urban area makes up a relatively small fraction of the European landscape even in the most densely built-up areas. However, when using the user s and producer s accuracy requirements set by the IMD layer, only 13 % of the country and country group regionalisations reach the minimum of 85 % when a 15 % threshold for built-up was selected. For the whole European area, the results were around 70 % for user s and producer s accuracies. Only 6 regionalisations exceeded the 85 % target and the poorest results were down to almost 30 %. Overall, the correspondence results in terms of user s and producer s accuracies are in the range of 30 to 90 %. The 95 % confidence intervals for both the user s and producer s accuracy produced a broad range with some below 0.1%, but also some extremely high values which require further investigation. As with the imperviousness layer, the stratification approach proved effective with acceptable results both at pan-european and disaggregated regionalisation. However, some PSUs with borderline (around the 30 % urban threshold) in the low risk omission stratum did cause some problems to the imperviousness results, but these were corrected in the assessment of EUGHSL2016. Further corrections were made where the PSUs were found to give results close to the 15 % threshold. In the future, perhaps CLC should be combined with other data such as imperviousness masks and / or Open Street Map to increase the extent of the high risk omission stratum to include borderline cases. GIO Land Product Validation Page 25 / 27