SIGMA thematic map validation

Transcription

1 LPVE

2 LPVE SIGMA GIS validation tools Thematic map validation plugin with 3 tools: Estimate sample size tool Estimate sample size using map class accuracies tool Build confusion matrix - tool

3 LPVE LPVE, Frascati / Italy, on New understanding of thematic Land Cover validation on basis of the SIGMA validation protocol and QGIS Plug-in Carsten Haub, Luca Kleinewillinghöfer & Sven Gilliams

4 LPVE SIGMA Stimulating Innovation for Global Monitoring of Agriculture EU contibution to GEOGLAM / GEOSS , ca. 11 mio Globale partner network led by VITO Focussing at the montoring of: o Agricultural land use change, intensification, and production versus o environmental impacts zum

5 LPVE Aim Aims of WP 33: Better integration of geographic information and statistics Improving the methodology for using remote sensing Supporting officers and operators in charge to organise relevant recent and reliable geospatial information for administration and decision making Through the collation of common principles and guidelines Objectives Therefore this SIGMA project component aimed to: Setting the scene through a data collection and validation protocol In order to provide practical guidelines that are in compliance with the requirements for sound statistical analysis of accuracy parameters of Earth observation derived thematic maps And to connecting this to dedicated GIS tools to support planning, surveying and analyzing in a convenient way

6 LPVE Initial problem Deficits between recent state of the science and technology related to appropriate validation techniques and principles versus the degree and appropriateness of validation in operational applications (Foody, 2002; Olofsson, 2013) Lack of systematic consideration during planning stages of EO applications, which might be possibly linked to limited resources, awareness or instruments In view of an increasing availability of EO data, products and services, e.g. via Copernicus

7 SIGMA thematic land cover validation LPVE Approach 1. Survey on common practice and approaches in EO data validation in frame of JECAM sites 2. Elaboration of a protocol and the collation of practical guidelines for appropriate field data collection for remote sensing applications in agriculture (and other land cover mapping activities) with dedicated accuracy requirements validation and accuracy assessment of land cover classification products 3. Development of applied Open Source GIS validation tools 4. Elaboration of applied training manual and practical guide

8 SIGMA thematic land cover validation LPVE Data collection and validation protocol Systematic collation and elaboration of the current state of science and technique with respect to: Commonly applied classic EO accuracy assessment approaches Required statistical measurements and an Outlook to modern object based classification methods Elaborated by JRC, FAO & EFTAS

9 SIGMA thematic land cover validation LPVE International standards for validation of remote sensing products. No clear standard for validation defined, different initiatives to standardize validation. Stages of validation from the Land Product Validation (LPV) sub-group of the CEOS Working Group on Calibration and Validation (WGCV) Stage 1 Validation Stage 2 Validation Stage 3 Validation Stage 4 Validation Product accuracy is assessed from a small (typically < 30) set of locations and time periods by comparison with in situ or other suitable reference data. Product accuracy is estimated over a significant set of locations and time periods by comparison with reference in situ or other suitable reference data. Spatial and temporal consistency of the product and with similar products has been evaluated over globally representative locations and time periods. Results are published in the peer-reviewed literature. Uncertainties in the product and its associated structure are well quantified from comparison with reference in situ or other suitable reference data. Uncertainties are characterised in a statistically robust way over multiple locations and time periods representing global conditions. Spatial and temporal consistency of the product and with similar products has been evaluated over globally representative locations and periods. Results are published in the peer-reviewed literature. Validation results for stage 3 are systematically updated when new product versions are released and as the time-series expands.

10 LPVE SIGMA GIS validation tools

11 LPVE Aims of WP 33: Better integration of geographic SIGMA WP 33 Task 3: Data collection and validation protocol information and statistics Improving the methodology for using remote sensing Area Frame Sampling Sampling design Validation Thematic Land cover Validation Collect Data Send Data Analysis Data Database GeoODK & GeoWiki Data collection

12 LPVE Validation tools Development of tools and routines to support thematic map validation based on open source technologies and on basis of DLV33.2 [Land cover validation protocol] Basic functionalities: Sample size estimation for probability sampling for validation with or without using a priori information Analysis of map and map class accuracies in confusion matrix General requirements: Open source technologies Offline environment Standard data formats User friendly and easy handling (for non statisticians)

13 LPVE Tools to estimate sample size Aim to provide a starting point to test and to plan reference data collection efficiently Calculator function to compare results, number of samples or expected accuracies under different scenarios How many samples to choose for statistical sound validation? Depends on: proportion (size) and number of the different map classes predefined (contractual) precision (confidence bounds) for accuracy measures available resources for reference data collection a priori information about map class accuracies

14 LPVE Estimate sample size tool Required input: Map classes and proportions (from raster / vector / manual entry) Desired confidence level concept of multinomial distribution (Congalton & Green 2009) Sample calculator Total number of samples to reach predefined precision Expected precision of OA with predefined number of samples *Precision = margin of error under selected confidence level

15 LPVE Estimate sample size using map class accuracies tool The tool builds on per class accuracies as input to estimate number of samples per class approach presented in (Olofsson et al., 2014) Required input: Map classes and proportions Desired confidence level Targeted precision of OA Expected map class accuracies (user s accuracy) 1. Output Total number of samples to reach predefined precision of OA

16 LPVE Estimate sample size using map class accuracies tool Assign samples to class Calculate expected precision of user s accuracy (margin of error) Fine tune per class sample size allocation Expected precisions of per class accuracies under sampling scenario Expected precision of OA Additional functions: Random stratified sampling from vector layer (sampling frame) Sampling plan output to csv

17 LPVE Build confusion matrix - tool Provide class related accuracies including its related confidence intervals Required input: ground truthing results, reference and map data at samples Map classes and proportions Output as.csv Confusion matrix, proportional matrix, Overall / user s / producer s accuracies, variances, standard errors, confidence bounds for 90, 95 and 99% interval

18 LPVE Build confusion matrix - tool Ensures: Full transparency on accuracy assessment Objective evaluation of the map product And ultimately the opportunities to explore how far decisions can be built on given map products.

19 LPVE Examples

20 LPVE Example 1: SIGMA Global crop map

21 LPVE Eaxample 1: SIGMA crop map (UCL) Assessments have been applied to: UCL crop/no-crop maps (based on GLC, Globecover 30) SIGMA global validation experiment (based on systematic stratified sampling) Achievements: Overall accuracy: 83,73% No. samples: 629 Confidence bounds: +/- 2,53%

22 LPVE

23 LPVE

24 LPVE SIGMA thematic validation - tests Adaptations where applied to: Accept slightly wider confidence intervals Increase targeted precisions Reduce no. of samples Outcomes: Confidence interval: <5% Related No. samples: 186 Confidence interval: ~1% Related No. samples: Related No. samples: 300 Confidence interval: ~3,6%

25 LPVE Example 2: Crop type classification NRW

26 LPVE

27 LPVE

28 LPVE Example 3: Sen2Agri Czech demonstrator

29 LPVE Example: EFTAS SIGMA validation plugin Class Proportion* Expected uers accuracy** Margin of error* Sample size ESA Czech Crop type map 2015, Czech Agriculture from space. Processed by UCL and validated by gisat** winter rapeseed 14.9% 95% +/- 4.3% 175 winter cereals 52.5% 89% +/- 3.9% 50 spring cereals 2.0% 85% +/- 8.3% 50 sugarbeet 2.4% 89% +/- 7.3% 50 maize 4.4% 84% +/- 8.6% 50 potatoes 0.9% 85% +/- 8.3% 50 fodder crops 0.9% 55% +/- 11.7% 50 other annual crops 21.8% 44% +/- 0.8% 105 Total 100% 87% +/- 2.8% 600 Source: *) Agricultural statistics of Czech Republic ( **) Validation results by GISAT s.r.o. (ESA Final report: Czech Agriculture National Demonstrator (CzechAgri) from ).

30 LPVE Current applications

31 LPVE IACS Monitoring pilot 2018

32 LPVE BfN Quantification of Grasland in Germany

33 LPVE EEA Copernicus Natura 2000 Habitate mapping

34 LPVE Conclusion

35 LPVE Legend and cartographic standard Sampling unit Analysis of the data Sampling design Data processing & documentation Survey guidelines

36 SIGMA thematic land cover validation LPVE Applied checklists Legend or land cover classification scheme Questions to be raised: Is there a clearly pre defined legend, which ensures: Sampling units o Full thematic match between map and reference data? Questions to be raised: o Clear rules for survey observations? Is there Sampling a clear design definition of the sampling unit which adequately suites the Questions most coarse to input be raised: data? Is this applicable Is there a clear for a field predefined data collection sampling in scheme, terms of which costs, accessibility feasibility Survey in terms guidelines Ensures of representative work load? and sufficient allocation of samples? Is it open to Questions also suite to other be raised: Allows to verify input the representativeness data in terms of data of gaps? the validation results? Are the clear observation rules and a defined field form which allow to: Documents a sound validation planning? odata Collect processing precise and and documentation complete field observations? Helps guiding the surveyors itinerary planning? oquestions Transparently to be raised: document the particular observation and the sample Is unit s there level? a dedicated digital data entry and digitization interface which allows to Analysis completely of the collate data the collected field results in a digital format for further Questions processing to be raised: and back up? Is there Is a there clear data a clear flow analysis mechanism functionality in order which to ensure is defined completeness on basis of the data transmission? previously defined parameters? Is it ensured Are the to explore results and data calculation loss at any steps processing transparently level? documented? Is it possible to provide access to the both data, field data and mapping product in order to allow independent assessments?

37 LPVE Conclusion We did not invent anything new but introduced well established concepts (see above) via R and shiny into an open source GIS environment Overall an interesting contribution to thematic map validations in order to facilitate planning and evaluation of mapping products Hopefully an opportunity to increase and maintain user acceptance Positive feedbacks from users in various countries Current limitations Performance limits of QGIS and the plugin (e.g. limited number of classes, data size,... to be further developed) Limited sampling design options current version targets random stratified sampling User friendliness and installation require additional development

38 LPVE Do you have questions? ESA, Frascati, EFTAS Fernerkundung Technologietransfer GmbH ı Oststr ı Münster ı Germany ı ı Tel.: +49 (0) ı Fax: +49 (0)