VERIFYING AND VALIDATING THE ACCURACY AND QUALITY OF IFSAR- DERIVED ELEVATION MODELS WITHIN AN INDEPENDENT QUALITY MANAGEMENT SYSTEM INTRODUCTION

Transcription

1 VERIFYING AND VALIDATING THE ACCURACY AND QUALITY OF IFSAR- DERIVED ELEVATION MODELS WITHIN AN INDEPENDENT QUALITY MANAGEMENT SYSTEM Lisa Mooney, IV&V Technician/GIS Specialist Stephen Cliffen, GIS Specialist Bob Richardson, IV&V Manager Intermap Technologies Corp. 2 Gurdwara Road, Suite 200 Ottawa, Ontario, Canada, K2E 1A2 lmooney@intermap.ca scliffen@intermap.ca brichardson@intermap.ca ABSTRACT The ever-increasing demand for highly-accurate digital elevation data means that industry leaders must adapt their operational procedures to ensure the accuracy of their data. Intermap provides high-resolution digital elevation datasets using Interferometric Synthetic Aperture Radar (IFSAR) technology for a wide variety of users and applications. With the launch of Intermap s countrywide program NEXTMap USA there is a greater need for an independent quality management system to ensure accuracy and consistency in the finished data. While internal quality control measures ensure quality of product during production workflow processes, it is necessary to conduct assessments from an independent and external perspective. The Independent Verification and Validation (IV&V) group provides company personnel with comprehensive analyses beyond what can be conducted by the internal quality control procedures. This requires using different raster analysis techniques in several GIS packages to manipulate data for further visual and statistical inspection. Of the many quality checks conducted by IV&V, quantitative analysis involves statistically comparing elevation data with accurate, publicly available, ancillary information. This ancillary information includes LIDAR (Light Detection and Ranging) data, verification checkpoints from the National Geodetic Survey, as well as 10m and 30m terrain data from the National Elevation Dataset (NED). After a brief introduction to the datasets and tools used during the IV&V processes, this paper will present an assessment of the importance of this independent quality management system IV&V to exist within the scope of the NEXTMap USA program. INTRODUCTION With the variety and volume of geospatial datasets available in the current geographic marketplace at an all time high, the quality of these available datasets has become the underlying premise of their value to stakeholders in the business world, as well as, to the end users. More specifically, it has become not only necessary to provide consistent and seamless digital elevation datasets that meet product specifications, but to also prove these geospatial data products are meeting the clients application needs. The ever-increasing demand for highly-accurate digital elevation data means that industry leaders must adapt their operational procedures to ensure the accuracy of their data. Quality assurance processes within the production workflow environment exist to ensure that product specifications are met and output is consistent. Such processes are necessary in order to measure how accurately the output geospatial data correlates with stated product standards. Historically, this has included the concept of Independent Verification and Validation (IV&V) to exist within the overall scope of producing quality geospatial products. Since 2003, Intermap Techologies has relied on the IV&V Department to provide timely, accurate reports on the quality of their digital elevation products, by analysing the data independent of the production process and production management. This paper will briefly discuss the tools and datasets used in IV&V analysis, to provide in more detail an assessment of how this Quality Management System exists within the scope of the NEXTMap USA program. In doing so, this paper will demonstrate how IV&V s place within the Customer Care Division at Intermap provides a new understanding of quality from a client-perspective. To show this, a case study demonstrating the IV&V process

2 will be conducted by comparing a variety of publicly available datasets to Intermap data through both qualitative and quantitative means. INDEPENDENT VERIFICATION AND VALIDATION (IV&V) AS A QUALITY MANAGEMENT SYSTEM Quality Management Systems set the standard for the policies and procedures that exist in an organization. The Quality Management System (QMS) is a system of continuous improvement which can lead to product improvement and all of the processes that help build it ( The Independent Verification and Validation Department operates in this manner, with the primary focus on verifying and validating the quality of the finished product, whether it is a Core product from the Intermap Store, or a Value-Added product. To achieve this, IV&V conducts analyses at key areas in the production workflow process, but acts independently so that production processes and procedures can be investigated from an objective and unbiased point of view. Conducting analysis under this philosophy allows IV&V to examine the quality of the data through alternative quality assurance methods, which often leads to new findings about the data, as well as identifying areas of improvement for the current production process. In the context of NEXTMap USA Intermap s on-going country-wide mapping initiative IV&V is involved in the production workflow process in several key areas: GPS Base Station and Corner Reflector (Control Point) Check, Motion and CORVEC Check, Pre-Edit Verification Check Point (VCP) Check, and the focus of this paper the Post-Edit VCP and Reporting Check. Figure 1 demonstrates IV&V workflow in conjunction with internal production processes. Ultimately, IV&V s goal is to provide internal production departments, sales, management, and clients with information about the overall quality of Intermap data as it is completed. Figure 1. Enterprise Workflow & IV&V Workflow

3 INTRODUCTION TO IV&V TOOLS Intermap has embarked on the ambitious under-taking of digitally mapping the continental United States and Hawaii, known as the NEXTMap USA program. Intermap is experienced in country-wide data collection, processing and editing of seamless core product datasets which include the Digital Surface Model DSM, Digital Terrain Model DTM, and ORI Ortho-rectified Radar Image. The DSM is a first surface product meaning that the radar platform recognizes the first surface of the terrain or a feature on the terrain. The DTM is a terrain product that is created by extracting features from the DSM so that only the terrain remains. The removal of residential and vegetative areas is conducted in the editing process to achieve this Bare Earth model. Table 1. Core Product Specifications Resolution RMSE 95 th Percentile Type of Accuracy DSM 5m Vertical DTM 5m Vertical ORI 1.25m Horizontal Source: Intermap Product Handbook and Quick Start Guide v.3.3 Due to the immense size of the NEXTMap USA program, the data is organized into missions or blocks, each roughly 200km by 200km in size, for easier acquisition and processing. IV&V conducts analysis on these blocks, as they move through the production process from acquisition by airborne RADAR (IFSAR) to client delivery. The reports created by IV&V present an assessment of the data at each check point along the production process. After completing IV&V checkpoints 1 through 3, IV&V uses a variety of tools, datasets, softwares, and analysis techniques to evaluate the quality of finished data, and presents this information in a final report. Essentially, the final report presents an assessment of the quality of the final NEXTMap USA block deliverable to the Intermap Store and subsequent clients. Each final report consists of several comprehensive reviews: a check for minimum elevations, block edge check, post-edit vcp analysis and an ancillary data comparison. Final Report Check # 1 Minimum Elevation Check A check for minimum elevation is conducted on 100% of the data within each NEXTMap USA block. A limitation of radar is that data may contain artifacts caused by system errors referred to as motion, which has been measured in Intermap s radar system to be +/- 30cm. In low-lying areas, this noise may produce invalid negative elevations, which are typically removed during the editing process. To verify that this potential problem has been managed, IV&V checks the data for invalid negative elevations in the Digital Surface Model (DSM) and Digital Terrain Model (DTM). Typically, negative elevations are eliminated in the edit process; however, there are features such as open-pit mines and low-lying coastal areas that may contain valid negative elevations. By using an Intermap-developed script, IV&V is able to report statistics on the minimum and maximum elevations in the DSM and DTM for each tile within a NEXTMap USA block. If there are any negative elevations in the DSM or DTM, they will be investigated and determined if they are valid. If invalid negative elevations are detected, feedback is provided to the editing group who will decide on an appropriate course of action, which may include rework, new tools, or re-training. Final Report Check # 2 Block Edge Check Depending on the acquisition order, adjacent blocks may be flown at the same time or acquired at different times, depending on corporate priorities. Therefore, even when adjacent blocks are acquired at different times, perhaps months apart, block edges must match to ensure the data is seamless from block to block. Automatic processes in the production workflow ensure edge matching occurs however there is one exception to this rule. Temporal change is a factor since varying acquisition dates may affect edited features such as agricultural features and water elevations that reside along the edge of the block. In the editing workflow, temporal changes are identified by the quality control group and are accepted by the edge matching process. To ensure quality of this process, IV&V will difference tiles in a block with all previously acquired adjacent data. In a final report analysis, difference masks created in ESRI s ArcGIS return the difference in elevation for the single overlapping row of pixels that the edge tiles share. If the result of the difference calculation is 0, then the edges are identical. If the result is +/- 0 then the edges are mismatched and not seamless as a result. This mismatch

4 should be caused by temporal change, but if temporal change is not the cause, this leads to more investigation of the edit process concerning the finished data. Final Report Check # 3 Post-Edit Verification Check Point (VCP) Analysis Perhaps the most significant statistical study conducted by IV&V on a block by block basis is the post-edit VCP analysis. IV&V uses ground control points available from the National Geodetic Survey (NGS) as ground truth in order to compare post-edit DSM, DTM, as well as, NED10m, and NED30m data. NGS points taken from GPS sites only are downloaded in shapefile format. These points can be clipped to the boundaries of the data block and an analysis of that block is conducted. Using these NGS points, the elevations in the DSM, DTM, NED10m, and NED30m data are sampled in a software called Global Mapper, so the datasets can be compared to the NGS elevation value and subsequently to each other. Differences in the elevations (residual values) are individually calculated between each DSM, DTM, NED10m and NED30m block dataset with the elevation of the NGS point. Statistical blunders (residual values that are 3 times the standard deviation) are removed from the statistical summary and reviewed to determine the true cause of the blunder. There are cases in which the NGS point may not have been sampled on the ground (i.e. NGS point sampled on top of a monument), so IV&V uses geographic analysis techniques and radar interpretation skills to determine the reason for the removal of each blunder point from the report. The specification of vertical accuracy for NEXTMap data is 1m for the DSM and DTM. However, residual values may be affected by radar limitations such as aircraft motion, and other radar artifacts. Therefore, IV&V interactively reviews all points +/- 70cm in order to determine if the point should be included or excluded from the analysis. VCPs are removed from the analysis if they are not suitable to measure the accuracy of the DSM or DTM, and a comment indicating the reason for its omission is added to the VCP report. If the VCP point is located on a water feature, slopes greater than 10 degrees, a bridge, or the datasheet for the NGS value indicates the point is not at ground level, these points will be omitted from the analysis. There are also specific criteria for removing points in the DSM and DTM individually. For example, if a VCP is located on or near features such as vegetation or cultural objects in the DSM, the point may be thrown out of the analysis. Since the DSM is a first surface model, it is not a fair comparison to use the VCP as truth in this area when the elevation of that VCP is likely referencing the ground. There are also exclusion rules that apply to the DTM. Since the current editing software algorithm may not remove cultural or vegetation clusters greater than 100m, these features may still exist in the finished terrain product. As a result, points in these areas would be removed from the analysis. Additionally, to compare all terrain models by the same criteria, points that are omitted from the DTM are also omitted in the NED30m. After the points are interactively reviewed and residual values are included or excluded based on where they reside in a DSM or DTM, statistics are generated from the remainder of residual values for each product. The most important measures of vertical accuracy are the Root Mean Square Error (RMSE) and the 95 th Percentile statistics. These statistical calculations are important indicators of the degree of vertical accuracy in the finished data. Core product specifications for the DSM and DTM indicate the RMSE must be less than 1.0m and the 95 th Percentile less than 2.0m. If this specification is not met, further investigation would have to be conducted to determine if the data is out of specification and if rework is required. Final Report Check # 4 Ancillary Data Comparison between Intermap DSM/DTM & NED Since the DTM and NED10m datasets are most alike in terms of terrain product and resolution, difference masks are created to determine where extreme highs and lows may exist between the DTM and NED10m data. It is in these areas that differences can be identified and explored in greater detail. IV&V conducts analysis and provides a few examples in the final report of where such differences exist. This is done by generating profiles in the same location on the DSM, DTM, NED30m, and NED10m surfaces to see how the elevations differ from product to product. These differences will be investigated and included in the data comparison section of the final report. CASE STUDY OF IV&V COMPARATIVE DATA ANALYSIS The case study area is located in North Western Texas, or NEXTMap USA Texas block This block contains many lakes, reservoirs, as well as the metropolis of Dallas. The results presented in this independent analysis reflect the quality of DSM and DTM data for block 3189 after it has passed all stages of the production workflow process.

5 Figure 2. Case Study Area NEXTMap Texas Block 3189 Check for Minimum Elevations There are minute tiles in NEXTMap USA Texas block The analysis was done on both the DSM and DTM data, therefore 608 tiles were checked for invalid negative elevations. Of the 608 tiles in this block, 0 tiles contained invalid negative elevations. Block Edge Check All tiles with adjacent edges to previously delivered data in other blocks were inspected for consistency. Two adjacent completed blocks were Oklahoma block 3167 and Texas block In the analysis, 14 DSM and 14 DTM tiles were inspected for consistency along the edge of the Oklahoma block and 16 DSM and 16 DTM tiles were inspected for the adjacent Texas block. Adjacent edges shared an identical row of pixels, ensuring the data was seamless between block edges in both the DSM and DTM products. Verification Checkpoint (VCP) Analysis VCPs were selected from GPS points in the National Geodetic Survey (NGS) database. A total of 166 NGS points fell within the block extents. All points where the elevation difference between the Intermap DSM or DTM and the surveyed point was greater than 70cm were reviewed to determine the cause of the difference. In this review, 57 points were identified as being significantly influenced by first surface features and/or statistical blunders in the DSM and excluded in the statistical analysis. Accordingly, 49 points were excluded in the DTM. Points excluded in the DTM were excluded in the NED.

6 Table 2. VCP Results for NEXTMap Texas Block 3189 DSM DTM NED30m* NED10m** Total VCPs Blunders/FS Included Points MEAN MAX MIN STDEV RMSE Percentile Blunders (3x STDEV) *NED 30m purchased from Global Mapper 2004 **NED 10m downloaded from USGS seamless database Nov-05 ( For NEXTMap USA Texas Block 3189 the quality of vertical accuracy is verified by two primary statistics: the RMSE and the 95 th Percentile. Compared to the VCP from the National Geodetic Survey, the difference between the DSM is 0.50m higher with 95% of the residual elevation values less than 0.87m. When comparing the DTM with the NGS point, the DTM has an RMSE of 0.45m, while the NED30m and NED10m have an RMSE of 1.54m and 1.41m respectively. The 95 th percentile figure for the DTM is lower by more than two meters when comparing it to the NED30m data, and 1.1m lower than the NED10m data. In this analysis, IV&V has verified that the DSM and DTM have met stated core product specifications, and that the DTM has a better vertical accuracy result than the NED data, based on the available points used in the statistical analysis. Figure 3. VCP Residual Value Distribution - NEXTMap USA Texas Block 3189 Figure 3 shows the distribution of the residual VCP values for the DTM, NED30m, and NED10m datasets. The histogram shows a normal distribution for the DTM, while the NED30m and NED10m data show an increased number of out-lying residual values in both directions. The DSM was excluded from this figure, since the DTM, NED30m and NED10m are all terrain products, with the same VCP points used for analysis.

7 Ancillary Data Comparison The figure below shows the difference masks calculated using ArcGIS for the case study area. The difference masks were loaded into Global Mapper and inspected for extreme differences. The legend shows the difference in elevation in meters between the DTM and the NED10m data. Notice the amount of red areas in Figure 4 below. These areas are typically obstructed (vegetation and urbanized) or high sloping areas. DTM NED Figure 4. Comparing Edits in DTM & NED10m The example below was identified from the difference image as an area of interest, resulting in a detailed investigation. This mining operation appeared dark blue in the difference mask or more than -10m lower in the DTM than the NED10m data. To determine why the difference was significant, the area was investigated and profiles were created on the surfaces. Below is the Intermap ORI with a profile line that identifies the area of investigation. Figure 5. Comparing Edits in DTM & NED10m The graph below shows a cross-section of elevations along three surfaces, the DSM, DTM, and NED10m. The DSM and DTM show very similar cross-sections. The dips in the graph are displaying the mining operations that are taking place deep in the terrain. The NED10m data is showing an undulating line, without any evidence of

8 elevation dips. This indicates that the NED10m data was collected before the mining operation began, and therefore the reason elevation differences exist in this area has been strongly affected by the time at which the data was acquired a temporal change. Figure 6. Comparing Edits in DTM & NED10m The second example of comparison between the Intermap data and NED data is this reservoir in Figure 7. The profile of the reservoir shows that spike in the Intermap DSM and DTM data in the location of the reservoir s dam. The flat area on the other side of the dam is the reservoir elevation. The corresponding NED10m data does not show this feature. Figure 7. Comparing Reservoirs and Dams in DTM & NED Figure 8. Profile of Reservoir

9 SUMMARY OF NEXTMAP USA PROGRESS TO DATE Although Intermap has already mapped many areas of the world, including Indonesia, United Kingdom, Philippines, Jamaica, and Vanuatu, the most challenging under-taking is the current NEXTMap initiative the NEXTMap USA program. NEXTMap USA is due to be completed by the end of 2008 and will encompass a seamless, national digital elevation dataset for the continental USA and Hawaii that meets stated quality objectives. Figure 9. NEXTMap USA Data Coverage July 31 st, 2006 Flown 22.3% IP 19.7% Edit 18.1% IV&V S Current Progress in NEXTMap as of July 31 st, 2006 The table below shows IV&V s reporting progress by state in square kilometres. As a department IV&V has been successful in providing timely, quality reports, as over 1,200,000km 2 of completed data or ~ 17% of the NEXTMap USA project has been reviewed. Table 3. IV&V Report Progress by State July 31 st, 2006 STATE Area (km 2 ) Reviewed by IV&V % of State California 428, Florida 142, Mississippi 123, Hawaii 16, West Virginia 29, Texas Alabama 43, Louisiana 37, Arizona Oklahoma 48, Michigan 28, Nevada 25,

10 STATE Table 4. DSM Residual Statistics July 31st, 2006 # of VCPs USED (DSM) MIN MAX MEAN STANDARD DEVIATION Table 5. DTM Residual Statistics July 31st, 2006 RMSE Table 6. Best Available NED Residual Statistics July 31st, TH PERCENTILE California Florida Mississippi Hawaii West Virginia Texas Alabama Louisiana Arizona Oklahoma Michigan Nevada Average STATE # of VCPs USED (DTM) MIN MAX MEAN STANDARD DEVIATION RMSE 95 TH PERCENTILE California Florida Mississippi Hawaii West Virginia Texas Alabama Louisiana Arizona Oklahoma Michigan Nevada Average STATE # of VCPs USED (DTM) MIN MAX MEAN STANDARD DEVIATION RMSE 95 TH PERCENTILE California Florida Mississippi Hawaii West Virginia Texas Alabama Louisiana Arizona Oklahoma Michigan Nevada Average

11 SUMMARY OF IV&V STATISTICS TO DATE The statistics calculated for each state verify the 1m vertical accuracy stated in Intermap s Core Product Handbook. For each state, all RMSE values are less than 1.0m in the DSM and DTM, and the 95 th Percentile value is well within the 2.0m accuracy statement. Compared to the best available NED, on average the RMSE of the Intermap data is lower by 98cm and the 95 th percentile is lower by 2.02m. This indicates that not only has IV&V been able to verify the accuracy of Intermap data specifications, but that the data has a better vertical accuracy than the NED10m data, and better than stated accuracy specifications. FUTURE PROGRESS COMPLETING NEXTMap USA IV&V will review almost half of the United States by the end of 2007, and will complete 90% of NEXTMap USA analysis by end of 2008, early By completion, IV&V will have conducted an analysis on an area that extends 55,000 US quadrangle tiles which make up the NEXTMap USA program. This includes the continental USA & Hawaii, as well as the US/Mexican and the US/Canadian border regions. Figure 10. NEXTMap USA Plan Flown 100% IP 94% Edit 89% CONCLUSIONS & RECOMMENDATIONS The competitive geographic industry has encouraged both government and private sector companies to focus on the quality of digital elevation datasets. The IV&V Department within the Customer Care Division at Intermap operates to verify and validate internal workflow processes and product specifications, but more importantly provides feedback to company personnel from a customer perspective. IV&V presents assessments of NEXTMap USA data through a continuous system of feedback, that operates externally and independent of the core operational procedures. Findings about the data and ways to improve operational methods are two key results of all IV&V analysis. IV&V has served and will continue to serve an important purpose within Customer Care, and ultimately to the customer. The value in verifying and validating data beyond the essential, yet typical quality control procedures ensures a higher level of quality. With the expansion of Intermap offices, it is imperative to continue to evaluate that the processes and products are successful with changing employees

12 and new additions to editing departments. This leads to a confidence in the processes that guide production workflow and the quality of the elevation products. The statistics presented in this paper show that IV&V has been successful thus far at verifying that the data meets, but exceeds core product standards. The statistics completed on the data so far verify that Intermap not only meets the stated vertical accuracy specifications of an RMSE of less than 1m and a 95 th Percentile value of less than 2.0m. For the DSM, the average RMSE is 0.61m with a 95 th Percentile of 1.25m, and 0.64m RMSE and 1.33m 95 th Percentile for the DTM accordingly. Going forward, it would be beneficial for IV&V to conduct more validation techniques. Often, the data is used by clients for such applications as flood risk mapping, hydrology studies, defence studies, and transportation studies. In order to validate the data, IV&V plans to conduct analysis on the application of this data as the client would, to validate that not only the data but also the specification will meet the client s intended application needs. The existence of other elevation datasets such as NED10m, NED30m, and LIDAR data encourage Intermap to continually search for new methods to verify that the quality of the data meets stated specifications, and better suits the needs of client s applications. The post-edit VCP analysis is the best method that IV&V can use to verify the vertical accuracy of the Intermap s DSM and DTM data, although other methods of statistical comparison could be investigated. Historically, IV&V has been given the freedom to be creative and explore new methods of data verification and validation. This culture encourages employees to think of new ideas, develop new methods of data verification and validation, as well as work within several GIS packages to develop new skills and techniques. This philosophy allows IV&V to develop a discipline of customer loyalty and customer satisfaction within the Customer Care Division (Kellman, 2006). Due to this philosophy, IV&V will continue to prove that its existence as an independent quality management system is significant in the NEXTMap USA program. REFERENCES Richardson, B., X. Li and T. Hutt (2004). Design and Implementation of Intermap s Independent Verification and Validation (IV&V) Program. In: Proceedings of ASPRS Annual Conference, May 2005, Baltimore, MD, USA. Baxter, Robbie Kellman. 7 Ways to a Customer-Focused Company. American Management Association. July Intermap Technologies Product Handbook and Quick Start Guide v3.3. Intermap Technologies Inc., (2004).