Report on Performance of DEM Generation Technologies in Coastal Environments

Transcription

1 CRC for Spatial Information Report on Performance of DEM Generation Technologies in Coastal Environments Clive Fraser and Mehdi Ravanbaksh Cooperative Research Centre for Spatial Information CRCSI is established and supported under the Australian Government s Cooperative Research Centres Program

2 Table of Contents Page Executive Summary 3 1. Introduction 5 2. Project Overview 5 3. Overview of DEM Generation Technologies Technology Options Accuracy Considerations LiDAR Photogrammetry IfSAR 8 4. Project Work Plan 9 5. Test Area Locations Specifications for DEM Data Sets Benchmark Elevation Data Permanent Survey Marks GPS Survey of Height Profiles Comparison of GPS and Ground Survey Elevations GPS Heighting versus LiDAR DEMs Analysis of Different DEMs Against LiDAR Reference DEM Discrepancies in Elevation SRTM DEM SPOT5 DEM Topo DEM (from 1:25,000 map data) Airborne IfSAR DEM ADS40 DEM Impact of Land Cover on DEM Accuracy Urban areas Open Rural Areas Forest/Bushland Areas Mixed Coastal Land Cover Influence of Terrain Slope Conclusions 40 2

3 Executive Summary Reliable digital elevation models (DEMs) are vital to better understand and prepare for the impacts of sea level rise and storm surges caused by climate change. A number of satellite and airborne remote sensing technologies can be used to generate digital elevation models, however each technology possesses its own advantages and limitations. The primary aim of this project has been to evaluate the performance of different technologies for the generation of digital elevation models, specifically in coastal environments. The accuracy characteristics of six such technologies have been assessed within four test areas on the mid north coast of New South Wales. These test sites were chosen as being representative of low-lying coastal zones of differing land cover, topography and geomorphology. The DEM technologies investigated were: Airborne LiDAR (airborne laser scanning) Airborne IfSAR (interferometric synthetic aperture radar) SPOT5 HRS satellite imagery 1-second SRTM-based national DEM Aerial photography: o with the DEM sourced from existing 1:25,000 digital topographic mapping o with the DEM derived from recent ADS40 digital imagery An objective of the project was to look beyond differences in vertical resolution, cost and productivity, and to consider the overall performance of different DEMs in the context of fulfilling anticipated requirements for fit-for-purpose elevation data in Australia s vulnerable coastal zones. Outcomes of the project can be used to inform the development of future guidelines covering optimal DEM generation technologies for programs such as UDEM and the National Digital Elevation Framework (NEDF). Recent forecasts of sea level rise are in the range of 0.5m to upwards of 1m over the remainder of this century. Digital elevation modelling in support of prediction and monitoring of the inundation impacts of sea level rise and storm surges will therefore require vertical resolution at the sub-metre and even decimetre level. A principal finding of this project has been to reinforce the prevailing view that LiDAR is the optimal DEM generation technology for this application. While DEMs produced photogrammetrically from aerial imagery can match the vertical resolution of LiDAR, namely around 10cm, they are invariably more expensive and exhibit significant shortcomings in bare-earth elevation modelling if automated classification and filtering are solely relied upon. Beyond highlighting the recognized superiorities of LiDAR, this project has identified DEM characteristics that are perhaps not as widely appreciated, but are nevertheless important in the context of producing accurate bare-earth DEMs of coastal terrain. One of these concerns the accuracy gap between LiDAR DEMs and those derived from airborne IfSAR and aerial photography. Comparing LiDAR accuracy (10cm) to IfSAR and ADS40 accuracy (50-100cm), one would expect LiDAR DEMs to be at least 5 times better. However the difference are accentuated by shortcomings in the automated classification and filtering of both vegetation and, to a lesser extent, man-made structures, within the process of producing a bare-earth DEM from the latter two technologies. Multiple-return LiDAR on the other hand displays significant advantages by way of last-pulse ground definition, which cannot be 3

4 matched in densely vegetated areas by radar and photogrammetry techniques, except through skill-intensive and expensive manual editing processes. The results obtained for DEM performance in open areas, largely free of trees and buildings, highlighted the fact that distinctions in DEM accuracy are as much due to different terrain and land cover, and consequently to filtering, as to differences in the basic metric resolution of DEM technologies. In the case of open pasture, sub-metre accuracy was obtained for the SRTM DEM while the 1:25,000 mapping, the IfSAR DEM and the aerial imagery DEM all displayed sub-half metre accuracy. Although the accuracy of all these lower-resolution DEMs exceeded specifications, they are nevertheless still not likely to fulfill requirements for fit-for-purpose high-resolution elevation models for decision support and risk analysis associated with sea level rise. 4

5 1. Introduction This report summarises the objectives, work plan, conduct and outcomes of the project Performance of DEM Generation Technologies in Coastal Environments, which formed a research project under the Urban Digital Elevation Modelling in High Priority Regions Program(UDEM). The focus of the analyses required to evaluate the performance of different Digital Elevation Model (DEM) technologies has been upon detailed assessments of the heighting accuracy produced by six DEM generation technologies within four test areas representing typical low-lying Australian coastal environments, with land cover types including urban, rural and forest. Outcomes of the project will provide an increased understanding of the characteristics of different elevation data technologies and how well they perform in Australian coastal environments. Results will therefore inform the development of guidelines covering optimal DEM generation technologies for vulnerable coastal zones. Results will also be of benefit to producers of DEMs, particularly to those producing elevation models under the National Digital Elevation Framework (NEDF) and UDEM. 2. Project Overview The objective of this project has been to investigate the performance of different DEM generation technologies within a range of coastal environments. The quality of DEMs is a function not only of the data acquisition and subsequent data processing, but also of the characteristics of the terrain being mapped, especially in regard to topography and vegetation, and to the presence of cities and urban land cover. DEMs produced from different imaging and ranging sensors need to be analysed in order to better understand their characteristics and accuracy, and also their cost-benefit ratios in relation to producing fit-for-purpose elevation models for coastal assessments. 3. Overview of DEM Generation Technologies 3.1 Technology Options It was initially envisaged that the research would investigate the accuracy capabilities of four categories of DEMs/DEM generation technologies, namely: Airborne LiDAR (Light Detection and Ranging technology). The new national Australian mid-resolution SRTM 1 DEM (derived from the Shuttle Radar Topography Mission IfSAR data). The mid-resolution SPOT DEM (derived from approximately 5m resolution stereoscopic SPOT5 HRS satellite imagery). High-resolution airborne IfSAR. In addition to comparing DEMs, and in cases DSMs (digital surface models), generated from these four data sources, the work plan was extended to also encompass analysis of DEMs derived from the following sources: Photogrammetrically derived DEMs from digital aerial imagery, and specifically automatically produced DEMs from the Leica ADS40 3-line scanning system. DEM data from current 1:25,000 topographic mapping, the data having been generated over several decades, originally from manual stereo-compilation of analog 5

6 aerial imagery and subsequently from digitisation of the resulting contour maps. This elevation model is referred to throughout this report as the Topo DEM. Photogrammetrically derived DEMs from 3-line ALOS PRISM satellite imagery. IfSAR derived DEMS from the TerraSAR-X/TanDEM-X satellite radar system. DEM data from six of the above technologies was successfully sourced for the project. LiDAR, ADS40, airborne IfSAR and Topo DEM data was made available by LPMA (NSW Dept. of Lands), and Geoscience Australia supplied the SPOT5 and SRTM DEMs. The project team was unable to source both ALOS PRISM and TanDEM-X DEM data. In the case of tandem-x, the system is still within its initial commissioning phase, with commercial operations not anticipated to commence for several months. The ALOS satellite unfortunately ceased to operate in late April, 2011, which lessened the imperative to examine this DEM data source. In the absence of PRISM and TanDEM-X data, it is still possible to infer to some degree the overall performance of these two technologies from the results obtained for the SPOT5 HRS and airborne IfSAR DEMs, respectively. However, in the IfSAR case, TanDEM-X is anticipated to produce vertical accuracies at the 2m level as opposed to the 0.5-1m expected accuracy for airborne IfSAR DEMs. Also ALOS PRISM DEMs should display higher overall accuracy than those from SPOT5 HRS, since PRISM has double the spatial resolution and 3- line scanner geometry. Prior to describing the methodology and workflow of the project, it is useful to recall the accuracy associated with each DEM technology and salient characteristics of the DEM sources considered. These are briefly summarized in the following sections. 3.2 Accuracy Considerations Associated with each DEM technology is an accuracy specification. This is generally expressed as a bound, since DEM accuracy is a function of both sensor and topographic/land cover characteristics, and in the case of LiDAR and photogrammetry it can vary according to project design requirements. More will be said in the following paragraphs about the accuracy specifications for each of the DEM technologies considered in this investigation, but it is initially useful to appreciate the range of accuracy anticipated, this range being shown in Figure 1. The figure shows representative 1-sigma accuracy bounds (68% confidence level) for each of the six DEM technologies. The bounds shown are indicative only, and their purpose is to highlight the order of magnitude and more difference between the representative 15cm vertical accuracy of LiDAR and the 8-9m accuracy of the SPOT5 HRS and SRTM DEMs. The considerable variation in vertical resolution needs to be kept in mind when comparing the merits of different DEM generation options. Another important aspect related to DEM quality is the presence or absence of height bias, which can be local, for example as a consequence of incomplete filtering of above-ground features in the DSM-to-DEM conversion process, or large-area, as a consequence of systematic errors in sensor positioning and orientation. Through reference to Figure 1 the reader can visualize that whereas a DEM may have high precision, of let us say a standard error (1-sigma value) of +/- 15cm, it may be inaccurate by the extent of the bias, which is indicated by the dashed line in the figure. 6

7 Figure 1. Representative 1-sigma vertical accuracy bounds for DEM technologies. 3.3 LiDAR Airborne laser scanning or LiDAR is today the clear technology of choice for the generation of high-resolution DEMs with post spacings of of 1-3m. The advantages of LiDAR centre upon its relatively high-accuracy of generally 10-15cm in height and around 1/2000 th of the flying height in the horizontal, and upon the very high mass point density of nowadays around 4 points/m 2. This high point density greatly assists in filtering out non-ground artefacts in the conversion from the directly acquired DSM to the final bare-earth DEM. Moreover, LiDAR has high productivity of around 300 km 2 of coverage per hour, and it can be operated locally, day or night. In practice, data acquisition is generally confined to daylight hours since most LiDAR units nowadays come with dedicated digital cameras (usually medium format), the resulting imagery being used both to assist in the artefact removal process and for orthoimage production. One of the most significant attributes of LiDAR is multiple-return sensing, where the first return of a pulse indicates the highest point encountered and the last the lowest point. There may also be mid pulse returns. As a consequence, LIDAR has the ability to see through all but thick vegetation. Whereas it might not be certain from where in the canopy the first pulse was reflected, it can be safely assumed that a good number of the last returns will be from bare earth. This greatly simplifies the DSM-to-DEM conversion process in vegetated areas. Whereas aerial photogrammetry techniques can yield DSMs of vertical accuracy equivalent to LiDAR, it is generally not economical to opt for photogrammetry over LiDAR for DEM generation at vertical resolutions of 10-20cm. Thus, in the context of UDEM, LiDAR stands alone in most practical respects as the most accurate and comprehensive means to produce highest resolution DEMs of coastal environments. For this reason, LiDAR has been chosen in 7

8 this project as the standard against which the other DEM generation technologies are compared. In order to quantify the accuracy of LiDAR against ground-truth, elevation data from a kinematic GPS survey of several thousand points has been used, along with data from permanent survey marks. 3.4 Photogrammetry As a tool for topographic mapping, photogrammetry has a long history. Traditionally elevation data was extracted from stereo aerial photography in the form of contours, as exemplified in this project by the Topo DEM, which was obtained from 1:25,000 topographic map data. DSM generation was automated with the advent of analytical stereoplotters and then further process automation accompanied the introduction of digital aerial imagery. The generation of a DSM from digital aerial or satellite imagery is today a fully automatic batch process, with the resulting elevation model often being employed to support orthoimage generation. Broad area DEM generation via photogrammetry is presently not the preferred approach, except in special circumstances such as very high accuracy DSMs for 3D city modelling. The latter is exemplified to some extent by current programs to create high definition, photorealistic models of major cities. For example, one approach employs the Vexcel Ultracam digital camera flown in a block configuration of 80% forward overlap and 60% side overlap at an imaging scale that yields a 15cm ground sample distance (GSD). DSM and subsequently DEMs to around 30cm vertical and 2-3m horizontal resolution can be generated with a high degree of automation through such a process. For the present project, DEM data generated from 50 cm GSD imagery recorded by LPMA s Leica ADS40 line scanning camera to a nominal vertical resolution of 0.5-1m has been adopted as representative of the capabilities of fully automated DSM production from digital aerial imagery, followed up with initial stage automated DSM-to-DEM conversion. However, it is noteworthy that the final stage, manually intensive classification and filtering was not carried out, and thus the ADS40 data should be thought of as constituting a bare-earth DEM over open terrain, and to some extent in urban areas, but only as a smoothed DSM for land cover comprising dense vegetation. Satellite imaging systems have gained popularity for DSM generation at vertical resolutions within the range of 1m to 10m. For example, the GeoEye-1 and World View-1 and -2 satellites have a 50cm GSD, which will support DSM extraction to around 1-2m vertical accuracy. Also, the dedicated DEM generation program of SPOT Image, which uses the SPOT 5 HRS system, yields DEMs with a nominal 5-10m height accuracy (1-sigma) and 20-30m horizontal resolution. All satellite imaging systems used for 3D terrain modelling use line scanner technology, with the 2.5m resolution ALOS PRISM satellite having a 3-line scanner geometry similar to that in the ADS40 aerial camera. DEMs produced from ALOS PRISM can be expected to display height accuracies of around 3-5m. 3.5 IfSAR Synthetic Aperture Radar (SAR) has been employed for a few decades as an imaging technology in remote sensing. Through an augmentation of a conventional airborne or spaceborne SAR system with a second receiving antenna, spatially separated from the first, it has been possible to utilise the principles of interferometry to extend SAR from a 2D imaging system to a 3D topographic modelling technology. The resulting Interferometric Synthetic Aperture Radar (IfSAR) system determines the relative heights of imaged ground points as a 8

9 function of the phase difference of the coherently combined signals received at the two antennas. The first commercial IfSAR system for DEM generation, the Intermap STAR 3i system, appeared in the mid 1990s and global focus was brought onto the capabilities of IfSAR to produce DEMs with the successful completion of the Shuttle Radar Topography Mission (SRTM) in There have been a number of refinements made to the SRTM DEM of Australia over the past few years, to the point where an updated DTED 2 DEM with a post spacing of 1 second (30m) and a nominal vertical accuracy in the range of 6-12m has recently been released. This new SRTM DEM data was accessed for the current project from Geoscience Australia. Beyond the heavily built-up areas of major cities and very rough mountainous areas, the Australian terrain can be characterized as being ideal for DEM generation via airborne radar. Intermap Technologies have recently completed a large project within the Murray Darling Basin with their STAR system and produced DSMs with a stated 0.5-1m vertical accuracy, and a post spacing of 5m. Moreover, use of stereo radar imagery as a complement to the process allows for semi-automated DSM-to-DEM conversion. Airborne IfSAR can record data at the very rapid rate of around km 2 per minute, which is some times the area acquisition rate of LiDAR (the IfSAR swath width is generally 8-20km). Moreover, data collection in not impeded by clouds. Over the past two or three years, there has been a considerable upsurge in 1m-accurate DEM generation via IfSAR, with national DEMs being commercially available through Intermap s Nextmap product line. A second source of radar DEMs is single-pass spaceborne IfSAR. Under the TanDEM-X program of Germany s DLR and the Infoterra company, the current TerraSAR-X satellite has been joined in space by a second X-band SAR unit. With the orbits of the two satellites being tightly controlled, single-pass IfSAR operation is possible, as is vegetation removal using new techniques for polarmetric radar interferometry. The intended elevation model product from TanDEM-X is a global DTED3 DEM of 12m post spacing and 2m vertical accuracy. As at late May 2011, full commercial operation of TanDEM-X had not commenced, though initial results from the system are reported as being very encouraging. From the standpoint of a DEM generation system that can economically provide 2m accuracy elevation models of Australia to horizontal resolutions of 10m, TanDEM-X has considerable potential. 4. Project Work Plan Shown in Figure 2 is the workflow designed for the DEM analysis. Given that a main focus of this analysis is upon DEM accuracy, it is useful to keep in mind that the DEMs being compared have accuracy ranges that differ by more than an order of magnitude. Recall that nominal vertical resolutions of the DEMS are: approximately 5-15m for SRTM and SPOT5 HRS, 3-5m for the 1:25000 Topo DEM (referred to here as the Topo DEM), 0.5 1m for airborne IfSAR and the ADS40 DEM, and 15cm for LiDAR. Thus, the principal aim of the analysis to be conducted is to better characterize the performance of these different DEM technologies within a typical Australian coastal environment, rather than to reinforce wellrecognised differences in resolution and accuracy. Also shown in Figure 2 is the work flow adopted for the production of the reference LiDAR DEM from the measured mass points. Automated classification and filtering was based upon analysis of the multiple-pulse returns, after which interpolation was adopted to generate the final grid of 2m horizontal spacing. 9

10 As can be seen from Figure 2, the initial step in the accuracy assessment and analysis of differently sourced DEMs of varying resolution against the reference dataset, which is taken to be LiDAR data, involves bringing all DEM datasets into a uniform reference coordinate system. Especially important is uniformity within the height datum. All current DEM data acquisition technologies utilize GPS for absolute positioning and consequently the DEM datum is initially referenced to the WGS84 ellipsoid. A height conversion from ellipsoidal to orthometric is then carried out using both geoid height information from AusGEOID09 and, where applicable and if known, the local relationship between AusGEOID09 and the Australian Height Datum (AHD) to facilitate a transformation of the DEM to AHD. In the conversion of height data recorded in the kinematic GPS survey conducted as part of the project, AusGEOID09 was employed to facilitate a one-step WGS84-to-AHD reference datum conversion. It is noteworthy that there can be discrepancies in actual local MSL and AHD amounting to 70cm or more as a consequence of sea-surface topography. However, localized distortions in AHD will have no significant impact in the accuracy analysis for two reasons. Firstly, height differences are being determined, which nullifies the effect of absolute biases in the datum, at least when all DEMs are nominally referenced to AHD. Secondly, the anticipated localized MSL versus AHD biases can be anticipated to be very small in relation to the overall error budget for all DEM data other than the LiDAR reference data. Figure 2. Project workflow. In order to compare height values from different DEMs at specific positions, interpolation is needed because of the multiple horizontal resolutions (post spacings) involved. Within the current project the principle adopted is that the interpolation should occur in the higher 10

11 resolution DEM. Thus, height comparisons require interpolation within the 2m horizontal resolution LiDAR DEM, this interpolation being bilinear as opposed to bicubic, in order to minimize smoothing effects. A result of this approach is that the number of sample points will vary proportionally to the horizontal resolution of the DEM being compared to the LiDAR reference data. In accordance with the different height resolutions of the DEMs being considered, different levels of initial artefact removal and filtering have been applied in the DSM-to-DEM conversion, with automated processes alone being largely relied upon. It is important to keep in mind that the characteristics of both the underlying terrain and the particular sensor technology will dictate the degree of complexity of the DSM-to-DEM process. Issues include, for example, the fact that photogrammetry techniques beneficially support manual artefact removal in a visual 3D environment, whereas removal of above ground features in LiDAR DSMs is greatly aided by both the high density and vertical resolution of the mass points and the provision of multiple returns (ranges) which allow penetration of the vegetation layer. Also, IfSAR DSMs can be accompanied by intensity images that support stereo visual interpretation to aid in the DSM-to-DEM conversion. 5. Test Area Locations Two criteria governed selection of geographic location for the DEM analysis: 1) suitability in the context of overall assessment of coastal zone vulnerability to climate change; and 2) availability of elevation model coverage from as many data acquisition sources as possible. Fulfilment of the latter criterion turned out to be the factor that most influenced the selection of test area locations, since the choice was essentially limited to the mid north coast of NSW, where there had been recent production of medium- and high-resolution DEMs from airborne IfSAR, LiDAR and photogrammetry (from ADS40 aerial imagery). Moreover, there was coverage from SRTM, 1:25000 topographic mapping and SPOT5 HRS. Following the selection of the general test area based on data availability, it was necessary to select specific test sites, which in combination fulfilled the following requirements: 1) Coastal zone with mixed vegetation, ranging from grassland to scrubland and forest. 2) Topographic variation, ranging from floodplains, to undulating low-level coastal sand dunes to low- and medium level hills. 3) Variation in landcover, from urban to rural to bushland and forests. 4) Containing extensive areas below 10m elevation and open to the coastline. A principal aim of the project was to assess the influence of both man-made structures in an urban environment, and different land and vegetation cover, on the accuracy and integrity of bare-earth DEMs. Although there have been a number of published reports on the performance of different DEM generation techniques in different topography, especially as a function of ground slope, this factor has only been briefly analysed here. The reasons for this are, firstly, that by its very nature the vulnerable coastal zone is low-lying, with only mild topographic variation; and, secondly, the metadata necessary to comprehensively consider slope and aspect for the IfSAR, ADS40 and LiDAR were not available. The analysis was thus limited to gridded DEM data only. Shown in Figure 3 are the four selected test areas: Area 1 (128 km 2 ) extends from South West Rocks to the Stuarts Point/Grassy Head area and comprises varied coastal topography and 11

12 vegetation cover. Area 2 (76 km 2 ), which is centred on the town of Kempsey, constitutes the sample low-lying urban area. Area 3 (24 km 2 ) covers Crescent Head and this was selected based on the varying terrain of the headland. Area 4 (72 km 2 ) was added to the initial three in order to provide further coverage of dense coastal forest areas, as well as an additional urban area, namely the settlement of Scotts Head. The DEMs within each of the test areas are shown in Figure 4, and Figure 5 highlights the areas below 10m elevation within each of the four test sites. Area 4 Area 1 Area 2 10 km Area 3 Figure 3: Test areas, with locations shown for 9 permanent survey marks used as GPS checkpoints. 12

13 (a) Area 1 (b) Area 2 (c) Area 3 (d) Area 4 Figure 4: LiDAR DEMs for each test area. 13

14 (a) Area 1 (b) Area 2 (c) Area 3 (d) Area 4 Figure 5: Areas below 10m elevation (black areas are >10m or outside area). 14

15 6. Specifications of DEM Datasets Shown in Table 1 is a summary of the specification for the different DEM datasets employed in the project. With the exception of the 1-second SRTM and SPOT5 data, which were made available by Geoscience Australia (GA), all DEM data was kindly provided to the project by the Land and Property Management Authority (LPMA) of the NSW Department of Lands. The project is indebted to LPMA and GA for this support, which was crucial to realization of the project objectives. Table 1. Specifications of DEM Datasets and GPS survey data. Dataset Technology Data format SPOT5 DEM ADS40 DSM Airborne IfSAR DEM SRTM DEM LiDAR mass points Topo DEM Space photogrammetry Aerial photogrammetry (50 cm GSD) Intermap STAR 3 & 4 IfSAR Space-borne IfSAR Airborne Laser Scanning Aerial photogrammetry, 1:25000 mapping ESRI binary 30m grid (.ADF) ERDAS 8 m grid (.IMG) 5 m grid (.BIL) ESRI binary 30m grid (.ADF) 2m grid (.LAS) ERDAS 25m Grid (.IMG) Horizontal accuracy (RMSE xy ) Vertical accuracy (RMSE z ) 10m 5-10m 0.5m 0.5-1m 1.5m 0.5-1m 7m 6-12m 0.3m 0.15m Ground check points Kinematic GPS ASCII 0.03m 0.03m 6m 3m 7. Benchmark Elevation Data 7.1 Permanent Survey Marks The reference elevation model against which DEMs from different data sources are compared is taken as the LiDAR DEM. In order to assess the quality of the LiDAR standard against ground survey data that is directly referenced to AHD to a nominal accuracy of better than 10cm, surveyed benchmark data within the test area was accessed. The elevations of nine benchmarks, the locations of which are indicated in Figure 3, were used in a comparison of GPS-derived AHD heights versus those of the permanent survey marks. It had originally been intended to employ additional benchmarks as ground checkpoints, however time constraints and difficulties imposed in locating the permanent survey marks beyond township areas meant that the number of checkpoints was restricted to nine. This number would be sufficient to indicate the presence of any localized biases in the AHD reference system that were not modeled via the AusGEOID09 Geoid model. 15

16 7.2 GPS Survey of Height Profiles Prior to the adoption of airborne LiDAR as the highest accuracy master elevation data set against which other DEM generation technologies are compared, it was necessary to validate the absolute accuracy of the LiDAR DEM. This is by no means a simple matter in practise, since the only available basis for comparison is elevation data acquired from ground surveys, either via GPS or standard surveying techniques of spirit or trigonometric levelling. As will be explained in a following section, there are practical limitations to utilization of thinly distributed benchmark and permanent survey mark data as an accurate base against which to assess LiDAR DEMs. Not only are there uncertainties of several cm in the height relationship between the ellipsoidal WGS84 and AHD reference systems, but there is also the inherent accuracy limitation, again several cm, of the ground surveyed elevations. The only feasible approach for assessing the absolute accuracy of LiDAR DEM data covering the UDEM test areas is through the provision of GPS surveyed bare-earth elevations. The most practical way of acquiring such data is through the use of real-time kinematic GPS (RTKGPS) surveying where a GPS receiver is mounted in a vehicle and 3D positions to an accuracy of a few cm are determined through the use of either a nearby radio-linked base station or a CORS network. For the present project, RTKGPS surveys were conducted in five areas: Scotts Head, Stuarts Point, South West Rocks, Kempsey and Crescent Head. The surveyed height profiles were mainly restricted to areas in or near townships, for two reasons. Firstly, the mode of operation was to utilize a base station that broadcast corrections to the vehicle-borne roving receiver via a radio link, and the effective maximum distance for radio reception was about 4km depending upon topography. Secondly, beyond townships, roads tended to be covered by overhanging trees, which blocked reception of the GPS signals. This accounts for most of the broken height profiles shown in Figures Notwithstanding these shortcomings, some 27,000 elevation readings at generally 3-5m intervals were made to 2-4cm accuracy over the roads indicated in the figures. One benefit of being restricted to open roadways was that heights to the same points would have been readily recorded within the LiDAR survey. An illustration of the problems posed by vegetation in the RTKGPS surveys is provided in Figure 12, which shows favourable and unfavourable areas for data collection within the Stuarts Point area. The vegetation cover is indicative of most of the native forest areas within the region, with only the low-lying test area around Kempsey (Area 2) being largely free of forest cover. 7.3 Comparison of GPS and Ground Survey Elevations In order to ascertain the absolute accuracy of the LiDAR DEM data, comparisons were to be made with the elevation data recorded within the vehicle-borne Real-Time Kinematic GPS (RTKGPS) Survey. Both technologies yield elevations, in the first instance, within an ellipsoidal height reference system, namely WGS84. In this sense, discrepancies between the RTKGPS heights and those determined from LiDAR yield an indication of the accuracy of the LiDAR system free of the effects of uncertainty in the relationship between the ellipsoidal and the orthometric height datums. For the conversion of both LiDAR and GPS surveyed heights to elevations referenced to AHD, it is necessary to apply a geoid correction, in this case via the AUSGeoid09 correction model. GPS surveyed AHD heights can then be directly compared to elevations of benchmarks (BMs) and permanent survey marks (PMs), which have traditionally been established via spirit leveling. 16

17 Scotts Head Stuarts Point SWRocks Kempsey Crescent Head Figure 6. Elevation profiles recorded by real-time kinematic GPS in the four test areas. Figure 7. Elevation profiles recorded by kinematic GPS in Scotts Head (Area 4). 17

18 Figure 8. Elevation profiles recorded by kinematic GPS in Stuarts Point (Area 4). Figure 9. Elevation profiles recorded by kinematic GPS in South West Rocks (Area 1). Figure 10. Elevation profiles recorded by kinematic GPS in Kempsey (Area 2). 18

19 Figure 11. Elevation profiles recorded by kinematic GPS in Crescent Head (Area 3). Figure 12. Constraints on kinematic GPS surveying: unfavourable vegetation conditions (left) and generally favourable conditions (right). A principal cause of discrepancies between GPS surveyed AHD heights and those for BMs/PMs can be anticipated to be localized biases in Geoid modeling. In the case of the test areas considered, the geoid correction value N varies by 1.1m over the 50km from Crescent Head to Scotts Head, from 30.7m to 31.8m, and by 0.4m over the 20km from Crescent Head to Kempsey. This fact, coupled with the anticipated accuracy (95% confidence) level of only 5-8cm for ground surveyed BM/PM elevations suggests that RMS discrepancies in the order of 10cm might well be expected between GPS and ground surveyed elevations. Table 2 lists the results of the GPS to BM/PM height data comparison for nine survey marks in the Kempsey, Scotts Head and Crescent Head areas. The overall RMS value of height discrepancies is 9.4cm and it is noteworthy that there is a systematic trend in the discrepancy values at two of the locations. These are indicative of either one or two factors: firstly, localized biases in AUSGeoid09 or, secondly, systematic errors in the BM/PM data. Either way, the results suggest that in order to independently ascertain the accuracy of the LiDAR data, it is more appropriate to use RTK GPS heights rather than benchmark data. 19

20 For the purposes of this study it suffices to note that the level of agreement between BM and PM data and RTKGPS is of a similar magnitude to the 1-sigma elevation accuracy anticipated from airborne LiDAR data, namely 10-15cm. Subsequent comparisons of the LiDAR DEM heights to ground surveyed data will utilize only RTKGPS data. Table 2. Comparison between GPS surveyed and published elevations for nine benchmarks/permanent survey marks (units are metres). GPS Point PM Ellipsoidal Height from GPS Geoid Separation Orthometric height ΔH from Levelling Height of PM from GPS True height of PM Error in Height (m) Kempsey GPS06 PM GPS10 PM GPS16 PM Scotts Head GPS10 PM GPS12 PM GPS13 PM Crescent Head GPS11 PM GPS14 PM GPS17 PM GPS Heighting versus LiDAR DEMs The LiDAR DEM has been adopted as the reference DEM in view of its significantly higher accuracy, and generally also resolution, as compared to the other DEM generation technologies. In order to validate, as far as was practical, the absolute accuracy of the LiDAR derived elevations, a comparison with the profiles of RTKGPS data described above was conducted. Within this process, some 27,000 individual RTKGPS height measurements were compared to elevations interpolated from the gridded LiDAR DEM via bilinear interpolation. The resulting discrepancies in elevation are summarized in Tables 3 and 4, where the heighting bias of LiDAR (-ve value indicates higher LiDAR elevation), the RMS discrepancy, the bias-free standard deviation of the discrepancies H and the size of the sample within each of the four test areas is listed. Table 3 shows results when all RTKGPS points are included, whereas Table 4 lists the corresponding results when height discrepancy values of greater than three times the standard deviation (ie 99% confidence level) of H values are omitted. In assessing the heighting discrepancies between the RTKGPS and corresponding points from the LiDAR DEM, it should be kept in mind that given the 2-3cm accuracy of the laser ranging component, and the fact that both data sets were transformed from ellipsoidal to orthometric heights via the AusGeoid09 geoid model, elevation differences will primarily be a function of: 20

21 discrepancies in the GPS surveying of platform positions, airborne and terrestrial; and errors in the filtering of the LiDAR data, ie in the removal of above bare-ground features. In many respects the comparison of elevations along roadways would be expected to yield optimal results, since the filtering issue is minimized. However, in the case of the test areas considered, there were instances were roadside vegetation appeared to influence localized filtering results. This issue will be addressed following a general summary of the results of the RTKGPS versus LiDAR comparison. Table 3. Comparison between RTK GPS surveyed elevations and those from the LiDAR DEM; all GPS points included (Units are metres). Test Area Mean elevation discrepancy (heighting bias) RMS elevation discrepancy Std. deviation of ΔH No. of points 1, Stuarts Pt , Kempsey , Crescent Hd , Scotts Hd Table 4. Comparison between RTK GPS surveyed elevations and those from the LiDAR DEM; GPS points where ΔH is greater than 3 times the standard deviation are omitted (Units are metres). Test Area Mean elevation discrepancy (heighting bias) RMS elevation discrepancy Std. deviation of ΔH No. of points % of points removed (ΔH >3 1, Stuarts Pt % 2, Kempsey % 3, Crescent Hd % 4, Scotts Hd % In the context of validating the LiDAR DEM via RTKGPS data, the results listed in Tables 3 and 4 are quite encouraging for two of the test areas, Areas 1 and 2, where neither the bias value nor the standard deviation of height discrepancies is significant given the 1-sigma accuracy of the LiDAR of around 15cm. However, the level of compatibility is less than expected within the remaining two areas, in Area 3 because of a higher than expected positive height bias for the LiDAR, and in Area 4 because of a high RMS discrepancy value. It is also noteworthy in Table 4 that, for the Stuarts Point and Kempsey test fields, only 1% of discrepancy values fell outside 3-sigma error bounds, which is consistent with a normal distribution. The corresponding figures for rejected points (ΔH >3 in Areas 3 and 4 are much higher at 11% and 8%, respectively. It is difficult to definitively establish the reasons for the larger mean LiDAR heighting bias in Crescent Head, though preliminary analysis suggests that it may in fact be due to a combination of both errors in the LiDAR DEM and lower than expected accuracy within the RTKGPS data. Shown in Figure 13 are plots of the positions of RTKGPS points, with the height discrepancy at each point being indicated by a coloured dot. White indicates within 1- standard deviation of ΔH (ie within 1-sigma), blue between 1- and 2-sigma, green between 2- and 3-sigma, and red greater than 3-sigma, the sigma values being those listed in Table 4. 21

22 Note in the upper two of the three images how GPS errors are suggested by distinctly different ΔH values being obtained in overlapping runs of the vehicle borne GPS survey. This is particularly apparent in the right-hand image covering a road roundabout. On the other hand, the lower image of Figure 13 shows systematic error in a double run along the edge of what is essentially a cliff face, and here one could infer that the heighting error is more likely to have arisen within the LiDAR processing. Figure 13. Sample discrepancies in LiDAR DEM versus RTKGPS elevation data, Crescent Head. White indicates within 1-sigma; blue, 1-2 sigma; green, 2-3 sigma; and red, >3 sigma. A further example of where the height discrepancies are more likely attributable to shortcomings in LiDAR classification and filtering is shown in Figure 14. Note how the discrepancies increase for a double-run RTKGPS survey exactly at the transition between an open urban area and a heavily forested area. The elevation cross section through the LiDAR DEM, at the position indicated by the yellow line, is also shown. A final example, which needs no explanation, is indicated by Figure 15. This shows the error arising when the vehicle borne GPS crosses a railway bridge, some 4-5m above the underlying DEM. It can be difficult to accurately attribute errors in the determination of absolute elevation to the LiDAR DEM versus the RTKGPS data. However, there is the consolation in this investigation that height discrepancies are of a sufficiently small magnitude where they are consistent overall with the 1-sigma vertical accuracy specification of around 15cm for the LiDAR DEM. Given that the next highest resolution DEM to be considered has a nominal vertical accuracy of 50cm, the LiDAR DEM can be safely taken as the benchmark against 22

23 which to assess the remaining DEM generation technologies. Notwithstanding the acceptance of this benchmark status, the RTKGPS versus LiDAR DEM analysis has highlighted practical issues that still hinder the acquisition of DEMs with vertical accuracies of better than, say, 10cm. This analysis has indicated that remaining shortcomings in the DSM-to- DEM conversion for LiDAR data are most apparent in the classification and filtering of vegetation as opposed to man-made, above-ground structures such as buildings. Figure 14. Sample discrepancies in LiDAR DEM versus RTKGPS elevation data, South West Rocks. White indicates within 1-sigma; and blue 1-2 sigma. Also shown is the cross section height profile corresponding to the yellow line. Figure 15. Discrepancies in LiDAR DEM versus RTKGPS elevation data at bridge crossing, Kempsey. White indicates within 1-sigma; and red greater than 3-sigma. 8. Analysis of Different DEMs against LiDAR Reference DEM 8.1 Discrepancies in Elevation Shown in Tables 5 and 6 are results from initial comparisons of DEMs against the LiDAR standard, for each different data acquisition technology investigated. The areas of comparison have been restricted to those indicated in Figure 5, ie to areas with an elevation of 10m or less, which are deemed most vulnerable to the impact of rising sea level and storm surges. The results represent an initial summary of overall accuracy in these regions, as quantified by both the Root Mean Square height discrepancy/error value (RMSE) and the estimated standard error ( h ), both being relative to the LiDAR DEM. The distinction between these two measures is that the RMSE includes the error arising from systematic height biases, whereas the h is free of the overall mean bias. Thus, h will always be equal 23

24 to or smaller than the RMSE, with the two estimates being equal when there is no mean height bias. Table 5. Accuracy evaluation result against LiDAR derived reference DEM. Only height differences below listed thresholds were included and those above removed, as per the %-removed column. Dataset SRTM DEM (Area 1, Threshold=15m) SRTM DEM (Area 2, Threshold=15m) SRTM DEM (Area 3, Threshold=15m) SRTM DEM (Area 4, Threshold=15m) SPOT5 DEM (Area 1, Threshold=15m) SPOT5 DEM (Area 2, Threshold=15m) SPOT5 DEM (Area 3, Threshold=15m) SPOT5 DEM (Area 4, Threshold=15m) Topo DEM (Area 1, Topo DEM (Area 2, Topo DEM (Area 3, Topo DEM (Area 4, IfSAR DEM (Area 1, IfSAR DEM (Area 2, IfSAR DEM (Area 3, IfSAR DEM (Area 4, ADS40 DSM (Area 1, ADS40 DSM (Area 2, ADS40 DSM (Area 3, ADS40 DSM (Area 4, Height bias (m) RMSE (m) h (m) Sample Size % removed The distinction between Tables 5 and 6 lies in the adopted threshold for classification of particular height discrepancy values as outliers, or gross errors. These are removed from the computation of the RMSE and standard deviation values. The outlier thresholds (cut-off values) in Table 6 impose a tighter tolerance on data acceptance than those of Table 5, and the different threshold values afford an indication of the extent of noise within each DEM data set. The cut-off height discrepancy values in Table 5 were set at 15m for SRTM and SPOT5 data, 10m for the Topo DEM and 5m for both the IfSAR and ADS40 DEMs. These values correspond roughly to multiples of three to five times the respective standard deviations. The area that was most noise-free, as expected, was Area 2 and the ADS40 DEM constituted the noisiest data. Some 40% of ADS40 data points in Area 4 were classed as 24

25 outliers, which is no doubt attributable to incomplete classification and filtering within forested areas. Table 6. Accuracy evaluation result against LiDAR derived reference DEM. Only height differences below listed thresholds were included and those above removed, as per the %-removed column. Dataset SRTM DEM (Area 1, SRTM DEM (Area 2, SRTM DEM (Area 3, SRTM DEM (Area 4, SPOT5 DEM (Area 1, SPOT5 DEM (Area 2, SPOT5 DEM (Area 3, SPOT5 DEM (Area 4, Topo DEM (Area 1, Topo DEM (Area 2, Topo DEM (Area 3, Topo DEM (Area 4, IfSAR DEM (Area 1, Threshold=3m) IfSAR DEM (Area 2, Threshold=3m) IfSAR DEM (Area 3, Threshold=3m) IfSAR DEM (Area 4, Threshold=3m) ADS40 DSM (Area 1, Threshold=3m) ADS40 DSM (Area 2, Threshold=3m) ADS40 DSM (Area 3, Threshold=3m) ADS40 DSM (Area 4, Threshold=3m) Height bias (m) RMSE (m) h (m) Sample Size % removed A feature to note is that due to the restriction of the analysis to elevations of less than 10m, there is limited initial consideration of DEM performance within urban environments, since most of the town of Kempsey, as well as significant parts Southwest Rocks, Crescent Head and Scotts Head, all lie at elevations above 10m. DEM performance in urban areas will be addressed in a later section of this report. The results in Tables 5 and 6, coupled with the plots in Figures showing height discrepancies above given thresholds, reveal a number of characteristics, some unique to particular DEM data acquisition technologies and others common to all. In the latter category, findings could be briefly summarizes as follows: 25

26 The accuracy associated with each DEM technology, as assessed via the RMSE and h values was basically consistent with or better than suggested by specifications. In the case of the SRTM data the RMSE values of around 2-4m were significantly lower than anticipated, whereas the standard error of the SPOT5 DEM displayed lower than expected standard error values of 2-3m, but a disturbing, persistent height bias of close to 5m. The accuracy of the Topo DEM was close to specifications, namely around 3m, whereas the IfSAR and ADS40 DEMs displayed an accuracy level in the range of 0.7m to 1.5m, which is equal to or slightly below expectations. As anticipated, both heighting biases and height RMSE values are generally larger for Areas 1, 3 and 4 than for Area 2. The lack of forest cover in the extensive open floodplain area around Kempsey accounts to a large degree for this characteristic, since the positive bias effect of the DEM being in reality more of a canopy DSM in forest areas is absent. This enhances the prospect for a better fit to the bare-earth LiDAR DEM. It can be seen that the bias and RMSE values follow this trend for the SRTM, IfSAR and ADS40 DEMs, but not for the SPOT5 and Topo DEMs. In the case of the SPOT5 DEM there is a relatively uniform bias of 4-5m across all three areas, with corresponding uniform RMSE values of m. Also as anticipated, the distribution of RMSE values and standard errors for each case are correlated to the presence or absence of forest. There should be an expectation that automated DSM-to-DEM conversion will yield better results for IfSAR versus photogrammetrically derived DEMs generated through image matching because of the ability of radar to penetrate vegetation, at least to a moderate extent. It is noteworthy that the mean biases for the SRTM and airborne IfSAR DEMs are 0.9m and 0.3m, respectively. In the case of the Topo DEM, where extensive manual filtering has been carried out, the systematic errors in DEM heights, although influenced by the presence of forest, tend to be concentrated in a small number of areas, as opposed to being distributed widely throughout forested regions. Based on results obtained in the foregoing analysis, as summarized in Tables 5 and 6, the following general summaries of DEM accuracy can be offered: 8.2 SRTM DEM When assessed against the basic accuracy specifications for the 1-second SRTM DEM, the achieved RMSE, standard error (1-sigma) and mean height bias values are very impressive. Instead of finding an RMSE in the range of 6-12m, the values instead range from 2.2m for Area 2 to 4.3m for the heavily forested Area 4. The corresponding 1-sigma values are 2.1m and 3.6m. The number of points with height discrepancies exceeding 10m (roughly 3-sigma) reaches 5.6% in the worst case (Area 4) and 0.3% in the best (Area 2). At a 15m or approx. 5- sigma threshold the number of rejected points falls below 1%. A further encouraging feature of the SRTM DEM, which can be seen for Areas 1, 3 and 4 in Figure 16, is that the distribution of height discrepancies exceeding the 10m threshold is characterized by concentrations in a few, mainly forested locations, with the majority of the area being free from rejected points. It is also noteworthy that there is a concentration of outlier points both within vegetated valley areas, which increase with increasing elevation, and along two watercourses. Heighting blunders exceeding 15m are confined to a small number of local vegetation clusters in Area 4. The main conclusion regarding the GA-supplied 1-second 26

27 SRTM DEM is that within the coastal areas considered it is more accurate than specifications would suggest, and it is free of significant height biases when assessed against RMSE values. Area 1 Area 2 Area 3 Area4 Figure 16: Points within the SRTM DEM with height discrepancies greater than threshold values when compared to the LiDAR reference DEM. Red areas representing a 10m threshold are overlaid by blue areas representing a 15m cutoff (areas not to scale). 27

28 8.3 SPOT5 DEM In the absence of height biases, DEMs generated from SPOT5 HRS imagery could be expected to show a standard error in elevation within the range of 5-10m. It is encouraging to see that with a point rejection threshold of 15m, the resulting standard errors for the SPOT5 DEM are 3m or just under in Areas 1, 3 and 4, and just below 2m in Area 2. This is well within specifications. Of concern, however, is the very significant height bias of over 4-5m in all four test areas, which results in RMSE values ranging from 4.7 to 6.0m. One can only speculate as to the cause of the systematic heighting error. For example, it is could arise in large part in this case from errors in the exterior orientation of the stereo satellite imagery, perhaps as a consequence of insufficient or inaccurate ground control within the block adjustment process. Alternatively, it might be attributable to shortcomings in the filtering of vegetation within the DSM-to-DEM conversion. The latter assumption is supported to some degree by the percentages of the rejected points where the height error exceeded a 15m threshold, there being over 8% in Area 1, 6% in Area 3, 10% in Area 4, and a predictably lower 0.4% in Area 2, which is largely devoid of forest cover. The rejections grow to greater than 10% in Areas 1 and 3, and to 18% in Area 4, when the threshold is reduced to 10m, with the distribution of the rejected points being shown in Figure 17. The rejected points are concentrated mainly in areas of dense coastal forest. Initial indications are that whereas the precision of relative heights is within specifications for SPOT5 data, the DEM exhibits degraded accuracy due to the presence of significant height biases, even in the absence of vegetation. 8.4 Topo DEM (from 1:25,000 map data) The vertical accuracy specification typically associated with 1:25,000 topographic mapping is 3m, corresponding to a third of the contour interval of 10m. Initial expectations for the Topo DEM would then be an RMSE at the 3m level, with localized occurrences of height biases as opposed to the area wide bias seen in the SPOT5 DEM. The mean height biases obtained for the Topo DEM, with a 10m removal threshold for height discrepancies, were 0.8m in Area 1, 2m in Area 2, 2.5m in Area 3 and 1.4m in Area 4. While the biases in Areas 2 and 3 are higher than one would anticipate for a 3m-accurate DEM, they are not viewed as significant given the corresponding 1-sigma values, which had a range of m. The number of points with height discrepancies greater than the 10m cutoff (nominal 3-sigma value) was 0.2% or less for all four areas. This is consistent with the expectation that the Topo DEM should have fewer filtering errors and thus fewer %-removals because of the map compilation process being based on manual stereoplotting from aerial photography. The higher %- removal values shown for a 5m cutoff in Table 6 can be discounted somewhat because the threshold is set too tight at only 2-sigma, but it is nevertheless interesting that the points removed are concentrated in localized, mainly forested areas, as shown in Figure Airborne IfSAR DEM With the relatively coarse rejection threshold value of 5m or approximately 5-sigma assigned to the airborne IfSAR DEM, resulting RMSE values were 1.4m in Area 1, 0.8m in Area 2, 1.1m in Area 3 and 1.5m in Area 4. The corresponding 1-sigma values were basically the same as a consequence of the modest bias values of 0.5m or less. Unlike the three lower resolution DEMs discussed above, the attained accuracy of the IfSAR DEM was not well within specifications. 28

29 Area 1 Area 2 Area 3 Area4 Figure 17: Points within the SPOT5 DEM with height discrepancies greater than threshold values when compared to the LiDAR reference DEM. Red areas representing a 10m threshold are overlaid by blue areas representing a 15m cutoff (areas not to scale). 29

30 Area 1 Area 2 Area 3 Area4 30

31 Figure 18: Points within the Topo DEM (1:25,000 map data) with height discrepancies greater than threshold values when compared to the LiDAR reference DEM. Red areas representing a 5m threshold are overlaid by blue areas representing a 10m cutoff (areas not to scale). Instead, the accuracy was generally consistent with expectations and even a little worse than anticipated. The accuracy indicators of RMSE and standard error changed marginally when the rejection threshold was lowered from 5m to 3m, and significantly more points were removed. The %-removal values climbed to 7% and 15% in Areas 1 and 4, respectively, and to 2% and 4%, respectively, for Areas 2 and 3. As can be seen in Figure 19, the regions with most rejected points correspond to hilly terrain with steeper slopes, and to a lesser extent to forested areas. Generally speaking, the results obtained with the IfSAR DEM were in accordance with accuracy expectations, with the technology performing best in low lying areas. 8.6 ADS40 DEM The DEM derived from ADS40 digital 3-line scanner aerial imagery was in fact a smoothed DSM that had undergone some initial automated classification and filtering. The first indication of the partial filtering of the ADS40 DSM is indicated in Figure 20, where it can be seen that the majority of the elevations within forested areas were rejected as outliers, their associated discrepancy values against the LiDAR data being greater than 5m or roughly 5-sigma. Some 40% of the height discrepancy values in Area 4 were rejected. The assumption that the RMSE values were inflated by an incomplete DSM-to-DEM conversion is reinforced by the results of the mostly forest free Area 2, where the RMSE value for the 5m threshold falls from the near 2m level of Areas 1, 3 and 4 to 1m, and the %-removal value drops from 30% or more to 4%. The height bias for Area 2 is also reduced to 0.3m from closer to 1m for the remaining areas. Given the incomplete filtering, it is difficult to characterize the accuracy of the ADS40 DEM (actually DSM), but it is encouraging to see results in Area 2 which are consistent with accuracy specifications, ie an RMSE value of less than 1m. 9. Impact of Land Cover on DEM Accuracy Based on the results obtained in the analysis of performance of the five DEMs against the LIDAR reference DEM, it is apparent that a significant factor limiting vertical accuracy in the generation of supposedly bare-earth DEMs is the automated classification and filtering in forest and urban areas, with vegetation cover appearing as a more significant issue than the presence of buildings and other man-made structures. In order to gain further insight into the impact of different land cover on the DEM technologies considered, analyses were carried out for samples of four specific land cover types: urban, forest/bushland, open farm land, and mixed coastal cover of vegetated dunes and housing. Once again, elevation bias, RMSE and standard error of height discrepancies were quantified using the LIDAR data as the reference DEM. 9.1 Urban Areas Figure 21 shows three sample urban areas: (a) a part of the coastal settlement of Scotts Head (taken from Test Area 4), (b) the commercial centre of South West Rocks (Area 1), and (c) a low-lying residential area of West Kempsey (Area 2). The results of the analysis for these three test sites are shown in Table 7, which has the same structure as the earlier Tables 5 and 6. 31

32 Area 1 Area 2 Area 3 Area4 Figure 19: Points within the airborne IfSAR DEM with height discrepancies greater than threshold values when compared to the LiDAR reference DEM. Red areas representing a 3m threshold are overlaid by blue areas representing a 5m cutoff (areas not to scale). 32

33 Area 1 Area 2 Area 3 Area4 Figure 20: Points within the ADS40 DEM with height discrepancies greater than threshold values when compared to the LiDAR reference DEM. Red areas representing a 3m threshold are overlaid by blue areas representing a 5m cutoff (areas not to scale). 33

34 (a) (c) Figure 21: Urban test areas, (a) Scotts Head, (b) South West Rocks and (c) West Kempsey. (b) Table 7. Accuracy evaluation result against LiDAR derived reference DEM for three Urban Test Areas. Only height differences below listed thresholds were included and those above removed, as per the %-removed column. Sample labels correspond with those in Figure 21. Dataset SRTM DEM (Area a, Threshold=15m) SRTM DEM (Area b, Threshold=15m) SRTM DEM (Area c, Threshold=15m) SPOT5 DEM (Area a, Threshold=15m) SPOT5 DEM (Area b, Threshold=15m) SPOT5 DEM (Area c, Threshold=15m) TopoDEM (Area a, Topo DEM (Area b, Topo DEM (Area c, IfSAR DEM (Area a, IfSAR DEM (Area b, IfSAR DEM (Area c, ADS40 DSM (Area a, ADS40 DSM (Area b, ADS40 DSM (Area c, Height bias (m) RMSE (m) h (m) Sample Size % removed

35 The first feature of note in Table 7 is that for the SRTM and SPOT5 DEMs, the bias value has increased over that listed in Tables 5 and 6. In the case of SRTM, it is safe to assume that this is attributable to an incomplete removal of buildings in the DSM-to-DEM conversion. The South West Rocks town centre, Figure 21b, is characterized by buildings taller than a single story and it is thus not unexpected to see a more significant bias being present. The bias value for SPOT5, at between 6m and 7m, is not at all consistent with a shortcoming in building classification and filtering. Instead, it is a gross positive height error likely attributable to a failure to utilize local ground control in the exterior orientation determination for the HRS imagery. Upon compensation for the bias, both SRTM and SPOT5 yield standard errors of height discrepancies in the range of 1.5m to 3m. The results achieved for the three urban areas for the Topo, IfSAR and ADS40 DEMs show an overall reduction in height bias, which is indicative of a more successful filtering of buildings in the automated DSM-to-DEM conversion. In terms of accuracy, the RMSE values obtained are largely consistent with those obtained in the full-area evaluations. 9.2 Open Rural Areas Figure 22 shows the three selected open rural area sites: (a) open grassland with thinly distributed houses and trees in West Kempsey, (b) open fields near Yarrahappini and (c) ploughed fields south of Stuarts Point. The first two areas are gently undulating, while the third is flat. The results of the analysis for these test sites are shown in Table 8, where it can be immediately seen that the DEM accuracy improves significantly when the need for extensive filtering is removed from the DSM-to-DEM transformation. (a) (c) Figure 22: Open rural test areas, (a) West Kempsey, (b) Yarrahappini and (c) Stuarts Point (b) Shortcomings in the DSM filtering required in the area shown in Figure 21a, which comprises a relatively small number of houses and trees, is enough to significantly inflate the RMSE value compared to that for the bare-ground areas of Figs. 21b and 21c, for all five 35

36 DEMs. In the open areas, the accuracy of SRTM, as expressed through the RMSE, is better than 1m, and the corresponding values for the IfSAR and ADS40 DEMs are between 0.4m and 0.8m. The absolute accuracy for all DEMs is within specifications for all three test sites. Table 8. Accuracy evaluation result against LiDAR derived reference DEM for three Open Rural Test Areas. Only height differences below listed thresholds were included and those above removed, as per the %-removed column. Sample labels correspond with those in Figure 22. Dataset SRTM DEM (Area a, SRTM DEM (Area b, SRTM DEM (Area c, SPOT5 DEM (Area a, SPOT5 DEM (Area b, SPOT5 DEM (Area c, Topo DEM (Area a, Topo DEM (Area b, Topo DEM (Area c, IfSAR DEM (Area a, Threshold=3m) IfSAR DEM (Area b, Threshold=3m) IfSAR DEM (Area c, Threshold=3m) ADS40 DSM (Area a, Threshold=3m) ADS40 DSM (Area b, Threshold=3m) ADS40 DSM (Area c, Threshold=3m) Height bias (m) RMSE (m) h (m) Sample Size % removed Given that the cultivated area shown in Figure 21c was likely bushland at the time the Topo DEM was produced, the probable reason for the bias figure of -2.3m is land clearing and subsequent earthworks to create the cultivated fields. Also exhibiting a large positive bias is, once again, the SPOT5 DEM. Given the largely insignificant height biases and RMSE values that are within specifications, it is not surprising to see so few points classified as outliers, with virtually all of these being found in the DEMs covering the scene with houses and trees. 9.3 Forest/Bushland Areas Figure 23 shows the three selected forest/bushland sites: (a) Dense tall (>10m) eucalypt forest at Yarrahappini, (b) Tall forest near Grassy Head and (c) scrubland covering a coastal dune at Stuarts Point, including an area of mangroves. The results of the analysis for these test sites are shown in Table 9. The table indicates a number of interesting features worthy of note. Firstly, in the heavily forested area, Figure 23a, the accuracy of the SPOT5 DEM is no better than 10m in absolute terms. Indeed, it can be seen that some 91% of the sample points are 36

37 rejected as outliers, meaning they are in error by more than 10m, the cause no doubt being a combination of the already referred to exterior orientation bias and an inadequate removal of vegetation from the DSM. (a) (b) (c) Figure 23: Forest/Bushland test areas (a) Yarrahappini, (b) Grassy Head Road, and (c) Stuarts Point Beach. Table 9. Accuracy evaluation result against LiDAR derived reference DEM for three forest/bushland areas. Only height differences below listed thresholds were included and those above removed, as per the %-removed column. Sample labels correspond with those in Figure 23. Dataset SRTM DEM ( Area a, Threshold=15m) SRTM DEM (Area b, Threshold=15m) SRTM DEM (Area c, Threshold=15m) SPOT5 DEM (Area a, Threshold=15m) SPOT5 DEM (Area b, Threshold=15m) SPOT5 DEM (Area c, Threshold=15m) Topo DEM (Area a, Topo DEM (Area b, Topo DEM (Area c, IfSAR DEM (Area a, IfSAR DEM (Area b, IfSAR DEM (Area c, ADS40 DSM (Area a, ADS40 DSM (Area b, ADS40 DSM (Area c, Height bias (m) RMSE (m) h (m) Sample Size % removed Another DEM showing a high bias value in thick forest was that from ADS40 imagery, though this was to be anticipated given the low level of filtering undertaken with this data. Contrasting to the poor accuracy of the SPOT5 and ADS40 DEMs is the result for the airborne IfSAR DEM in the same area, where agreement with the LIDAR DEM is 0.5m RMS. 37

38 Also noteworthy in Table 9 are the negative height biases of the SRTM DEM in the coastal scrubland and mangrove environment of Figure 23c and the Topo DEM in the tall bushland of Figure 23b. Neither systematic error is immediately explainable. Overall, the results for the forested areas are consistent with expectations, namely that the RMSE is higher than specifications for the DEM technologies would suggest, with the achievable accuracy being inversely proportional to vegetation density. 9.4 Mixed Coastal Land Cover The final land cover type sampled could be characterized as mixed coastal dunes, scrubland, bush and built-up urban area. The chosen test area shown in Figure 24 is representative of much of the low-lying coastal environment along Australia s eastern seaboard that is vulnerable to sea level rise and storm surges. Figure 24: Coastal area of mixed land cover, Arakoon. The results shown in Table 10 show largely the same characteristics as those presented in Table 5 for the full test areas. The RMSE of the SRTM elevations is a commendable 3m, with a modest bias, whereas the SPOT5 elevations show an RMSE of 6m, largely due to the now quite familiar large bias of also close to 6m. The Topo DEM also has a larger than expected bias given that the test area is right on the coast, with its RMSE value being marginally higher than expected. Both the ADS40 and IfSAR DEMs have small biases, though RMSE values which are outside specifications by approximately 0.7m. Given these results it is observed that the particular combination of land cover types does not reveal any distinctive performance characteristics which might not be apparent in the data covering the broader test areas. Table 10. Accuracy evaluation result against LiDAR derived reference DEM for coastal area of mixed land cover, as shown in Fig. 24. Only height differences below listed thresholds were included and those above removed, as per the %-removed column. Dataset Smoothed SRTM DEM (Sample 1, Threshold=15m) SPOT5 DEM (Sample 1, Threshold=15m) Topo DEM (Sample 1, IfSAR DEM (Sample 1, Smoothed ADS40 DSM (Sample 1, Height bias (m) RMSE (m) h (m) Sample Size % removed