A GIS BASED ACCURACY EVALUATION OF A HIGH RESOLUTION UNMANNED AERIAL VEHICLE DERIVED DIGITAL TERRAIN MODEL

Transcription

1 A GIS BASED ACCURACY EVALUATION OF A HIGH RESOLUTION UNMANNED AERIAL VEHICLE DERIVED DIGITAL TERRAIN MODEL A Research Project Presented to The Vancouver Island University GIS Graduate Program In Partial Fulfillment of the Degree Master of Geographic Information Systems By Jennifer Rolph August, 2017 Submitted to: The VIU Advisory Committee David Cake Michael Govorov Paul Zandbergen 1

2 ABSTRACT This project presents an evaluation of the accuracy and error distribution of a high resolution Unmanned Aerial Vehicle (UAV) derived Digital Terrain model (DTM). This exercise involved determining a flight plan to collect imagery from which to extract a georeferenced point cloud containing elevation values through digital photogrammetric processing, resulting in a Digital Surface Model (DSM). The DSM was refined to a DTM reflecting only bare earth values through an editing process of point classification. From the DTM, the surfaces of slope and aspect were derived and assessed using spatial statistics. The accuracy assessment of this spatial data presents a characterization of the error in the differences between the UAV derived elevation, slope and aspect surfaces, referenced to elevations of GNSS RTK Survey Points, as compared to a DTM of 2-meter resolution derived from digital photogrammetry based on large scale aerial survey (SCOOP Data, OMNR 2013). Statistical analysis included an examination of the distribution of error based on distribution plots, descriptive statistics and normalcy testing, reporting of Root Mean Square Error (RMSE), percentiles and confidence intervals, both with and without major outliers. Significance testing of results showed no significant difference in means or distributions between UAV and SCOOP derived DTMs, though the UAV DTM showed lower RMSE and percentile statistics at a higher frequency than the SCOOP DTM in stratified accuracy testing. Statistical analysis and accuracy reporting is stratified based on Non-Vegetated (NVA, Fundamental) and VVA (Vegetated, Supplementary) stratums, in adherence to ASPRS 2014 positional Accuracy Standards for Digital Geospatial Data. This study has shown how a lightweight UAV can contribute to terrain based studies for small geographic extents, by collecting high density elevation data with good accuracy. The UAV DTM is represented by 0.25 m NVA accuracy at 95% confidence, and 0.10 m accuracy based on 95% RMSE, the level at which the data exhibits a Normal Distribution. In the VVA the accuracy equates to 0.08 m accuracy based on the 90 th percentile statistic. ACKNOWLEDGMENTS The completion of this project comes with much gratitude for the knowledge and guidance provided by the VIU faculty and in particular my advisor, David Cake. I am also thankful to Ian Jeffry of the Ganaraska Region Conservation Authority for his assistance with the GNSS RTK Survey, and to Norm Lamothe of Deveron Resources for providing valuable knowledge about UAVs and their emerging use in resource management in the early stages of this project s development. 2

3 Table of Contents TITLE PAGE...1 ABSTRACT.. 2 ACKNOWLEDGEMENTS...2 LIST OF FIGURES...5 LIST OF TABLES..7 LIST OF ILLUSTRATIONS.. 7 LIST OF ABBREVIATIONS. 8 CHAPTER 1: INTRODUCTION 1.0: An Overview of Unmanned Aerial Systems : UAV Derived Digital Elevation Models : An Overview of Mapping Standards : An Overview of the Acquisition of Terrain Data : Problem Statement : Research Objectives.16 CHAPTER 2: LITERATURE REVIEW 2.0: Terrain Modelling : Remote Sensing Platforms and the Role of the UAV : Digital Photogrammetry and the UAV Based DEM : DTM Derivatives of Slope and Aspect : DTM Resolution and Accuracy : Statistical Analysis and Reporting of Terrain Data Accuracy.24 CHAPTER 3: METHODS 3.0: Study Area : Data Sources : Terrain Data Acquisition : UAV Flight : Ground Control Points : GNSS Survey : Slope and Aspect Field Survey : Digital Photogrammetry Workflow : Image processing and Bundle Block Adjustment : Dense Geometry Reconstruction and Inclusion of GCPs : DTM Construction in GIS Software : Statistical Analysis and Reporting 38 CHAPTER 4: RESULTS 4.0: Descriptive Statistics and Distributions of Error : Normal Testing : Regression Modelling : Accuracy Reporting : Statistical Significance Testing.46 3

4 CHAPTER 5: DISCUSSION 5.0: Statistical Analysis of Elevation Error Distribution : UAV NVA Elevation Errors UAV VVA Elevation Errors : SCOOP NVA Elevation Error : SCOOP VVA Elevation Error : Statistical Analysis of Slope Error Distribution : UAV NVA Slope Error : UAV VVA Slope Error : SCOOP NVA Slope Error : SCOOP VVA Slope Error : Statistical Analysis of Aspect Error Distribution : UAV NVA Aspect Error : UAV VVA Aspect Error : SCOOP NVA Aspect Error : SCOOP VVA Aspect Error..78 CHAPTER 6: CONCLUSION 6.0: Hypothesis Testing and Study Objectives : Potential Sources of Error : UAV Regulations in Canada and the U.S : Future Applications of UAV Based Terrain Modelling : Summary.87 CHAPTER 7: REFERENCES.89 4

5 LIST OF FIGURES Figure 1: Study Area, Hamilton Township, Ontario, Canada..26 Figure 2: GCP and Accuracy Checkpoint Layout..30 Figure 3: AgiSoft GCP RMSE Calculation 30 Figure 4: 3D Model showing locations of aerial photos and GCP s...33 Figure 5: 3D terrain model in Agisoft; Point Classification in AgiSoft..34 Figure 6: TIN derived from UAV Las data.35 Figure 7: Hillshade revealing problem areas over surface features remaining after initial Point Cloud Classification.36 Figure 8: UAV DTM shown in 3D from west-east (left) and east-west (right) 37 Figure 9: SCOOP and UAV DTMs overlaid with signed error magnitudes in meters at GNSS survey locations.47 Figure 10: SCOOP vs UAV Elevation Error (m) at GNSS survey points overlaid on SCOOP-UAV elevation error surface.48 Figure 11: UAV NVA Elevation Error Histogram and QQ Plot.49 Figure 12: UAV NVA Absolute Elevation Elevation Error Magnitudes and Outliers 50 Figure 13: 95% Histogram and Regression Distribution plots of true vs. UAV elevation values in the NVA Figure 14: UAV VVA Elevation Error Histogram and QQ Plot...52 Figure 15: UAV VVA Absolute Elevation Elevation Error Magnitudes and Outlier 52 Figure 16: 90% Histogram and Regression Distribution plots of true elevation values vs. UAV values in the VVA 53 Figure 17: SCOOP NVA Histogram and QQ plot...54 Figure 18: SCOOP NVA Elevation Elevation Error Magnitudes and Outliers..55 Figure 19: 85% Histogram and Regression Distribution Plots of true elevation values vs. SCOOP values in the NVA.55 Figure 20: Histogram and Normal QQ Plot of the SCOOP DTM VVA data 56 Figure 21: SCOOP VVA Absolute Elevation Elevation Error Magnitudes and Outlier 57 Figure 22: 90% Histogram and Regression Distribution plots of true elevation values vs SCOOP values in the VVA...57 Figure 23: UAV and SCOOP Slope Surfaces overlaid with slope under and over predictions at clinometer survey locations.59 Figure 24: SCOOP minus UAV Slope Error Surface overlaid with UAV vs SCOOP absolute slope error values (Degrees slope)..60 Figure 25: The high resolution 10 cm slope surface reveals error due to surface feature removal..61 Figure 26: UAV NVA Slope Error Histogram and Normal QQ Plot.61 Figure 27: UAV Slope Error Magnitudes (Degrees Slope) and Outliers.62 Figure 28: 95% Histogram and Regression Distribution Plots of true slope values vs. UAV slope values in the NVA..62 Figure 29: Histogram and Normal QQ Plot of UAV Slope Errors in the VVA..63 Figure 30: UAV Slope surface overlaid with slope error magnitudes and outliers in the VVA..64 Figure 31: 90% Histogram & Regression Distribution plots of true values vs. UAV slope values in the VVA..64 Figure 32: Histogram and Normal QQ Plot of SCOOP Slope Error Values (Degrees slope) in the NVA 65 Figure 33: SCOOP Slope Error Magnitudes and Outliers in the NVA 66 Figure 34: 95% Histogram and Regression Distribution Plots of clinometer vs SCOOP Slope in the NVA.66 Figure 35: Histogram and Normal QQ Plot of SCOOP slope error in the VVA 67 Figure 36: SCOOP Slope Error Magnitudes in the VVA.. 68 Figure 37: Regression modelling of clinometer slope values vs SCOOP values in the VVA...68 Figure 38: UAV vs SCOOP Absolute Aspect Error Magnitudes (Degrees Error)

6 Figure 39: SCOOP ASPECT vs UAV ASPECT Maps at 2 m Resolution...71 Figure 40: Histogram and Normal QQ Plot for UAV Aspect (Degrees Error) in the NVA...72 Figure 41: UAV NVA/VVA Aspect Error Magnitudes (degrees aspect)...73 Figure 42: 85% Histogram and Regression Distribution Plots of UAV Aspect vs. Compass Aspect in the NVA...73 Figure 43: Histogram and Normal QQ Plot of Aspect Error in the UAV VVA data...75 Figure 44: 90% Histogram and Regression Distribution Plots of UAV VVA Aspect Error...75 Figure 45: Histogram and Normal QQ Plot of SCOOP Aspect Error in the NVA...76 Figure 46: SCOOP Aspect Error Magnitudes and Outliers in the NVA...77 Figure 47: 85% Histogram of SCOOP NVA Aspect Data...77 Figure 48: Regression residual distribution plots and Spatial Autocorrelation report on trimmed residual data...78 Figure 49: Histogram and Normal QQ Plot of SCOOP VVA Aspect Error...79 Figure 50: Regression modelling of compass aspect vs. SCOOP aspect in the VVA...79 Figure 51: UAV Ortho Photo at 1: Figure 52: UAV Ortho Photo at 1: Figure 53: UAV Ortho Photo at 1:

7 LIST OF TABLES Table 1: Data Sources...28 Table 2: Descriptive Statistics of SCOOP and UAV DTM Elevation Error Datasets...40 Table 3: Descriptive Statistics of SCOOP and UAV DTM Slope Error Datasets...40 Table 4: Descriptive Statistics of SCOOP and UAV DTM Aspect Error Datasets...41 Table 5: Normalcy Testing P values of Full Residual Datasets...42 Table 6: Normalcy Testing P values of 95% Residual Datasets...42 Table 7: Normalcy Testing P values of 90% Residual Datasets...42 Table 8: Normalcy Testing P values of 85% Residual Datasets...43 Table 9: Ordinary Least Squares (OLS) Regression Modelling Results...44 Table 10: RMSE, Percentiles and 95% confidence Values for Elevation, Slope and Aspect Error...45 Table 11: T Tests and Wilcoxon Tests for Statistical Significance between Datasets...46 LIST OF ILLUSTRATIONS Illustration 1: UAV Configuration Types...9 Illustration 2: Example of Aspect Area Calculations...21 Illustration 3: PHANTOM 3 Advanced UAV...28 Illustration 4: 10 cm UAV Aspect Map Shows Micro typing of Aspect

8 LIST OF ABBREVIATIONS ASPRS: American Society of Photogrammetry and Remote Sensing ASTER: Advanced Space Bourne Thermal Emission and Reflection Radiometer CCD: Charged Coupled Device CHM: Canopy Height Model DEM: Digital Elevation Model DTM: Digital Terrain Model DSM: Digital Surface Model EXIF: Exchangeable Image File Format FAA: Federal Aviation Administration FGDC: Federal Geographic Data Committee FVA: Fundamental Vertical Accuracy GIS: Geographic Information System GLONASS: Global Navigational Satellite System; Russia GNSS: Global Navigational Satellite System GPS: Global Positioning System; American Ha: Hectares IDW: Inverse Distance Weighting IR: Infrared ISA: International Society of Arboriculture IMU: Internal Measurement Unit IMS: Internal Measurement System LAS: Log ASCII Standard LiDAR: Light Detection and Ranging LIO: Land Information Ontario NEEA: National enhanced Elevation Assessment NDEP: National Digital Elevation Program NDVI: Normalized Difference Vegetative Index NIR: Near Infrared NSSDA: National Standard for Spatial Data Accuracy NMAS: National Map Accuracy Standard NVA: Non-Vegetated Area PDOP: Percent Dilution of Precision PPK: Post Processed Kinematic RTK: Real Time Kinematic SCOOP: South-Central Ontario Orthophotography Project SD: Standard Deviation SfM: Structure from Motion SGM: Semi Global Matching SRTM: Shuttle Radar Topography Mission SVA: Supplemental Vertical Accuracy TLS: Terrestrial Laster Scanning UAV: Unmanned Aerial Vehicle UAS: Unmanned Aerial System USGS: United States Geological Survey VLOS: Visual Line of Sight VVA: Vegetated Area 8

9 CHAPTER 1 INTRODUCTION 1.0: An Overview of Unmanned Aerial Systems The use of Unmanned Aerial Systems (UAS) is becoming more common in many industries for a wide range of scientific, civil, industrial and recreational purposes. The Unmanned aerial system is composed of an aircraft, a Global Navigational Satellite System (GNSS) receiver, an Internal Measurement System (IMS), which records and coordinates pitch, roll and trajectory and tags photos with precise geographic locations, a stabilizing platform (gimbal) and a payload, the equipment that captures data. Payloads can be action, inferred, or thermal cameras, high precision barometers or multispectral, LiDAR, or hyperspectral sensors (Rock et al., 2011). Commonly known as drones, Unmanned Aerial Vehicles are abbreviated as UAV, and refer to the aircraft itself; when the aircraft is used as a data collection platform, the process of capturing the data and then correcting and processing it to generate a range of terrain products and maps, composes the entirety of the Unmanned Aerial System. While there are many variations of the vehicle configuration; UAVs are generally sorted as either fixed wing or multicopters (Illustration 1). Illustration 1: UAV Configuration Types (Source: There are many advantages linked to the use of UAVs as data collectors; they are cheaper to operate than larger airborne platforms, they pose minimal risk to human life and property, are easier to deploy and retrieve and provide a temporally current means of remote sensing; data can be collected with high repeatability and relatively low cost (Kung et al., 2011). Many types of UAV platforms exist, and the variability and range of models is growing quickly. Size and power of the UAV are generally the distinguishing factors among models; these two variables dictate the payload, operating altitude, and range (Anderson and Gaston, 2013). UAV models can be categorized in many different ways, for example as fixed wing or multicopters, or by range or payload capacity, or by size, which generally can be categorized as large and medium, small, mini, micro or Nano, or fixed wing or rotor (Anderson and Gaston, 2013). The cameras on most UAV s can achieve high ground sample distance (GSD) and resolution regardless of their platform. While Survey grade models will often conduct their own accuracy assessments, many of the consumer grade, lightweight models are not accompanied by any reference to positional point cloud 9

10 accuracy, or report only a generalized internal GPS accuracy of typically approximately 10 m. These accuracies can be greatly improved through the use of ground control, and in many cases accuracies approaching the centimetric range of survey grade models can be reached with ground calibration and point cloud editing (Kung et al., 2011). Flight time and altitude are often the parameters of interest when it comes to choosing a UAV model for a specific application. Resolution can be very high across most models, and accuracy can generally be acceptable to very high; however, smaller, lighter models are limited by sub hour flight times, shorter battery life, lower altitude ranges, and vulnerability to environmental conditions such as strong winds (Anderson and Gaston, 2013). Survey grade UAV s are becoming available which are capable of producing 2-3 cm location accuracy without the need for Ground Control Points (Sense Fly, 2016). These models integrate Real Time Kinematic (RTK) and Post Processing Kinematic (PPK) techniques to create terrain products of high accuracy and resolution without ground control requirements. This can be highly valuable in geographic areas that are difficult to access or navigate, or where time constraints limit the ability to place Ground Control Point s (GCP) in the field. RTK operations either require a stationary base station located at a known control point and depend on a dynamic link to the receiver on the UAV, or make use of a mobile antenna which receives real time corrections from a cellular network. In the PPK System, GNSS positional data is stored within the onboard system and camera locations are corrected in the post processing phase through corrections derived from network base station data. A number of studies have assessed the accuracy of UAV derived elevation models (Rock et al., 2011; Lucieer et al., 2013; Kung et al., 2011; Ritter 2014). These studies have found that the accuracy of a model created through image matching software using data collected by RTK-UAV can reach sub centimeter accuracy, approaching the accuracy of Total Station Survey and RTK GNSS Survey. An increasing number of studies are showing that UAVs can collect data of adequate to high accuracy for mapping, surveying and engineering applications at a lower cost, and lower turnaround time than other survey options, such as Light Detection and Ranging (LiDAR) or Semi Global Matching (SGM), a digital photogrammetric process based on fixed wing high altitude flights (Uysal et al., 2015), with the added benefits of very high point cloud density and high temporal currency and repeatability. As electronic instruments, such as GNSS receivers, microcomputers and sensor systems become smaller, their integration with UAVs is growing. These platforms are gaining popularity in the geological sciences for their ability to capture high resolution data though cost-effective means (Rock et al., 2011). Though there are studies documenting many aspects of UAV applications, the wide variability in platforms, sensors, and technical specs of different UAV s and the equipment they carry create gaps in which to uniquely assess certain characteristics and accuracies of this method of data collection. While RTK and PPK GNSS equipped UAVs are excellent solutions for commercial use where larger geographic areas need to be covered (typically up to about 8 km 2 /40-minute flight) and funding is sufficient to cover the cost of the aircraft, there is also strong potential to apply the UAV platform for scientific data collection for smaller geographic areas at lower costs, factors that are valuable to the small research group. Consumer grade multicopters are usually priced at a few thousand rather than tens of thousands of dollars, as compared to RTK and PPK survey models, and by combining the data collected from these smaller units with digital photogrammetric technologies, accuracies are being proven to approach those of larger airborne systems equipped with high end sensors and differential GNSS receivers, as well as LiDAR Systems (Kung et al., 2011). One of the challenges with using UAVs as tools for the collection of scientific data has been finding a way to mitigate the problem of the low absolute accuracies typically associated with raw UAV data. Unlike larger aerial vehicles such as fixed wing planes, UAVs are subject to wide variation in position due to their small mass. Variation in image scaling, illumination, rotation and altitude are factors that affect UAV photo acquisition; depending on climatic conditions such as wind and lighting, these factors exhibit an unpredictable range. Altitude often fluctuates by a few meters over the course of a flight, and distance to the ground is influenced by terrain variation. 10

11 Changes in view point, image scale and resolution are complexities that require specific processing to resolve (Mancini et al., 2013). The Internal Measurement System (IMS) in the UAV records the exact camera position at each photo, and location coordinates are written to the photo s Exchangeable Image File Format (EXIF) header in a process known as geotagging. UAV imagery is subject to large rotational angles between photos due to the movement of the aircraft; this results in a lack of predefined relationships between adjacent images (Mingyao et al., 2015), as would be the case in traditional imagery taken from a large airplane with a straight and level trajectory. In addition, the large volumes of photos captured by the small footprint of the low altitude camera, and the large margins of overlap required to minimize edge distortion, provide challenges for traditional photogrammetric processing methods. 1.1: UAV Derived Digital Elevation Models The integration of computer vision techniques, mainly involving a process known as Structure from Motion (SfM), has become a critical mitigating factor in reducing the errors caused by the low accuracy of the onboard GNSS receiver on most consumer grade UAVs (Westoby et al., 2013). Aerial triangulation must be implemented to continuously assess the changing positions of the camera throughout the data capture process, and Structure from Motion technology is gaining popularity as an efficient and accurate means of generating 3D point cloud imagery through automated feature matching. Images create 3D Point clouds and are tied to coordinate systems using either Ground Control Points, which are identified automatically within the point cloud, or direct georeferencing through estimated camera positions (Turner et al, 2012). Ground Control Points are very important to accurate georeferencing in UAV photogrammetry. When the point cloud is initially created in SfM software, it exists in model space but is not yet grounded to a coordinate system. SfM software is capable of solving the collinearity equations necessary to align photos in an arbitrarily scaled environment (Westoby et al, 2013). Camera positions are relatively accurate to one another, but do not achieve absolute accuracy until they are calibrated with ground control points, specific visible locations where coordinates are measured with a high degree of accuracy. Ideally these positions are located through a highly accurate ground survey technique such as Real Time Kinematic Global Navigational Satellite System (RTK GNSS) survey, but can also be integrated in the data processing stage to lower, but reasonable accuracy (defined by the accuracy of the GCP collection method), by identifying point locations and elevations in Google Earth or a similar mapping platform. The SfM software is integral to data processing to remove error associated with geometric distortion from UAV imagery. This solution is proving to be a robust means of terrain extraction capable of producing high resolution and accuracy from large volumes of photos with variable camera locations and angles; it produces a georeferenced point cloud where each point in the cloud is projected to a unique location on the ground. The georeferenced point cloud is then used to generate Digital Terrain Models (DTM), representing bare earth terrain surface, and Digital Surface Models (DSM) to represent height of tree canopies, buildings and other above ground objects (Ritter, 2014). Together, these elevation products are referred to as Digital Elevation Models (DEM) (ASPRS, 2014). The applications of the DEM extend across many fields; it is an important component of visibility analysis, erosion modelling, surface hydrology, watershed modelling, geomorphology, agriculture and ecosystem modelling. Various derivatives can be calculated from the DEM including measures of terrain roughness, curvature, slope and aspect as well as more involved values like the topographic wetness index. These derivatives are important to the process of modelling energy, water, sediments, nutrients and pollution distributions. Slope and aspect are the most widely used DEM derivatives, and serve as important components of most environmental hazard assessment (Miles, 2013). Environmental hazard mapping, for example in the case of slope stability assessments, is highly dependent on thresholds of slope and aspect. Slope failure is often associated with the angle of repose, the steepest angle at which the slope remains stable. Incorrect slope measurements can have serious consequence if slide prone slopes are overlooked in landslide hazard assessments (Miles, 2013). Aspect influences the angle of 11

12 repose, due to changes in sun exposure and illumination which influence temperature, moisture, soil type and stability, and the type of vegetative communities present, which also contribute to the degree of stability a slope has. Slope and aspect errors can result from a number of factors, and studies have been completed linking slope uncertainty to DEM errors derived from data precision, grid resolution, and grid orientation, as well as the slope algorithms employed in calculations (Zhou & Liu, 2004). Therefore, it is important to assess the performance of a DEM s derivatives as well as the quality and accuracy of the DEM itself, and to have a consistent standard on which to base measures and reporting methods of accuracy, so that the usability of data can be determined. 1.2: An Overview of Mapping Standards The evolution of mapping over the past few decades has been driven by technological advances and computational abilities that have demanded a subsequent and ongoing adjustment in standards and best practices. In 2014, the American Society of Photogrammetry and Remote Sensing (ASPRS) released an updated set of standards which attempts to provide a current framework for the consistent production, management and reporting of geospatial data. The ASPRS Positional Accuracy Standards for Digital Geospatial Data is a widely renowned and utilized best practice standard, and the 2014 release is an updated attempt to reflect current technologies in mapping, by which accuracy of positioning on the earth s surface can now be achieved to millimetre accuracy and spatial data exists independent of scale (ASPRS, 2014). Digital spatial data is now the foundation of cartography, but much of the mapping industry has been tied to older map accuracy standards which are no longer applicable. These include the National Map Accuracy Standard (NMAS) of 1947, created for hand drawn or plotted cartographic products, and governed by the concept of hardcopy map scale. This measure of accuracy no longer applies to spatial data, where computerized displays render fixed scales obsolete (ASPRS 2014; Zandbergen, 2011). It also includes the ASPRS 1990 Accuracy Standard for Large Scale Maps, which established vertical and horizontal accuracy measures as a function of RMSE, yet was also created to apply to printed maps with fixed scales and contour intervals; and the National Standard for Spatial Data Accuracy (NSSDA), published in 1998 by the Federal Geographic Data Committee (FGDC) in the U.S, developed to report digital data RMSE accuracy at 95% confidence, independent from scale and contour interval, but dependent on a Normal Distribution of errors, a condition which is not always met by spatial data (ASPRS, 2014). The past two decades in the mapping world have been characterized by a transition from hardcopy to soft copy environments, yet most standards which incorporate measures of Ground Sample Distance into accuracy measures were inherited from photogrammetry based on scanned film. In the 1990 s Softcopy Photogrammetry allowed large format film scanners to create digital imagery from film, with a resulting need for new guidelines that could relate the scanning resolution of the film to the map scale and contour interval according to older Legacy Standards (ASPRS, 2014). The introduction of digital large format metric mapping cameras in 2000 revolutionized data capture and have since become the main aerial image acquisition system for the collection of geospatial data. With current technology onboard, these platforms have allowed for the large-scale collection of highly accurate data at higher altitudes and longer flying times than what was previously available with older technologies. The advancement of image based terrain extraction techniques, such as the development of robust automated aerial photogrammetry software, have improved the quality of terrain data over the last decade (Westoby et al., 2012). Improvements in computer power have also added to the efficient processing and management of mass amounts of geospatial data. These technological advances demanded a relative evolution in standards and a number of revisions have been made to mapping standards in the past two decades. These include the National Digital Elevation Program (NDEP) 2004 guidelines which recognized the Non-Normal distribution of errors in vegetated areas and distinguished Fundamental Vertical Accuracy (FVA) from Supplemental Vertical Accuracy (SVA), and utilized percentiles as accuracy statistics in vegetated areas (ASPRS, 2014), and the ASPRS 2004 guidelines for LiDAR Data, which reinforces the stratifications of the NDEP guidelines; however neither of these standards provide accuracy 12

13 thresholds or quality levels (ASPRS,2014). In 2012 the USGS published its LiDAR base Specification, which is not a standard but a specification for LiDAR data acquisitions, as well as releasing its report on the National Enhanced Elevation Assessment (NEEA) which stipulates quality levels and thresholds for digital spatial data. Along with this report the USGS developed the 3D Elevation Program (3DEP) based on Level 2 Quality data for most of the country, with the exception of Alaska, and this data is expected to become the industry standard for digital elevation data (ASPRS, 2014). In spite of these attempts to create and implement revised mapping standards, outdated practices are often still applied in the mapping world due to a lack of clear guidelines on how to manage digital geospatial data. The ASPRS 2014 guidelines recognize that a wide variety of factors contribute to spatial data accuracy, and these factors now include an array of equipment including the quality of camera calibration, quality and size of the Charged Coupled Device (CCD) in the digital camera, amount of image overlap, GNSS accuracy, quality and density of Ground Control, quality of aerial triangulation parameters, quality of the photogrammetric processing, and quality of the interpolation used to create the final DEM (ASPRS, 2014). These standards provide a much more current mapping framework; however, implementation of these guidelines is still in its early stages. According to ASPRS 2014 Standards, a more developed quantitative characterization of the spatial distribution of error is necessary than what is currently presented, and until this is achieved, a general recommendation is made regarding the quantity and distribution of checkpoints, which requires Elevation Datasets of <500 km 2 to contain at least 20 vertical checkpoints in Non-Vegetated Areas (NVA), and at least 5 checkpoints in Vegetated Areas (VVA). Vertical checkpoints should be surveyed on flat or uniformly sloped terrain, and should avoid vertical artifacts or abrupt changes in elevation (ASPRS, 2014). Root Mean Square Error (RMSE) represents the square root of the mean of squared errors for a sample. If the dataset being measured exhibits a Normal Distribution of error, suggesting minimum bias (trends in the data) and systematic error, RMSE is the statistical equivalent of the Standard Deviation (SD) (ASPRS, 2014). The Standard Deviation is a statistic that indicates the distribution or spread of the elevation errors around the mean, and indicates data precision (OMNR, 2014). Elevation errors produced in Non-Vegetated Areas (NVAs) are assumed to follow a Normal Distribution, and according to ASPRS 2014 Standards, accuracy of NVA areas will be reported as a function of RMSE at a 95% Confidence Level. The Normal Distribution predicts that 63% of values will be within one standard deviation of the mean error, 95.5% will be within 2 Standard Deviations of the mean error, and 99.7% of values will fall within 3 Standard Deviations of the mean. Error distributions in Vegetated Areas (VVA) are not expected to follow a Normal Distribution, and according to ASPRS Standards, are better presented by 95 th percentile statistics. However, the use of RMSE alone as a reporting method is insufficient as it does not account for bias; trends in under and overpredictions in the data are not represented by the absolute nature of the RMSE statistic. In addition, there is a growing body of literature documenting the frequent occurrence of Non-Normal distributions in the residuals of elevation data (Hohle & Hole, 2009; Zandbergen, 2008; Aguilar et al., 2007; Oksanen & Sarjakoski, 2006). These studies, among others, give weight to the literature that is emerging in support of alternative methods of reporting elevation data accuracy which are independent from the assumption of an underlying Normal distribution. Outliers are common in elevation data acquired through LiDAR or digital photogrammetry, often related to inaccurate point classification or incomplete surface feature extraction. These errors are known as blunders and are a frequent occurrence in digital terrain products (Hohle & Hohle, 2006). Non-stationarity and spatial autocorrelation are also contributors to Non-Normal distributions in terrain products (Zandbergen, 2008). Non-stationarity refers to a trend where the model performs differently in different parts pf the study area; for example, the behavior resulting from variability due to land cover (Gorokhovich & Voustianiouk, 2006). Spatial autocorrelation refers to a tendency for sample points that are close in proximity to exhibit more similar values than samples that are farther apart. In terrain products for example, this would be expected in the data as altitudes at neighboring sample points are likely to be similar. Spatial autocorrelation can violate assumptions about the independence of samples, which 13

14 is problematic in statistical testing where a random distribution of residuals is required. Most ecological and environmental data exhibits spatial autocorrelation, and statistical tests need to recognize and minimize this effect to produce models with meaningful results (Hohle & Hohle, 2006). In addition, systematic errors can be introduced in photogrammetric processing if insufficient texture or structure exist for accurate feature matching to occur, and positional errors can result from inaccuracies in sensors or onboard GNSS equipment (Hohle & Hohle, 2006; Zandbergen, 2008). With a wide variety and high frequency of factors collaborating to introduce error into elevation data, the underlaying assumption of a normal, Gaussian distribution is often violated, even in the NVA, and when this is the case, RMSE is an insufficient reporting statistic. ASPRS currently recommends the additional reporting of descriptive statistics such as mean, median, standard deviation, skewness, kurtosis, and quantiles to more fully describe error distributions, and there are a variety of non parametric reporting methods such as confidence intervals, quantiles, and coefficients of variation (Hohle & Hohle, 2006) being suggested in the literature as alternative reporting methods for the quality of elevation products in both National and International Standards. 1.3: An Overview of the Acquisition of Terrain Data Remote sensing is considered to be a crucial part of the geographic sciences (Miles, 2012) which has allowed for unprecedented monitoring of global resources and patterns of environmental change. At local scales, remote sensing platforms include fixed wing planes, helicopters, kites, balloons and UAV s. At a global scale, remote sensing platforms are most often based on earth orbiting satellites (Miles, 2012). Satellite sensors are classified as either active or passive; active sensors emit electromagnetic energy and use the strength of the return to measure location, and passive sensors only receive electromagnetic radiation. To bridge the gap between local and global, some of the more common methods for the collection of large scale data include Light Imaging and Detection (LiDAR), Synthetic Aperture Radar (SAR), and Semi-Global Matching (SGM) implemented through digital photogrammetry based on aerial photography. LiDAR data is an industry standard as a reliable and accurate means of collecting high density, high accuracy elevation data compared to sources of elevation data such as topographic maps, softcopy photogrammetry, and most satellite sensing (Liu, 2012). Using laser pulses, LiDAR actively illuminates objects and measures the time between laser emission and return to record a range, which is then combined with GNSS receiver and IMU information to generate point clouds of precise geographic location with three dimensional spatial attributes (Liu, 2012). Synthetic Aperture Radar (SAR) uses radar mounted on a moving platform (usually aircraft or spacecraft) and measures the time between pulse emittance and return, to record geographic information. SGM relies on automated photogrammetric processing of imagery captured with a large scale digital sensor from high elevation aerial surveys. Although there are increasingly available sources of spatial data on the market, organizations face challenges in the form of high costs generally associated with collecting and processing this data into a consistent and reliable dataset (GRCA, 2015). As a result, agencies often collaborate with government bodies to acquire reliable spatial data. An example of this is the South-Central Ontario Orthophotography Project (SCOOP, 2013), part of the Ontario Imagery Strategy which involves a multi year phased acquisition of orthoimagery and its derived terrain products in Ontario. It is funded through the Ontario Ministry of Natural Resources (OMNR) and a collaboration of conservation authorities and agencies across the province, and takes a provincially coordinated approach to wide scale Geospatial data collection. The raw Unclassified Point Cloud from the SCOOP acquisition is available free of charge to the public via the Land Information Ontario (LIO) website with 0.4 m Point Spacing. In addition, the OMNR and partners are currently implementing stage 2 of a province wide acquisition of high resolution LiDAR data, which will become publicly available in early This LiDAR acquisition has not yet received publicity and little documentation exists about its deliverables. Half of the acquisition was completed in November of 2016, and has only been selectively delivered to stakeholders. 14

15 However, elevation data and orthoimagery will become increasing reliable through widespread coordinated efforts such as the SCOOP and LiDAR acquisitions. UAV s as data collection platforms cannot compete on the scale of large data acquisitions such as highaltitude flights or satellites, and their role should be to complement these efforts, with some benefits to smaller geographic study areas. UAV s are most appropriate for use in smaller area, larger scale mapping projects. In this capacity, they provide several important benefits that larger efforts do not; temporal currency, and very high resolution. UAV s flown from low altitudes generally provide very high Ground Sample Distance (GSD) and resulting resolution, often in the centimeter level. Increased resolution provides an opportunity to model complex ground surfaces with higher precision than previously possible with lower density point clouds. Because higher point density equates to lower distances between points, less of the elevation surface needs to be interpolated in DTM production, resulting in a higher quality product where more surface details and micro features can be modelled (ASPRS, 2014). High resolution and the ability to fly on demand allow UAV s to capture rates of change with much higher temporal currency than what is provided by large data acquisitions, which often experience many years between collections. The SCOOP 2013 project is representative of this, and is the first phase in the 5- year refresh cycle. Thus, if an organization requires updated imagery mid-cycle, they would need to rely on satellite imagery or private acquisitions (OMNR, 2013). UAV technology can provide temporal currency which can serve important roles in monitoring environmental changes such as erosion, land use changes, flood water mapping following storm surges, and sensitive habitat monitoring. 1.4: Problem Statement If UAV s can produce reliable, high quality data at similar accuracies to accepted standards, yet with the benefits of temporal currency and high resolution, they can serve as an integral supplementary product in the evolution of elevation mapping procedures. This study proposes to use GIS to assess the absolute accuracy of the UAV derived high resolution DTM through point error based statistical comparison to Real Time Kinematic Global Navigational Satellite System (RTK GNSS) Survey, and analysis of the key DTM derivatives slope and aspect. As a basis of comparison, the error distribution and accuracy reporting statistics of the UAV elevation model and its derivatives will be compared to the Ontario SCOOP 2-meter DTM. The integral use of Digital Elevation Models in many applications and the rise in the popularity of Unmanned Aerial Vehicles as a data source for the generation of these products constitutes a basis for more studies to assess the accuracy of the UAV derived Elevation Model. Many studies are suggesting that UAV derived products such as the DTM, processed in automated photogrammetric software, can rival or outperform LiDAR and Terrestrial Laser Scanning in terms of accuracy and resolution, while being more affordable, and temporally current. It is hypothesized in this study that a UAV generated DTM will support this and be capable of producing elevation point cloud data and a derived DTM with accuracy equal to or better than the error reporting statistics (RMSE, percentiles and 95% confidence) of the SCOOP dataset, which will be calculated in this project, with the benefits of temporal currency, low relative acquisition cost, and high resolution. It is also hypothesized that the accuracy of derived slope and aspects, measured by the same error statistics, will be better than or equal to the SCOOP DTM, and that there will be a statistically significant correlation between errors in elevation and corresponding errors in derived slope and aspect. It is also hypothesized that all variables being tested (elevation, slope and aspect) should follow a Normal Distribution in the NVA and a Non-Normal Distribution in the VVA. 15

16 1.5: Research Objectives In attempt to assess the accuracy and usability of elevation data produced by digital photogrammetry based on UAV imagery, the objectives of this study aim to answer to following questions: 1. How accurate is the Unmanned Aerial Vehicle (UAV) derived Digital Terrain Model in predicting true elevation, slope and aspect values, as indicated by the error distribution of the differences between GNSS RTK survey points and UAV values? 2. How do the error distributions of elevation, slope and aspect differ between the South-Central Ontario Orthophotography Project (SCOOP) 2m DTM and the UAV DTM, and are these differences significant? 3. How does Land Cover Type, as defined by Non-Vegetated (NVA) and Vegetated (VVA), influence the magnitude and distribution of error in the UAV DTM? 4. Are there significant correlations in the data between elevation error and slope and aspect error, or between slope and aspect error? CHAPTER 2 Literature Review 2.0: Terrain Modelling The DEM and its classified forms of DSM and DTM are the foundations for a wide range of geographic modelling, such as flood plain mapping, avalanche management, terrain visualization, geological mapping, meteorological studies, hydrogeological mapping, watershed management, forest health monitoring, soil mapping and erosion control (Singh et al., 2015). Digital Terrain Models (DTM) reflect bare earth values and the term is often used interchangeably with Digital Elevation Model (DEM); however, the DTM is usually associated with a gridded ground class terrain model where points are distributed at equal interval, and a DEM should be used to refer in a broader sense to terrain products which may include classified or unclassified elevation data, essentially a DTM or bare earth product and/or a Digital Surface Model (DSM) which may include the heights of surface features such as forest canopy or buildings (Garnett, 2010). A raster based, gridded DTM provides a continuous elevation matrix on which map algebra can be performed (ESRI). The raster DTM represents the world as regular arrangements of cells and is the foundation of systematic analysis of relationships between mapped features. There are various ways to generate a DTM from a set of input points, all based on the interpolation of unknown locations between known locations. Interpolation predicts values for a population from sample points. In a Geographic Information System (GIS), common interpolation methods are Inverse Distance weighting (IDW), Kriging, Natural Neighbor, Spline, and a range of tools to convert a variety of data formats to raster. The choice of interpolation method to use is dependent on the structure of the data being modelled. For example, the Spline function is a smoothing method that passes a surface directly through a set of input points, resulting in a minimization of surface curvature. It is appropriate for continuous data, such as elevation and temperature (ESRI, interpolation methods). DTMs are historically contour based, and are generally either inaccurate and/or time consuming and costly to produce (Uysal et al., 2015). DTMs are now most often created using a continuous, gradient raster surface on which cell statistics are calculated. From DTMs, important topographical derivatives can be analyzed; these 16

17 include slope length and percent, surface roughness, aspect and curvature (Rock et al., 2011). These derivatives are inputs into many types of environmental models, and have often been identified as sources of uncertainty or inaccuracy in model outputs (Lucieer et al. 2013). The resolution and accuracy of the DTM are important components to model stability, and these variables depend both on the data collection method and the processing used to interpolate the model (Tilly et al., 2016). There is an increasing availability of software used to manage point cloud data through automated photogrammetric processing. This point cloud can be exported in various formats including feature class or shapefile, PDF and Log ASCII (LAS), at variable degrees of accuracy and resolution, and incorporated into GIS software for spatial analysis to produce a variety of models (Ritter, 2014). Studies have considered various applications of UAV derived DTMs in environmental management models such as soil erosion models (d Olieire-Oltmanns et al., 2012; Lucieer et al., 2013; Bazzoffi, 2015), landslide hazard assessments (Turner et al., 2015), forest resource inventories (Ritter, 2014), costal erosion mapping (Mancini et al., 2013), and bio erosion models, which attempted to characterize rooting and nesting patterns in trees using UAV s (Genchi et al., 2015). The Digital Surface Model (DSM) represents the reflective surface; the surface including any above ground objects. DSMs can be generated by LiDAR as well as by UAV photogrammetry. To calculate Canopy Height Models (CHM) the Digital Elevation Model is subtracted from the DSM. A 2014 study by Brian Ritter investigates the use of UAV data for creating Canopy Height Models for Urban tree inventories and processes the LAS data in GIS, in addition to providing a comparison of spatial accuracy and tonal balance in point cloud mapping and image correction from two different software applications, Drone Mapper and AgiSoft Photoscan. The author successfully used UAV imagery to create a model predicting tree heights which suggests that UAV products can be used in place of traditional data to obtain tree inventories and produce Canopy Height Models (Ritter, 2014). 2.1: Remote Sensing Platforms and the Role of the UAV Some of the most common means of acquiring geospatial information through remote sensing include Light Detection and Ranging (LiDAR), Synthetic Aperture Radar (SAR) and Semi Global Matching (SGM) based on aerial imagery, an algorithm used in many automated photogrammetric software suites, including SfM Photogrammetry. Semi Global Matching (SGM) has proven to be a reliable process of photogrammetric terrain extraction based on ortho imagery to obtain high resolution, accurate DTMs, using global and local stereo models for accurate pixel matching at low run time (GRCA, 2015; Hirschmuller, 2011). SGM/SfM methods produce a classified data point cloud in LAS format. Digital photogrammetric platforms are not limited to the UAV and plane or helicopter and also include remote sensing platforms such as the hot air balloon, the kite, or hand-held poles (Snavely et al., 2008). Studies that conduct accuracy assessments on satellite elevation data produced by SAR include statistical investigations of DEMs created by the Shuttle Radar Topography Mission (SRTM) and Advanced Space Bourne Thermal Emission and Reflection Radiometer (ASTER) (Miles, 2013; Singh et al., 2015). LiDAR uses light in the form of a pulsed laser to measure variable distances to the earth and generate precise 3D point clouds that can be classified into surface cover classes based on the return and intensity of the pulse. LiDAR can be collected quickly, day or night, is highly accurate and does not rely on the placement of GCPs, can be collected in dense forest, and contains no geometric distortion (ESRI, 2014). Studies that deal specifically with LiDAR data and its accuracy or that use LiDAR as reference data are abundant in the literature (Liu & Sherba, 2012; Zandbergen, 2011; Aguilar & Mills, 2008; Ritter, 2014). Drawbacks to LiDAR are usually related to cost of acquisition and consideration of the risk involved in manned aerial survey. In this sense, the growing use of Unmanned Aerial Vehicles to generate DTMs through aerial imagery provides some benefits over airplane mounted LiDAR, especially when the area of interest has a small geographic extent, including less risk and lower cost involved in acquisition. While LiDAR is generally mounted on fixed wing aircraft or helicopters, it can also be attached to a UAV, providing possible combinations of technology to efficiently gather data (Liu & Sherba, 2012). 17

18 The use of LiDAR and UAV survey methods have resulted in an increase of available high-resolution data, often in the range of m resolution from LiDAR and as high as 1-3 cm from UAV images. Many of the studies completed on the influence of DTM resolution on environmental models use 10 m DTM s as the minimum resolution input (Dehvari, 2014; Zhao, 2008), but the influence of higher resolution data in resource modelling has been less studied. As more data becomes available for market applications, it is important to understand the influence that resolution can have on predicting topography in order to choose the most applicable data to achieve the desired end result. Studies show that topographical and hydrological parameters such as average slope, depressions, channel networks and stream lines derived from remote sensing provide improved estimations of these extracted parameters due to the high resolution of the data (Zhao, 2008), as compared to models of 2 dimensional, contour based mapping. Much of this type of remotely sensed environmental data has a temporal requirement where change needs to be continuously monitored. UAV s lend themselves well to this kind of data collection, where change may need to be identified quickly following a flood event or mass movement of land. Elevation models can be created from LiDAR LAS point cloud datasets through GIS processing, and a LAS point cloud can also be derived from geotagged Red-Green-Blue (RGB) imagery (often in conjunction with ground control points and collected by aerial survey) using photogrammetric software. In deriving DTMs from imagery, both accuracy and resolution depend on the camera and lens used, as well as the type of processing applied and the software used. With high quality equipment and calibration by ground reference points, centimetric detail and accuracy can be achieved in UAV derived DTMs (James and Robson, 2014). However, it is important to consider the various types and scales of error that can result from variation in equipment and processing when creating Digital Terrain Models from imagery. UAV s are capable of capturing very high-resolution data, often in the sub decimeter range, but due to the small footprint of their flight paths, large volumes of photos are captured to achieve the necessary spatial coverage required by many applications (Turner at al., 2012). Due to the high overlap required for image processing, commonly 60-80%, UAV s often need to capture hundreds to thousands of images. These images require processing to correct and georeference them in a photogrammetric software program. UAV imagery is subject to light distortion, particularly at the edges of photographs, rotational and angular variations between images, perspective distortions, inaccurate or unknown exterior orientation parameters, and high variation in illumination and occlusions (Turner et al, 2012). However, modern photogrammetric software is capable of dealing with some of the error of data collection and is considered to be robust against changes in rotation, scale, and translation between images (Lingua et al., 2009). 2.2: Digital Photogrammetry and the UAV Based DEM Digital Terrain Models based on UAV Imagery often require additional filtering methods than what is used to process LiDAR point clouds, which are most often classified by the vendor. The UAV Point cloud initially includes all surface points; these surface objects are filtered out to create a bare earth DTM. While Digital Surface models serve important purposes for forestry applications, in the generation of forest canopy models and vegetation typing, for most environmental applications, it is necessary to derive a bare earth model in order to model water flow, soil conditions, and terrain variability. The UAV is challenged by this bare earth model construction (Serban et al., 2015). Unlike LiDAR, which uses multiple returns to penetrate a forest canopy and reflect back pulses indicative of multi story and bare earth elevations, UAV photogrammetry is capable of producing only Digital Surface Models in raw point cloud form. Extra processing must be applied to remove surface objects through point classification techniques and editing. Many different software platforms are capable of performing this function, to varying degrees of completeness and accuracy. However, even with multiple passes and editing, usually some artifacts of surface structures remain in imagery derived DTMs, and this is considered a drawback to the UAV DTM (Dowling & Gallant, 2013). Further studies into how the UAV performs over different land cover types are important to 18

19 quantifying error; however, it is generally assumed that error from imagery derived elevation models in vegetated areas do not follow a Normal Distribution and usually exhibit a higher measure of RMSE than from bare earth geography (ASPRS, 2014). Most DTMs are gridded, continuous raster datasets, and with accurate surface feature removal, elevation values should reflect the variability of the terrain itself. Structure from motion (SFM) software (Snavely et al., 2008) is emerging as a means to derive point cloud data from still two-dimensional imagery taken from various locations and orientations. SfM produces a point cloud utilizing the algorithms of Semi-Global Matching, with the distinction lying in the variation of orientations managed by SfM as compared to the relatively stable orientation provided by a high-altitude platform most commonly used for SGM. SfM has been successfully implemented for assessing processes such as soil and coastal erosion from terrestrial and manned airborne platforms (James and Robson, 2012). Studies that have assessed the accuracy of elevation values extracted through SfM software have suggested that these 3D scene reconstruction and image correlation techniques can provide effective means of mapping erosion processes (Lucieer et al., 2014). The highlydetailed 3D images that can be reconstructed through SFM algorithms have shown to be comparable in accuracy to Terrestrial Laster Scanning (TLS), airborne LiDAR and on ground GNSS surveys, with error ranges less than 0.1 m in X, Y and Z attributes (James and Robson, 2012; Fonstad at al., 2013). SfM technology is a region detector that uses image matching in a Bundle Block Adjustment to correct for exterior orientation of photographs. Images are matched using an autocorrelation matching technique based on aggregates per pixel. Studies suggest that UAV technology used in conjunction with SfM software provides effective land scape monitoring solutions due to the very high resolution imagery (1-20 cm), the ability of the UAV to fly on demand without putting pilots and third parties at risk, cost effectiveness compared to LiDAR and ground based surveys, the ability to carry and utilize multiple different sensors, and the cost effective yet accurate automated post processing options now available (Lucieer et al., 2012; Kung et al, 2011). Images used for photogrammetric processing require both forward lap and side lap to mitigate camera tilt and distortion that comes from the edge of photographs; typically for UAV photogrammetry the value of forward lap should be about 70%. The forward lap is a percentage of total image coverage that is introduced between successive photos on a flight line. Large overlap can help account for the substantial effects of camera rotation due to the lightweight frame of the aircraft (McGraw Hill, 2014). Side lap describes the amount of overlap between images from adjacent flight lines; for UAV photogrammetry, this should be at least 50%. Image ground coverage must also be taken into account; it is determined by the focal length of the camera, the size of the CCD array, and the flying altitude above ground elevation (Westoby et al., 2012). SfM software typically exports data in a range of formats including TIFF, JPG, or LAS format which integrate easily with most GIS software. The Log ASCII Standard (LAS) dataset is an industry standard binary format for storing LiDAR data. LiDAR LAS point attributes are maintained for each laser pulse and include: horizontal and vertical location (X, Y, Z) information, GPS time stamp, intensity, return number, number of returns, point classification values, scan angle and direction, RGB values, scan direction, edge of flight line, user data, point source ID and waveform information (ArcGIS Resources). There are freely available standalone options for processing LAS data such as LASTools (Isenberg, 2014), a subset of which can be used interchangeably in mapping applications like ESRI s ArcGIS. In addition, ArcMap has a set of LAS processing tools to further analyze LAS data. Point cloud data produced by UAV photogrammetry can be processed using the same basic processing as would apply to LiDAR data in GIS to produce and analyze DTMs. 19

20 2.3: DEM Derivatives of Slope and Aspect Slope and aspect are important and widely used derivatives of the DTM (Bolstad & Stowe, 1994), crucial to mapping any terrain driven phenomena such as flooding and water management, avalanche monitoring, and landslide risk assessments. Numerous studies assess DTM grid size influence and conclude that slope estimates from DTMs decrease with increasing grid cell size; higher resolution has a smoothing influence on the landscape. Many studies have shown correlations between slope errors and steeper slopes (Bolstad & Stowe, 1994; Miles, 2013) and that slope differences are greatest in steep areas (Ashraf et al., 2011; Dehvari, 2014). Numerous studies are focused entirely on the accuracy assessment of DTM derivatives like slope and aspect, as these two variables play important roles in land analysis (Weih and Mattson, 2004). Studies have tested GIS derived slopes using different methods. Weih and Mattson ground truthed by using mathematical models as a surface, and by field checking using a clinometer. Accuracy assessments of derivatives like slope are important because it has been shown that errors in elevation data can propagate in spatial analysis applied to derivative surfaces (Bolstad & Stowe, 1994); the error will carry over and even be amplified in derivatives. In addition, more research that addresses slope and aspect can help to understand variability that comes from the limits of field checking slope and aspect values with clinometers and compasses, particularly in forested or complex terrain where line of sight can be difficult to establish (Bolstad & Stowe, 1994). Weih (1991) showed that results of deriving slope from different methods can vary by up to 40%. In ArcGIS, the Slope tool calculates the maximum rate of change in value from that cell to its neighbors. The maximum change in elevation over the distance between the cell and its eight neighbors identifies the steepest downhill descent from the cell (ESRI surface toolset). If the slope method used in GIS does not accurately capture reality, conclusions derived from that model may be incorrect. Topography determines areas of source and sink, and the error of the surface used to measure water flow can have impacts on natural resources, agriculture, and infrastructure. These errors can carry over into estimates of derivatives, like slope and aspect, and influence sensitive models (Singh et al., 2015). Erosion models, such as the Revised Universal Soil Loss Equation (RUSLE, Renard et al., 1997), and the recently upgraded, process based Water Erosion Prevention Program (WEPP, USDA 1985), are often used in GIS based analysis incorporating slope derived from DTMs, and have proven to be effective at estimating the intensity of erosion (Fernandez et al, 2003). All erosion models require a set of inputs, and are sensitive to the accuracy of the input data. Elevation data provides the material with which to build DTMs, a critical component to Erosion Modelling and one that can significantly influence predictions of erosion and deposition. Many studies have assessed the sensitivity of erosion models to DTM accuracy (Renard & Farreira, 1993; Rock at al., 2011; Uysal et al., 2015; Vaze et al., 2010) as well as to DTM resolution (Habtezion et al., 2016; Wu et al., 2005; Zhang et al., 2005; Zhao et al., 2010). Yet, there is still uncertainty regarding the effect of using very fine resolution (<5m) DTM data to estimate topographical parameters at the small watershed and agricultural field levels. Dehvari 2014 points to results that suggest insignificance in model discrepancies when using 1 m and 5 m DTM s compared to cheaper, more available 30 m resolutions. However, Zhang et al., 2005, suggest that using finer resolutions may actually introduce bias into models by exaggerating micro features in the terrain. Though Zhang and co authors conclude that 10 m LIDAR DTM s are the most appropriate topographical input source for soil erosion modelling using the WEPP model, the exaggeration of terrain that they observe may be beneficial in some cases, and should be better understood. For example, in Ontario, the Soil Erosion Rating is derived from the RUSLE model and is used in the calculation of the Phosphorus Index for a field, to assess risk for phosphorus contamination into a watershed. Flow maps and erosion risk maps are important components of agricultural management, and the input data into these models is becoming increasingly available and more precise (Patil et al., 2015). Phosphorus losses from agricultural zones need to be reduced to prevent eutrophication of water. Phosphorus distribution exhibits high spatial variability across terrain, and studies have shown that cost effective 20

21 countermeasures need to be taken to target parts of the topography that are most susceptible to losses. In the case of phosphorus management, over prediction of high resolution DTM s may provide a desirable effect as a safety margin (Djodjic & Villa, 2015). Aspect identifies the downslope direction of the maximum rate of change in value from each cell to its neighbors and it indicates the slope direction. The values of each cell in the output raster indicate the compass direction that the surface faces at that location. Like the Slope tool, the Aspect tool fits a plane to the z-values of a 3 x 3 cell neighborhood around the processing or center cell. The direction the plane faces is the aspect for the processing cell (ESRI, Surface toolset). Studies that focus on accuracy assessments of aspect in GIS are often the ones that also focus on slope (Bolstad & Stowe, 1994; Weih, 1991; Miles, 2013); these two derivatives tend to go together, since aspect can be considered a byproduct of slope surfaces. However, accuracy assessments of aspect values are more complicated than for slope due to the face that aspect values are measure from degrees, on a scale where 0 and 360 are the same value. If actual data and observed data are on opposite sides of the 0 line, this is a problem for using the data in any sort of automated algebra, for example, for the processing of an aspect surface. For example, if the observed value is a bearing of 5 degrees and the actual value is 359 degrees, the error is 6 degrees, not 354 degrees. To accommodate this in a large dataset Miles (2013) split the aspect data into two halves based on different equations to negate the effect of the 0-degree line from their calculations (Miles, 2013; Illustration 2). The cyclic nature of aspect data must be accounted for in accuracy assessment or error surface creation; either by manually assessing the values to ensure they do not cross the zero line the wrong way, or by applying equations to sort the data into correct aspect error classes. Illustration 2: Example of Aspect Error Calculations for Cyclic data for larger datasets (Miles, 2013) 2.4: DTM Resolution and Accuracy Resolution refers to the grid cell size and the point spacing of the data used to create a specified raster, such as a DTM. Modelling of water flows and hillslope will require a grid scale much smaller than the scale of the hillslope, but the question of how much smaller depends on the application and model (Vaze et al., 2010). The resolution of a Digital Terrain Model is a crucial factor in watershed modelling which can cause significant variability in the representation of topographical factors (Habtezion et al., 2016). Many studies have shown that as DTM resolution becomes coarser, surface depressions are reduced, topographical features such as slopes are 21

22 decreased, and drainage networks are underestimated (Habtezion et al., 2016; Wu et al., 2008; Zhang and Montgomery, 1994). Both accuracy and resolution have implications on the topographical values derived from DTMs. Some studies show that the detail maintained from resampling high resolution to lower resolution DTMs is much higher than the detail available from more commonly available, contour derived, lower resolution DTMs (Vaze et al., 2010). As higher resolution DTM s become more available through LiDAR and UAV sources, more research is being directed towards examining the need to update conventional DTM s using higher resolution, and more accurate DTM s (Zhao et al., 2008). However, there are mixed results surrounding the impact of resolution on hydrologic and erosion modelling. Some studies show that resolution below 10 m does not produce significantly better results than higher 1 or 5 m DTMs (Dehvari, 2016), some suggest that very high resolution DTMs create bias in models by exaggerating microfeatures in terrain (Zhang et al., 2005), and others suggest that water flow lines produced from 1 m LiDAR derived DTMs are considered to be better at accurately modelling terrain, especially in the ability to accurately model flow diversion terraces (Zhao et al., 2008). In their 2008 study on the impacts of DTM resolution on agricultural erosion control, Zhao and co-authors found that at a 10-m resolution, neither conventional or LiDAR derived DTMs could identify the impacts of soil conservation structures, such as diversion terraces, on erosion to the same capacity as 1 m DTMs. The spatial resolution for UAV imagery from conventional platforms is typically <20 cm/pixel, and a UAV can collect data at a detail as high as 1 cm/pixel when flown at low altitudes. This kind of resolution can be applied to many assessments, such as calculating vegetation and soil cover (Johnson et al., 2003), monitoring agriculture and calculating Vegetation Indices (Turner et al., 2012). Accuracy is a measure of how closely the measured values represent the true values. DTM accuracy refers to how well a digital terrain model captures the variations in landscape. It is dependent on many factors including source and technique of measuring elevations, the density and distribution of sample points, the interpolation methods, and the landscape complexity (Singh at al., 2015). Studies that assess the ability of the UAV generated DTM to accurately reflect elevation values show that data derived from UAVs, post processed and calibrated with GCP s, is comparable to the values captured from RTK GPS data (Uysal et al., 2015; Rock et al., 2011). Technology is advancing, in terms of high resolution DTMs from LiDAR and UAV sources becoming more available, and computing abilities becoming more capable of handling large and detailed datasets. While accuracy and resolution are related, the two do not measure the same factors. Improvements in technology have had positive impacts on refining both accuracy and resolution, but the factors that improve resolution are not the same factors that improve accuracy. In the case of LiDAR, for instance, the factors that create a high-resolution dataset include high pulse rate or multiple overflights. The factors that improve accuracy are closer base stations, narrow swath width, and good Percent Dilution of Precision (PDOP) which is based on the various atmospheric factors what can interfere with data collection (Uysal et al. 2015). Different end results have correspondingly different parameters. If the spatial details are more important than exact elevation values, such as in vegetative typing, high resolution will be more of a requirement than highly accurate elevation data. However, usually, as the cell size in a raster decreases, and resolution increases, accuracy generally improves as well. As the cell size decreases, there is a likelihood that elevation variability within the cell decreases, so more meaningful conclusions can be drawn regarding accuracy (Zhao et al., 2008), as well as conclusions regarding data quality, by which higher resolution usually generates a more detailed model of terrain micro features (ASPRS 2014). In the collection of UAV elevation data, location measurements taken by the internal GNSS device must be georeferenced, and the accuracy of these results varies depending on the method of post processing. While resolution of UAV imagery can be very high, often in the centimeter range, the accuracy of their orientation estimates is comparably low (Kung et al., 2011). In their 2011 study, Kung and co authors suggest that since extremely precise positioning of the landscape cannot yet be achieved directly from UAV imagery, post processing is key to accurate georeferencing in the production of UAV DTMs. They compare two variants of georeferencing, 22

23 the first being an aerial triangulation algorithm using 3D modelling software to create an automated output consisting of an orthomosaic and DTM derived from the internal geotags of the UAV and no Ground Control Points. The second method uses GCP s to correct for geotag inaccuracy and improve geolocalization accuracy. They conclude that for a resolution of 40 cm, location accuracy is between m using GCPs and 2-8 m without. Their study suggests that light weight UAVs can take reasonably accurate images with sufficient overlap while covering areas of a few square kilometers per flight. They suggest that the use of recent computer vision techniques for post processing overcomes the issue of low location accuracy typically expected from UAV data, and enables a range of decision makers from various fields to create and use this data more accurately. Accuracy depends on ground resolution or Ground Sample Distance (GSD), overlaps, geotagging accuracy and ground control. Positional accuracy has become acceptable in most UAVs carrying an onboard GNSS system; these now commonly access both GPS (American) and GLONASS (Russian) satellites and generate positional accuracies around 1 3 x the pixel size, when ground control points are used (Sense fly, 2015). Other studies have assessed the influence of number of ground points and flying heights on accuracy of UAV derived DTMs (Rock et al., 2011) and the accuracy of elevation models produced using direct georeferencing from UAV log files in the absence of ground control points (Blaha et al., 2011; Li et al., 2011). In a 2012 study by d Olieire-Oltmanns and co-authors which addressed the ability of using a UAV for soil erosion mapping in Morocco, the authors investigated two different approaches for geo referencing based on high precision GCPs or the UAV flight log with exterior orientation values. They maintain that UAV based remote sensing bridges the gap in resolution and scale between ground surveys and imagery from satellite and manned aircraft. The accuracy of the UAV DTM is also influenced by land cover. Accuracy is based on the assumption that errors follow a Normal Distribution, however in vegetated areas this is not the case (ASPRS, 2014; Singh et al., 2015). The UAV produces a Digital Surface Model, and surface filtering is required in any areas of built up structure, vegetation, or forest cover. The error of the photo derived DTM is susceptible to filtering error to a variable extent. Studies that have investigated the influence of land cover on DTM creation have shown that DTM accuracy is terrain class dependent (Miliaresis and Paraschou, 2005; Singh et al, 2015). Often it is the influence of forest cover that results in a reduced DTM accuracy, and UAV terrain mapping does not have the ability to penetrate the forest canopy with a frame camera. LiDAR is a much better choice for measuring ground elevations in areas of dense forest cover and complex vegetation (Liu & Sherba, 2012). Multiple returns allow for multi story measurement within and below the forest canopy. However, photo derived elevation models processed with surface filtering techniques can achieve surface feature reduction for smaller geographic extents (Mancini et al., 2013), and the success achieved by the surface filtering is dependent on the editing input and software used. One benefit to mapping complex terrain with a UAV is that they typically fly at low elevations, and as a result are capable of capturing very high Ground Sample Distances (GSDs) in their point clouds. It is common for a flight of 50 m to capture <2 cm/pixel GSD in a highly dense point cloud. This density is often equated with equally high accuracy; however, this is not always the case. While higher densities yield higher measurement precision, they do not necessarily yield higher accuracies due to the fact that accuracy is limited by systematic error; error in the components of the system itself, including the IMU, GNSS device or scanner. These errors cannot be changed by increasing the resolution or scale, and are not improved at higher resolutions (ASPRS, 2014). However, increased resolution does provide an opportunity to model complex ground surfaces with higher precision than previously possible with smaller scale, less dense point clouds. Because higher point density equates to lower distances between points, less of the elevation surface needs to be interpolated in DTM production, resulting in a higher quality product where more surface details and micro features can be modelled (ASPRS, 2014). Increased point density, as is supplied by LiDAR and UAV, does not improve the accuracy of each individual point, but it does 23

24 improve the accuracy of the derived DTM at locations between points, and is therefore desirable for the mapping of complex terrain. 2.5: Statistical Analysis and Reporting of Terrain Data Accuracy Root Mean Square Error (RMSE) is a widely used measurement of data accuracy in the geographic sciences (ASPRS, 2014). However, to correctly represent the error in a dataset, RMSE depends on the Normal distribution of residuals. It is also widely accepted that error analysis should not be reduced to a single number (Oksanen and Sarjakoski, 2006), and various approaches have been used in error analysis to better understand the distributions of error such as percentiles, standard deviation, confidence intervals, or trimmed datasets to achieve normal distributions (Hohle & Hohle, 2006). These statistics compose a characterization of error more through in nature than single measures of accuracy such as RMSE, and present information about the spatial structure and distribution of the data (Shortridge, 2001; Oksanen & Sarjakoski, 2006). In the case of DTM errors, it is often found that non-stationarity and spatial autocorrelation invalidate statistical assumptions of an underlying Normal Distribution and that the correct characterization of DTM error requires methods that also describe bias, consider spatial autocorrelation of residuals, and deal with outliers which tend to be prevalent in digital elevation data (Hohle & Hohle, 2006; Zandbergen, 2008). Studies that attempt to characterize the spatial behavior of error often conclude that RMSE is not a comprehensive enough means to report on error that is often biased and non-normal in nature. These include the characterization of error bias in LiDAR (Zandbergen, 2011; Aguilar, 2008; Oksanen and Sarjakoski, 2006) in which non-normal behaviour of spatial error is common and resistant to representation with a single statistic such as RMSE. Non-stationary behaviour of error is also found to be common and influenced by land cover. It is understood that non-open terrain influences the accuracy and distribution of error in DTMs and error can be introduced by interference in the data collection method, and/or from data processing though filtering, gridding and point classification manifesting in high kurtosis and skewness resulting from systematic error (Flood, 2004, Aguilar, 2006). In vegetated areas, overestimations of terrain are common due to the high frequency of nonground objects such as fallen logs and low brush being included in interpolation (Clark, 2004). Both the NSSDA and ASPRS, by incorporation of NSSDA guidelines, provision for the non-normal nature of error through the recommendation of different statistical reporting according to land cover (ASPRS, 2014). In non-open terrain, where error is assumed to be non-normal, supplemental vertical accuracy is calculated using the 95 th percentile. This is a non-parametric based statistic which can be used in non-normal distributions, and states that 95% of the values will have errors of equal or lesser value, and that 5% of errors will be higher. Fundamental Vertical Accuracy is considered to be the accuracy in open terrain, represented by the 95% confidence level of RMSE, which equates to RMSE x 1.96 (ASPRS, 2014). The error is assumed to be Normal or close to Normal in the case of fundamental Vertical Accuracy. However, studies show that this assumption is not always valid and that non-normal behaviour of LiDAR data in open terrain is common (Zandbergen, 2011; Oksanen and Sarjakoski, 2006; Aguilar, 2011; Hodgson et al., 2008). In attempt to better characterize non-normal error, Aguilar 2011 recommends the use of confidence intervals based on the RMSE; Zandbergen 2011 reports percentiles, confidence intervals, and RMSE of both full and trimmed datasets. Many of these studies investigate the fundamental and supplemental accuracy of LiDAR derived data and it is well understood that accuracy varies with landcover (Bowen and Walermine, 2002; Adams and Chandler, 2002; Zandbergen, 2011). Studies that focus on the error distribution of UAV DTMs frequently use RMSE as a measure of accuracy in combination with other statistics, such as Standard Deviation (SD) (Rock et al, 2011), confidence intervals (Bazzoffi, 2015), t tests to measure differences between means (Bazzoffi, 2015; Ritter, 2014) or the coefficient of variance from the error distribution (Aguilar, 2007). ASPRS recommends the reporting of all additional descriptive statistics such as mean, median, standard deviation, kurtosis, skewness and quartiles to supplement accuracy statistics (ASPRS, 2014). Hohle and Hohle (2009) attempt 24

25 to find a better characterization of non-normal distributions of residuals in remote sensing data through robust means of deriving accuracy statistics in spite of factors such as high kurtosis or skew that are influencing the distribution. They recommend calculating accuracy directly as quantiles of the error distribution and use the nonnormal error of DTM s produced by LiDAR and aerial photogrammetry to exemplify this. Other studies that investigate the error of remote sensing derived DTMs include the classification of the residuals of elevation, slope and aspect from the Advanced Space borne Thermal Emission and Reflection Radiometer (ASTER) and the Shuttle Radar Topography Mission (SRTM) into error classes for topography in Nepal (Miles 2013); the error analysis of ASTER and SRTM data in the Himalayas using a high-resolution aerial photogrammetric DTM as reference (Singh et al, 2015); and the assessment of the accuracy of ASTER and STRM data in Idaho (Garnett, 2010). In these studies, additional parameters are used to describe error which include RMSE and detailed error analysis according to elevation error, aspect and slope classes. From the literature, it is understood that RMSE is too simplified a statistic to represent the error of DTMs, especially when the distribution strays from Normal. Non-Normal behavior can often be attributed to one of the following reasons: the frequent occurrence of blunders or large errors, large positive spatial autocorrelation in residuals, and non-stationary behavior in the residuals (Zandbergen, 2008). Regression is another technique that can be employed to investigate the accuracy of elevation models and determine the predictive value of a DTM for determining true elevation. Gorokhovich and Voustianiouk (2006) use bilinear regression to investigate the relationships between true and predicted elevations in remote sensing data as well as to model the influence of elevation and elevation error on slope and aspect and their residuals. Mancini and co authors (2013) use bilinear regression to model the correlation and relationship between TLS and UAV elevation models. However, regression depends upon normal distribution of residuals, again emphasizing the importance of examining data to determine the character of the distribution before assuming statistical indicators to be accurate. Ordinary Least Squares (OLS) Regression models can be run in GIS software and incorporate measures for dealing with the spatial tendencies of errors, including spatial autocorrelation and non-stationarity. OLS regression is a global model that requires the residuals to follow a Normal Distribution and returns a Jarque- Bera statistic indicating the probability of a Normal Distribution of the residuals. If this p value is statistically significant, the Null hypothesis of a Normal Distribution is rejected. OLS regression requires stationarity and an absence of spatial autocorrelation in the residuals; the data cannot behave differently in different parts of the study area, and each sample must be considered independent. The Moran s Index statistic indicates spatial autocorrelation, and run on the residuals, can help to determine if spatial autocorrelation is contributing to the significant Jarque-Bera Statistic. CHAPTER 3 METHODOLOGY 3.0: Study Area The study area is located at the east end of the Oak Ridges Moraine in Southern Ontario, Canada ( N, E), a prominent landform of glacial deposition that serves as an important aquifer for the metropolis of the Greater Toronto Area (GTA). This area was settled in the mid 1800 s, and is now predominantly an agriculture zone, important to the production of corn, soy, and wheat. The study area consists of 11 Hectares (Ha) of agricultural crop land (Figure 1), unplanted at the time of data acquisition. It is divided by a small watercourse and two prominent bands of disturbed tree cover consisting of Acer negundo (Manitoba Maple), Pinus strobus (White Pine), Quercus rubra (Red Oak), Quercus alba (White Oak), and Betula papyrifera (White Birch). It exhibits gentle 25

26 terrain variability with a number of small undulations, especially in the zone where agriculture transitions into Riparian, and experiences a rise of about 40 m within its extent. It was chosen as a basis for this project due to accessibility of the location for field efforts, as well as to streamline the process of obtaining an exemption from a Safe Flight Operations Certificate (SFOC) from Transport Canada. Because it is private, rural land, it offered a low risk flight zone free from traffic, residences, and civilians, which made adhering to the provisions stated in the SFOC exemption straightforward. Transport Canada has specific regulations in place to manage the safe use of UAV s; these will be discussed to a greater extent later in this paper, but in brief, for any non-recreational use of UAV s, pilots must obtain an SFOC proving, among other stipulations, operations within Visual Line of Sight (VLOS), 20 km away from any classified airspace, with provisions for protecting third party injury of utmost importance. An exemption from an SFOC can also be obtained, for example in the case of a single low-risk flight to collect research data, as is the case in this paper. However, even in the case of an SFOC exemption, an agreement is made between the pilot and Transport Canada to operate only under controlled and low risk circumstances. This made choosing a low risk study area critical; the situation of this study area, on private, rural land, which also exhibits the terrain variability necessary for stratified testing, fulfilled these purposes. In addition, the access to the study area needed to be convenient and unrestricted, as multiple field efforts were made to gather data over the course of six months. In terms of terrain attributes, it offers varying slope and aspect, but not to the extent that these attributes interfere with the placement of accuracy checkpoints, which should be on flat or gently sloping land (ASPRS, 2014). The size of the study area was dictated primarily by the limits of the Phantom 3 UAV; it is the maximum geographic size that the Phantom 3 can cover on a single flight. The flight used approximately 65% of the Phantom s battery, and the recommendation from DJI is that the aircraft should return to home when it has drained 70% of its battery, to ensure that it lands safely and in a controlled manner. Larger geographic extents could be covered, but would require additional batteries and processing to stitch together footprints and rectify changes in illumination resulting from flights occurring at different times. Figure 1: Study Area, Hamilton Township, Ontario, Canada 26

27 The transition zone around the riparian is a source of potential surface runoff, when erosion at the edges of the agricultural field creates pathways where contaminants could potentially be introduced into the watercourse. These highly exposed areas are vulnerable to the practice of fall tilling, following which the field is typically free of snow cover until later in the winter. Soil exposed to the elements is at greater risk for erosion, and when combined with slope breaks flowing into a watercourse as well as disturbed Riparian vegetation, these areas are potential point sources of contamination. In addition, these areas are likely to change more quickly over time due to influence by agents of erosion, and as a result are likely not reflected in less current elevation models. Studies in UAV Based Riparian Mapping are limited, but as UAV photogrammetry techniques advance, some studies are showing that the UAV can provide a unique dynamic link to river morphology and bank stability (Tamminga et al., 2015). 3.1: Data Sources Two datasets were used for comparison to RTK GNSS survey point elevations in this project; the UAV high resolution (0.08 m) DTM and the SCOOP (2 m) DTM. UAV Data: The Phantom 3 UAV collected 337 geotagged photos to achieve coverage of the Study Area. These photos were aligned and processed in digital photogrammetric software (Agisoft Photoscan), and exported as a LAS point cloud. The LAS data was then imported into a GIS (ESRI, ArcMap) and used to build the UAV DTM. The method of interpolation used to fill in the gaps between points in the cloud; essentially to guess the unknown elevations from the known; introduces variable error depending on the method used. An attempt was made to minimize this error by interpolating a raster directly from the raw LAS files using ArcMap s LAS to Raster function (LAS Tools). A Triangulated Irregular Network (TIN) was also created as an initial spatial product. No interpolation is required in the construction of a TIN, which is built on true point locations joined by lines. This TIN then offered a baseline for accuracy to ensure that gross interpolation errors were not occurring in the process of raster creation. SCOOP Data: The South-Central Ontario Orthophotography Project (SCOOP, 2013) is part of multi year acquisition of orthoimagery which aims to provide a large scale, high quality dataset for provincial ortho imagery and its derived products, including elevation data. This data is available from the Land Information Ontario Geoportal (LIO), but requires transfer though a hard drive mailed to the OMNR. There is also a 2m raster DTM version of the SCOOP data available for direct download from the LIO site. This data is currently the highest resolution elevation data available in Southern Ontario free of charge, and is a logical choice to fulfill the elevation data requirement for many local natural resource and agricultural related projects. The 2 m DTM is a generalized terrain product that has undergone a process of classification to remove surface objects from the raw LAS data through a steam roller algorithm in unspecified photogrammetry software by an imagery contractor. The DTM was delivered to MNRF as derivative product as part of the imagery contract (OMNRF, LIO 2015). A DSM is also available that represents above ground objects as well; however, it is important to be aware that the DTM does not represent a full bareearth elevation surface. While the steam-rolling algorithm has allowed for some raised features to be reduced closer to bare-earth elevations (e.g. small buildings, small blocks of forest cover), many features are still raised above ground surface, such as larger buildings, larger forest stands and other raised features. For the study area of this project, it is evident in some areas that vegetation has not been completely removed; however, the buildings in the study area do appear to have been captured by the surface removal process. The SCOOP dataset has a published accuracy assessment which reports NVA accuracy of 0.36 m at 95% confidence and 2.86 m VVA 95 th percentile accuracy (OMNR SCCOP 2013 DTM-DSM Accuracy Assessment). 27

28 The data produced by and used in this project is summarized in table 1. From the DTM surfaces, derivatives of slope and aspect were also extracted from each dataset. Table 1: Dataset Name and Source Name Source Resolution UAV DTM UAV LAS point cloud converted to raster in ArcMap 0.08 m UAV TIN TIN created from UAV LAS point cloud in ArcMap N/A SCOOP DTM Land Information Ontario Geoportal (OMNR, 2013) 2 m 3.2: Terrain Data Acquisition 3.2.1: UAV Flight The calculation of requirements for UAV image volume, overlap and resolution are controlled through settings within most Flight Control Applications. Many different apps are readily available for download and provide the map frame on which photo acquisition plans are based. For the purpose of this study, a few different apps were tested for redundancy and ease of use, and the final data was collected using the DroneMapper Pilot App. This IOS app is free for download, and calculates required front and side overlap, image quantity and frequency, flight speed and GSD for a given geographic area and flight altitude. Flights were undertaken using Altizure, but the connection with the UAV was unreliable and the app was prone to crashing. Flights were also tested with the PIX4D app, which initially resulted in up to 100m discrepancies in elevations without ground control. This app was not tested with ground control, because DroneMapper proved to reflect lower raw elevation errors, as well as providing a user-friendly interface, and was designed specifically to be integrated with the DJI Phantom series. The data collection platform is a DJI Phantom 3 Advanced Quadcopter (Illustration 3) carrying a Sony EXMOR 1/ megapixel camera with lens specs of f/2.8, with a 94 0 Field of View. Illustration 3: DJI Phantom 3 Advanced Quadcopter 28

29 The sensor size is 6.16 mm wide and 4.62 mm tall. The DJI Phantom 3 is a lightweight UAV retailing for around $900 CAD. It has a battery life of approximately 25 minutes and can reach a trajectory of 16 m/s, though this study was flown at 9 m/s. The Phantom 3 easily achieves a GSD of 2 cm when flown at 50 m altitude. The DJI Phantom is a popular model for both recreational users and environmental monitoring. It is ready to fly out of the box, and DJI produces in depth tutorials on piloting the UAV, processing UAV data, and producing terrain products from the photos. All of these components conspire to make the DJI Phantom series a logical choice for individuals or small research groups looking for the functionality of a mapping drone at a reasonable cost. Its primary drawback is a shorter battery life than fixed wing survey models, a factor that was improved upon with the release of the Phantom 4. The data acquisition occurred in a series of 12 flights between February and April, 2017 (11 for testing purposes and fine-tuning parameters, and a final refined flight for data collection in April), in leaf off canopy conditions. Leaf off conditions are recommended by the OMNR for photo acquisitions, due to the visibility of ground objects that is not present with full vegetative cover. However, the window for this type of acquisition is narrow, as the DJI Phantom battery drains quickly in the cold and is not recommended for use below 0 degrees Celsius. Therefore, flights had to occur strategically in the window between cold temperatures and green up. Parameters tested for best performance included overlap, time of day, altitude and flight speed. The final dataset was collected on April 14 at 10:30 AM on a slightly overcast day to minimize shadows. The flight altitude was 50 m and a total of 337 photos were captured in 15 minutes with a side lap of 70% and a front lap of 80% to minimize edge distortion, according to the minimum recommendations from DJI which recommend at least 60% and 70% side and front overlap. More overlap produces better results, with the downfall of longer processing times to align photos and construct terrain products : Ground Control Points GCPs are generally of 2 types: 1) Photo Identifiable (any ground feature that is fixed, such as a hydro pole or tree; can be surveyed post flight). 2) Pre-marked (must be created pre-flight). For the purpose of this study, Pre-marked GCPs were laid out on a UTM grid in ArcMap according to GCP recommendations in AgiSoft Photogrammetric Software, which recommends GCP s per study area, regardless of size. Due to the GNSS based aerial triangulation process that takes place, a low density of GCPs are typically required. However, more GCP s are generally correlated with higher accuracy, but must be balanced by the effort required to place them (Rock et al., 2011). Twenty GCP s were chosen for this study area, to strike a balance between coverage and field effort. GCPs were distributed to give coverage across the entire study area, especially around the boundary of the geographic extents, as recommended by AgiSoft (Figure 2). These GCPs must be of suitable size and colour to be clearly visible in the UAV imagery; often painted aluminum disks are used with a clearly defined centroid where Horizontal (X, Y) and Vertical (Z) values are measured. The GCPs for this project were constructed from Vinyl LP Records, spray painted bright pink and blue and labelled GCPs were staked to the ground through the center of the LP, and this stake served as the location for RTK GNSS measurement. GCPs are integrated in post processing to serve as tie points for georeferencing in photogrammetric processing. Each GCP is identified manually in the UAV photos, and then photos are filtered by each marked GCP to correct any automated placement error. When all GCPs are in place, geographic location and elevation information are imported as a text file derived from GNSS survey which identifies GCP ID, Lat/Long in Decimal Degrees, and Elevation. For each GCP, RMSE values are automatically calculated to assess the discrepancy between UAV and true values (Figure 3). 29

30 Figure 2: GCP (red) and Accuracy Checkpoint (green) Layout. The VVA is represented by the inner polygon outlined in red. Figure 3: AgiSoft GCP RMSE Calculation. 30

31 3.2.3: GNSS Survey RTK GNSS Survey was completed using an Ashtech ProMark 800 (RTK rover) with Ashtech Mobile Mapper 10 (data logger) connected to the Can-Net Virtual Reference Station Network (cellular RTK corrections network) and to the U.S and Russian satellite Systems, GPS & GLONASS. This model uses a roving handheld antenna, and does not require a base station for operation. The GNSS device was set to an accuracy tolerance of 10 cm, so no points would be collected over this threshold without a low accuracy warning. However, the device averaged 18mm-3cm accuracy on all points. RTK GNSS survey is a logical choice for ground truthing a small study area. Other options include Terrestrial Laser Scanning (TLS) and LiDAR, but neither were available for this study. In addition, RTK GNSS usually offers higher accuracy than both TLS and LiDAR, is relatively straightforward to carry out by an individual, and has a lower cost association than both TLS and LiDAR. The ASPRS 2014 Standards require the accuracy of a dataset to be determined by an independent source of higher accuracy, and that this source be at least three times the required accuracy of the data product being produced (ASPRS, 7.9 Checkpoint Accuracy and Placement Requirements, 2014). The GNSS survey satisfied this component of the Standards, obtaining geospatial coordinates of checkpoints at < 3 cm of accuracy. Accuracy assessment points were placed with an even distribution over the VVA and NVA types (figure 1). Points were taken according to a 50-m grid, laid out as follows: Points were placed at 50 m grid intervals in ArcMap and the shapefile was uploaded to the GNSS Device. The Point of Commencement (POC, point 1) was tied by tight chain and compass from a known RTK survey point at the corner of the property line. All points were laid out in succession using a 50-m forestry tight chain and compass, and adjustments were made at each point by GNSS receiver to ensure the point lined up with the grid, and that any error associated with compass bearing deviations was not carried over to the next point. Each plot center was marked with a stake and ribbon for visibility. GCPs were laid out at the same time with independent and more deliberate placements to ensure good distribution over the study area and visibility from the air. The GNSS survey followed the course of laid out plots and took points at each accuracy assessment plot and GCP center, resulting in a total of 45 accuracy assessment points and 20 independent GCPs. The 2014 ASPRS Accuracy Standards (ASPRS Positional Accuracy Standards for Digital Geospatial Data, 2014) outline requirements for accuracy assessment methods and horizontal and vertical accuracy checkpoints. Due to expected variance in accuracy of remote sensing methods over differing land cover types, the study area was stratified according to ASPRS Standards into Non-Vegetated Area (NVA) and Vegetated Area (VVA). In accordance with ASPRS Standards (ASPRS 4.0 Accuracy Assessment Procedures) a minimum of 20 checkpoints are required for a NVA study area, with a minimum of 5 points in the VVA stratum. Check points in each stratum should also reflect the ratio of NVA: VVA. In this case, 8/45 points composed 18% of checkpoints in the VVA, and the VVA similarly composed 2.1/11.8 Ha, or 18% of the study area. Stratified testing against GNSS survey test points will determine if land cover type influences elevation data capture accuracy, and if errors in accuracy exhibit spatial patterns based on vegetative cover : Slope and Aspect Field Survey A second field effort was undertaken to verify slope and aspect measurements. At established GNSS survey points, a Suunto clinometer was used to measure slope, and a compass bearing was taken in the direction of the steepest downhill slope to measure aspect. 31

32 To measure slope with the clinometer, a distance of 2 m was decided upon. This is the resolution of the SCOOP data, and also a reasonable distance over which to calculate slope. While the UAV DTM can produce a slope map of 0.08 m resolution which has many applications such as micro typing terrain, field verification with a clinometer at an 8-cm distance is unrealistic, and in addition, most applications relying on slope inputs need to generate an idea of terrain variability over larger distances than sub meter measurements. To field verify the slopes produced by both the UAV DTM and the SCOOP DTM at two meters, the greatest downhill slope was identified at each known GNSS survey point. Because it can be difficult to determine the direction of greatest slope on gently undulating terrain, the clinometer was used to confirm the greater degree of slope that appeared to be close in measurement, and at the same time measurements were taken at 1 m on both sides of the GNSS checkpoint and marked with stakes. These stakes were also marked at 1 m high with tape. Clinometer readings were than taken in degrees, using a Suunto clinometer positioned at the 1 m high on the upslope stake, aimed at the 1 m marker on the downslope stake, with the GNSS survey point in the middle. To measure aspect, this downslope direction was also measured with a compass bearing lined up with the stakes. These slope and aspect values were recorded and added to a feature class in ArcMap to be used for slope and aspect verification. This method was determined to be the most applicable method to generate accurate measurements in what is generally gently sloping terrain. This method is also what is recommended by United States Geological Survey Methodology (USGS). Some studies (Weih and Mattson, 2004; Weih, 1991) field verified slope by shooting a clinometer reading away from the checkpoint both upslope and downslope at a distance of 20 feet on either side of the point, and the slope of the point was considered to be the average of the two. However, in relatively flat terrain and over a short distance, it is difficult to determine visually the direction of greatest descent, and slope had to be confirmed from several angles before a maximum descent was identified. 3.3: Digital Photogrammetry Workflow There are various options available to process and correct the imagery produced by a UAV. Pix4D and AgiSoft Photoscan are two of the more highly regarded and utilized technologies. ESRI has recently introduced Drone2Map, which integrates into the ArcMap platform and offers a range of photogrammetric processing options. Professional versions of Pix4D and AgiSoft offer GCP incorporation and outputs of LAS, DTMs, and DSMs. Basic versions offer photo alignment but without the use of GCPs. Based primarily on availability, the software used for the purposes of this study was AgiSoft Photoscan Professional The DTM was exported in LAS format using basic workflow as follows: 3.3.1: Image processing and Bundle Block Adjustment A detailed description of the process of Bundle Block Adjustment can be found in Turner et al., 2012 and Westoby et al., Essentially it is a type of aerial triangulation performed on a block of imagery to account for relative orientation of the photos to each other. Absolute Orientation refers to the process of levelling the model with respect to a reference plane or ground control points. If direct georeferencing is to be used, the UAV must be equipped with an internal IMU and GNSS receiver capable of collecting data with sufficient accuracy to use in bundle adjustment. If this can not be achieved, Ground Control Points need to be incorporated into the model. Direct georeferencing without GCPs is useful in inaccessible areas and can still produce DTMs with acceptable accuracy (Li et al., 2011; Blaha et al., 2011), but this accuracy varies widely with the model of UAV in use. A sparse point cloud is generated from the Bundle Block Adjustment from which noise and other obvious errors can be edited prior to dense cloud construction. 32

33 3.3.2: Dense Geometry Reconstruction and Inclusion of GCPs If GCPs are established and measured with GNSS survey they can be used in the Block Adjustment to root the model in real world coordinates. GCPs can also be used as weighted factors in conjunction with internal UAV measurements to georeference the data. With large amounts of overlap, the edges of the photos, which are subject to more distortion, can be excluded from the adjustments (Turner et al, 2015). GCPs are added after the sparse cloud is created. Markers are identified in a single photo, and then the photos are batch filtered by markers so that adjustments between actual marker location and predicted marker location can be made; this process is repeated for each GCP. GCP location coordinates and elevation values are imported into the pre-placed markers in AgiSoft through a text file specifying a common point name, x and y coordinates in Decimal Degrees, and Elevation values. This text file was exported from the GNSS RTK Survey feature class attribute table in ArcMap. RMSE values are automatically calculated and reported as individual point error and overall dataset error. If outliers of extraordinary error are observed, these points can be manually edited or removed from the calibration process. Camera optimization weights the GCPs with a user defined accuracy value, determined by the accuracy of the GNSS device; in this case, an average of 3 cm accuracy. Based on the mesh generated by this process, a dense cloud can be constructed. This is computationally intense and processing time is based on computer hardware specs; quality settings range from low to very high, and determine the resolution of the output. Outputs of this dense reconstruction can be LAS files, a DSM, DTM, or orthorectified mosaic. For this data, the dense reconstruction from 331 photos processed on the High Accuracy and Aggressive Depth Filtering settings took approximately 7 hours of processing time (Figure 4). Figure 4: 3D Model showing locations of aerial photos (overlap not shown for visual purposes) and GCP Placement. The Phantom 3 Professional UAV collects data at an average GSD of 2 cm/pixel represented as an Unclassified Point Cloud. Surface objects need to be removed to generate a bare earth Digital Elevation Model from the Digital Surface Model. To undergo statistical analysis in ArcMap, a Classified Point Cloud was created in AgiSoft Photoscan (figure 5) through the Classification Tools to remove points beyond a specified search threshold for values that are uncharacteristically dissimilar to neighboring points, indicative of above ground objects. The resulting point classes can be further edited manually. When an acceptable degree of filtering is met, the DTM is regenerated using only the Ground Point Class, and/or the LAS point cloud is exported based on only the Ground Class. 33

34 Figure 5: 3D model with Surface features (buildings, trees) evident in the upper figure; gaps in the riparian show areas where trees have already been classified and removed by a previous iteration. The lower figure shows point classification in AgiSoft; Ground class (brown) and Surface features (white); in this case the filtering was slightly too aggressive, and some ground objects are misclassified as surface objects, such as in the upper right corner. Manual editing is used to reduce these errors and refine the model further : DTM Construction in GIS Software In ArcMap (ESRI), a LAS dataset was created (ArcMap, LAS Tools) and UAV Classified ground LAS files exported from AgiSoft Photoscan were imported. The LAS files should remain in their native projection upon export from Agisoft and re-projected only in ArcMap if necessary. While Agisoft allows for export of files in any 34

35 chosen projection, error in elevation values occur if the LAS files are projected in anything other than their native format, in this case WGS 84. If re-projection is required, this step can be taken in a GIS. From this LAS dataset, a Triangulated Irregular Network (TIN) was created from UAV Point Cloud (LAS to TIN, LAS Tools, figure 6). A temporary raster and Hillshade were derived from the TIN (TIN to Raster, 3D Analyst; Hillshade, 3D Analyst). The hillshade gives visibility to problem areas so that they can be identified by digitizing polygons around them, and removed using LAS Clip (Isenberg, 2014). These problem areas tend to be relics of surface features that were not entirely removed during point cloud classification in AgiSoft. Visible in figure 7 below, in the South west corner, some of these relics are evident over buildings that were not completely removed. In addition, errors are evident around the perimeter of the study area where photo overlap is insufficient. The large error in the south west is evident of a few surface points left on the perimeter of a gap in the data where a large building was removed in surface filtering. The computer attempts to fill this gap which is mostly absent of data, resulting in a large blur; it is corrected by removing the few remaining points on the perimeter. Figure 6: TIN derived from UAV LAS Point Cloud 35

36 Figure 7: Temporary Hillshade revealing problem areas over surface features remaining after initial Point Cloud Classification Editing of the point cloud can take many iterations to achieved a desired end result. There is a wide range of point cloud classification software available to remove surface features and classify LAS data, all with variable results and user involvement. For the purpose of this study, a reasonable effort was made to remove surface objects, first by automated filtering in Agisoft to remove all but ground points, then through additional manual editing in Agisoft, and manual point editing in ArcMap. An attempt was made to keep manual editing time reasonably low to maintain the relatively low-cost factor associated with a UAV DTM. LiDAR has an advantage to UAV terrain mapping over vegetation; last returns penetrate a canopy and are capable of measuring multiple levels of vegetation and timber heights (Dehvari, 2014). UAV and Fixed wing RGB sensors are notoriously problematic for mapping elevation over vegetated areas, where range of error is unpredictable and generally does not follow a Normal Distribution (ASPRS, 2014). Many terrain products produced by one of these measures maintain a disclaimer regarding the likely existence of surface object remnants that were not completely removed during processing. When a point cloud of adequate classification has been produced, descriptive statistics, confidence intervals, percentiles and RMSE can be calculated from differences between accuracy assessment values and UAV predicted values. Elevation values were extracted from the TIN (Add surface information, 3D Analyst) and appended to the GNSS RTK survey point feature class. From this, the RMSE for the entire dataset, as well as the stratified NVA and VVA was calculated in ArcMap s field calculator by adding and calculating fields for Elevation Differences (UAV-GNSS/SCOOP-GNSS), squared differences, and sum and square root of squared differences to produce an RMSE value for each category. In addition, descriptive statistics and distribution plots can be viewed and analyzed (Geostatistical Analyst). To answer questions about distributions of errors over the entire dataset, and to produce the derivatives of slope and aspect, it is necessary to produce an interpolated raster surface from the LAS points (figure 8). Raster surfaces allow map algebra to take place, and because they offer a matrix of interconnected cells, they are the logical choice for modelling a wide range of terrain driven processes (Miles, 2013). Interpolation methods vary, and a range of these are available in GIS and depend on the type and character of the data being modelled. For example, the spline function fits a surface through fixed points and is appropriate for continuous data such as elevation and temperature. However, LAS data challenges most standard interpolation methods in ArcMap, which were designed for use with lower density data. UAV and LiDAR Points clouds are typically very high density with 36

37 millions of points; attempting to interpolate a surface using standard methods like Spline or Inverse Distance Weighting (IDW) will typically fail. To deal with this, point clouds need to be heavily thinned, or another interpolation method must be used. Thinning is undesirable because the reduction means omitting points in the cloud and potentially mis-representing the data. The functionality offered by the LAS to Raster Tool in ArcMap is therefore a logical choice for DEM processing; it creates a raster surface using the entire LAS dataset, interpolating directly from the points. A DTM in raster format was generated from ArcMap s LAS to Raster Tool at 0.08 m resolution (parameters of this tool suggest a sampling value of 4x the original point spacing of 0.02m). From the raster, RMSE was calculated again to determine if there is any change in accuracy resulting from the process of interpolation used to create the DTM (Extract Values by Points, Spatial Analyst). In this case the TIN and Raster displayed an equal overall RMSE of 0.15 m, so the interpolation process was assumed to be representative of the point cloud. When an acceptable DTM was generated, an error surface between the SCOOP and the UAV DTM was produced using the Minus Tool in ArcMap s Raster Calculator, with the output resolution set to.08m, the minimum of the inputs, so that no error in the UAV DTM would be minimized by resampling to the 2m resolution of the UAV DTM. Figure 8: 3D UAV DTM in Arc Scene (ESRI) from west-east (left) and east-west (right). The derivative surfaces of slope and aspect were calculated from the UAV DTM (Spatial Analyst, Surface Toolset), at 0.08 m and 2 m resolution. The high-resolution slope raster helps to highlight micro changes in the terrain, and sources of error such as surface objects that were not fully removed; however, the field verification of slope at 10 cm was impractical with a clinometer, so a slope of 2 m is used for accuracy assessment through field verification and comparison to the SCOOP data. Comparisons of aspect are also conducted at the maximum of the inputs, 2 meters. The cell statistics contained in the raster provide the basis for the Slope tool to calculate the maximum rate of change in value from the cell to its 8 surrounding neighbors. Aspect is calculated as a by product of the slope tool fitting a plane to the z values around the processing cells (ESRI). The direction the plane faces is the aspect of the cell. The output of the slope can be in degrees or percent rise; for this study degrees was used. Accuracy assessment of the Slope and Aspect measurements was also conducted through RMSE calculations, percentiles and descriptive statistics. Feature classes were created for both derivatives and slope and aspect measurements were extracted from both DTMs using the Extract Values to Points Tool. Field data was added to additional fields recording the clinometer slope values and compass bearing in the direction of the measured 37

38 slope. RMSE values were calculated through the differences between field survey values and computer-generated values, and stratified by VVA and NVA as they were for elevation, and the same reporting methods of descriptive statistics, distribution plots, percentiles and confidence intervals are applied in addition to RMSE. 3.4: Statistical Analysis and Reporting The point error descriptive statistics of Elevation, Slope and Aspect provide an initial means for investigating patterns in the data such as outliers and clustering of highs or lows in the errors. Some degree of the statistics become evident when calculating RMSE, and the distribution plots and statistics describing the structure of the data (Mean, median, skew, kurtosis, standard deviation) were analyzed in ArcMap s Geostatistical Analyst. RMSE as a single indicator of accuracy is insufficient; in addition to RMSE, descriptive statistics were generated for each dataset, and histograms and Quantile Quantile (QQ) plots were analyzed to characterize the distribution and any bias that may be present in the form of under or over predictive tendencies. The data was also tested for Normalcy using the Shapiro Wilk and the Kolmogorov-Smirnov with the Lilliefors Correction Normal tests. These tests are considered to be the most robust for detecting non-normal behaviour in small samples (Zandbergen, 2008). If samples returned significant probabilities of exhibiting non-normal distributions due to outliers, outliers were removed based on absolute magnitude, and re examined and tested again. A Normal Distribution is a requirement for the RMSE indicator to be considered robust, and is a requirement for many other statistical parameters such as regression modelling. Accuracy reporting statistics were determined for elevation (Z, m), slope (degrees) and aspect (bearing) by VVA and NVA statistics for the following categories: A) RMSE at 100%: Root Mean Square Error for the full datasets. B) RMSE 95%: Root Mean Square Error of 95% of the values, omitting 2 of the outliers of highest magnitude, on the NVA datasets. C) RMSE 90%: Root Mean Square Error of 90% of the values, omitting 4 outliers in the NVA and 1 outlier in the VVA. D) 95 th percentile: the maximum error value of 35 points, omitting 2 points with the highest error in the NVA. E) 90 th percentile: the maximum error values of 33 points in the NVA and 7 points in the VVA, omitting 4 points of highest error in the NVA, and 1 point of highest error in the VVA. F) 95% confidence level: 100% RMSE x Histograms, QQ plots and Normalcy Testing were investigated to report through characterizations of error distributions, and descriptive statistics of mean, median, skew, kurtosis, maximum and minimum values and 1 st and 3 rd quartiles are reported. Cluster and outlier mapping (Spatial Statistics Toolbox) was implemented to look for outliers and clustering of high and low values of error, showing point locations where the UAV under or overpredicted elevation in relation to the GNSS values. In the case of elevation values, autocorrelation exists in the data; near values are usually closer to each other than they are to values far away, and this spatial autocorrelation needs to be accounted for. However, spatial autocorrelation in the errors can reveal clustering of high or low values of errors in certain areas, such as over a certain land cover type or on a particular aspect or feature, or alternately, reveal total randomness in the patterns of error. Relationships between the error values of Elevation, slope and aspect might be expected; elevation errors could produce incorrect slopes and corresponding incorrect aspects (Weih & Mattson, 2004). These relationships can be modelled though regression analysis. Spatial regression tools integrate information about the spatial structure of the data into their processing, and model relationships among variables associated with geographic features (ESRI, Modelling Spatial Relationships Toolset). The strength of these relationships can also be measured through correlation coefficients to determine the magnitude of influence elevation error has on slope and aspect. Ordinary Least Squares (OLS) Regression tool generates outputs including a map of the regression residuals and a summary report with coefficient information and diagnostics. Bilinear regression was used to model true elevation, slope and aspect values against model values, elevation error values against slope and aspect error values, and slope error values against aspect error values, for both the UAV and SCOOP datasets (ArcMap, Spatial Statistics, 38

39 OLS Regression). In order to model elevation vs. slope and aspect values, and slope vs. aspect values using regression, values were normalized in ArcMap so that units of comparison would be equal. All elevation, slope and aspect values were normalized by their relative maximum value, so that the scale of error could be represented as a decimal between 0-1 for all attributes. This was achieved by adding a Normal field to each feature class in ArcMap to hold Normalized values, calculated by dividing all of the values in each dataset by their respective maximum value. For each stratified dataset of elevation, slope and aspect error, statistical analysis is based on the following procedure: visual analysis of the point error distributions and error surfaces, descriptive statistics and distribution plots, normal testing, regression modelling, accuracy reporting and significance testing of difference between sample means. Results are presented in Chapter 4 with a summary of each statistical method applied, and a more through analysis of error distributions based on distribution plots and a detailed investigation of descriptive statistics and Normalcy Testing is discussed by stratified dataset in Chapter 5. CHAPTER 4 Results 4.0: Descriptive Statistics and Distributions of Error Descriptive statistics provide an initial overview of potential bias (under or over estimates) and allow inferences to be made about the distribution of the data. Tables 2-4, below, provide descriptive statistical summaries for all of the DTM datasets based on comparison to checkpoint accuracy. Central tendency is measured by mean, median and mode, and dispersion is measured by standard deviation, range, kurtosis and skew. In a Normal dataset, the mean and median are close together near 0, standard deviation is low, and minimum and maximum values are balanced. A perfect Normal Distribution has a skew of 0 and kurtosis of 3. Deviations from these statistics indicate Non-Normal distributions; high maximum and minimum values, high kurtosis, high skew, or discrepancies between means and medians can be indicators of outliers which influence the data away from a Normal Distribution. Based on descriptive statistics, none of the NVA elevation data appears to be normal at 100%, and in the VVA, UAV elevation and aspect appear to be the only non-normal distributions at 100% with the remaining VVA data apparently Normal. Most of the Elevation (table 2), Slope (table 3) and Aspect (table 4) values in both stratums suffer from one or a combination of factors influencing their distributions; high kurtosis, high skew, and high maximum and/or minimum values are characteristics of most of the 100% data. These distributions appear Normal when trimmed to either 95%, 90% or 85% by removing the outlier(s) with the highest magnitude of error. A trimmed data set to 95% normalizes most distributions in the NVA, but at this level the SCOOP elevation and both SCOOP and UAV aspect appear to be resistant to a Normal distribution, and improve at 90% with the removal of 4 outliers. In the elevation datasets, skewness and high kurtosis are problematic for all data except the SCOOP VVA data which shows only moderate negative skew and a kurtosis of 1.58 (table 2). Kurtosis is often indicative of outliers, and as the datasets are trimmed, kurtosis values improve in all cases. Similarly, skew also improves with the omission of influential outliers. High kurtosis and skew are evident in the descriptive statistics of all of the full error datasets of Slope (table 3) and Aspect (table 4) as well. As is the case with the elevation data, these statistics improve in the trimmed data. All full datasets also show discrepancies between median and mean, and maximum error values higher than the mean in all cases. Maximum error values are higher than their respective standard deviations in all cases except the UAV VVA Aspect, where standard deviation is much higher than the 100% maximum value due to the presence of one very large outlier (table 5). Descriptive statistics are 39

40 interpreted for each individual sample dataset in Chapter 5, along with an investigation of distribution plots, with reference to tables 2-5, an interpretation of these plots in relation to Normal Testing Scores, and a consideration of the factors that may be driving error through cartographic assessments of outliers and regression modelling. However, this summary shows how the presence of influential outliers is characteristic of all of the sample data, and a similar trend of improvement in descriptive statistics is evident across all datasets as outliers are removed. Table 2: Descriptive Statistics of SCOOP and UAV DTM Elevation (Z) Error datasets (Unit = meters with the exception of kurtosis and skew for all data). UAV DATA SCOOP DATA NVA (N=37) NVA (N=35) VVA (N=8) VVA (N=7) NVA (N=37) NVA (N=33) NVA(N=31) VVA(N=8) VVA (N=7) Mean Median ST DEV Minimum Maximum Skewness Kurtosis st quartile rd quartile Table 3: Descriptive Statistics of SCOOP and UAV DTM Slope Error datasets (unit = degrees). UAV DATA SCOOP DATA NVA (N=37) NVA (N=35) VVA (N=8) VVA (N=7) NVA (N=37) NVA (N=35) VVA (N=8) Mean Median ST DV Minimum Maximum Skewness Kurtosis st Quartile rd Quartile

41 Table 4: Descriptive Statistics of SCOOP and UAV DTM Aspect Error datasets (unit = degrees). UAV DATA SCOOP DATA NVA (N=37) NVA (N=35) NVA (N=31) VVA (N=8) VVA (N=7) NVA (N=37) NVA (N=35) NVA (N=30) VVA (N=8) Mean Median ST DEV Skew Kurtosis Minimum Maximum st Quartile rd Quartile : Normal Testing The data was tested for Normalcy using the Shapiro Wilk (SW) and the Kolmogorov-Smirnov (KS) with the Lilliefors Correction Normal tests. If samples returned significant probabilities of exhibiting non-normal distributions due to outliers, and appeared to be non-normal from descriptive statistics and distribution plots, outliers were removed based on absolute magnitude, and re examined and tested until the tests did not return a p value indicative of a non-normal distribution (p < 0.05). At 100%, the tests do not indicate non-normalcy for the SCOOP VVA elevation (Z), the UAV VVA Slope, the SCOOP VVA Slope, and the SCOOP VVA Aspect (table 5). At 95%, a trimmed dataset shows that only the SCOOP NVA Z, UAV NVA Aspect and SCOOP NVA Aspect remain resistant to a Normal Distribution according to Normalcy Testing (table 6). At 90% (table 8) only the UAV and SCOOP NVA Aspect data fail Normalcy testing, and reflect an insignificant p value only after reduction to 85% (table 8). Insignificant p values do not indicate that the data is Normal, only that the tests do not detect a significant possibility of a Non-Normal Distribution. In addition, the small sample size of the VVA data may cause a non- Normal distribution to be more difficult to detect, and so a through examination of descriptive statistics and distribution plots must accompany the interpretation of the results of Normal tests. Normalcy must also be based on descriptive statistics and distribution plots of the trimmed data (table 10), which presents a characterization free from major outliers. The distributions of the stratified data, both with and without outliers, will be examined thoroughly in Chapter 5, and error will be discussed in the context of potential sources. 41

42 Table 5: Normalcy Testing Results of Full Residual Dataset (* Indicates a statistically significant p Value; critical p value = 0.05) Data N value Stratum Shapiro Wilk P Value Kolmogorov-smirnov P value UAV Z 37 NVA * UAV Z 8 VVA * * SCOOP Z 37 NVA * * SCOOP Z 8 VVA UAV Slope 37 NVA * UAV Slope 8 VVA SCOOP Slope 37 NVA * * SCOOP Slope 8 VVA UAV Aspect 37 NVA * 0.044* UAV Aspect 8 VVA * SCOOP Aspect 37 NVA * * SCOOP Aspect 8 VVA Table 6: Normalcy Testing P values of 95% Residual Dataset Data N value Stratum Shapiro Wilk P Value Kolmogorov-smirnov P value UAV Z 35 NVA SCOOP Z 35 NVA * * UAV Slope 35 NVA SCOOP Slope 35 NVA UAV Aspect 35 NVA * SCOOP Aspect 35 NVA * * Table 7: Normalcy Testing P values of 90% Residual Dataset Data N value Stratum Shapiro Wilk P Value Kolmogorov-smirnov P value UAV Z 7 VVA SCOOP Z 33 NVA UAV Aspect 33 NVA * UAV Aspect 7 VVA SCOOP Aspect 33 NVA * * 42

43 Table 8: Normalcy Testing P values of 85% Residual Dataset Data N value Stratum Shapiro Wilk P Value Kolmogorov-smirnov P value UAV Aspect 31 NVA SCOOP ASPECT 30 NVA : Regression Modelling Ordinary Least Squares (OLS) Regression tool generates outputs including a map of the regression residuals and a summary report with coefficient information and diagnostics. Bilinear regression was used to model true elevation, slope and aspect values their relative computer modelled values, normalized elevation error values against normalized slope and aspect error values, and normalized slope error values against normalized aspect error values, for both the UAV and SCOOP datasets (table 9). Normalized values were used to scale the data equally for comparison and were generated by dividing each value by its respective maximum value to create a scale of error between 0-1, so that all data being modelled were represented at a consistent scale. High statistically significant correlations between true and modelled values and excellent fit of models are found in the elevation, slope and aspect data for both datasets and stratums; most of these relationships are nearly a 1:1 fit of observed to actual values. The exception is the SCOOP VVA slope values, which display a very low correlation and poor model fit between clinometer slopes and modelled slopes. Some moderate correlations are found between the UAV Elevation error and UAV Aspect error in the VVA, SCOOP Elevation error and SCOOP Aspect error in the VVA, and both UAV and SCOOP Slope error and respective Aspect error in the VVA. However, none of these moderate correlations prove to be statistically significant. 43

44 Table 9: Ordinary Least Squares (OLS) Regression Modelling results Dependent Variable Independent Variable Data percent N Multiple R Squared Value Correlation Coefficient Regression Coefficient GNSS Z UAV NVA Z GNSS Z UAV VVA Z GNSS Z SCOOP NVA Z GNSS Z SCOOP VVA Z Clino Slope UAV NVA Slope Clino Slope UAV VVA Slope Clino Slope SCOOP NVA Slope Clino Slope SCOOP VVA Slope Compass Bearing UAV NVA aspect Compass Bearing UAV VVA Aspect Compass Bearing SCOOP NVA Aspect Compass Bearing SCOOP VVA Aspect UAV Z Error UAV Slope Error NVA UAV Z Error UAV Slope Error VVA SCOOP Z Error SCOOP Slope Error NVA SCOOP Z Error SCOOP Slope Error VVA UAV Z Error UAV Aspect error NVA UAV Z Error UAV Aspect Error VVA SCOOP Z Error SCOOP Aspect Error NVA SCOOP Z Error SCOOP Aspect Error VVA UAV Slope Error UAV Aspect Error NVA UAV Slope error UAV Aspect error VVA Scoop Slope error Scoop Aspect Error NVA Scoop Slope error Scoop Aspect error VVA

45 4.3: Accuracy Reporting Based on 100% RMSE error, the UAV outperforms SCOOP in all categories with the exception of aspect error in the VVA (table 10). However, the 100% RMSE is not appropriate in most cases due to violation of the Normal distribution. At 95% RMSE, the UAV outperforms SCOOP in all categories in the NVA, and at 90% RMSE, the UAV outperforms SCOOP in all categories except for Slope in the VVA and Aspect in the VVA. 95% confidence statistics also show the UAV to have higher accuracy measures in all categories except for aspect in the VVA. The 95 th and 90 th percentile statistics reflect maximum absolute error values for their respective data percentages. 95 th percentile statistics show the UAV to outperform SCOOP in all NVA categories. 90 th percentile statistics show the UAV to outperform SCOOP in all categories with the exception of slope and aspect in the VVA. With the exception of SCOOP Aspect error, all NVA data shows higher magnitudes of differences between the 95 th and 90 th percentiles than between their respective 95% and 90% RMSE statistics. In all categories, the 95 th and 90 th percentile statistics are higher than the respective 95% and 90% RMSE statistics, reflecting the differences in data ranges used in calculations; the RMSE considers the full distribution of error in the sample data, while the percentiles are reflective of the maximum error within the same dataset. Reporting of accuracy statistics for stratified data is expanded upon in Chapter 5, but from table 10, it is evident that different reporting statistics reveal very different ideas about the accuracy of the data presented here. Table 10: RMSE, Percentiles and 95% confidence values for elevation (Z) (m), slope (degrees) and aspect (degrees) error 100% RMSE 95% RMSE 90% RMSE 95th percentile 90th Percentile 95% Confidence UAV Z NVA UAV Z VVA 0.18 N/A 0.04 N/A SCOOP Z NVA SCOOP Z VVA 0.48 N/A 0.43 N/A UAV SLOPE NVA UAV SLOPE VVA 4.21 N/A 2.02 N/A SCOOP SLOPE NVA SCOOP SLOPE VVA 5.61 N/A 1.32 N/A UAV ASPECT NVA UAV ASPECT VVA N/A N/A SCOOP ASPECT NVA SCOOP ASPECT VVA N/A N/A

46 4.4: Statistical Significance Testing In spite of apparent differences in reporting statistics, an independent Paired T-Test showed no significant difference between the SCOOP and UAV derived models, and neither model showed significant difference from true elevation values. No significant differences were found between datasets for slope and aspect values. The paired t-test is appropriate because the samples were not completely random, but instead derived from the same geographic locations. However, the T-test relies on the assumption of Normal Distribution of the residuals. Results show frequent non-normal behaviour across all datasets, and data trimming is required to reach a Normal Distribution. Central Limit Theorem states as that as the Sample Size Increases, a Gaussian Distribution becomes easier to achieve, and non-normal behaviour exerts less of an influence. T tests on the full datasets (N=45) of slope, aspect and elevation show non-normal behavior in the residuals, and while a sample size of over 30 points should render the tests on these datasets valid, in addition to T-tests, the Wilcoxon test was used. This test is nonparametric and can be used to test significance of difference between distributions of non-normal data, using the median instead of the mean. Wilcoxon Tests showed no significant difference between SCOOP and UAV elevation, slope and aspect datasets (table 11). Table 11: T Tests and Wilcoxon Tests for Statistical Significance between Datasets (Critical p value = 0.05) Test of Statistical Significance P Value Paired Difference T Test Elevation Paired Difference T Test Slope Paired Difference T Test Aspect Wilcoxon Test Elevation Wilcoxon Test Slope Wilcoxon Test Aspect CHAPTER 5 DISCUSSION 5.0: Statistical Analysis of Elevation (Z) Error Distribution The SCOOP and UAV DTMs display a similar distribution of elevation values, though the range is slightly higher at 29 m in the UAV DTM vs 26 m in the SCOOP DTM; the UAV DTM records a slightly lower and higher range of values (figure 9). Both DTM Maps show the gully feature of the VVA, though the high resolution of the UAV DTM delineates a greater range of detail within the riparian, where the narrow channel branches of the creek are evident. In addition, the narrow depression in the Southwest corner is clearly defined by the UAV DTM but appears only vaguely in the SCOOP data. The high resolution also causes error in the UAV DTM to be more evident; surface features that remain are also more clearly outlined by the high resolution of the UAV DTM. 46

47 Figure 9: SCOOP (upper) and UAV (lower) DTMs overlaid with signed error magnitudes in meters at GNSS survey locations. 47

48 Figure 10: SCOOP vs UAV Absolute Unsigned Elevation Error Magnitudes (m) at GNSS survey locations overlaid on SCOOP- UAV elevation error surface Upon visual inspection of the error surface (figure 10), both models produce a majority of values with neutral differences. Red areas are places where SCOOP has predicted higher elevations than the UAV, and blue indicates areas where SCOOP elevations are lower. Areas of greater difference can be compared to bar height of absolute elevation errors for a qualitative assessment of DTM accuracy. Errors are highest around Vegetated areas, which is to be expected and results from error in surface feature reduction, and the UAV data shows some higher error around the perimeters, particularly in the North-East and South-West corners, due to insufficient overlap on the outermost flight lines. SCOOP in particular struggles to represent true values in the Vegetated area, confirmed by a high RMSE value in the VVA (table 10). Transition zones between vegetated and open areas are also problematic, due to terrain variation and shadows. Some of these variations may actually result from land change over time. The SCOOP data is 3 years older than the UAV data, and in this time the land has changed, especially around the Riparian tree cover, where brush clearing resulted in an increase in open ground, and some piling and mounding of earth adjacent to the riparian. In these areas, along the edge of the riparian where SCOOP has predicted lower than both the UAV and true values (is inaccurate) elevation differences could result from recent build up of ground, as well as deposition from upslope erosion. Areas where SCOOP has inaccurately over predicted appear to be concentrated within the riparian, under tree cover. This is likely due to incomplete surface feature removal, perhaps due to full canopy conditions. Area where the UAV elevations are less accurate are 48

49 typically lower in magnitude than SCOOP, and appear mostly random apart from 3 higher values in the North of the study area, and some higher error in the S-W corner, likely due to the distortions of edge effect from insufficient photo overlap on the perimeter of the UAV flight : UAV NVA Elevation Errors Descriptive Statistics, Distribution Plots & Normalcy Testing Table 2 reports descriptive statistics for the 100% UAV NVA elevation data. The mean and median values are not close together, providing an indication of a non-normal distribution. The standard deviation is 0.13 m, and the IQR is.17 m with a minimum value of m and a maximum value of 0.42 m, which is much higher than the mean and indicative of positive outliers. The mean value shows a slight tendency towards overprediction, but this does not agree with the negative median. Skewness is 1.06 suggesting a slight positive skew to the right reaching out to the maximum value of 0.42 m, and kurtosis is 4.3, indicating a leptokurtic distribution producing more outliers than would be expected from the Normal Distribution. Visual analysis of distribution plots gives further evidence to a Non-Normal distribution (figure 11). Figure 11: UAV NVA Elevation Error Histogram (left) and QQ Plot (right) show a non-normal Distribution. The UAV data is Unimodal but appears asymmetrical and skewed with a tail to the right, influenced by a few outliers as suggested by the descriptive statistics. The majority of values are clustered around the mean to the right. Normal testing confirms a Non-Normal Distribution with the Shapiro-Wilk Normality Test, but does not show significant variation from normal with the Kolmogorov-Smirnov test (table 5). The QQ plot reinforces the nearly Normal distribution with the majority of the values following the slope of the graph, with a few outliers swaying the data at the upper limit. The data becomes Normal according to both Normal tests at 95%, with the removal of 2 outliers (table 6). Table 2 reports improved values for skewness and kurtosis using a 95% dataset which omits two positive outliers. Figure 12 shows the magnitude of absolute errors and the location of two positive outliers, both on the perimeter of the study area, that are causing a strong skew to the right. The Moran s I statistic for the full dataset also shows positive spatial autocorrelation with residuals of overpredictions significantly clustered (p= ). 49

50 Figure 12: UAV NVA Absolute Elevation Elevation Error Magnitudes (m) and Outliers At 95 % the mean and median values are now both slightly negative and closer together, reinforcing the initial median value and showing a tendency towards a slight underprediction of elevation values with the UAV. With the removal of two positive outliers (figure 12) the maximum value is reduced from 0.4m to 0.2m, and the IQR and standard deviation are also reduced. Skewness and kurtosis values are also improved, with a kurtosis value of less than 3 (table 2). The histogram for the trimmed dataset shows improved symmetry with a more even distribution of values above and below the mean (figure 13). Spatial autocorrelation is corrected by outlier removal and the Moran s I value now indicates a random error distribution. 50

51 Figure 13: 95% Histogram (left) shows an improved distribution and Regression Distribution plots (right; Histogram and scatter) of true vs. UAV elevation values in the NVA show a strong positive relationship. Regression Modelling UAV NVA Elevation Error: The Normal Distribution of residuals is a condition that must be satisfied to produce meaningful, significant regression models. Regression modelling gives further insight into the correlation between model variables and the factors that drive changes in them. Bilinear regression results are reported in table 9. Using GNSS elevation values as the dependent variable and UAV elevation values as the independent variable, the regression model shows a positive and statistically significant relationship between true values and UAV derived elevation values as displayed by distribution plots in figure 13; the histograms show very similar distributions, and the scatterplot shows a strong positive relationship indicated by the steep ascending slope of the line and the plotted values which follow it closely. Regression using normalized (by maximum value) elevation error as the dependent variable and normalized slope and aspect error as explanatory variables show weak statistically insignificant positive correlations between elevation (Z) error and errors in slope and aspect. Accuracy Reporting UAV NVA Elevation Error The assumption that data in the NVA follows a Normal Distribution does not hold for the UAV NVA dataset; it is nearly Normal at 100%, and Normal at 95%. The 100% RMSE of 0.13 m is sensitive to outliers; the 95% RMSE of m and the similar 90% RMSE of m reflect the improvement of removing major outliers (table 10). The absolute percentiles and 95% confidence levels still reflect smaller outliers and are higher in value than the RMSE at 0.18 m (95 th percentile) and 0.16 m (90 th percentile). The 95% confidence value of 100% RMSE x 1.96 is 0.25 m, again reflecting higher error values present in the tails of the distribution. In the UAV NVA, the 100% RMSE statistic fails to represent the negative bias of the UAV; though small, the negative mean and median at 95% signify a tendency for the UAV to underestimate true elevation value. The 100% RMSE also fails to tell the whole story regarding the presence of frequent major and minor outliers within the dataset; these outliers are evident in the higher error values represented by the 95 and 90 th percentiles and the 95% RMSE : UAV VVA Elevation Errors Descriptive Statistics, Distribution Plots & Normalcy Testing In the VVA, the mean and median are both negative but approximately 5 cm different. The minimum value is m and the maximum is 0.48 m, suggesting a positive outlier of high magnitude. Skewness is positive and kurtosis is high, indicating a Non-Normal distribution (table 2). 51

52 Figure 14: UAV VVA Elevation Error Histogram (left) and QQ Plot (right) show a non-normal distribution. The histogram confirms this (figure 14), showing a bimodal, non-normal curve with higher values tending to cluster towards either end of the distribution. Though the distribution is limited with only 8 values, the QQ plot shows deviation from the curve and a non-linear pattern to the residuals. To reach a Normal curve, the dataset has to be reduced to 90%, which involves the removal of the high positive outlier noted at the upper end of the QQ plot. Using 7 variables, the mean and the median are equal, the standard deviation is reduced, the maximum reach is reduced to 0.08 m and skewness and kurtosis are cut in half (table 2). Normal tests at 90% reveal nonsignificance for both the KS and SW statistics, retaining the Null hypothesis of Normality (table 6), and showing a histogram that looks much closer to Normal with a unimodal peak and a more equal distribution of values around the center (figure 16). The removal of the outlier located in the SW corner corrects the abnormal distribution. This outlier can be explained by the confounding factors of shadows from tree cover and a break in terrain where a deep, narrow gully causes a distinct elevation change. The high resolution of the UAV DTM reveals this gully in full detail, however the UAV measured elevation at its top bank where grass cover obscured the true ground of the gully bottom. It should also be noted that this point was not random, but was placed to test the UAV s ability to distinguish this gully feature; its placement on a break line of distinct slope change violates the ASPRS checkpoint placement guidelines and as a result, the data is much more fairly described without it. Figure 15: UAV VVA Absolute Elevation Elevation Error Magnitudes (m) and Outlier 52

53 Figure 16: 90% Histogram (left) shows improved distribution and Regression Distribution plots of true elevation values vs. UAV values in the VVA show a strong positive relationship. Regression Modelling UAV VVA Elevation Error: Regression in the VVA shows a strong positive correlation between true elevation values and UAV estimations; histograms show similar distributions (although the UAV VVA histogram is bimodal even after the outlier is removed), and the scatterplot shows the strong positive relationship (figure 16). The Multiple R squared value of indicates that almost all of the variability in elevation is captured by the modelling of the UAV values (table 9). Neither the Koenker (BP) statistic or the Jarque-Bera (JB) Statistic are significant, indicating that there is no autocorrelation in the model, and that the residuals are Normal. Accuracy Reporting UAV VVA Elevation Error: The UAV VVA distribution is affected by one major outlier of 0.5 m, and at 100% is non-normal as expected by ASPRS guidelines. 100% RMSE reporting would not be an expected reporting method in a VVA dataset, and instead 95 th percentiles would be used. However, ASPRS recommends a minimum of 5 sample plots in a VVA stratum (ASPRS, 2014). This small number of samples makes any further reduction in data difficult. With 8 samples, reducing the data by one outlier equals a 10% reduction, making 95 th percentile reporting unrealistic. In the UAV VVA, the single outlier of 0.5 m is much higher in magnitude than the mean value of 0.06 m. The 100% and 90% RMSE values reflect this difference in their respective magnitudes of 0.18 m and 0.04 m (table 10). The 90 th percentile is close to the 90% RMSE at 0.07 m, while the 95% confidence level is influenced by the high outlier at 0.34 m. Again, in the VVA, though error assumptions are assumed to be Non-Normal, different reporting methods tell different stories about the data, which is on average higher in accuracy than the NVA, but is susceptible to extreme outliers due to variation in land cover and the influence of shadows and variable vegetative cover. While error in the VVA is better than might be expected in vegetated terrain, this is likely due to a number of error reducing factors such as the fact that most of the accuracy assessment check points were placed in open ground within the vegetated area due to the leaf off canopy conditions at the time of the study. The riparian is composed mostly of deciduous trees, so canopy cover was not likely as much of an influence to the error budget as it would have been following green up. In addition, the low altitude of the UAV flight resulted in higher resolution and less interpolation, which may contribute to higher accuracies in the UAV VVA data than in the SCOOP data. 53

54 5.0.3: SCOOP DTM NVA Elevation Error In the SCOOP NVA data, the mean is negative, the median is slightly positive, and the standard deviation is quite high for NVA values at.25 m (table 2). The range of minimum and maximum values extend from m to 0.74 m, exhibiting a much wider range of values than the UAV dataset, with high values reaching almost a meter of error at both ends (figure 17). Skewness is negative and kurtosis is very high at 9.3, the strongest indicator of Non- Normalcy. Figure 17: SCOOP NVA Histogram (left) displays high kurtosis and QQ plot (right) reinforces non-normal distribution. The histogram is unimodal and fairly symmetric, but a very high kurtosis value is displayed by the distinct peak near the mean, which declines rapidly and spills into a heavy tailed leptokurtic distribution reaching out to outliers on both sides. Following the removal of two outliers the descriptive statistics improve (table 2) but still the data fails the Normal KS and SW tests and the kurtosis value remains high at Reduced to 90% with the removal of 4 outliers, skew and kurtosis are improved, but kurtosis remains high at 3.821, and the results of Normal testing (table 6) show significant deviation from Normal. At 85%, the data shows insignificant results in Normalcy Testing (table 7) and kurtosis falls below 3 (table 2). When the 6 outliers of highest error magnitudes are removed, the improved descriptive statistics show a distribution much closer to Normal; the standard deviation is reduced to 0.08 m, the minimum value is m, the maximum value is balanced at 0.17 m and skew is reduced to 0.04 (table 2; figure 18). In attempt to avoid reducing the data to this extent, log transformation was also applied to the full dataset. Log transformation has been shown to improve Non-Normal Distributions (Zandbergen, 2008) and in this case, a log transformation improves the distribution so that the 95% dataset passes both the KS and SW tests. However, regression modelling of the SCOOP Z NVA 95% log data identifies a statistically significant Jarque-Bera statistic (p<0.0000) indicating that the residuals are still not Normal. Running spatial autocorrelation on the residuals indicates a Moran s Index Statistic of and a random distribution. The non-normal distribution is driven by outliers rather than from spatial autocorrelation at the 95% level. 54

55 Figure 18: SCOOP NVA Elevation Elevation Error Magnitudes (m) and Outliers Figure 19: 85% Histogram (left) shows improved distribution, and Regression Distribution Plots (right) of true elevation values vs. SCOOP values in the NVA show a strong positive relationship. Regression Modelling SCOOP NVA Elevation Error: Regression at both the 90% and the 95% level on the log transformed GNSS elevations (dependent) and SCOOP elevations results in statistically significant Jarque-Bera p values, though p is improved to 0.04 with the log transformed 90% dataset. While spatial autocorrelation reveals no clustering of values, there does appear to be a range of higher error values along the North-east side of the riparian (figure 18). This could be due to actual land 55

56 change rather than systematic error, where the processes of erosion and deposition over the past 4 years resulted in real changes of elevation values. In addition, this area has been cleared of vegetation and the riparian has been reduced to create more crop land. When the SCOOP data was collected, this area was likely more vegetated and would have been considered part of the VVA, factors which could be contributing to the heavy tailed leptokurtic distribution. A closer look at the differences in the elevation measurements in this area could reveal correlation between value changes over time and the processes of erosion and deposition at the edge of the field, as well as the influence of land change from riparian to agriculture. While Normal tests of the 90% NVA dataset return p values large enough to keep the Null Hypothesis, it takes a reduction to 85% data (removal of 6 outliers) to achieve an insignificant Jarque-Bera value in regression modelling of true elevation values in the NVA vs. SCOOP elevation values (table 9, figure 19). At this level, the regression model shows a strong positive correlation between true and estimated elevation values with a correlation coefficient of 0.99 (table 9), evident in the regression distribution plots in figure 19. Accuracy Reporting SCOOP NVA Elevation Error: The assumption that the NVA data follows a Normal Distribution is problematic and violated for the SCOOP data, which actually requires more trimming than any other elevation dataset in this study. The extremely high kurtosis resists a Normal Distribution due to outliers, and 15% of these must be removed to meet the expectation of Normality. At 100%, the RMSE is 0.25 m, increasing to.48 m at 95% confidence (table 10). The 95% RMSE is improved to 0.15 m, and the 90% RMSE shows even more improvement at 0.10 m, equivalent to the 90% RMSE of the UAV data. The 95 th percentile statistic of 0.5 m reflects high outliers, and the 90 th percentile is improved, but still high at 0.31 m. The SCOOP data shows large discrepancies in the magnitudes of reporting errors depending on the method. The 85% trimmed dataset provides the highest accuracies, and follows a Normal distribution, and this is only represented by the 95% and 90% RMSE. All other reporting methods retain the influence of the large outliers present in the data. However, given the temporal lag present in this data, which was collected in 2013, it would be unwise to assume that outliers are in fact representative of gross blunders in data collection. They could very well be reflective of true changes in elevation over time as a result of land use change, erosion, and deposition, and warrant further investigation as to the significance of their role in the dataset, and may be indicators of true environmental processes at work : SCOOP DTM VVA Elevation Error In the VVA the discrepancy and difference in signage between the median of 0.05 and the mean of initially indicates a non-normal distribution (table 2). The standard deviation is quite high at 0.5 m, but the minimum and maximum values are fairly symmetrical around the mean at -0.7 m and 0.5 m. Skewness is slightly negative, and kurtosis is low at 1.2, relative to the high kurtosis in the NVA. Figure 20: Histogram (left) and Normal QQ Plot (right) of the SCOOP DTM VVA Data show a bi-modal, non-normal distribution. 56

57 The histogram, however, does not appear Normal. It is bimodal with the bulk of its values on the outside peaks (figure 20). The QQ plot reveals fluctuation above and below the line. The histogram slightly improves (figure 22) with log transformation on the absolute values and the removal of one outlier (figure 21), which is uncharacteristically of higher accuracy (0.03 m) than the bulk of the other values. This point is in a very open part of the VVA, which explains why its accuracy is closer to typical NVA accuracy. Figure 21: SCOOP VVA Absolute Elevation Elevation Error Magnitudes (m) and Outlier. However, the log transformation produces other issues such as asymmetry about the mean, strong negative skew and higher kurtosis. In spite of its apparent Non-normalcy, when tested with the KS and SW test, the 100% dataset of N=8 does not show significant deviation from Normal. Small sample size can sometimes result in untrustworthy p values when N is too small for the test to detect deviations from Normal. However, in medcalc Statistical software a message will usually be returned indicating the small sample size, and no message was returned with the VVA Normal testing. Figure 22: 90% Histogram (left) shows improvement but retains skew, and Regression Distribution plots (right) of true elevation values vs SCOOP values in the VVA show a fairly strong positive relationship. 57

58 Regression Modelling of SCOOP VVA Elevation Error: Regression modelling of the VVA indicates no significant bias through the Jarque-Bera statistic, and no autocorrelation through the BP statistic. A significant positive relationship is found through regression modelling of true elevation values and SCOOP values (figure 22) with a correlation coefficient of 0.97 and a multiple R squared value of 0.94 (table 9). Accuracy Reporting SCOOP VVA Elevation Error: In the VVA, the assumption that the data does not follow a Normal Distribution does not hold. Accuracies here are lower than they are in the UAV VVA data; 100% RMSE of 0.5 m increases to 0.9 m at 95% confidence (table 10). Both 90 th percentile statistics and 90% RMSE value remain high, though improved, at 0.6 m and 0.4 m respectively. The SCOOP VVA data accuracy appears to be affected by land cover; lower accuracies are likely a combined result of errors due to canopy presence and variable vegetative cover during data collection (the data was collected in 2013 in leaf off conditions, but the exact time of year is not known), as well as errors in surface feature extraction (the process of which is considered proprietary to the vendor and is undisclosed). In addition, the lower data accuracy may reflect true changes in elevation along the perimeters of the VVA, as it likely does in the NVA. 5.1: Statistical Analysis of Slope Data Upon initial observation, the UAV map shows a higher range of slope, with high end values at 45 degrees compared to the SCOOP upper end value of 23 degrees (figure 23). At least some of these high-end slopes are likely error caused by surface features remaining in the vegetated riparian. Though the UAV slope data is resampled from 10 cm to two meters, it retains more feature distinction than the SCOOP data, and residual surface features such as tree cover, low brush and building footprints are more clearly delineated. The original high resolution of the UAV data creates higher feature visibility, especially within the riparian, where slope changes are more clearly defined and even the shallow, thin banks of the river bed are evident. The point error of UAV slope measurements shows higher measurements in the southern end of the study area, close to extracted surface features, and along the edges of the riparian where tree cover likely skewed slope measurements. High error in the NVA is capped at 2 degrees, but in the VVA a possible outlier of 10 degrees is evident. The Scoop slope error has a higher range in the NVA to 6 degrees, but a slightly lower range in the VVA to 8 degrees. High error points appear to be situated within or close to the Riparian, in areas which at the time of acquisition were likely part of the riparian as well. 58

59 Figure 23: UAV (upper) and SCOOP (lower) Slope Surfaces overlaid with slope under and over predictions at clinometer survey locations. 59

60 Figure 24: SCOOP minus UAV Slope Error Surface overlaid with UAV vs SCOOP absolute slope error values (Degrees slope). In the NVA the error surface produces a surface where SCOOP predominantly slightly overpredicts slope compared to the UAV (figure 24), but the magnitude of the slope differences is closer to neutral in the NVA than is the case along the transition of the stratums and within the VVA. The edges of the riparian show strips of higher SCOOP slopes containing bands of lower SCOOP slopes within the riparian. These bands of high slopes could be partially contributed to inaccurate point classification resulting from the existence of riparian forest cover which has since been cleared, resulting in an expansion of the NVA since the SCOOP acquisition. SCOOP slope error magnitude at survey points in the VVA also appear to be higher than the UAV error, though error frequency at point locations in the VVA is approximately equal with the SCOOP slope showing higher error at 4/8 survey locations (figure 24). Both SCOOP and UAV data appear to be most incorrect in the riparian, which also includes the majority of high slopes, and high error; indicators that incomplete surface feature extraction in both datasets could be contributing to high error here (Illustration 3). It is difficult to measure the accuracy at the points of highest slope error because most of the GNSS survey locations were placed on open ground within the riparian. However, both datasets appear to have about the same frequency of higher error, with the UAV retaining lower error magnitude more often with the exception of one major outlier. 60

61 Figure 25: The high resolution 10 cm slope surface reveals error due to incomplete surface feature removal UAV NVA Slope Error Descriptive Statistics, Distribution Plots and Normal testing: The mean and the median are about equally asymmetric around 0, but of different signs (table 3). The standard deviation is 0.69 degrees, with minimum and maximum errors balanced around 0. There is a slight positive skew, and kurtosis is slightly high at The 1 st quartile is and the third quartile is The histogram appears to be nearly normal (figure 26); it is unimodal and close to symmetrically distributed, but its high kurtosis indicated by the high peak relative to long tails, especially on the right, suggest non-normality. The QQ plot shows some fluctuation around the line of slope, and a statistically significant Moran s I value indicates spatial autocorrelation. Normal testing of the full dataset shows a significant p value for the KS value, but nonsignificance for the SW p value (table 5). Figure 26: UAV NVA Slope Error Histogram (left) and Normal QQ Plot (right) show nearly Normal distributions at 100%. The reduction of the dataset to 95% resulting from the removal of two outliers on the SW and SE perimeters (figure 27) improves the skew and reduces the standard deviation, but does little to improve the discrepancy between mean and median and actually increases the kurtosis, as evidenced in the 95% histogram (figure 28). However, the dataset shows no significant p value for either normal test at the 95% level. The outliers are on opposite perimeters of the study area and are of opposite signs. There is nothing initially evident that would 61

62 be contributing to the occurrence of these errors in slope other than elevation error; the SW point is also an outlier of elevation error, and the Eastern point has a higher elevation error as well (figure 27). However, regression modelling of normalized elevation error vs slope error reveals no statistically significant correlation between datasets and a poor fit of model (table 9). Figure 27: UAV Slope Error Magnitudes (Degrees Slope) and Outliers. Figure 28: 95% Histogram (left) shows improvement and Regression Distribution Plots (right) show a strong positive relationship of true slope values vs. UAV slope values in the NVA. 62

63 Regression Modelling of UAV NVA Slope Error: Regression modelling of true clinometer measured values vs. the UAV DTM derived slopes at 100% data indicates spatial autocorrelation of the residuals with a significant KB value, and significant non-normality with a significant JB value (table 9). Regression modelling of the 95% dataset shows that a high degree of true values are found in modelled values with an a Multiple r squared value of 0.876, an absence of significant spatial autocorrelation, and a Normal Distribution of residuals. (figure 28, table 9). Reporting of UAV NVA Slope Accuracy: Overall, the UAV NVA slope data is of good accuracy and at 100% follows a nearly Normal distribution. The removal of two minor outliers allows Normalcy to be reached at 95%, but overall the data displays slope error of less than one degree at 100%, 95% and 90% (table 10). The 100% RMSE therefore differs little in magnitude than the 95% and 90% RMSE, which are 0.7, 0.6, and 0.5 degrees respectively. The percentiles at 95 and 90 reflect the influence of the outliers, at 1.5 and 1.1 degrees; the 95% confidence level is similar in magnitude at 1.3 degrees error. The slope data is slightly positively biased, but at 0.01 degrees this bias is not drastic, and so the use of RMSE as a reporting statistic does not misrepresent the UAV NVA slope data : UAV VVA Slope Error Descriptive Statistics, Distribution Plots and Normal testing In the VVA, a higher range of error values is evident. Here an outlier of 10 degrees skews the data positively. The mean and median are close together, and the minimum value is much lower than the maximum (table 3). Kurtosis is high at 4. The histogram shows asymmetry of values with a high frequency of values occurring to the left of the centre, and the highest peak at these values to the left (figure 29). The QQ plot shows a failure of values to adhere to the slope of the line. At 100%, the VVA shows no significant variation from normal in both tests, however this may be being influenced by small sample size (table 9), because the results of Normal testing conflict with the suggestion of non-normalcy in the distribution plots. Figure 29: Histogram (left) strays from a Normal Bell Curve, and Normal QQ Plot (right) of UAV Slope Errors in the VVA show non-normal distributions at 100%. Reduction to 90% with the removal of the large positive outlier reduces the kurtosis, balances the positive skew with a slightly negative skew, cuts the standard deviation in half, and changes the sign of the mean to negative, suggesting an overall underprediction of slopes (table 3). The histogram (figure 31) is now improved in that it becomes Unimodal, but still shows an unbalanced distribution. The removed outlier was a particularly 63

64 difficult point where a large brush pile obscured the North side of the point (figure 30). It is likely that the slope measured at this point measured the slope of this brush pile, in addition to other obstructions created by Creekside vegetation. Figure 30: UAV Slope surface overlaid with slope error magnitudes (degrees) and outlier in the VVA. Figure 31: 90% Histogram (left) shows some improvement but maintains skew, and Regression Distribution plots (right) of true values vs. UAV slope values in the VVA show a strong positive relationship. 64

65 Regression Modelling of UAV VVA Slope Error: Regression on the 90% slope dataset with clinometer values as the dependent variable showed no statistically significant autocorrelation or deviation from normalcy, with the model reporting a Multiple R squared value of 0.89 and a correlation coefficient of 0.95 (table 9; figure 31), indicating a good fit of model and a positive correlation as evidenced in the regression plots in figure 31. Regression modelling between normalized elevation error and normalized slope error revealed no statistically significant correlation between the two variables (table 9). Reporting of UAV VVA Slope Error: In the VVA, the degrees of slope error are much higher, and the data does not conform to a Normal distribution until it is trimmed to 90% (table 7, figure 33). The 100% RMSE statistic of 4 degrees is an inappropriate representation due to the non-normal nature of the data, and at 90% the RMSE is cut in half to 2 degrees (table 10). The 90 th percentile statistic of 3.05 degrees (table 10) is one degree higher than the 90% RMSE value of 2.02 degrees. At 95% confidence, the slope error increases to 8 degrees. While regression reveals no significant relationship (table 9) between elevation error and slope error, the higher magnitude of slope error in the VVA still likely reflects errors in the DTM; surface objects clearly remain evident in the DTM, and remnants of trees and vegetation exist in areas between sample points where neither elevation or slope were measured; these locations likely reflect both false elevations and false slopes although they are not considered in the descriptive statistics, because the point accuracy of these locations was not directly tested : SCOOP NVA Slope Error Descriptive Statistics, Distribution Plots and Normal testing In the SCOOP NVA data, the mean and median are not close together, with a median close to 0 and a mean of.25 degrees. The standard deviation is 1.5 degrees error of slope, and the maximum value is higher than the minimum, reflected in a positive skew (table 3). Kurtosis is high at 9.8, a strong indicator of non-normal distribution. The histogram reflects this and is unimodal but with a very high peak in relation to its tails. The Normal QQ plot shows a flattening trend across the center point of the line of slope (figure 32), but the Moran s I statistic does not indicate autocorrelation. The 100% dataset exhibits statistically significant non-normalcy on both the KS and SW tests (table 5). Figure 32: Histogram (left) shows high kurtosis and skew, and Normal QQ Plot of SCOOP Slope Error Values (Degrees slope) reinforces the apparent non-normal distribution in the NVA. The 95% dataset shows a mean that is still higher than the median, but the maximum and minimum values are more symmetric and skew and standard deviation are reduced (table 3). Kurtosis is strongly reduced to 65

66 3.2, and the histogram exhibits a much more symmetrical distribution (figure 34). At 95% the NVA slope error shows no statistically significant deviation from normal in either test (table 6). The removed outliers are both situated on the transition of the VVA and are of similar magnitude and opposite signs. Likely ground obstruction due to riparian vegetation contributed to the higher slope errors. In addition, true changes over time may have resulted in discrepancies between true present values and SCOOP slopes (figure 33). Figure 33: SCOOP Slope Error Magnitudes (degrees slope) and Outliers in the NVA. Figure 34: 95% Histogram (left) shows reduced kurtosis and Regression Distribution Plots (right) of clinometer vs SCOOP Slope in the NVA show a fairly strong positive relationship. 66

67 Regression Modelling of the SCOOP NVA Slope Error: Regression shows statistically significant non-normalcy at 100%, but not at 95%. The trimmed data shows a strong positive correlation with a multiple R squared value of 0.88 and a correlation coefficient of 0.94 (table 9, figure 34). Regression on the slope errors and elevation errors show no correlation between errors in elevation and errors in slope measurements (table 9). Reporting of SCOOP NVA Slope Accuracy: The Scoop NVA slope values trimmed to 95% fit a Normal Distribution, and overall are of good accuracy. The 100% RMSE is 1.4 degrees (table 10) and the 95% and 90% RMSE are improved to less than a degree error at 0.9 and 0.7 degrees respectively. The 95% confidence level reflects higher values in the tails of the distribution at 2.7 degrees, and the 95 th and 90 th percentiles also reflect these higher error values at 2.24 and 1.66 degrees error : SCOOP VVA Slope Error Descriptive Statistics, Distribution Plots and Normal testing: Figure 35: Histogram (left) and Normal QQ Plot (right) of SCOOP slope error in the VVA show a nearly Normal distribution. The VVA data (figure 35) achieves a nearly normal curve at 100%. The data shows a positive mean and median with a small discrepancy between them, a standard deviation lower than the minimum and maximum values which are spread fairly symmetrically around 0, a slight negative skew, a kurtosis of 2.71 and a first and third quartile that match in value but differ in signage (table 3). The histogram is unimodal and kurtosis is low. The QQ plot reveals values that mostly mirror the slope of the line. Although one outlier was identified in cluster and outlier analysis, identified in figure 36, this outlier does not significantly influence the distribution of the dataset away from Normal, as evidenced by the Normalcy tests, descriptive statistics and distribution plots. Though the SCOOP VVA data can be considered Normal at 100%, it lacks accuracy, and this is also reflected in weak relationships between true and observed values in regression modelling (table 9). 67

68 Regression Modelling of the SCOOP VVA Slope Error: Figure 36: SCOOP Slope Error Magnitudes in the VVA. Regression modelling of true clinometer values vs SCOOP slope values on the 100% dataset reveals no correlation between variables, and a trimmed 90% data set modelled with regression shows a very weak and insignificant negative correlation between variables with an r squared value of and a correlation coefficient of 0.07 (table 9, figure 37). The relationship between true and derived slope values in the VVA is erratic and a source of high error; this is reflected in the lack of positive relationship between true and estimated slope values, as well as in the relatively high error statistics for the 90% RMSE and the 95% confidence values (table 10). Regression modelling between elevation error and slope error in the VVA reveals no significant correlation between variables (table 9). Figure 37: Regression Distribution Plot of Clinometer values vs SCOOP Values in the VVA show a weak and insignificant negative relationship. 68

69 Reporting of SCOOP VVA Slope Accuracy: The VVA statistics show high deviations in the 100% RMSE value of nearly 6 degrees, which increases to 11 degrees error at 95% confidence (table 10). The 90% RMSE shows much improvement with a value of 1.3 degrees. The 90 th percentile statistic here provides the most generous estimate of VVA accuracy at 0.6 degrees, but to use this statistic alone as ASPRS suggests for VVA reporting would misrepresent the variation present in the data. In the VVA, slope errors likely result from incomplete surface feature extraction, however, as in all SCOOP data, errors may also be indicative of true change over time and errors of this nature should not be dismissed without investigation. Regression provides some clues to the nature of these errors, and the lack of significant relationship between slope error and elevation error does suggest that slope errors in the VVA are a result of inaccurate point classification rather than true change in slope over time, but this observation does not prove causality. 5.2: Statistical Analysis of Aspect Data Figure 38: UAV vs SCOOP Absolute Aspect Error Magnitudes (Degrees Error. The calculation of aspect poses more challenges than other derivatives, because both 0 and 360 describe the same value. As a result, incorrect values can result when values cross 0 in the wrong direction during 69

70 subtraction to calculate aspect differences. To rectify this within a small dataset, each of the difference values produced from the subtraction of UAV/SCOOP values from true compass aspects was manually examined to ensure that it did not cross 0 the wrong way, resulting in large Type 1 error. For example, the difference between 2 degrees and 359 degrees is should be 3 degrees, not 357. This issue also posed problems for producing an error surface between the SCOOP and UAV aspect layers, and as a result of values crossing 0 the difference surface produced would be unreliable and was omitted from the discussion of aspect error. The SCOOP Aspect map does not differ greatly from the UAV map upon visual inspection (figure 39). Less micro-typed aspect variation is evident in the SCOOP data, especially within the riparian, but overall SCOOP appears to report the same general aspects as the UAV. The UAV aspect map shows less change in aspect over the buildings in the SW; in the SCOOP data, the footprints of the buildings are elevated and display a purely southern exposure rather than W-SW. To some degree this effect is also evident in the UAV data, and occurs more strongly in the UAV data over the remaining tree cover on the western side of the study area. Both datasets show similar error across the terrain, but the UAV data reports at least one major outlier. Both datasets appear to be less accurate within the VVA (figure 38). In both datasets, aspect error distributions are prone to frequent influential outliers. Visual inspection of the UAV aspect map (figure 39) shows an accurate representation of the cardinal landscape, with some micro typing of aspect clear within areas of more consistent orientation, where terrain changes below the 2-meter resolution baseline determine resulting aspect changes. The 10-cm aspect map produced by the UAV micro types so many competing aspects that it is not of practical use for most applications; changes in aspect within corn rows is evident at this scale (Illustration 4). All of the aspect data requires trimming to eliminate major outliers and achieve a normal curve (table 4). Aspect errors are measured in terms of their absolute differences. Illustration 4: Screen Shot of 10 cm UAV Aspect Map; the high-resolution results in micro typing of variable aspect within corn rows. Error over tree cover is also evident. 70

71 Figure 39: SCOOP ASPECT (upper) and UAV ASPECT (lower) Maps at 2 m Resolution 71

72 5.2.1: UAV NVA Aspect Error Descriptive Statistics, Distribution Plots and Normal testing: Figure 40: Histogram (left) and Normal QQ Plot (right) for UAV Aspect (Degrees Error) in the NVA show a nearly Normal distribution. The NVA data shows a nearly normal distribution, with only a slight difference between the mean and median. The standard deviation is high relative to the mean, there is a slight positive skew, and kurtosis is moderately high at 3.8 (table 4). The first quartile is -0.5 degrees and the third quartile is 3.0 degrees, suggesting asymmetry about the mean. The histogram shows a heavier distribution of values on the right threatening the unimodal nature of the data, and the QQ plot shows a general adherence to the line of slope, with some deviation above and below the line in the middle of the dataset (figure 40). The UAV NVA aspect data does not fulfill normal distribution requirements for either the KS or SW test at 100% or at 95% (tables 5 and 6). At 90% it retains a statistically significant KS p value, and requires a reduction to N=31 (85%) to be considered Normal by both tests (table 8). In the 85% dataset, the mean and median are closer together, the standard deviation is reduced by half, skew becomes slightly negative, kurtosis is acceptable at 3, and the IQR is reduced (table 4). The Histogram shows improved kurtosis with lower relief between the peak and tails (figure 42). Outliers seem to be more of an issue around the south and east perimeter of the study area, though these are not areas where aspect abruptly changes or where surface features or shadows should play a major role (figure 41), and so this suggests distortion from edge effect due to insufficient photo overlap. However, the range of error is only approximately 20 degrees, much lower than the range in the VVA. 72

73 Figure 41: UAV NVA/VVA Aspect Error Magnitudes (degrees aspect). Figure 42: 85% Histogram (left) shows improved kurtosis and Regression Distribution Plots (right) shows a strong positive relationship of UAV Aspect vs. Compass Aspect in the NVA 73

74 Regression Modelling of UAV NVA Aspect Error: Regression between true and estimated values of UAV NVA aspect, using compass bearings as the dependent variable, shows strong positive correlation and excellent model fit with an R2 value of 0.99 (table 9, figure 42), and no significant clustering or non-normal distribution of the residuals. Regression modelling of normalized aspect error vs normalized slope error reveals no significant relationship between the two variables. While many studies show correlation between these two variables (Weih, 2004; Bolstad, 1994), the lack of relationship displayed here may result from the localized terrain variation, where a large error in aspect doesn t necessarily correspond to large changes in slope; the two are not matched in magnitude. This is more evident in the VVA, where a large aspect error value records the wrong aspect in a concave gully feature of similar slopes (figure 41). Reporting of UAV NVA Aspect Accuracy: Aspect values require a high degree of trimming to reduce the NVA error values to 85% and achieve a normal distribution. However, error magnitudes in the NVA are reasonable for the UAV data, with a 100% RMSE of 4.8 degrees, a 95% RMSE of 4 degrees, a 90% RMSE of 3.2 degrees, and a 95% confidence level of 9.3 degrees (table 10). However, since the data is not Normal at any of these percentages, none of these statistics are completely appropriate representations. At 85%, when all outliers contributing to Non-Normalcy are removed, the RMSE is greatly improved to 0.5 degrees error. To consider the 100% dataset through non-parametric statistics, the 90 th and 95 th percentile remain high in value at 7 and 12 degrees. Aspect values in the NVA are clearly very prone to both high and moderate outliers, suggesting that any assumption of Normality on which to use parametric based accuracy statistics should be thoroughly examined first : UAV VVA Aspect Error Descriptive Statistics, Distribution Plots, Normalcy Testing: In the VVA the UAV data reveals one major outlier of high magnitude (168 degrees difference, figure 41). The mean is much larger than the median, and the negative skew and high kurtosis reflect the outlier. The standard deviation is 60 degrees, and the minimum value of -168 is much higher than the maximum value of 20 degrees error (table 4). In this case the UAV heavily underestimated the aspect and recorded the polar opposite of the true bearing, turning a true aspect of 340 degrees into a south facing aspect of 172 degrees. This could be explained by the concave nature of the gully, where the UAV measured the aspect of the incorrect bank, however upon closer inspection, the gully is too wide and should not have created this issue with the high GSD of the UAV. Instead, this is likely a blunder resulting from shadows in the tree cover. Slope and elevation error are only moderate at this point, however the slope would not necessarily change drastically based on aspect in the relatively flat gully bottom, so aspect error here appears to be independent of other errors. The histogram and QQ plot reflect this outlier, and the histogram is skewed left with a major negative outlier (Figure 43). 74

75 Figure 43: Histogram (left) shows negative skew and Normal QQ Plot shows a non-normal distribution of Aspect Error in the UAV VVA data. Trimming this value improves the distribution, resulting in a unimodal peak with a more even distribution of values around it (figure 44). The mean and the median become closer together, but the median is now positive and the mean remains negative. The standard deviation is reduced by two thirds to 20 degrees, the skew is improved, the kurtosis is cut in half, and the range is reduced (table 4). At 90%, the VVA data shows no significant chance of falsely rejecting the Null hypothesis of normality and the data appears normal according to both the KS and SW p values (table 7). Figure 44: 90% Histogram (left) shows improved skew, and Regression Distribution Plots (right) show a fairly strong positive relationship of UAV aspect vs compass bearing in the VVA. Regression Modelling of UAV VVA Aspect Error: Regression modelling shows the UAV aspect values to have a high positive correlation with the true aspect values (figure 44). Modelling od elevation error vs aspect error shows a weak positive and insignificant correlation (table 9), and regression between the NVA aspect and slope error shows a very weak, insignificant correlation. Regression modelling of aspect error in relation to elevation error in the VVA shows a statistically insignificant negative correlation, suggesting that as elevation error declines, aspect error rises. However, this does not signify causation that declining aspect error is caused by increasing elevation error and is only representative of a small number of points in the VVA. Slope and aspect error show a moderate positive correlation of 0.65 in the VVA (table 9). In this area, results initially suggest that as slope error increases, so does aspect error; however, the probability is not considered significant by the model. A higher number of samples would help to draw out a more significant relationship from these variables, if one does exist; using only 7 variables, only limited conclusions can be drawn about model results. 75

76 Reporting of UAV VVA Aspect error: In the VVA, the UAV aspect data follows a nearly normal distribution at 100% and a normal distribution at 90% after the removal of one extreme outlier. This value records the polar opposite of the true northern aspect and is a blunder which heavily influences the distribution. As a result, the 100% RMSE and 95% confidence statistics are very high, at 62 degrees and 122 degrees (table 10). The 90% RMSE is improved to 21 degrees and is the most generous representation of aspect error, but due to the non-normal nature of the 100% data, is not the most appropriate means of reporting. Following ASPRS guidelines, the 95 th percentile should represent the VVA at 43 degrees. The significant positive relationship between slope and aspect error here suggest that incorrect slopes influence the aspect, as would be expected due to the interference of surface features creating their own slopes and aspects independent of terrain : SCOOP NVA Aspect Error The SCOOP aspect values prove to be resistant to the Normal distribution and require the most trimming of all of the data to reach a normal curve at 85% (table 4). At first glace the full dataset appears to be nearly normal, with a mean and median which are both positive with a discrepancy of less than two degrees. The standard deviation is 10 degrees, the skew is slightly positive, but the kurtosis is high at 10 and the maximum value of 45 degrees is much lower than the minimum error value of -15 degrees (table 4). The histogram displays the high peak and low tails of the high kurtosis value, with the positive values stretching out towards the 45-degree outlier. The QQ plot shows a tendency for values to meander around the line of slope with a flattening trend levelling the central values (figure 45). Normal testing shows very small p values reflecting low chances of incorrectly rejecting the Null hypothesis of normality (table 8). Figure 45: Histogram (left) shows high kurtosis and positive skew, and Normal QQ Plot (right) reinforces the non-normal distribution of SCOOP Aspect Error in the NVA. These p values remain small as the dataset is reduced to 95% and 90%, likely still vulnerable to high kurtosis. At 85%, with an N value of 30, the data finally reaches Normalcy, and shows low but acceptable p values at 95% confidence (table 8). The histogram shows more symmetry, the kurtosis is still high at 3.8 but much improved, the minimum and maximum values are more balanced, and the standard deviation is reduced (figure 47). The outliers removed from the data are dispersed throughout the study area (figure 46) and do appear to be related to stratum in that half of the outliers are located on the edge of the riparian. As with slope and elevation error, aspect error could be partially attributed to real changes over time, in areas where the VVA was reduced and the NVA increased, leading to actual changes in aspect. One of the outlying aspect errors is also a slope outlier, and a separate outlying point of aspect error is also an elevation error outlier. Two outliers are located in the center of the study area, where an aspect change form SW to SE could contribute to error. 76

77 Figure 46: SCOOP Aspect Error Magnitudes and Outliers in the NVA. Figure 47: 85% Histogram of SCOOP NVA Aspect Data shows improved skew and reduced kurtosis. 77

78 Regression Modelling of SCOOP NVA Aspect Error: Regression between compass bearings and SCOOP aspect at 85% data shows a high positive correlation and an excellent model fit (table 9). However, a statistically significant BP value suggests autocorrelation of the residuals. When investigated by running the spatial autocorrelation tool on the OLS residuals, a statistically insignificant Moran s I value indicates that the residuals are in fact stationary and error pattern is random (figure 48). Regression modelling of Aspect error vs elevation error reveals a weak and insignificant positive correlation. A weak insignificant positive correlation also exists between slope and aspect error (table 9). Figure 48: Regression residual distribution plots (left) show a strong positive relationship between true and SCOOP derived aspects in the NVA, and Spatial Autocorrelation report (right) of trimmed residual data indicates a random distribution of residuals at 85%. Reporting of SCOOP NVA Aspect Error: In the NVA, the SCOOP aspect is resistant to Normalcy and must be reduced to 85%. The 100% RMSE in this case is not an appropriate representation of the data at 10 degrees, nor are the 95% and 90% RMSE at 9 and 6 degrees (table 10). At 85%, the RMSE is reduced drastically to 1.2 degrees following the removal of 7 influential outliers. The 95% confidence statistic of 20 degrees reflects the pull of these outliers, and the 95 th and 90 th percentiles do as well at 17 and 16 degrees. While the RMSE statistics on the full dataset are not appropriate representations due to the Non-Normal distribution, the outliers they reflect should not necessarily be dismissed as errors; positive relationships between elevation error and slope and aspect error do suggest that true change over time could contribute to deviations in derived aspect from true aspect. Although these relationships are weak and do not prove to be statistically significant in the regression models, their presence nonetheless warrants further investigation before outliers are dismissed as error : SCOOP VVA Aspect Error Descriptive Statistics, Distribution Plots, Normalcy Testing: At 100% the Scoop data in the VVA shows a positive mean that is slightly above the negative median, a standard deviation of 18 degrees, a slight positive skew and a kurtosis value of 2. The minimum value of -21 degrees is relatively matched to the maximum value of 29 degrees, and the first and third quartile are relatively 78

79 symmetrical at -12 and 15 degrees (table 4). The data shows normality with non-significant p values in both the KS and SW tests (table 5). However, the histogram does not appear to be normal; it is skewed to the right by a positive outlier (figure 49). In this case the small sample size may be influencing the results of the Normal testing, though no warnings were generated during processing to allude to this. When trimmed to 90%, the data actually becomes less normal due to an increase in kurtosis. Figure 49: Histogram (left) shows positive skew, and Normal QQ Plot shows some deviation from the Normal slope in the SCOOP VVA Aspect Error data. Regression Modelling of SCOOP VVA Aspect Error: Regression modelling in the VVA shows an insignificant JB statistic, supporting the normalcy of the residuals. A large number of true values are modelled by the SCOOP aspect values with a strong positive correlation and an R squared value of 0.96 (table 9. Figure 50). Weak negative and statistically insignificant correlations are found between elevation error and aspect error and slope error and aspect error in the SCOOP VVA data (table 9). Figure 50: Regression Modelling of SCOOP Aspect vs. compass aspect in the VVA shows a fairly strong positive relationship. Reporting of SCOOP VVA Aspect Error: The SCOOP aspect error values in the VVA follow a Normal distribution, contrary to what is expected by most reporting standards. The 100% and 90% RMSE are fairly close together at 17 and 14 degrees, with the 95% confidence level increasing to 34 degrees and a 90 th percentile statistic of 26 degrees (table 10). Overall, the SCOOP VVA data shows lower accuracy than the UAV VVA aspect data, but more consistency in reporting values, and a normal distribution at a higher sample size, requiring less reduction than the UAV aspect data does. 79

80 6.0: Hypothesis Testing and Study Objectives CHAPTER 6 CONCLUSION This project has presented an empirical based case study of the accuracy of a Digital Terrain Model produced by a UAV. This involved elevation data acquisition through UAV flight; ground truthing through clinometer, compass and RTK GNSS field survey; digital photogrammetric processing to align, correct and georeference photos; point classification and editing to create a bare ground surface; interpolation in a GIS to create a DTM from a point cloud; extraction of surface derivatives slope and aspect, and a GIS based analysis of the spatial statistics and distributions of error. The results of the project addressed the objectives and hypothesis of this study as follows: 1) How accurate is the UAV derived DTM in predicting true elevation, slope and aspect values, as described by the error distribution and reporting statistics of RMSE, percentiles and 95% confidence level of the differences between GNSS RTK survey points and UAV values? The UAV DTM is of good accuracy. The project resulted in a 0.08 m DTM covering approximately 11 Ha with a 95% confidence level RMSEz of 0.25 m in open terrain; a 95% RMSEz of 0.10 m in open terrain; a 90 th percentile elevation error statistic of 0.34 m in vegetated terrain; a 95% confidence level of 1.33 degrees slope error in NVA; a 95% RMSE of degrees slope in the NVA, a 90 th percentile of degrees slope error in the VVA; a 95% confidence level of 9.31 degrees aspect error in the NVA; a % RMSE aspect in the NVA; and 90 th percentile aspect error of 43 degrees in the VVA. This VVA aspect error represents atypical error in the UAV data and is swayed by some high outliers in the data. 2) How do the error distributions of elevation, slope and aspect differ between the SCOOP 2m DTM and the UAV DTM, and are these differences significant? The error measures of the UAV data are lower than those of the SCOOP data for most reporting statistics, with the exceptions of slope and aspect error in the VVA. Therefore, the UAV data on average shows higher accuracy at a higher resolution than the SCOOP data. However, these differences are not considered to be significant in T Tests or Wilcoxon Tests, so it can be concluded form the evidence presented in this project that a lightweight UAV can create a DTM of approximately equal accuracy to the SCOOP data, with the benefits of higher resolution, temporal currency, and low acquisition cost, with the limits of being appropriate only for small study areas. The error distributions for all attributes tested required some amount of trimming to achieve a Normal Distribution and to be represented appropriately by error statistics. While some of the NVA data is nearly Normal at 100%, both major and minor outliers are a common and reoccurring problem in all of the data and require consideration in the analysis and characterization of error. 3) How does Land Cover Type influence the magnitude and distribution of error in the UAV DTM? The UAV DTM displays higher accuracy in the NVA than in the VVA for all attributes (elevation, slope and aspect) and for all reporting statistics with the exception of the 90 th percentile value, where the VVA value is less than half of the NVA error (table 10). On average, the UAV DTM is influenced by land cover, but not to the same extent as the SCOOP DTM, which reports higher discrepancies between NVA and VVA statistics. The error budgets of both NVA and VVA statistics are vulnerable to outliers and resist a Normal Distribution until these outliers are identified and removed, with the exception of slope error in the VVA. The NVA error does not follow a Normal Distribution at 100% as might be expected in open terrain, and in contrast to what may be expected in vegetated terrain, the UAV slope error in the VVA does follow a Normal Distribution at 100%. 80

81 4) Are there significant correlations in the data between elevation error and slope and aspect error, or between slope and aspect error? While most data show good model fit and strong positive relationships between true and modelled values (table 9), there are no significant correlations between elevation error and slope or aspect error, or between slope and aspect error. There are some moderate correlations between slope and aspect error in the VVA, and elevation error and aspect error in the VVA, but these relationships are not considered to be significant in regression modelling (table 9). The hypothesis of this project set out to prove that a lightweight UAV generated DTM will be capable of producing elevation point cloud data and a derived DTM with accuracy equal to or better than the error reporting statistics (RMSE, percentiles and 95% confidence) of the SCOOP dataset, with the benefits of temporal currency, low relative acquisition cost, and high resolution. It is also hypothesized that the accuracy of derived slope and aspects, measured by the same error statistics, will be better than or equal to the SCOOP DTM, and that there will be a statistically significant correlation between errors in elevation and corresponding errors in derived slope and aspect. It is also hypothesized that all variables being tested (elevation, slope and aspect) should follow a Normal Distribution in the NVA and a Non-Normal Distribution in the VVA. To this end, the project can conclude the following from the observations: The Phantom 3 Professional lightweight UAV used in the project can create a DTM of equal accuracy to the SCOOP data with the benefits of high resolution and temporal currency, as well as low cost acquisition for the small study area. The error reporting statistics can therefore result in a rejection of the Null Hypothesis and a verification of the Hypothesis presented in the objectives of this study regarding UAV DTM Accuracy. However, the hypothesis that there will be statistically significant relationships between elevation error and slope and aspect error, and between slope and aspect error, was not verified by this project, and the Null Hypothesis which assumes no significant relationships between these attributes should not be rejected. In addition, the Hypothesis which asserts an expectation of Normal Distributions in the NVA is not verified by this project, and in all cases outliers must be removed to achieve a Normal Distribution. In the VVA, the hypothesis that data follow a Non-Normal distribution is verified for all cases except UAV Slope in the VVA, which is Normal at 100%. The results of this project show that assumptions about the characterization of error in a UAV DTM should not be made without empirical evidence of the distribution from the dataset, and that in most cases, some trimming of data is required to appropriately and fairly represent the error. 6.1: Potential Sources of Error Error can be introduced to DTM construction in data acquisition, from the variable error budgets of the sensors and GNSS receivers used and from unfavourable climatic conditions, as well as in post processing, through inaccurate or incomplete point classification, inaccuracies in georeferencing, and interpolation error. Attempts were made to minimize these errors as follows: UAV Flight occurred in mid morning during leaf off canopy conditions on a slightly overcast day with low winds to minimize error resulting from shadows and positional variation. A highly accurate GNSS survey was conducted to measure GCPs and georeference the model with the highest accuracy possible. Digital photogrammetric processing was run at high quality levels in the photo alignment and dense cloud construction phases, and point classification was completed at both automated and manual levels in attempt to reduce error. Further point editing was undertaken in a GIS by utilizing the Hillshade function to identify errors in the initial elevation surface and further remove surface artifacts. To assess initial RMSE of the elevation errors, a TIN was derived form the point cloud. TINS are a commonly employed 81

82 methodology for interpolating elevations from point cloud data, since the point of the cloud is unlikely to fall directly on the survey point location (ASPRS, 2014). To create a raster surface, processing was undertaken using the LAS to Raster function in ArcMap. This method required the least amount of interpolation to be used in estimating the values between sample points, and resulted in a Raster surface derived directly from the point cloud with an RMSE equal to that of the TIN. Error can also be introduced through field survey methods, creating a scenario where the reference data is of questionable accuracy. In this project, the RTK GNSS survey was of high accuracy with an average value of 3 cm, so this source is considered to be error free for the purposes of this study. However, the same cannot be said for clinometer and compass verification of slope and aspect. The accuracy assessment of these attributes involves more subjectivity, and therefore a higher likelihood of error. Human error can be introduced in the reporting of clinometer values and compass bearings, and though effort was taken to carefully record these values, an undefinable amount of error is likely present in these verification methods. In addition, clinometer and compass are measured in to one decimal place, as any precision beyond this is impractical in the field, while the same values in computer models contain many decimal places. Therefore, error can be introduced in the rounding of computer generated values. Another point of consideration is the low density of checkpoints relative to the point cloud. RTK GNSS survey is time consuming and can be costly. While a higher number of checkpoints is desirable, this is not always practical. In this case, the field verification was limited by availability of survey equipment to a two-day campaign, in addition to the time it took to lay out check points, and lay out and measure Ground Control Points. One of the problems with DEM verification is the often-sparse nature of reference data; conclusions regarding accuracy must be drawn from a relatively low number of field truthed sample points (Oksanen and Sarjakoski, 2006). For example, in this project, the high-density point cloud produced points, while the RTK GNSS survey produced 45 reference points. This is obviously extremely unproportional, and could only be improved by using LiDAR or a similar type of high density reference data instead of RTK GNSS survey. However, GNSS survey is considered to be highly accurate (Rock et al., 2011) and is often of higher accuracy than LiDAR. So, a compromise between accuracy, density and availability of reference data must be reached. For this survey, due to availability, RTK GNSS survey was implemented with high accuracy but low density. Therefore, other types of reference survey may produce different results in accuracy assessments than what was shown by this study. Recommendations to improve on this study are as follows: 1. Use a higher density of accuracy assessment points. For the purposes of this study, conclusions had to be drawn from a very small number of sample points relative to the digital point cloud. This gave an estimation of accuracy in open and partially vegetated stratums. However, it did not provide a good assessment of places where the DTM was known to be wrong; for example, points directly over tree cover in the riparian that were observed to represent the height of the surface feature and not the bare earth below, due simply to the fact that no accuracy checkpoints existed that measured the tree height. The lack of high quality reference data has been identified as a limiting factor in the generation of reliable elevation data (Oksanen & Sarjakoski, 2006). LiDAR has been shown to be a reliable source of high density reference data (Oksanen & Sarjakoski, 2006; Rock et al., 2011) and is especially applicable for modelling and assessing larger extents. For this study area, reference data was limited by a lack of LiDAR data for the area. The OMNR and partners are currently in the final stage of LiDAR data collection for the Southern Ontario area under which this study area falls, however until this data is publicly available, reference data will continue to be limited. RTK GNSS is a highly accurate means of assessment and has been used in many studies as reference data (Turner et al., 2015). To improve upon this research using RTK GNSS, it would be desirable to increase the density of sample points to capture more variation in the landscape, and to test the accuracy of the DTM at areas of expected deviation such as at slope breaks, under shadows, and over vegetation. An investigation of relative positional accuracy of the UAV-DTM would also be a worthy undertaking. This project 82

83 looked at absolute positional accuracies in elevation (Z) but did not address error in X and Y. To assess this, field level terrain feature measurement can be used to assess the location accuracy in terms of positions of X and Y by mapping features such as the beginning and ending of ridges, stream banks or discrete objects such as trees or buildings, to provide a more through understanding of the error present at multiple levels in a UAV derived DTM or ortho product. 2. An additional avenue of research to pursue based on this study would be to test the ability of the UAV in a wider variety of land cover types. This study limited the defined stratums to Vegetated and Non-Vegetated, however within the VVA there are many micro types that could be stratified and tested, such as high and low riparian vegetation, open grassland, flood plain, and deciduous tree cover. The VVA accuracy of the UAV presented in this paper is higher than what may be achieved in more variable or complex vegetated terrain, and since UAV s have been identified as being challenged by the collection of data in vegetated terrain (Serban et al., 2015), it is a topic of interest to further investigate the performance of the UAV over variable surface types (ASPRS, 2014). Studies that have investigated this application are present in the literature (Rock et al., 2011; Tamminga et al., 2015) and UAV s have been shown to perform better than what may be expected, due partially to the advancement of digital photogrammetric processing, in areas such as river bank and stream reach mapping (Tamminga et al., 2015) where the ability of the UAV to map submerged topography in a vegetated river bed was documented. 6.2: UAV Regulations in Canada and the United States The Digital Elevation Model and its classified forms of DSM and DTM has been considered to be the most fundamental variable in GIS (Atkinson, 2002; Oksanen & Sarjakoski, 2005). As a critical layer to such a wide variety of applications and industry, the demand for high quality spatial data should maintain a parallel increasing demand for consistent and current standards of data quality. There are now a wide variety of ways to accurately map the earth with high detail and currency, and data becomes increasingly available. Evolving standards must keep up with technological advancements which may demand adjustments and revisions of less applicable models, as is the case in the advance of mapping standards to incorporate the current ability to remotely map the world as a high resolution, three-dimensional digital geographic surface. UAV s have potential as an emerging reliable platform for the collection of low altitude, high resolution aerial images, which can be used to generate high quality elevation surfaces and ortho rectified photos with the advantages of being a flexible, low cost option for small survey areas (Mingyao et al., 2015). While this study and a growing number of others have shown UAV s to serve a valuable purpose in scientific data collection, there are obstacles to consider regarding their application beyond the limits of their engineering, and the main one currently limiting their widespread use in industry is legislation. In the United States, national Air Space (NAS) is governed by the Federal Aviation Administration (FAA), under which the use of UAV s for civilian and commercial use is limited (Ritter, 2014). A detailed investigation is currently underway to revise FAA regulations to accommodate UAV applications, and in 2012 then-president Obama signed an appropriation to channel financial support to streamline the process of regulation revision in the FAA, and to streamline the system of permit application. Some of the concerns around lifting restrictions on UAV use include the need to better develop UAV sense and avoid technology, and concerns for national security and privacy. In 2016 the FAA released new guidelines for UAV s based on Visual Line of Site (VLOS) operations for lightweight aircraft. As of 2017 FAA revisions, commercial pilots must obtain a remote airman certificate by demonstrating aeronautical knowledge on a FAA approved exam, and conform to a number of provisions including an agreement to fly at low altitude and low speed. Recreational pilots are not required to obtain this certificate, but must operate under safe operation procedures outlined by the FAA and must register the aircraft if it is over.55 lbs (FAA, 2017). 83

84 In Canada, the UAV industry has not been supressed by regulation to the same extent as was experienced in the U.S, however, Transport Canada has its own set of regulations regarding UAV use, and the process by which administration evolved to keep up with the growing demand for safe flight permits has been a source of frustration to some commercial vendors. Lightweight UAV s are not required to be registered, and recreational use does not require any formal documentation, however to fly a UAV for commercial use or research, pilots must obtain a Safe Flight Operations Certificate (SFOC), which requires submission of an application and Site Plan Map to Transport Canada outlining risk management strategies, study area information, airspace class, and proximity to local airports, among other details. The application for an SFOC is becoming more streamlined with the introduction of an online application in January, 2017, and the July, 2017 release of updated regulations. Aviation insurance is also required to obtain an SFOC, and the rates for this insurance in Canada are subject to wide variation based on location and vendor. In the U.S, a very convenient an affordable on demand insurance policy can be purchased through an online app from a company called Verifly; single use to annual policies can be purchased to cover a range of applications for as low as $10 USD for a single flight, but unfortunately this option is not available in Canada, and pilots requiring an SFOC will likely be subject to paying expensive insurance rates until market competition begins to drive rates down. The alternative to an SFOC is to apply for an exemption from it; currently processing turnaround for SFOCs in Canada can be quite high, and exemptions allow for safe single flights to take place according to a declaration signed in an online application process through Transport Canada which agrees to abide by a series of provisional guidelines for safe flight operations. For the purposes of this study, an exemption from the SFOC was obtained from Transport Canada. 6.3: Future Applications of UAV Based Modelling While there are a wide variety of both established and emerging applications of UAV technology, there seem to be some key areas of interest in the field of natural resource management that could be well served by UAV technology. In the process of writing this paper, it became evident that there is a growing body of literature dedicated to the emerging use of UAV s as platforms for the collection of high quality scientific data. This paper set out to investigate only a few objectives selected from many avenues of potential research, and this meant navigating away from some objectives in order to pursue others. Some of these research paths are outlined below, as areas of noteworthy future research to pursue, based on the empirical evidence presented in this paper and the body of literature used to support it. 1) The use of UAV s to build Digital Surface Models, and an assessment of the accuracy of the UAV DSM. This type of investigation has been undertaken and documented in the literature (Ritter 2014) as an interesting application of forest management and arboriculture. The DSM is required in forestry to build Canopy Height Models (CHM) and obtain accurate biostatistics regarding tree height and growth patterns. The use of the UAV for this purpose is well suited; for creating spatial tree inventories, monitoring forest health and acquiring aerial imagery for spatiotemporal analysis (Ritter, 2014). UAV s are very well suited to obtain repeated imagery for purposes where this repeatability renders other means of image acquisition, such as high altitude aerial survey, impractical and costly. In his 2014 paper, Ritter demonstrates the cost and time savings generated by utilizing a UAV to generate a spatial tree inventory and complete what is defined by the International Society of Arboriculture (ISA) as a Level 1 Tree Risk Assessment, which is basically comprised of identification of the target tree and a limited visual inspection, and was shown to be successfully completed using UAV generated imagery. Since the UAV DTM often suffers from inaccuracies due to remaining surface features, a targeted application of the UAV-DSM is worthy of investigation to better understand the accuracy involved in intentionally mapping above ground surface objects. 84

85 2) Similar to this application of the UAV-DSM for forestry is the application of the UAV for generating vegetative health indexes such as the Normalized Difference Vegetative Index (NDVI) and to monitor nitrogen accumulation, crop yield and health, among other factors (Bareth et al., 2016). UAV based sensing is proving to fulfill a need in the agricultural industry to return valuable information about crops during the growing season, when vehicle and manually based surveys are not possible due to crop growth. The miniaturization of UAV based sensors applies to hyperspectral data as well as RGB imagery; vegetation indices can be generated from the Infrared (IR) and Near Infrared (NIR) spectrums to monitor growth and yield. Vegetative Indexes can be calculated directly from NDVI sensors, and also from ortho imagery using the Red-green-blue Vegetative Index (RGB-VI) (Bareth et al., 2016). Similarly, UAVs have been used for purposes such as crop biomass predictions (Caruso et al., 2017), mapping leaf area index (Tian et al., 2017), monitoring and managing crop pests (Salvatore et al., 2016), and forest pests (Lehmann et al., 2015), and to assess water stress (Gago et al, 2015). Emerging research is also showing UAV monitoring to be promising in determining areas of soil compression in agriculture, and even to predict the levels of organics present in soil (Deveron Resources, 2016). 3) Building on the lines of UAV image based data extraction, future research should also be directed towards accuracy assessments and applications of the high resolution ortho image produced by lightweight UAVs. The ability to zoom in at very close range to detect high resolution detail in the landscape holds potential for image based classification, and UAVs have been shown to provide high quality, accurate high resolution images for applications like vegetative typing (Ahmed et al., 2017), urban vegetative mapping (Van Iersel et al., 2016), juvenile tree typing (Huang et al., 2017) and weed control (De Castro et al., 2017). Many studies have focused on the utilization of the UAV for these resource monitoring purposes, but less have focused on the accuracy assessment of these georeferenced images. Figures 51-53, below, provide an example of the high resolution (0.10 m) ortho produced by this project at 3 different scales; 1:2500 shows a detailed and crisp ortho image where most terrain features can be easily distinguished (figure 51); 1:250 shows the detail retained by the high-resolution image when zoomed in (figure 52); 1:100 (figure 54) shows a discrepancy between a GCP (pink) and the marked RTK GNSS point, which equates to approximately 0.09 m. If relative positional error is inherent to the ortho photo, it would be assumed to carry over into other terrain products, since the ortho is built on the same georeferenced point cloud as the DSM. In the case of this project, the approximately 0.09 m absolute elevation error of the NVA DTM mirrors the 0.09 m relative positional error which is evidenced by the ortho photo. Classification based on a UAV ortho image could therefore be subject to an unpredictable amount of both relative and absolute positional accuracy, and a characterization of these errors could be considered an important component to any image based analysis. 85

86 Figure 51: 1:2500 UAV Ortho Figure 52: 1:250 Ortho shows the ability to identify terrain features in high detail. 86

87 Figure 53: 1: 100 scale show discrepancy of 0.09 m evident between RTK GNSS Survey point (red) and purple GCP, representative of relative positional error. 6.4: Summary The connection of industry and resource management to geography is intrinsic; technology which allows us to better understand and positively develop this connection by providing improved geographic models is highly valuable. Remote sensing has played a leading role in allowing us to monitor and map parts of the earth that were previously inaccessible due to a variety of factors ranging from political instability to geographic inaccessibility. It has also allowed us to document evidence of change in variables over time to a precedent setting extent; from remote sensing, the scientific community now has a better ability to predict, understand, and respond to phenomena such as flooding, landslides, volcanoes, and glacial recession, and to map the world with an unprecedented amount of detail and currency. The development of satellite monitoring has also allowed us to monitor and model social and ecological phenomena and interactions between people and the environment; visualize population dynamics, monitor and manage land use, land change, and biodiversity (Miles, 2013), at both global and local scales. UAV s have a role to play in this process, as supplementary platforms for the collection of high resolution local data. As technology develops, and remote sensing systems become increasingly accessible and affordable, ongoing accuracy assessments of error are important to the evolution of standards by which to align industry and product with best practices, and to drive changes in legislation and regulation that will, in turn, allow industry to grow. The empirical evidence presented in this paper presents the UAV-DTM as a reliable and accurate means of acquiring low cost, high resolution, and temporally current elevation data and its derivatives at the field scale. It also presents a thorough characterization of the error inherent to this data through investigations of distribution plots, descriptive statistics, normalcy testing, regression modelling, and both parametric and nonparametric accuracy statistics. Our physical world has been modelled in detail and our geography is now well defined, understood, and monitored; however, there are vast horizons on which to explore the applications of 87