Author(s): Document owner: Reviewed by: Workgroup: MESH action: Action 5.2: Case histories on the applications of habitat mapping

Transcription

1 Title: Author(s): Document owner: Reviewed by: Workgroup: MESH action: Case Study: The redundancy of acoustic and video ground truth survey data Bob Foster-Smith and Ian Sotheran (Envision Mapping, Ltd.) Bob Foster-Smith Clare Fitzsimmons, Envision Mapping, Ltd. Action 5.2: Case histories on the applications of habitat mapping Version: 0.1 DRAFT Date published: 06/02/2007 File name: ENV CS11 Redundancy of survey data.doc Language: English Number of pages: 17 Summary: Reference/citation: Keywords: During repeated surveys of Loch Maddy SAC, an area of high heterogeneity, different acoustic and ground truthing methods were tested. The analysis of video ground truth samples and AGDS acoustic data, and the scales at which the survey attempts to operate, may be major sources of variability in the final map products. This case study focuses upon lessons learned regarding the optimal deployment of video systems for ground truthing acoustic data (AGDS), and the use of AGDS itself, during this period. It examines the apparent redundancy of acoustic and video ground truth survey data gathered, and makes recommendations for future cost-effective survey design accordingly. ISBN or full reference. Sampling strategy, survey strategy, video sampling, ground truthing, acoustic ground discrimination, AGDS, biotope mapping, data redundancy Bookmarks: Related Links to other sections Links to related pages

2 information: MESH metadata: MESH metadata template: MESH

3 Case Study: The redundancy of acoustic and video ground truth survey data 1. Introduction 1.1. Goals of Mapping The study Trialling of AGDS and video monitoring techniques in Loch Maddy was part of a larger project Loch Maddy monitoring trials which contributed to the UK marine SACs LIFE project. This aimed to establish an appropriate programme of surveillance and monitoring for the Special Areas for Conservation (SAC) and to trial monitoring methodologies and protocols for the features of a site. The Loch Maddy project aimed to assist in the development of best practise for future monitoring and reporting programmes for other marine SACs in the UK and Europe. The major part of the study concentrated on the use of acoustic ground discrimination systems (AGDS) for biotope mapping, and made extensive use of previous mapping studies undertaken over a four year period, and provided valuable guidance for future studies on repeatability of mapping. This case study focuses upon lessons learned regarding the optimal deployment of video systems for ground truthing acoustic data (AGDS), and the use of AGDS itself, during this period. It examines the apparent redundancy of acoustic and video ground truth survey data gathered, and makes recommendations for future cost-effective survey design accordingly Pilot Sites Loch Maddy is representative of fjardic sea lochs on the coast of north-west Scotland. This site is exceptionally complex. It is predominantly shallow, with deeper water only in its entrance, and wave exposure grades from moderately exposed to extremely sheltered in the inner basins. There are numerous rocks and islands and at least 22 shallow sills and associated basins. The fjardic marine biotopes on this site are more diverse than on any other known site in the EU. There is a particularly wide variety of shallow tide-swept reef and sediment habitats and communities. Some of the holothurian species found in abundance on soft mud in the inner basins, such as the sea cucumber Labidoplax media, are considered rare elsewhere. There are dense beds of knotted wrack Ascophyllum nodosum ecad mackaii and a variety of kelp forest types that illustrate the wave exposure gradient in the loch. There are transitions to a complex system of lagoons, which have been selected in their own right as Annex I type Methods Summary Equipment for field survey: Video systems The system comprised of a surface TV unit, a co-axial umbilical (strengthened by polypropylene rope) which linked to the sub-surface unit. A Sony Hi8 video camera housed in a SeaPro housing (Greenaway Marine) was mounted in a stainless steel cage with RoHo video lights fitted. This system allowed real time video to be viewed on the surface during operations and the camera recorded the video footage. This system was used as the primary ground truthing tool deployed on sites from the 1996 and 1998 surveys. The position of the video during a tow was estimated from the position of the boat (stored on computer) linked to accurate times (on video), with allowances made for layback of the video sledge. The 1998 survey found that a series of short drops was preferable to a long tow because it became difficult to accurately determine the position of the camera on the seabed as Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 3 of 17 Last saved: 06/02/ :56

4 the boat drifted away from the initial drop point. A strategy of short, frequent video drops was therefore used to ground validate the acoustic data Equipment for field survey: Acoustic ground discrimination systems Acoustic ground discrimination systems (AGDS) are based on single beam echo sounders and are designed to detect different substrata by their acoustic reflectance properties. The strength of the echo and the way it decays with time produces a complex signal whose shape depends to a large degree on the nature of the sea floor and this is the basis upon which echo sounders have been used for sea floor classification. The extent to which sound is absorbed or reflected by the sea floor depends upon the hardness of the seabed: Hard surfaces produce strong echoes whilst soft surfaces (and this may include rock substrata that are acoustically softened by overgrowth of biota) results in a weak signal. The sound energy that spreads away from the centre of the cone produces a weaker echo. This wave energy takes slightly longer to reach the seabed because of the extra distance travelled and this time lag increases with increasing angular distance away from vertical axis of the transmission pulse. Rough surfaces will produce an echo that decays slowly since sound spreading some distance from the vertical may reflect off inclined surfaces angled towards the transducer (a property termed backscatter ) whilst flat surfaces will reflect sound away from the transducer. The decaying echo may also contain an element that depends on the reflectance of sound from subsurface features. This is particularly the case for low frequency sounders where there is greater penetration through soft surface sediment. The shape of this returning pulse or first return forms the basis for AGDS systems that map acoustic seabed properties to physical seabed properties The RoxAnn system uses analogue signal processing hardware to select two elements from the echo and measure signal strength (in millivolts) integrated over the time. The first selected segment of the echo is the decaying echo after the initial peak. This measure of time/strength of the decaying echo is termed Echo 1 (or E1 ) and is taken to be a measure of roughness of the ground. The second segment is the whole of the first multiple echo and is measured by the RoxAnn processor as Echo 2 (or E2 ). The two paired variables (E1 and E2) can be displayed on a Cartesian XY plot, and this is the basis of the RoxAnn real-time display as used in the data logging and display system Microplot. The AGDS used was a single frequency RoxAnn system operating at 200kHz with a Koden CVS-8112 echo sounder. The track spacing was set at 50m intervals, provided by EMU. A test area was used for calibration during all surveys. The RoxAnn and DGPS outputs were logged using Microplot. Microplot saved data every 2 seconds, and an average value for E1, E2 and depth were calculated over the two second interval and logged together with the ships position at the moment when the values are saved Analytical systems The data have been exported from the above logging software as ASCII text files and be imported into Excel. Data resulting from other analyses (e.g., accuracy assessment) will also imported into Excel for statistical analysis. MapInfo Professional is the geographic information system (GIS) in which most of the spatial display (including map design), the creation of vector layers (such as coastlines), spatial editing, spatial query and the creation of vector buffers is performed. Data can be imported from or exported to Excel, Idrisi and other programmes.

5 Case Study: The redundancy of acoustic and video ground truth survey data The data were imported into Surfer for Windows for interpolation of the track point data. This method was used where any quantitative analysis was performed on gridded values or where they were exported to the image processing and classification procedures. Classification was performed using Idrisi for Windows. Other operations were also performed in Idrisi, such as image enhancement. 2. The characteristics of the habitat type Previous surveys of the Loch had identified areas of high uncertainty and this was taken to be a measure of the heterogeneity of Sponish Harbour rather than simply a result of poor survey. It was suggested that heterogeneity was at a scale below the spatial resolution of the survey system and that a more statistical approach to sampling and the interpretation of survey data was required. Heterogeneity creates problems for remote survey when the patchiness of biotopes approaches or falls below the spatial resolution of the survey. The combined inaccuracies of AGDS and ground truth sampling means that there will be a wide spread of acoustic data associated with the life forms resulting in poorly defined acoustic signatures. Recognising these limitations, this case study aims to describe optimal techniques (e.g. deployment and analysis) for video ground truthing in such habitats. 3. Track spacing and redundancy of acoustic data Purpose To investigate the effects of increasing track spacing on the interpolated acoustic image to assess redundancy in the data Methods Apart from track redundancy, a supplementary analysis of the track data was performed (the calculation of the variogram) which has a direct bearing on interpolation Track redundancy Subsets of single AGDS data sets from the outer Loch and Sponish harbour were selected for this study since there was a high density of data for both of these areas. However, because of constraints imposed during the survey, the tracks were not regularly spaced nor did they necessarily run parallel to each other. In order to reproduce the effect of taking out selected parallel tracks on interpolation, the data set was first divided into a series of strips 50m wide running in a north/south orientation. Thus, the full data set against which redundancy was compared included the data from all of the strips (treatment 1). Alternate strips were selected or eliminated to simulate the effect of doubling the track spacing (treatment 2) and one in three strips were selected whilst the two were eliminated to simulate trebling the track spacing (treatment 3). Two data sets were generated for treatments 2 and 3. Standardised E1 and E2 and corrected depth were interpolated using inverse distance square interpolation, and a 25m grid spacing. A 200m search and display radius was used and any interpolated data that lay outside the boundaries of the test area were subsequently eliminated ( blanked ). This was necessary to ensure that every treatment resulted in the same number and arrangement of pixels, despite the variation in track data. Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 5 of 17 Last saved: 06/02/ :56

6 The differences in interpolated values between the treatments were calculated by subtracting the values from either treatments 2 or 3 from treatment 1. The overall differences in the values (summed for all of the grid values) are summarised in Table 1, expressed as percentages. The differences in the interpolated values can have an unpredictable effect on the biotope signatures and the maximum likelihood classification. The interpolated values from treatment 1 and examples from treatments 2 and 3 were classified using unedited signatures created using the drop down video ground truth samples that fell within the test areas in the outer Loch and Sponish Harbour (see Figure 1). A buffer of 25m radius was used around each ground truth point for extracting acoustic data for signature calculation. This was found to be large enough with the 25m gridded values to encompass sufficient data to run the maximum likelihood classification whilst not being so large that values too distant from the ground truth position were incorporated into signatures. It should be noted that unedited signatures were used for this Task (to standardise treatments). These unedited signatures will not produce interpretations that are directly comparable to other results. Crosstabulation was used to compare the similarity between the treatments ( Table 3) The Variogram A graph showing the variance between points at increasing separation illustrates the nature of the spatial correlation within the data and can be used to justify interpolation and/or indicate those areas where it is likely that the estimated values are simply the local average. These graphs are termed variograms. The average of the square of the differences between values at different lags (the term used to denote the distance that separates two points) are plotted against the lag distances. This has been calculated for the track data by first calculating the average distance between consecutive points and then finding the differences between n - n+1 for lag1; n - n+10 for lag 10 etc where lag 1 = the average inter-point distance; lag 10 = 10 x the average separation etc. This is shown in Figure Results and discussion The differences between the complete data set (treatment 1) and the reduced data sets are more marked as distance increases between the tracks, as would be expected. Again as might be expected, the largest differences are where there are gaps between tracks. These become especially marked in Sponish Harbour and the variable ground in the south east part of the outer Loch. The largest differences are localised to the edges of the data or those areas where the track spacing is actually greater than the inter-strip distance because of the irregular tracks of the original data set. Thus, the overall performances as indicated in Table 1 are probably conservative. However, this does point up the fact that the inclusion or exclusion of relatively isolated data can have a large impact on interpolated values.

7 Case Study: The redundancy of acoustic and video ground truth survey data Treatment 1 (track spacing~25 m) Treatment 2 (track spacing 50 m) Treatment 3 (track spacing 100 m) Biotopes Algal mat Mounds & burrows Sea-pens Turf on sand Crusts & turf Maerl/kelp L hyp forest L hyp park L sac forest L sac park Projection: Ordnance Survey 1936 Units: Metres Surveyors: R.Walton, I.Sotheran Date of survey: 05-19/10/1998 Survey site: Loch Maddy SNH project code: BAT/98/99/48 SNH nominated officer: A. Davison Kilometres Scale: 1:30,000 Figure 1: Interpreted biotope distribution for the comparison of different track spacing treatments. The full data set against which redundancy was compared included the data from all of the strips (treatment 1). Alternate strips were selected or eliminated to simulate the effect of doubling the track spacing (treatment 2) and one in three strips were selected whilst the two were eliminated to simulate trebling the track spacing (treatment 3). Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 7 of 17 Last saved: 06/02/ :56

8 Table 1: Percentage Error with increasing track spacing for E1and E2 Percentage Difference of E1 Percentage Difference of E2 Treatment 2 (A) 8.54% 11.26% Treatment 2 (B) 10.29% 14.22% Treatment 3 (A) 16.37% 22.51% Treatment 3 (B) 13.78% 18.93% The three examples of interpreted data are broadly similar in the outer Loch, but differences in Sponish Harbour are more obvious. If the classification using the full data set (treatment 1) is used as the yardstick, it can be seen from Table 2 that the overall similarity (taking all biotopes into consideration) between images falls from 78% when one half of the data are removed (simulating a doubling of track spacing) to 66% when two thirds of the data are removed (spacing is trebled). A similarly low similarity (64%) is found when treatment 2 is compared with treatment 3. Note that these similarities may seem high as compared with those obtained from the comparison of the 1996 and 1998 biotope maps. However, this study concentrated on a small sub-area and did not have the full range of biotopes found in the survey area as a whole. Table 2: Similarity (expressed as proportions) between interpreted biotope distributions. Treatment1 (approx. 25m track spacing) Treatment 2 (50m track spacing) Treatment 2 (50m track spacing) Treatment 3 (100m track spacing) 0.64 These similarities give a useful overall estimate of match between two maps. However, the match for individual biotopes is more complex and contains much valuable information about the degree of similarity between different biotopes. Inspection of the crosstabulation matrices (and example shown for a treatment 2A compared with treatment 1 in Table 3) indicates that similarities are more marked with some classes of biotopes than others. For example, the biotopes classed as mounds and burrows and seapens were similarly classified for treatments 1 and 2. These biotopes appear to form large homogeneous areas. Other biotopes, such as algal mats and many of the kelp biotopes have a low similarity value and they were often classified as a range of other biotopes in treatment 2. These biotopes with poor similarity values would appear to be smaller and/or have more complex shapes than the consistently classified biotopes. In other words, the area in which the former were (mounds and burrows and seapens), were found to be more heterogeneous. As a generalisation, those biotope classes that show the greatest discrepancy as track spacing increases are those with a large proportion of the pixels situated on or close to boundaries between classes.

9 Case Study: The redundancy of acoustic and video ground truth survey data Table 3: Example crosstabulation matrix comparing treatment 1 and 2A with values in percentages comparing similarities (the diagonal cells) for the different biotopes. Values in other cells are dissimilarities and indicate where confusion has taken place between biotopes between treatments 1 and 2A. Treatment 2 Algal mat Mound & burrows Sea-pens Turf/sand Treatment 1 Algal mat Mound & burrows Sea-pens Turf/sand Crust/turf Maerl Lhyp forest Lhyp park Lsac forest Lsac park Crust/turf Maerl Lhyp forest Lhyp park Lsac forest Lsac park Sponish Harbour and the outer Loch have very different characteristics regarding biotope composition and spatial heterogeneity. Large areas of the outer Loch would appear to be uniform whilst Sponish harbour is composed of a wide range of biotopes occurring in small patches. Are the similarities between the treatment different in the two areas? Sections of the interpreted images corresponding to Sponish Harbour and the outer Loch were separately compared using crosstabulation and the results confirm that the similarities are much higher for the outer Loch than for Sponish Harbour (Table 4 and Table 5). Table 4: Similarity between interpretations for Sponish Harbour. Treatment 1 (approx. 25m track spacing) Treatment 2 (50m track spacing) Treatment 2 (50m track spacing) Treatment 3 (100m track spacing) 0.45 Table 5: Similarity between interpretations for the outer Loch. Treatment 1 (approx. 25m track spacing) Treatment 2 (50m track spacing) Treatment 2 Treatment 3 (50m track spacing) (100m track spacing) Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 9 of 17 Last saved: 06/02/ :56

10 The Variogram The following graph (Figure 2) is based on data from (a) Sponish Harbour (where the ground is presumed to be variable over short distances) and (b) the outer Loch (where the ground is more homogeneous). These indicate that the spatial correlation is quite high over small distances and increases exponentially over moderate distances. The variance then levels off towards a maximum for the data sets (the sill ) and this point on the graph indicates the maximum range. It is probably reasonable to assume that this along-track variogram would also hold true for points between tracks and can be used to justify distances over which interpolation can be taken to give more information than simply the local mean. Athough the maximum range is similar for both areas (approximately 500m) the variance is much higher for the track in Sponish harbour. However, the spatial correlation is low at the maximum range and a smaller distance would be desirable. For example, it would be reasonable to doubt the reliability of interpolated data where the expected variance is one half of the maximum variance, giving a useful range of about 100m in each case. If very reliable estimated values are required (e.g., 0.2 x maximum variance), then a range of about 25m would be required for Sponish Harbour but 50m for the outer Loch. These ranges are useful as a means of expressing areas where the uncertainty in the interpolated values might be expected to be high because of distance away from the track. The differences in the variances also gives an indication on the variability in the acoustic reflectivity of the ground, being greater in Sponish Harbour than for the outer Loch. It is likely that this variability is also mirrored by variability in ground types Variance Sponish Outer Distance (m) Figure 2: Variogram for Sponish Harbour and Outer part of Loch Maddy 3.3. Conclusions and recommendations The minimum size of a pixel reflects the limits of the maximum resolution of the biotope map. There are technical limitations to this dimension that depend largely on position accuracy of the ship and the ground truthing methods. A track spacing which is greater than the pixel dimensions may lead to unacceptably large errors in mapping biotope distribution if a large proportion of biotopes are either small (represented by a few pixels only) or a complex shape (where a large proportion of the biotope area lies within one pixel of a boundary).

11 Case Study: The redundancy of acoustic and video ground truth survey data In heterogeneous areas, such as in Sponish Harbour where more than 50% of pixels lie on or close to a biotope boundary, increasing the distance between tracks to twice the pixel size (50m) significantly degrades the accuracy of the biotope distribution, although the overall pattern of biotope distribution remains much the same. When the track spacing is increased further (100m) the discrepancies in the biotope distribution become very obvious. In more homogeneous areas where much fewer than 50% of pixels lie on a biotope boundary, a wider track spacing has a less marked effect on biotope distribution. When deciding what track spacing to use, the following questions must first be answered: What are the smallest sized biotopes which are to be mapped and what is the maximum resolution required for their detection? How many of these small features could pass undetected by a survey without seriously detracting from the usefulness of the map (a function of heterogeneity)? How accurately do biotope boundaries need to be mapped? If the tolerance for error is small, then a track spacing of less than twice the pixel size is required. If a larger margin of error is acceptable, then a track spacing of less than three times the pixel size would give a reasonably consistent representation of biotope distribution patterns over heterogeneous ground although larger track spacing could be contemplated for more homogeneous areas. Track spacing has cost implications for survey. However, the contribution that track spacing has to overall survey costs is very dependent upon the size of the survey area. For small areas, such as the Loch Maddy trial area, the cost of mobilising and setting up the survey equipment is a large part of the total AGDS survey costs. Once on site and running, there is very little additional cost to intensively tracking the area. In such circumstances, perhaps the most efficient survey design is to err on the cautious and track intensively. For larger areas a compromise needs to be reached between accuracy and the cost of a tracking survey. This might entail nested survey. With such a survey design the broader scale tracking could detect and describe (in general terms) areas of high and low heterogeneity and representative sub-areas of high heterogeneity selected for more intensive survey. 4. Redundancy of ground truth data Purpose: To investigate the effects of decreasing the number of ground truth samples on the accuracy of the interpreted biotope distribution Methods The same subsets of single AGDS data sets from the outer Loch and Sponish harbour were selected for this task. The ground truth data used were supplemented by data from diver and ROV records to ensure that there were sufficient sample data within the test area to engineer different combinations of ground truth data used for classification of the image. The acoustic data were interpolated using a 25m grid spacing, as described previously. The subsets of the total ground truth data were generated for comparison with the full data set by grouping samples by life-form codes and eliminating the requisite proportion of records. Subsets with three quarters, one half and one quarter of the records were generated. The images were classified using a maximum likelihood classifier and are shown in Figure 3 and Figure 4. The classified images were subjected to (1) an internal accuracy matrix test, (2) Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 11 of 17 Last saved: 06/02/ :56

12 an external accuracy test, using the remaining ground truth data not used in the classification if appropriate, and (3) a comparative crosstabulation to assess image similarity. All Samples 3/4 of samples (A) 3/4 of samples (B) Biotopes Algal mat Mounds & burrows Sea-pens Turf on sand Crusts & turf Maerl/kelp L hyp forest L hyp park L sac forest L sac park Projection: Ordnance Survey 1936 Units: Metres Surveyors: R.Walton, I.Sotheran Date of survey: 05-19/10/1998 Survey site: Loch Maddy SNH project code: BAT/98/99/48 SNH nominated officer: A. Davison Kilometres Scale: 1:34,000 Figure 3: Interpreted biotope distribution for the comparison of different ground truth treatments.

13 Case Study: The redundancy of acoustic and video ground truth survey data Half of samples (A) Half of samples (B) Quarter of samples Biotopes Algal mat Mounds & burrows Sea-pens Turf on sand Crusts & turf Maerl/kelp L hyp forest L hyp park L sac forest L sac park Projection: Ordnance Survey 1936 Units: Metres Surveyors: R.Walton, I.Sotheran Date of survey: 05-19/10/1998 Survey site: Loch Maddy SNH project code: BAT/98/99/48 SNH nominated officer: A. Davison Kilometres Scale: 1:34,000 Figure 4: Interpreted biotope distribution for the comparison of different ground truth treatments. The results of the above analyses raised many questions as to heterogeneity and uncertainty (accuracy) of the interpreted acoustic images. Additional analyses were also performed to address the specific questions:- (1) What are the causes of uncertainty in classification? (2) How could this uncertainty be expressed? (3) How could the biotopes be shown, given the uncertainty of their classification? Analysis used the module Bayclass in Idrisi, which measures uncertainty and calculates the likelihood of each biotope being associated with all pixels in an image. The signatures created for the drop down video was used for these analyses. Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 13 of 17 Last saved: 06/02/ :56

14 4.2. Results and discussion Internal accuracy (Table 6) is variable but, although there is insufficient data for confirmation, it would seem that smaller data sets give a higher internal accuracy. This is perhaps not surprising since the signatures generated by a small data set may not have the spread of values of a larger data set. Thus, whilst the resulting interpretation may be internally accurate, the signatures do not fully represent the biotopes. Internal accuracy assessments are only useful as a check on the performance of the analyses and not the relative accuracy of different data sets. The accuracy measured against an independent data set gives an alternative indication of the relative performance of the different ground truth data sub sets. They show that there is a steady decrease in accuracy as the sample numbers are reduced. The exception to this is the comparison of the image as classified by the drop down video ground truth data with the ROV/diver observations. However, since the latter were confined largely to the heterogeneous Sponish Harbour area, they were not fully representative of the range of biotopes found in the whole area. There is also the possibility that these records were not as closely associated (spatially) with track data as the drop down video. In other words, the ROV/diver records are more reliant on data that has been interpolated at some distance from real track data. This points up the need for careful use of external data for accuracy assessment. Note that the overall accuracy assessment, although appearing to be low, gives a much better prediction than would be had by chance alone. The factor by which the interpreted map outperforms chance prediction can be expressed as a percentage (see Table 6) which represents the likelihood that the distributions were not accounted for by chance. Thus, values close to 100% indicate that the distributions are highly unlikely to be due to chance predictions whilst values close to 0% indicates that the distributions are no better predictors than those generated by chance. Whilst the percentages may seem to be low for many of the analyses, it must be remembered that near misses and possible confusion between very similar biotopes has not been taken into consideration. Ground truth subset Table 6: Accuracy (expressed as proportions) of the interpreted biotope distributions. Number of samples Internal accuracy Data used for assessing external accuracy All data N/A (all data used in Classification) Accuracy against external data N/A Three quarter (A) One quarter remaining (B) % Three quarter (B) One quarter remaining (C) % Half data (A) Half data (B) % Half data (B) Half data (A) % Quarter data (A) Three quarter remaining % Quarter data (B) Three quarter remaining % Drop down video ROV/diver records % Predictive capability The similarity between interpretations as calculated using crosstabulation (Table 7) indicates very little overall change takes place, as compared to the use of all the data, until over half of the ground truth samples have been removed from the treatments. N/A

15 Case Study: The redundancy of acoustic and video ground truth survey data Table 7: Similarity (expressed as proportions) between interpreted biotope distributions. ¾ (A) ¾ (B) ½ (A) ½ (B) ¼ All data ¾ (A) ¾ (B) ½ (A) ½ (B) 0.39 An alternative view of the effect of reduction in the number of ground truth samples can be gained by looking at the interpreted images (Figure 3 and Figure 4, above). There seems to be a convergence between interpretations in the overall patterns in the biotope distributions as more samples are incorporated into the analysis. However, with small numbers of ground truth points some obvious discrepancies between the interpretation of the acoustic data emerge. For example, kelp was predicted in the outer Loch where, with the benefit of larger numbers of ground truth points, sea pens might have been expected. This illustrates the unpredictable effects that low ground truth numbers can have on signature development and subsequent image classification. There is no denying that there is a high degree of uncertainty about the distribution of many biotopes over areas of high spatial heterogeneity. This uncertainty can be expressed as the probability of a hypothesis that a particular biotope is associated with a set of pixel values is incorrect. It can be calculated using Bayesian statistical classification techniques (the BAYCLAS module in Idrisi). The results have been plotted in Figure 5 and this indicates that there is a high degree of uncertainty about the interpretation of the data for Sponish Harbour whilst the uncertainty is low for much of the outer Loch. Uncertainty: probability scores Very high (uncertain) High Moderately high Moderately low low Very low Extremely low (certain) Projection: Ordnance Survey 1936 Units: Metres Surveyors: R.Walton, I.Sotheran Date of survey: 05-19/10/1998 Survey site: Loch Maddy SNH project code: BAT/98/99/48 SNH nominated officer: A. Davison Kilometres Scale: 1:10,000 Figure 5: Uncertainty of Classification Plotting the likelihood of particular biotopes gives a good picture of why this uncertainty exists. The Laminaria hyperborea forests and parks on bedrock overlap to a large extent. Similarly, the Laminaria saccharina forests and parks also overlap (Figure 6). There is also a substantial Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 15 of 17 Last saved: 06/02/ :56

16 overlap in Sponish Harbour between the Laminaria saccharina park and forest biotopes and the predicted distribution of maerl/kelp. However, there does seem to be a more or less distinctive distribution for the algal crusts/faunal turf in the centre of Sponish Harbour and soft mud (mounds and burrows) close to the south west shore. Laminaria hyperborea forest Laminaria hyperborea park Mixed turf on clean sand Sea-pens on fine sand Laminaria saccharina forest Laminaria saccharina park algal crust & sparse faunal turf Mounds & burrows on soft mud Projection: Ordnance Survey 1936 Units: Metres Surveyors: R.Walton, I.Sotheran Date of survey: 05-19/10/1998 Survey site: Loch Maddy SNH project code: BAT/98/99/48 SNH nominated officer: A. Davison Kilometres Scale: 1:35,000 Likelihood: probability scores Likelihood: probability scores Highly likely Very likely Likely Likely Highly likely Very likely Likely Likely Moderately unlikely Very unlikely Highly unlikely Extremely unlikely Moderately unlikely Very unlikely Highly unlikely Extremely unlikely Projection: Ordnance Survey 1936 Units: Metres Surveyors: R.Walton, I.Sotheran Date of survey: 05-19/10/1998 Survey site: Loch Maddy SNH project code: BAT/98/99/48 SNH nominated officer: A. Davison Kilometres Scale: 1:35,000 Figure 6: Probability distribution for individual biotopes Likelihood: probability scores Highly likely Moderately unlikely Very likely Very unlikely Likely Highly unlikely Likely Extremely unlikely 4.3. Conclusions and recommendations Internal accuracy measures are not a good measure of the predictive capability of maps. An external data set should be used for this purpose. If, as is usually the case with surveys, it is difficult to arrange a separate ground truth data collection exercise, then a proportion of samples should be removed from the classification process for this purpose. These data can be returned to the data set, a second batch removed and the classification repeated. The aim of this process would be to establish a convergence in the broad patterns of the biotope distributions between the different interpretations. Sufficient ground truth samples should be taken to ensure this. It would appear from the results of this task, although they are by no means conclusive, that increasing the number of samples per biotope category over a minimum does not greatly increase the overall predictive capability of the biotope maps. However, increasing confidence can be placed in the maps through establishing a convergence in the broad biotope distribution patterns. Even when a clear pattern emerges, it is likely that a residual variability will persist. In this study this figure would seem to be about 20% overall.

17 Case Study: The redundancy of acoustic and video ground truth survey data The root cause of this is likely to be the fine scale heterogeneity of some of the areas, especially Sponish Harbour. Two samples taken very close together can reveal different biotopes. The scale over which this occurs can be close to the spatial resolution of the acoustic data and near the limits of position accuracy of the ground truth methodology. This reinforces the conclusions presented in the previous section, that the required spatial resolution should be clearly established within what is technically possible. Issues of spatial heterogeneity, if at a finer scale than which is feasible to survey, must be addressed in some other way. The options are:- 1. Map heterogeneous areas as a special biotope group with a given proportion of biotopes in its composition; 2. Map areas by the likelihood of finding the different component biotopes within it 3. Specialist positioning methodologies for small representative areas where the position of the ground truth samples and the acoustic data can be recorded within the resolution required. If there is a high degree of uncertainty despite intensive sampling (as is the case in Sponish Harbour), then this is likely to be because many different biotopes could be predicted to occur. In such cases, uncertainty reflects heterogeneity rather than poor survey methods. It is possible that plotting the likely occurrence of individual biotopes may prove more useful than simply mapping the distribution of the most likely biotope, despite the extra maps involved. 5. Conclusions The closest track spacing that can be realistically achieved is of the order of 10-25m. However, there is little scope for increasing track spacing beyond 25-50m if the ground is heterogeneous without running the risk of increasing variability beyond acceptable levels. Areas the size of the Loch Maddy trial area can be covered at this track intensity relatively quickly and there would appear to be no cost advantages in increasing track spacing for such small areas. More homogeneous ground can be tracked with a greater track spacing. Thus, a spacing of 25-75m resulted in very little information loss for homogeneous ground, such as the majority of the outer Loch. Alternative strategies are suggested for larger areas where the ground is variably heterogeneous. The number of ground truth samples does not, above a minimum, increase the accuracy of a biotope map. The limits are set by ground heterogeneity, rather than the quantity of the data. It would be better to place effort and time into the accurate positioning of samples (if a categorical map of predominant biotopes is required) or use a more statistical description of biotope heterogeneity. Classification techniques using a maximum likelihood classifier are compared with alternative probability distribution maps and it is suggested that the latter help explain the apparently poor correspondence between biotope maps that are sometimes found. This is further discussed in case study ENV CS09 Prior Probability Classification.doc. 6. Acknowledgements These surveys were completed by Envision Mapping, Ltd (Formerly SeaMap Research Group of the University of Newcastle) for Scottish Natural Heritage, funded under the EC LIFE programme. Author: Bob Foster-Smith and Ian Sotheran Partner: Envision Mapping, Ltd. File: ENV CS11 Redundancy of survey data Page 17 of 17 Last saved: 06/02/ :56