Sampling Method and Sample Placement: How Do They Affect the Accuracy of Remotely Sensed Maps?

Transcription

1 Sampling Method and Sample Placement: How Do They Affect the Accuracy of Remotely Sensed Maps? Lucie Plourde and Russell G. Congalton Abstract The accuracy of remotely sensed forest stand maps is tra- ditionally assessed by comparing a sample of the map data with actual ground conditions. Samples most often comprise clusters of pixels within homogeneous areas, thereby avoiding problems associated with accurately mapping edges (e.g., transition areas between two forest types). Consequently, they may well overestimate accuracy, but the degree of overestimation is unknown. This paper examines two important factors regarding accuracy assessment that are not well studied: the effect on estimates of accuracy of (1) the sampling method and (2) the exact placement of the samples. Overall accuracy, normalized accuracy, and the KHAT statistic are computed from error matrices generated from simple random sampling, stratified random sampling, and systematic sampling using totally random sample placement and samples chosen from homog- eneous areas only. The results indicate that Kappa appears to be as appropriate to use with systematic sampling and stratified random sampling as it is with simple random sam- pling, but suggests that sample placement may have more of an effect on estimates of accuracy than sampling method alone. Introduction Once the sample map data are collected, they can be checked for accuracy against the reference data. A widely accepted procedure for comparing these data is the generation of an error matrix (Card, 1982; Congalton et al., 1983; Story and Congalton, 1986; Congalton, 1991). An error matrix is an especially effective accuracy assess- ment tool because it provides a starting point for a series of sta- tistical techniques to further examine accuracy (Congalton and Green, 1999). One such analytical technique is the Kappa anal- ysis, a discrete multivariate technique for comparing error ma- trices (Congalton et al., 1983; Hudson and Ramm, 1987; Con- galton, 1991; Ma and Redmond, 1995; Stehman, 1996; Steh- man, 1999; Congalton and Green, 1999). Kappa analysis, which assumes a multinomial distribution, generates a KHAT statistic that measures the difference between actual and chance (or ran- dom) agreement between the map and reference data. It can also be used to test for significant differences between two error matrices. The only sampling method that satisfies Kappa s assumption of a multinomial model is simple random sampling. The effect of other sampling schemes on the outcome of the Kappa analysis has not been well studied. In addition, samples are of- ten chosen only if they occur within the interior of homogeneous pixel groupings in order to avoid problems with sampling along boundaries of several cover types or in areas of mixed pixels. Limiting the sampling to homogeneous areas of vegetation may inflate the accuracy measure for the map, but the magnitude of this inflation is unknown. Therefore, the specific objects of this project were to (1) evaluate the effects of sampling method on map accuracy, specifically the Kappa sta- tistic, and (2) explore various sample placements to investigate the effect on map accuracy. All results will be evaluated by gen- erating an appropriate error matrix and Kappa analysis. The demand for remotely sensed imagery to develop forest maps for natural resources continues to increase. Remotely sensed forest stand maps are used in decisions regarding land use, resource treatments, water quality, forest health, and wild- life habitat, to name only a few. In order to make effective, intelligent decisions, the data must be accurate and reliable. For this reason, accuracy assessment has become an increasingly important component of any map generated from remotely sensed data. The problem is that it remains expensive and time consuming. Determining not only what level of accuracy is acceptable, but how effective various assessment methods are Methods will be key to solving this problem. Study Area The most reliable way to check the accuracy of a remotely This research focused on two study sites in a forested environsensed forest vegetation map would be by comparing all the ment in southeast New Hampshire, together comprising more data in the remotely sensed map with actual ground conditions. than 3500 hectares. The entire area is in the transition hard- However, this would require a detailed inventory of ground wood-white pine zone, with the most common species includdata, and if such a dataset already existed, the remotely sensed ing the following: white pine (Pinus strobus), hemlock (Tsuga data would be unnecessary. In reality, then, a sample of data canadensis), red oak (Quercus rubra), red maple (Acer rubrum), from the classified imagery (map) must be checked with ground American beech (Fagus grandifolia), sugar maple (Acer sacconditions (reference data). Various sampling schemes have charum), shagbark hickory (Carya ovata), white ash (Fraxinus been suggested for selecting the sample data from the remotely sensed map, from simple random sampling to stratified systematic unaligned sampling (Hord and Brooner, 1976; Ginevan, 1979; Fitzpatrick-Lins, 1981; Congalton, 1988; Stehman, Photogrammetric Engineering & Remote Sensing 1992). Vol. 69, No. 3, March 2003, pp Department of Natural Resources, University of New Hampshire, Durham, NH (russ.congalton@unh.edu) /03/ $3.00/ American Society for Photogrammetry and Remote Sensing PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

2 americana), white oak (Quercus alba), black oak (Quercus velu- the image. Homogeneous areas were then identified in the imtina), black birch (Betula lenta), yellow birch (Betula lutea), age, using ERDAS IMAGINE s AOI and SEED tools, to be used as and some areas of red pine (Pinus resinosa). training areas to train the classifying algorithm. A total of 118 One of the study sites occupies three-quarters (1670 ha) of forest polygons were selected. Pawtuckaway State Park. This site has been described as wild forest (Pugh and Congalton, 2001), protected from development and used only lightly by people (Irland 1982). The terrain Data Exploration/Image Processing in this study area ranges from rocky outcrops to low-lying wetband Before performing any image analysis, the thermal infrared lands, with a maximum elevation of some 305 m and a minithis was removed from the raw seven-band image because mum elevation of approximately 76 m. The second study site band has such coarse resolution (120 m) and has not been consists of 1870 ha of privately owned land northeast of and shown to add essential information for forest vegetation delinpartially adjacent to the public land area. Most of this private eation. Five bands were then generated and added to the six- land study site is rural forest with some open fields, while band raw TM image to determine whether these would be useful about a quarter is industrial, owned by a lumber company. for delineating the ten forest classes identified for this project. The five bands were (1) vegetative index (TM band 4 TM band 3), (2) normalized difference vegetation index (NDVI), (3) the ra- Intensive ground reference data were collected for the entire tio of TM band 5 to TM band 4, (4) the ratio of TM band 4 to TM study area and were available in a relational database. Pugh band 3, and (5) the first principal component of the visible and Congalton (2001) inventoried both sites by first delineating bands. The NDVI band was scaled from 0 to 255 to simplify the forest stand boundaries on color infrared National High-Alti- differences between vegetation density/greenness and bare tude Photography (NHAP) with a 1:58,000 nominal scale en- soil/developed areas (Klöditz et al., 1998). The ratios of band 5 larged to 1:15,800. They then performed field observations to 4 and 4 to 3 were scaled from 0 to 164, where 164 was the from the summer of 1994 to the winter of 1995 to acquire over- maximum value among all the raw bands. The histograms for story species composition. The inventory was performed by all 11 bands were analyzed, and visual pairwise comparisons walking transects through forest stands delineated from NHAP were made to ascertain potentially useful band combinations and recording the overstory species composition in 2-ha mini- for discriminating forest stands. Next, signatures for the fieldmum mapping units. The classification system used was a verified training areas were recorded from the 11-band image modified version of that used by the Society of American For- and examined empirically using spectral pattern, histogram, esters (SAF), the system currently used by the State of New and bi-spectral plot analyses. Finally, a divergence (separabili- Hampshire. This classification scheme is totally exhaustive, ty) analysis was used to statistically determine the bands that mutually exclusive, and hierarchical. Ten forest cover classes could best discriminate the ten forest classes. Together, these were identified using this system (Table 1). analyses suggested that visible band 3 (red), the two middle in- Forest stand types (polygons) were then entered into a geofrared bands (5 and 7), and the NDVI were the best combination graphic information system (GIS) and the forest inventory varifor maximizing the classification based on the training areas. ables (attributes) were entered into a database management system. The result was a map layer for each study area that included spatial data and forest inventory attributes. These reference Classification maps were then registered to the New Hampshire State Plane A combination of supervised and unsupervised classifications Coordinate System (North American Datum (NAD) 1983) and (Chuvieco and Congalton, 1988) was performed on the July converted from vector to raster format using ERDAS IMAGINE 1996 TM image. First, a supervised classification of the image (v.8.2), with a pixel size equivalent to approximately 30 m 2 was performed on the four best image bands as determined (Plate 1). from the data exploration analyses, using maximum likelihood Classification of Satellite Imagery with a first-pass parallelepiped algorithm. The signature statistics from this classification were saved and entered into SPSS (v. Data Acquisition 6.1.3), a statistical software package. Second, an unsupervised A July 1996 Landsat Thematic Mapper (TM) scene was acquired classification was performed with 70 classes and 20 iterations for the study area, and a subset comprising both study sites was or a 0.95 convergence threshold. The statistics from the resultcreated. Using ERDAS IMAGINE (v.8.2), the imagery was then regsification, and a cluster analysis was performed using the ing signature file were added to those from the supervised clas- istered to the New Hampshire State Plane Coordinate System based on Pugh and Congalton s (2001) reference imagery, with squared Euclidean distance (Chuvieco and Congalton, 1988). A an RMS error of 0.13, or less than 4 m. Next, various visual comcomplete linkage, or furthest distance method (Romesburg, dendrogram was produced from the cluster analysis using a posites and histogram stretches were explored in order to select the best composite to visually delineate homogeneous areas in 1984; Chuvieco and Congalton, 1988). From this dendrogram, unsupervised clusters were labeled, classes that were not spectrally unique were merged, and poor training areas were deleted. TABLE 1. FOREST VEGETATION CLASSES IDENTIFIED IN STUDY AREA Three iterations of the cluster analysis were performed, resulting in a training statistics signature file that contained 67 Forest Vegetation Class Abbreviation training areas. This signature file was used for the final supervised White Pine WP classification. White Pine-Hemlock WH Hemlock HE Other Conifer OC Post-Processing Red Maple RM After the two study sites were classified, a 5 by 5 majority filter Mixed Forest MX was applied to each (Plate 2). By aggregating pixels and reduc- Oak OAK ing speckle in the image, the filter effectively increased the Beech BH minimum mapping unit (MMU) from 900 m 2 to 22,500 m 2 -a Other OTH better match to the 2-ha minimum mapping unit of the refer- Nonforest NF ence data. 290 March 2003 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

3 Plate 1. Forest vegetation reference images. Plate 2. Forest vegetation maps from classification of July 1996 Landsat TM image with 5 by 5 majority filter applied. Accuracy Assessment case) for all categories is then selected as the desired sample size (Congalton and Green, 1999). A 95 percent confidence Sample Size level was chosen, and the absolute precision was set at The The number of samples for each study site was chosen by using value for B was obtained from a chi-square table, where (1, /10) the following equation from the multinomial distribution: For the public land study site, the largest n was 607, so 600 was chosen as the desired sample size. For the private n B i (1 i )/b 2 i (1) land study site (a slightly larger area), the largest n was 737, so 700 was chosen as the desired sample size. Both sample sizes where B is the upper ( /k) 100th percentile of the 2 distriburule-of-thumb were greater than 500 close to Congalton and Green s (1999) tion with one degree of freedom, II i (i 1,..., k)isthe proportion of a minimum of 50 samples per class. A 3 by 3 cluster of pixels was selected as the sample unit instead of a single pixel in an effort to minimize misregistra- tion issues and to better match the MMU of the reference data. of the population in the ith category, and b is the absolute precision of the sample (Congalton and Green, 1999). This equation is calculated for each of k categories, and the largest n (i.e., worst PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

4 Multivariate Analysis From each of the error matrices (the six sampling schemes plus the total enumeration) for each of the two study sites, overall accuracy was calculated. Next, a Kappa analysis was performed on all the matrices and a KHAT statistic was generated for each. Because the KHAT statistic is asymptotically normally distrib- uted (Congalton and Green, 1999), it could be tested for significance based on the standard normal distribution (i.e., to assure that the classifications were better than a random classification). Pairwise analyses were also performed to test for signifi- cant differences between the error matrices. Overall accuracy, normalized accuracy, and the KHAT statistic for each of the seven error matrices in each study area were then compared and analyzed for within-matrix and between-matrix differences. Results Overall accuracies for the public land study area ranged from 42 percent to 64 percent (Tables 3 and 4), while overall accuracies for the private land study area were generally lower, ranging from 30 percent to 58 percent (Tables 5 and 6). For both study areas, the lowest accuracy (i.e., poorest agreement between the map and reference data) resulted from the total enumeration of pixels. The best accuracy (i.e., the highest agreement between the map and reference data) resulted from sampling in homogeneous areas, with simple random sampling in the private land study area, and stratified random sampling in the public land study area. Sample Methodology In order to explore the effect of sample placement, two criteria or rules for sample acceptance were established. One rule, called the majority rule, compared the majority value of the 3 by 3 sample unit to the majority value of the same unit in the reference data. Using this rule, sample placement was arbitrary; that is, no consideration was given to whether a sample fell along an edge/boundary or in an area of mixed pixels. The second rule was the homogeneous rule, in which samples were limited to homogeneous areas: i.e., all nine pixels in the sample unit had to be equal in the map data. This value was then compared to the majority value of the sample unit in the reference data. Both rules were applied to three sampling methods simple random sampling, stratified random sampling, and systematic sampling so that, in effect, six sampling schemes were employed (Table 2). In addition to the six sampling schemes, a total enumeration of the map and reference data comparing each pixel in the classified maps to the respective pixel in the reference data was also generated. This total enu- meration was then available to compare with the other six sampling results. The sampling program was run ten times for each of the six sampling schemes, and an error matrix was generated for each trial. Sampling Method The Kappa analyses of error matrices for all the sampling schemes indicated that the classifications for both the public land and private land study areas were better than random clas- sifications (Table 7). Variances of KHAT were much the same for all sampling schemes for both study areas; confidence intervals for the KHAT values among sampling schemes also showed lit- tle variation. The 95 percent confidence interval for KHAT for all six sampling schemes for both study areas ranged between 0.04 and Further, the Kappa analysis of the repeated trials of simple random sampling, stratified random sampling, and systematic sampling using both the majority rule and the homogeneous rules revealed small variances in the Kappa statistics, confidence intervals, and variances for each sampling scheme. For example, ten trials of simple random sampling using the ma- jority rule in the public land data set produced similar esti- mates of KHAT (ranging from 0.31 to 0.37), with confidence in- tervals for all ten Kappa statistics always approximately 0.05, and their variance always approximately (Table 8). Indeed, the largest variance in KHAT for ten trials of Kappa (resulting from stratified random sampling with the majority rule in the private land data set) was only The 95 percent con- fidence interval was nearly always between 0.04 and 0.06, with a negligible variance (approximately ) in confidence intervals for all sampling schemes in both study areas. Similarly, variances for KHAT statistic variance overall were small less tha in most every case (Table 8). This replication of Kappa analyses further demonstrated Kappa s appropriateness with sampling schemes other than simple random sampling. Sample Placement For the public land study area, the KHAT value from the total enumeration (0.31) indicated the poorest agreement between the map and reference data, followed by systematic sampling and simple random sampling using the majority rule (Table 9). The remaining four sampling schemes resulted in KHAT values that indicated moderate agreement. The KHAT values for the private land study area were lower than those for the public land, with values from the total enumeration and sampling methods using the majority rule all indicating poor agreement between map and reference data. On the other hand, all three TABLE 2. SAMPLING SCHEMES EMPLOYED TO ASSESS THE ACCURACY OF THE LAND-COVER MAP GENERATED FROM THE JULY 1996 LANDSAT TM IMAGE Sample size Public Private Sample placement Sampling method Sample unit land land Homogenous rule (sample limited to uniform areas/stands) Simple random sample 3 3 pixel cluster Stratified random sample pixel cluster Systematic sample pixel cluster Majority rule (random sample placement) Simple random sample 3 3 pixel cluster Stratified random sample pixel cluster Systematic sample pixel cluster (not applicable all pixels are compared with reference data) Total enumeration single pixel The desired sample sizes calculated from the multinomial distribution for the public and private land study areas were 600 and 700, respectively. This desired sample size could not always be attained within the constraints of stratified random sampling and systematic sampling. Therefore, sample sizes as close to 600 and 700, respectively, were accepted. 1 Stratifed by forest class, with strata sample sizes chosen proportionately by area represented by each forest class on the map and overall desired sample size. 2 Sampling interval was every third column and fourth row. 3 Sampling interval was every fifth column and every sixth row. 292 March 2003 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

5 TABLE 3. ACCURACY ASSESSMENT OF CLASSIFIED JULY 1996 LANDSAT TM IMAGE USING A TOTAL ENUMERATION OF PIXELS: PUBLIC LAND STUDY AREA Map Data WP WH HE OC RM NF MX OAK BH OTH Row Total WP WH HE OC RM NF MX OAK BH OTH Col. Total Overall Accuracy: 0.42 TABLE 4. ACCURACY ASSESSMENT OF CLASSIFIED JULY 1996 LANDSAT TM IMAGE USING STRATIFIED RANDOM SAMPLING LIMITED TO HOMOGENEOUS 3 BY 3 CLUSTERS: PUBLIC LAND STUDY AREA Map Data WP WH HE OC RM NF MX OAK BH OTH Row Total WP WH HE OC RM NF MX OAK BH OTH Col. Total Overall Accuracy: 0.64 TABLE 5. ACCURACY ASSESSMENT OF CLASSIFIED JULY 1996 LANDSAT TM IMAGE USING A TOTAL ENUMERATION OF PIXELS: PRIVATE LAND STUDY AREA Map Data WP WH HE OC RM NF MX OAK BH OTH Row Total WP WH HE OC RM NF MX OAK BH OTH Col. Total Overall Accuracy: 0.30 sampling methods that used the homogeneous rule resulted in KHAT values that indicated moderate agreement (Table 9). Overall accuracies calculated for all six sampling schemes and the total enumeration for both study areas agreed with the KHAT values in relative order of accuracy (Table 9). Estimates of normalized accuracy, however, were not ranked in quite the same order as estimates of overall accuracy and KHAT. Nevertheless, in all but one case, all three measures of accuracy indicated that using the homogeneous rule resulted in the highest estimates of accuracy (see boldface in Table 9). Pairwise comparisons of KHAT values for all six sampling methods were nearly exactly the same for both study areas. Z test statistics indicated that with only one exception the results of the total enumeration were always significantly different from those of any other sampling method (Figure 1). In addition, in all but one of 18 tests for significant difference between sampling schemes (nine for each study area), the KHAT values from sampling methods using the homogeneous rulewerestatistically significantlydifferentfromthoseusing the majority rule (Figure 2). Conversely, when all sampling methods that used the homogeneous rule were tested against each other for significant difference, in all but one case they were not significantly different. Similarly, in all but two cases of pairwise comparisons of sampling methods using the ma- PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

6 TABLE 6. ACCURACY ASSESSMENT OF CLASSIFIED JULY 1996 LANDSAT TM IMAGE USING RANDOM SAMPLING LIMITED TO HOMOGENEOUS 3 BY 3CLUSTERS: PRIVATE LAND STUDY AREA Map Data WP WH HE OC RM NF MX OAK BH OTH Row Total WP WH HE OC RM NF MX OAK BH OTH Col. Total Overall Accuracy: 0.58 TABLE 7. KAPPA ANALYSIS RESULTS FOR ERROR MATRICES GENERATED WITH VARIOUS SAMPLING SCHEMES (NOTE: AT THE 95 PERCENT CONFIDENCE INTERVAL, THE CRITICAL VALUE FOR THE ZTEST IS 1.96.) Public Land Study Site Private Land Study Site Sampling Scheme KHAT Variance Z Statistic KHAT Variance Z Statistic Total enumeration Majority Rule Simple random sampling 95% confidence interval Stratified random sampling 95% confidence interval Systematic sampling 95% confidence interval Homogeneous Rule Simple random sampling 95% confidence interval Stratified random sampling 95% confidence interval Systematic sampling 95% confidence interval TABLE 8. VARIANCE OF KHAT VALUES, 95 PERCENT CONFIDENCE INTERVALS, AND VARIANCES FOR TEN TRIALS OF SIMPLE RANDOM SAMPLING, STRATIFIED RANDOM SAMPLING, AND SYSTEMATIC SAMPLING USING BOTH THE MAJORITY RULE AND THE HOMOGENEOUS RULE Public Land Study Area Private Land Study Area KHAT 95% Conf. Int. Variance KHAT 95% Conf. Int. Variance Simple random sampling (10 trials) Majority rule Min-Max: Variance E E E E-10 Homogeneous rule Min-Max: Variance: E E E E-11 Stratified random sampling (10 trials) Majority rule Min-Max: Variance: E E E E-10 Homogeneous rule Min-Max: Variance: E E E E-10 Systematic sampling (6 trials) Majority rule Min-Max: Variance: E05 1.6E E E-10 Homogeneous rule Min-Max: Variance: E E E E-08 Overall variance E E E E-09 jorityrule, theestimatesofaccuracywerenotsignificantlydifferent (Figure 2). Discussion than simple random sampling specifically stratified random sampling and systematic sampling have little effect on the Kappa analysis, despite Kappa s assumption of a multinomial model. This may be good news for mapping professionals who are bound by specific sampling methodologies. At the same time, limiting sample placement to homoge- Consistently small confidence intervals and little difference in variances for KHAT values suggest that sampling schemes other 294 March 2003 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

7 TABLE 9. COMPARISON OF OVERALL ACCURACY, NORMALIZED ACCURACY, AND KHAT FROM ERROR MATRICES GENERATED WITH VARIOUS SAMPLING SCHEMES Public Land Study Area Private Land Study Area Overall Normal. Overall Normal. Sampling Scheme Accuracy Accuracy KHAT Accuracy Accuracy KHAT Total enumeration Majority rule : Simple random sampling Stratified random sampling Systematic sampling Homogeneous Rule : Simple random sampling Stratified random sampling Systematic sampling in homogeneous areas only, even if the forest vegetation and reference maps are misregistered, the samples are more likely to fall within the same vegetation stand boundary (Figure 3). Misregistration and mixed-pixel problems may also expound error in estimates of accuracy produced from a single pixel, and indeed may contribute to the low estimates of accuracy produced by the total enumeration. The ability to locate exactly a 30-m Landsat TM pixel on the ground is complicated by a multitude of issues, including inaccurate map coordinates, GPS error, data conversion (e.g., vector to raster), and forest-stand boundary overlap. Therefore, examining single-pixel accuracy, such as with the total enumeration in this experiment, may not be the most meaningful method for assessing accuracy. What may be more useful to mapping professionals is how well forest types or stands can be mapped not how accu- Figure 1. Pairwise comparison of KHAT values for sampling rately individual pixels can be mapped. For these applications, schemes and total enumeration. Bars to the right of the selecting samples within homogeneous areas can be further vertical line indicate KHAT values that are statistically signifijustified. cantly different from a total enumeration; bars to the left Nevertheless, mapping professionals must be cautioned to indicate KHAT values from sampling schemes that are not consider the mapping objective when choosing the appropriate significantly different from the total enumeration. Lengths sampling scheme and sample unit. For example, if the goal of of the bars indicate magnitude of significant difference. the mapping project is to map vernal pools within forests, then sampling in homogeneous areas may not be appropriate. Rather, a more specialized sampling scheme that involves sampling probable areas may be better suited to assessing the accuracy neous areas resulted in the highest accuracies. That is, the of the map. Further, because vernal pools are small, a 3 by 3 highest estimates of overall accuracy, normalized accuracy, and cluster of 30-m pixels would be inappropriate. (Indeed, 30-m KHAT resulted from using the homogeneous rule. A Z-test on Landsat TM data would not likely be suitable for such a project; the KHAT statistics indeed indicated that sampling in homoge- finer resolution data would be required.) In other words, samneous areas produced significantly higher estimates of accura- pling in homogeneous areas can be justified in virtually any cy than did random sample placement, suggesting that sample landscape or geographic area as long as the goal of the mapping placement has more of an effect on accuracy than does the sam- project is in fact to identify homogeneous areas such as forest pling method. In other words, regardless of whether simple ran- stands. However, if the map objective is to identify anomalies, dom, stratified random, or systematic sampling was used, the rareties, or small features that do not necessarily comprise hohomogeneous rule consistently produced the highest estimates mogeneous areas, sampling in homogeneous areas may well of accuracy. not be appropriate. Systematic sampling and simple random sampling appeared to produce more conservative estimates of accuracy than did stratified random sampling. Consequently, one might Conclusions consider using either of these with the majority rule in an effort This research project has shown that overall accuracy, normalto avoid overstating accuracy by using the homogeneous rule. ized accuracy, and the KHAT statistic are equally effective at es- Systematic sampling in general, however, can be problematic timating accuracy. Moreover, the Kappa analysis appeared to be for selecting samples from a remotely sensed map if the interval equally appropriate with simple random sampling, stratified coincides with periodicity of error in the map. In addition, sys- random sampling, and systematic sampling, suggesting that tematic sampling can coincide with spatial autocorrelation of Kappa is as effective with simple random sampling as with forest-cover types, and either over- or undersample (or commultinomial model. sampling schemes that do not meet the assumptions of the pletely omit) certain forest types from the sample. Further, using the majority value in a sample unit may introduce error if Overall and normalized accuracy as well as the KHAT statistic indicated that sampling only in homogeneous areas prothe map and reference data are misaligned. For example, if a 3 by 3 cluster is sampled in a classified forest vegetation image duced the highest estimates of accuracy, suggesting that sample map and checked against reference data that are not properly aligned with the map, the sample may be incorrectly recorded as an error (Figure 3). On the other hand, if samples are selected placement has more of an effect on estimates of accuracy than does the sampling method. Regardless of sampling method: i.e., whether systematic sampling, simple random, or stratified PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March

8 Figure 2. Pairwise comparison of KHAT values for sampling schemes. Top third of graph tested KHATs from sampling methods using the homogeneous rule against other sampling methods using the homogeneous rule. Middle third of graph tested KHATs from sampling methods using the majority rule against other sampling methods using the majority rule. Bottom third of graph tested KHATs from sampling methods using the homogeneous rule against sampling methods using the majority rule. Bars to the right of the vertical line indicate statistically significantly different KHAT values; bars to the left indicate KHAT values that are not significantly different (length of the bars indicate magnitude of significant difference). For example, the topmost bar in the graph indicates that KHATs for stratified random sampling and systematic sampling resulting from sampling with the homogeneous rule were significantly different from one another. Figure 3. Comparison of results of sampling using the majority rule and the homogeneous rule with misregistered classifications and reference maps. In the top example, the majority value of the sample unit in the map does not match the majority value of the corresponding sample unit in reference data because the two are misaligned (a). This comparison would result in a false error report. When samples are taken from homogeneous areas, however, the sample units usually fall within the same forest stand even if the map and reference data are misregistered (b). In the second example, because a single pixel is used as the sample unit and the map and reference data are misaligned, each sample is found to be in error (c). However, when a 3 by 3 cluster is expanded around this single pixel (d), the samples from the map and reference data agree. 296 March 2003 PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING

9 random sampling was used, estimates of accuracy from samsat Congalton, R.G., R.G. Oderwald, and R.A. Mead, Assessing Land- pling in homogeneous areas only were statistically significantly classification accuracy using discrete multivariate statistical different from total enumeration estimates and from sampling techniques, Photogrammetric Engineering & Remote Sensing, 49(12): with random sample placement (i.e., using the majority rule). However, random sample placement was shown to be potenly Sensed Data: Principles and Practices, Lewis Publishers, Boca Congalton, R.G., and K. Green, Assessing the Accuracy of Remotetially problematic when map and reference data are misregis- Raton, Florida, 137 p. tered. Therefore, an effective option for best representing map accuracy may be to choose a combination of sampling schemes. Fitzpatrick-Lins, K., Comparison of sampling procedures and data analysis for a land-use and land-cover map, Photogrammetric For example, systematic sampling in homogeneous areas could Engineering & Remote Sensing, 47(3): be chosen as a first pass sampling scheme and then augment- Ginevan, M.E., Testing land-use map accuracy: Another look, ed by random sampling using the majority rule. This addition Photogrammetric Engineering & Remote Sensing, 45(10):1371 of a random sample to a systematic sample may also minimize the omission of some forest categories when the systematic in- Hord, R.M., and W. Brooner, Land-use map accuracy criteria, terval coincides with the spatial autocorrelation of forested Photogrammetric Engineering & Remote Sensing, 42(5): areas. Hudson, W.D., and C.W. Ramm, Correct formulation of the Kappa Once the sampling scheme has been chosen, Kappa can be coefficient of agreement, Photogrammetric Engineering & Remote used regardless of whether the scheme meets the assumptions Sensing, 53(4): of the multinomial model. While only simple random sampling Irland, L.C., Wildlands and Woodlots: The Story of New England s completely satisfies these assumptions, this project has shown Forests, University Press of New England, Hanover, New Hampthat the effect on the Kappa analysis of using stratified random shire, 206 p. sampling or systematic sampling is not significant. Klöditz, C., A. van Boxtel, E. Carfagna, and W. van Deursen, Acknowledgments Estimating the accuracy of coarse scale classification using high scale information, Photogrammetric Engineering & Remote Sens- The authors would like to acknowledge funding for this re- ing, 64(2): search from the University of New Hampshire Agricultural Exment Ma, Z., and R.L. Redmond, Tau coefficients for accuracy assess- periment Station under McIntire-Stennis grant MS-33. This of classification of remote sensing data, Photogrammetric publication is Scientific Contribution Number 2055 from the Engineering & Remote Sensing, 61(4): New Hampshire Agricultural Experiment Station. Also, thanks Pugh, S.A., and R.G. Congalton, Applying spatial autocorrelation to Dr. Stephen Stehman for his review and comments on this analysis to evaluate error in New England forest cover type maps work. Finally, we would like to thank the two anonymous rederived from Landsat Thematic Mapper data, Photogrammetric Engineering & Remote Sensing, 67(5): viewers whose comments helped improve this paper. Romesburg, H.C., Cluster Analysis for Researchers, Lifetime Learning Publications, Belmont, California, 334 p. References Stehman, S., Comparison of systematic and random sampling for estimating the accuracy of maps generated from remotely Card, D., Using known map category marginal frequencies to sensed data, Photogrammetric Engineering & Remote Sensing, improve estimates of thematic map accuracy, Photogrammetric 58(9): Engineering & Remote Sensing, 48(3): , Estimating the Kappa coefficient and its variance under Chuvieco, E., and R.G. Congalton, Using cluster analysis to imstratified random sampling, Photogrammetric Engineering & Reprove the selection of training statistics in classifying remotely mote Sensing, 62(4): sensed data, Photogrammetric Engineering & Remote Sensing, 54(9): , Comparing thematic maps based on map value, International Journal of Remote Sensing, 20(12): Congalton, R.G., A comparison of sampling schemes used in generating error matrices for assessing the accuracy of maps generperspective, Photogrammetric Engineering & Remote Sensing, Story, M., and R.G. Congalton, Accuracy assessment: a user s ated from remotely sensed data, Photogrammetric Engineering & Remote Sensing, 54(5): (3): , A review of assessing the accuracy of classifications of (Received 16 October 2002; accepted 04 April 2002; revised 11 July remotely sensed data, Remote Sensing of Environment, 37: ) PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING March