INT. J. REMOTE SENSING, 1995, VOL. 16, No.3, 581-587 Conservative bias in classification accuracy assessment due to pixelby-pixel comparison of classified images with reference grids D. L. VERBYLA and T. 0. HAMMOND Department of Forest Sciences, School of Agriculture and Land Resources Management, University of Alaska Fairbanks, PO Box 757200, Fairbanks, AK, U.S.A. 99775-7200 Received 14 July 1994; in final form 22 September 1994) Abstract. The use of reference grids derived from aerial photography for a pixel-by-pixel comparison with classified images can yield conservative estimates of classification accuracy. Even if the class assignment of each polygon is 100 per cent correct, and there is no change in cover type due to temporal differences between the reference data and the classified image, conservative bias in estimates of classification accuracy are still possible. In this letter, we discuss two major sources of this potential bias: I. positional errors, and 2. difference between polygon minimum mapping unit (MMU) and pixel size of the classified Image. I. Introduction The error matrix and kappa coefficient have become a standard in assessment of classification accuracy (Skidmore and Turner 1989, Oicks and Lo 1990, Congalton 1991, Lunetta et al. 1991, Janssen and van der Wel 1994). The error matrix (also termed contingency table or confusion matrix) compares predicted cover class, (classified data), with observed cover class (reference data). The error matrix can describe overall classification accuracy as the percentage of pixels correctly classified. In addition, for each cover class, accuracy can be described in terms of both omission and commission errors using the error matrix (Arnoff 1982, Story and Congalton 1986). The error matrix can also be used with discrete multivariate statistical techniques (Congalton et al. 1983) to statistically compare classifications. Reference data are the benchmark of comparison in the error matrix. These data are usually collected using ground-based sampling or by visually interpreting aerial photography (Congalton and Biging 1992, Jakubauskas et al. 1992). For reference data collection, aerial photography has the following potential advantages over field-sampled data: I. less expensive and faster, especially in remote s; 2. larger sample size is possible if polygons from aerial photographs are converted to pixels for accuracy assessment; 3. photography can be acquired for the same time period as the classified image; and 4. the accuracy of the entire can be assessed (instead of ground-based sampling within easily accessible s). With coarse-resolution imagery, such as Advanced Very High Resolution Radiometer (AVHRR) images, comparison with reference digital maps may be the only feasible method for accuracy assessment (Nelson and Horning 1993, Kasischke et al. 1993). One approach in developing reference data for accuracy assessment is to first interpret aerial photography for delineation of reference land cover polygons (Jakubauskas et al. 1992, Hudson 1987). Next, these polygons are typically either 0143-1161/95 $10.00 (!;) 1995 Taylor & Francis LId
582 D. L. Verbyla and T. 0. Hammond (1000 ROWS :x 1000 COLS)~ Figure Design for assessing conservative estimates of overall classification accuracy due to imperfect co-registration between classified image and reference grid. scanned or digitized into a vector-based geographical information system and then converted to a reference grid using vector-to-raster conversion software. Classification accuracy is then generally assessed on a pixel-to-pixel basis by comparing the co-registered classified image with the reference grid (Campbell 1987). The objective of this letter is to demonstrate that this approach will yield conservative estimates of classification accuracy. All analyses were conducted using a 22 June 1991, Landsat Thematic Mapper image and a 4 July 1990, multi-spectral image from Bonanza Creek Experimental Forest, near Fairbanks, Alaska, USA. This is part of a circumpolar bank of taiga and is described by Van Cleve et at. 1983 and Viereck et at. 1986. 2. Sources of conservative estimates of classification accuracy 2.1. Bias due to positional errors The importance of accurate co-registration of data has been demonstrated for accurate change detection (Townshend et al. 1992). In accuracy assessment, the reference data and classified data are assumed to be perfectly co-registered and any differences between these data are assumed to be exclusively due to classification errors (Campbell 1987). However, because both the reference data and classified data have inherent positional errors, it is impossible for them to be perfectly co-registered. To demonstrate the effect of imperfect co-registration on classification accuracy assessment, we classified the and Landsat- TM images using the ISODATA clustering procedure (Ball and Hall 1967). These classified images were reserved as reference data. To simulate positional error, each image was then copied and shifted one pixel in each cardinal direction. This produced four images that were treated as classified images to be evaluated for classification accuracy assessment (figure 1). Since there was no classification error, an unbiased classification assessment would indicate 100 per cent classification accuracy. However, because of the effects of imperfect co-registration, estimates of overall classification accuracy ranged from 85 to 64 per cent (table 1) The conservative bias increased as the
Remote Sensing Letters 583 Table I Estimates of overall classification accuracy with shift between perfectly classified images and corresponding reference grids.. Landsat-TM 5-Classes: North-shift South-shift East-shift West-shift l-pixel 84-7 84-7 84-9 84-9 77.4 77.4 JO-Classes: North-shift South-shift East-shift West-shift 76-6 76'6 77-0 77-1 77.5 77.5 25-c/asses: North-shift South-shift East-shift West-shift l-pixel l-pixel 63.2 63.3 64.0 64.1 63.8 63.8 62.7 62.7 number of spectral classes increased. This was due to a higher spatial heterogeneity occurring with a greater number of spectral classes. With 25-spectral classes, there were many single-pixel class aggregates that influenced the conservative bias due to the single-pixel shift. In practice, the co-registration error between the reference data and a classified image is unknown and certainly not constant. To assess the potential of positional error under more realistic conditions, we delineated 30 ground control points for the and Landsat- TM images. Ground control points were delineated using a vector GIS coverage of roads and skid trails that were recorded in the field using a global positioning system (GPS) receiver, and from digitizing features such as small water bodies from a 1 :63360 USGS topographic map. For the image, we randomly selected 15 of these ground control points. The 10 best of these ground control points, in terms of model residual error, were retained for development of a rectification model. Each classified image was then rectified to a 20-m Universal Tranverse Mercator (UTM) grid using nearest neighbour resampling. We then repeated the rectification process using the remaining 15 ground control points to develop a linear model for rectifying the same classified images which we treated as reference data. For the Landsat- TM image, we selected 30 different ground control points at road intersections and small water bodies. A linear rectification model was initially developed using these 30 points. Another linear model was developed after removing the 20 worst ground control points based on model residual error values. This model had a final RMS error of 0.23 and was used to rectify each classified image being treated as 'reference' grids. Another model was developed after removing the 20 best ground control points based on residual error values. This model had a final RMS error of 0.82 and was used to rectify each classified image. Both the classified and reference images were rectified to the same UTM grid. The only difference between the reference and classified data was due to slightly different model coefficients resulting from use of different ground control points for
584 D. L. Verbyla and T. 0. Hammond developing each rectification model. We then estimated the overall classification accuracy by comparing each classified image, on a pixel-by-pixel basis, with the associated rectified reference grid. Since the classified images were rectified to the same UTM projection, using two different transformation models, the true overall classification accuracy was 100 per cent. Any estimate less than 100 per cent would result from positional errors between the rectified classified and reference images. As expected, the overall classification accuracy estimate was consistently conservative (table 2). The estimates of overall classification accuracy became more conservative as the number of classes increased because of a corresponding increase in spatial heterogeneity. 2.2. Conservative bias due to minimum mapping unit ( MMU ) Aerial photography is usually stereoscopically interpreted to some MMU. This MMU is often larger than the classified image pixel size and therefore can cause bias in estimating classification accuracy. To demonstrate the potential effect of MMU, we converted the 5-class, 10-class, and 25-class classified images to vector polygons using a raster to vector conversion programme. We then generalized five different reference vector coverages with MMU s ranging from one to five ha by eliminating all polygons less than these s. This range of MMU s is comparable to those used in other studies (Hudson 1987, Moore and Bauer 1990, Dicks and Lo 1990, Bauer et al. 1994). The generalized reference polygon coverages were then converted to raster data to serve as reference grids in assessing the classification accuracy of the original classified images (figure 2). Thus, the accuracy assessment compared each original classified image with a corresponding generalized reference grid. This would be analogous to comparing a perfectly classified image with a reference database derived from aerial photography with no positional errors and no interpretation errors. The only difference between the classified images and the corresponding reference grids was due to size of MMU and raster-to-vector/vector-to-raster Table 2. Estimates of overall classification accuracy using classified image and reference grids that were rectified using slightly different rectification models. (a) Overall classification accuracy: 5-classes IO-classes 25-classes 70.2 58.6 42.9 Landsat- TM 81.4 78.6 59.0 (b) Rectification model root mean squared error: Classified images Reference images 1.31 1.01 Landsat-TM 0.82 0.23
~ Remote Sensing Letters 585 CLASSIFIED IMAGE Figure 2. Design for assessing bias in estimate of overall classification accuracy due to minimum mapping unit (MMU) of reference data. conversions. Therefore the actual overall classification accuracy was 100 per cent. However, the overall classification accuracy estimates were consistently conservative, with estimates ranging from 90 to 48 per cent (table 3). Once again, because of increased spatial heterogeneity, as the number of classes increased, the estimate of classification accuracy became more conservative. 2.3. Other potential sources of conservative bias In these simulations, we examined the potential effect of only two sources of conservative bias in classification accuracy assessment. A critical assumption with any reference data is that they are 100 per cent correct (Congalton and Biging 1992, Congalton and Green 1993). Any errors in assignment of reference data classes will be reflected as conservative estimates of classification accuracy. Likewise, temporal differences between the classified image and reference data can be another source of conservative estimates of classification accuracy (Nelson and Homing 1993, Joshi and Sahai 1993). 3. Conclusion Aerial photography, existing vegetation maps, or accurate vector-based GIS coverages are attractive alternatives compared to time-consuming and expensive field-collection of reference data for classification accuracy assessment. However, because of inherent problems with reference grids such as imperfect coregistration with classified images, relatively large MMU s, photo interpretation errors, and temporal differences between classified images and reference grids, classification accuracy estimates are likely to be conservative when reference grids are used.
586 D. L. Verbyla and T. 0. Hammond Table 3. Estimates of overall classification accuracy with perfect classification and reference data generalized to MMU. Landsat-TM 5-Classes: I-ha MMU 2-ha MMU 3-ha MMU 4-ha MMU 5-ha MMU 90.5 88.0 86.4 85.1 83.9 89.4 85.3 83.0 81.0 79.8 lo-classes: I-ha MMU -2-ha MMU 3-ha MMU 4-ha MMU 5-ha MMU 83.6 79.1 75.1 73.3 89.3 85.6 83.2 81.6 80.4 lo-classes: I-ha MMU 2-ha MMU 3-ha MMU 4-ha MMU 5-ha MMU 65.8 58.3 53.5-50.6 47.9 73-4 67.1 63-2 60-7 58-4 Acknowledgments We thank Ken Dean, Donna Grindle, Pete Wolter, John Yarie and the anonymous referees for offering suggestions that helped us to improve the manuscript. This research was partially supported by the National Science Foundation Long- Term Ecological Research Program. References ARNOPP. S., 1982, Classification accuracy: A user approach. Photogrammetric Engineering and Remote Sensing, 47, 1299-1307. BALL, G. H., and HALL, D. J., 1967, A clustering technique for summarizing multivariate data. Behavioral Science, 12, 153-155. BAVER, M. E., BURK, T. E., EK, A. R., COFFIN, P. R., LIME, S. D., WALSH, T. A., WALTERS, D. K., BEPORT, W., and HEINZEN, D. F., 1994, Satellite inventory of Minnesota forest resources. Photogrammetric Engineering and Remote Sensing, 60,287-298. CAMPBELL, J., 1987, Introduction to Remote Sensing (New York: Guildford Press). CONGALTON, R. G., 1991, A review of assessing the accuracy of classifications of remotely sensed data. Remote Sensing of Environment, 37, 35-46. CONGALTON, R. G., and BIGING, G. S., 1992, A pilot study evaluating ground reference data collection efforts for use in forest inventory. Photogrammetric Engineering and Remote Sensing,58, 1669-1671. CONGALTON, R. G., and GREEN, K., 1993, A practical look at the sources of confusion in error matrix generation. Photogrammetric Engineering and Remote Sensing, 59,641-644. CONGALTON, R. G., ODERWALD, R. G., and MEAD, R. A., 1983, Assessing Landsat classification accuracy using discrete multivariate analysis statistical techniques. Photogrammetric Engineering and Remote Sensing, 49, 1671-1678. DlCKS, S. E., and Lo, T. H. C., 1990, Evaluation of thematic map accuracy in a land-use and land-cover mapping program. Photogrammetric Engineering and Remote Sensing, 56, 1247-1252. HUDSON, W. D., 1987, Evaluating Landsat classification accuracy from forest cover-type maps. Canadian Journal of Remote Sensing, 13, 39-42.
Remote Sensing Letters 587 JANSSEN, L. F., and VAN DER WEL, F. J. M., 1994, Accuracy assessment of satellite derived land-cover data: a review. Photogrammetric Engineering and Remote Sensing, 60, 419-426. JAKUBAUSAKAS, M. E., WHISTLER, J. L., DlLLWORTH, M. E., and MARTINKO, E. A., 1992, Classifying remotely sensed data for use in an agricultural non-point source pollutionmodel. Journal of Soil and Water Conservation, 47, 179-183. Josm, M. D., and SAHAI, B., 1993, Mapping of salt-affected land in Saurashtra coast using Landsat satellite data. International Journal of Remote Sensing, 14, 1919-1920. KASISCHKE, E. S., FRENCH, N. H. F., HARELL, P., USTIN, S. L., and BARRY, D., 1993, Monitoring wildfires in boreal forests using AVHRR large s coverage imagery. Remote Sensing of Environment, 45,61-71. LUNETTA, R. S., CONGALTON, R. G., FENSTERMAKER, L. K., JENSEN, J. R., MCGWIRE, K. C., and TINNEY, L. R., 1991, Remote sensing and geographic information system data integration: error sources and research issues. Photogrammetric Engineering and Remote Sensing, 57,677-687. MOORE, M. M., and BAVER, M. E., 1990, Classification of forest vegetation in North-Central Minnesota using Landsat Multispectral Scanner and Thematic Mapper data. Forest Science, 36, 330-342. NELSON, R., and HORNING, N., 1993, AVHRR-LAC estimates offorest s in Madagascar, 1990. International Journal of Remote Sensing, 14, 1463-1475. SKIDMORE, A. K., and TURNER, B. J., 1989, Assessing the accuracy of resource inventory maps. In Global Natural Resources Monitoring and Assessments. Proceedings of an International Conference and Workshop 24-JO September 1989, Fondazione G. Cini, Isle of San Giorgio Maggiore, Venice, Italy (Bethesda, MD: American Society of Photogrammetry and Remote Sensing). pp. 524-535. STORY, M., and CONGALTON, R. G., 1986, Accuracy assessment: a user's perspective. Photogrammetric Engineering and Remote Sensing, 52, 397-399. TowNSHEND, J. R. G., JUSTICE, C. 0., GURNERY, C., and McMANus, J., 1992, Impact of misregistrations on change detection. I.E.E.E. Transactions on Geoscience and Remote Sensing, 30, 1034-1060. VAN CLEVE, K., DYRNESS, C. T., VIERECK, L. A., Fox, J., CHAPIN III, F. S., and oechel, J., 1983, Taiga ecosystems in interior Alaska. Bioscience, 33, 39-44. VIERECK, L. A., DYRNESS, C. T., and V AN CLEVE, K., 1986, Forest ecosystem distribution in the taiga environment. In Forest Ecosystems in the Alaska Taiga, edited by K. Van Cleve, F. S. Chapin 111, P. W. Flanagan, L. A. Viereck and C. T. Dyrness (New York: Springer-Verlag), pp. 22-43.