Assessing Limits of Classification Accuracy Attainable through Maximum Likelihood Method in Remote Sensing

Transcription

1 Asian Journal of Water, Environment and Pollution, Vol. 1, No. 1 & 2, pp Assessing Limits of Classification Accuracy Attainable through Maximum Likelihood Method in Remote Sensing R.K. Gupta, D. Vijayan, T.S. Prasad and P.M. Bala Manikavelu National Remote Sensing Agency, Balanagar Hyderabad , India * gupta_rk@nrsa.gov.in Received February 26, 2004; revised and accepted May 22, 2004 Abstract: The signal reaching the satellite is mix-up of contributions from varied features on the ground surface besides atmospheric errors. Getting back the information on ground features from satellite-based data is an illdefined inverse problem and is solved by applying statistics-based maximum likelihood (ML) type constraints. Here, the limits of classification accuracy attainable, using (in full) the professionally prepared GIS vector overlay for giving comprehensive training sets (approach a ), and thereafter fortifying this further even by starting with a priori probability worked out using relative fractional distribution under each class (approach b ), and improvement of accuracy in iterative mode were studied. Both GIS vector overlay-based approaches a and b gave nearly similar results for the area under different thematic classes and nearly matched with area(s) under input GIS overlay for dense, mixed and teak forest, grassland and water, and did not match for settlement, podu/blank, bamboo and semi-evergreen forests. In non-gis based ML classification, near matching with GIS vector overlay was achieved in initial classification for dense forest, semi-evergreen forest, settlement and water. Overall accuracy and kappa coefficients were favouring GIS vector overlay-based technique giving overall accuracy in % range (and not 100%) while for non-gis overlay approaches it was about 73%. Introduction The Remote Sensing signal which reaches sensor onboard the satellite is the complex aggregation of signals (in agriculture field for example) from soil (with all its variations such as colour, texture, particle size, clay content, organic and nutrition content, inorganic content, water content etc.), plant (height, architecture, leaf area index, mean canopy inclination etc.), canopy closure status and atmospheric effects, and from this we want to find say, characteristics of vegetation. If sensor on-board the satellite makes measurements in n-bands (n of n 1 dimension) and number of classes in an image are c (f of c 1 dimension), considering linear mixture modeling the pixel classification problem could be written as n = m f + e, where m is the transformation matrix of (n c) dimension and the e represents the error vector (noise). The problem is to estimate f by inverting the above equation and the possible solutions for such problem are many. Thus, getting back individual classes from satellite data is an ill-posed inverse problem for which unique solution is not feasible and this puts limit to the obtainable classification accuracy. Maximum Likelihood (ML) is the constraint mostly practiced in solving such a situation which suffers from the handicaps of assumed Gaussian distribution and random nature of pixels (in fact there is high auto-correlation among the pixels of a specific class and further high auto-correlation among the pixels in subclasses where the homogeneity would be high among pixels). Due to this, achieving of very high accuracy in the classification of remote sensing images is not a straight proposition. Improving of classification accuracies by choosing appropriate clustering algorithms (Duda & Canty, 2002) and through this imposing training sets

2 100 R.K. Gupta et al. (Churieco & Congalton, 1988), by calculating spectral mixture within pixels (Casals-Carrasco et al., 2000; Erol, 2000) by identifying reduced set of features that minimize the errors of the classifier (Bruzzone & Serpico, 2000) and through many other approaches including use of GIS (Janssen, et al., 1990) had been tried. With the availability of the GIS vector layer for the area under study, (i) the purity of training sets for different thematic classes could be better ascertained and (ii) a priori probability for different classes could be assigned to ML classifier in more realistic terms. To what extent this could improve the accuracy of classification in ML classifier had been the point of examination in this research contribution. Scope of improving the classification accuracy by using the fraction area under different classes in the classified image for giving a priori probability for next classification in iterative manner is also examined in this paper. The research was carried out over the forest region, with scattered patches of narrowly differing forest classes, for which a groundbased GIS vector overlay defining nine thematic classes had been available. The paper demonstrates that even use of full area under each thematic class in giving training sets (to account for full variance) and assigning of a priori probability based on the fraction area under each class in the GIS vector overlay could not give the accurate classification for each class. This brings out the inherent limitation of maximum likelihood method as classification is an ill-defined inverse problem. Data Sets The IRS-1C LISS-III image of 11th February, 1999 (Figure 1) providing coverage over Antilova tropical moist deciduous reserve forest in Andhra Pradesh (India) at its border with Orissa was used. The used image segment is bounded by to N in latitude and to E in longitude. Intensive use of remote sensing and GIS technologies have been practiced over this forest by the forest officials who have also worked out the GIS vector overlay for it (Figure 2) using ground-based information and visual interpretation of remote sensing image. Methodology The GIS vector layer prepared by forest officials from the satellite image contains nine thematic classes viz., dense, semi-evergreen, mixed, bamboo and teak forests, grassland, podu/blank (barren area), settlement and water. Figure 1: IRS/1C LISS-III image of the study area. Figure 2: GIS vector overlay for Antilova reserve forest. The satellite image was classified, using maximum likelihood method, by using (a) the entire area for a given thematic class under the GIS vector layer as training sets, (b) same as in (a) but additionally with the assignment of a priori probability for each class based on the ratio of the area under the thematic class to the total area as in GIS vector layer, (c) 2 to 3 training sets (by selecting within GIS vector overlay) for each thematic class, and adding of additional training sets, to the ones used in approach (c) above, under dense and mixed forest categories to include wider range for variance (approach d ). This was necessitated after analyzing the results of above mentioned approach (c) to work out mechanism to improve the accuracy; details are presented in Analysis, Discussions and Results section. For the first classification (zeroth iteration) equal a priori probability was assigned to each class for the classification approaches except in the approach (b). Thus, the difference in area under each theme in the classified images between approaches (a) and (c) and

3 Assessing Limits of Classification Accuracy Attainable through ML Method in Remote Sensing 101 between approaches (a) and (d) were primarily due to full variance taken into account for the training sets under approach (a) as compared to representative samples based variance used in approaches (c) and (d). Approach (b) involves total use of GIS vector overlay for training sets as well as in assigning a priori probability based on the fraction of area under each class to the total area for all the classes in the image for the zeroth iteration itself. Difference between the classification output under approach (b) and the GIS vector overlay conveys the inherent limitation of maximum likelihood technique. To examine the role of a priori probability in classification accuracy, iterations were carried out by using ratio of area under a particular class to the total area under all the classes in the image of the previously classified output (to work out a priori probability assignment for the iteration) under approaches (a) through (d); in approach (b) the change during iterations was low as expected. Here, iterations were continued till the areawise change under iterations in the thematic class was, in general, less than 2% and ensuring of stability was also the criteria with reference to previous iteration. Convergence was observed in the fifth, third, fourth and fourth iterations for approaches (a) through (d), respectively. The a priori probability put under approach (b) is based on GIS overlay which refers to facts as worked out with ground-based human verification and intelligence. Even with this [approach (b)], wherein full accounting for variance in spectral signatures is also taken care of, area under different classes obtained even in third iteration based classification could not match with the respective areas under each theme as available under GIS vector overlay. This brings out the inherent limitation of maximum likelihood classification algorithm. Further, deviations from GIS overlay based areas in the nine thematic classes under approach (c) where the training sets do not represent the population, brings out the limited degree of improvement through iterations which is feasible using a priori probability computed from the classified image under the previous iteration. Besides comparing the areas under different classes with those available under GIS vector overlay, classification accuracies were assessed with the use of 200 randomly distributed sample points using stratified random sampling technique. Thereafter, confusion matrices were developed which were used to work out errors of commission and omission; producer s and user s accuracies for each thematic class; and the overall classification accuracy and the kappa coefficient. Analysis, Discussions and Results Area Based Accuracies Assessments Table 1 (a) through (d) gives the percentage area under each thematic class for maximum likelihood classification(s) carried out with equal (except for approach b ) a priori probability (zeroth iteration), and thereafter with a priori probability worked out with fractional area under each class in the previous iteration based classified image till convergence was arrived at in iterative mode, for the approaches (a), (c) and (d). The percentage area as per GIS vector overlay has been included in Table 1 at column 2 for ease of comparison. Figure 3 gives the graphic presentation of results at Table 1 for the last iteration under approaches (a) through (d). Figure 3: Percentage of area under GIS vector overlay and for approaches (a) through (d). Ignoring second decimal places variations, it is observed (Table 1 a ) that except for the water (for decrease after third iteration), bamboo (decrease during first three iterations and thereafter marginal increase) and dense forest (decrease in first and fourth iterations); no mentionable changes for settlement, podu/blank, teak and grassland; and increase in area for mixed and semievergreen forests with iterations were observed in case of classification using full GIS vector overlay for giving training sets. The figures in Table 1(a) indicate that classification, in general, was of stable nature. Similarly tables 1 (b) through (d) could be analysed. Table 2 gives the deviations, with reference to GIS vector overlay, for classifications carried out with and without usage of GIS vector overlay. Here + indicates overestimation while indicates underestimation. It could be seen from Table 2 that bamboo is always

4 102 R.K. Gupta et al. Table 1: Percentage area under each thematic class under classification approaches (a) through (d) for initial classification (IT 0 ) and thereafter classifications under iterations (IT) identified by IT iteration number (e.g. IT 1 for first iteration) (a) (b) Approach (a) Approach (b) Thematic Classes % Area as IT 0 IT 1 IT 2 IT 3 IT 4 IT 5 IT 0 IT 1 IT 2 IT 3 per GIS overlay Dense Forest Semi- evergreen Mixed Forest Bamboo Teak Grassland Podu/Blank Settlement Water (c) Approach (c) Approach (d) Thematic Classes % Area as IT 0 IT 1 IT 2 IT 3 IT 4 IT 0 IT 1 IT 2 IT 3 IT 4 per GIS overlay Dense Forest Semi- evergreen Mixed Forest Bamboo Teak Grassland Podu/Blank Settlement Water (d) Table 2: Percentage of area under field survey based GIS vector overlay and deviations from it in classified images with and without usage of GIS vector overlay under approaches (a) through (d) Description Ø Class* Æ Base % value as per GIS vector overlay % Deviation for approach (a) % Deviation for approach (b) % Deviation for approach (c) % Deviation for approach (d) (0th iteration) % Deviation for approach (d) (4th iteration) * 1. Dense forest, 2. Semi-evergreen forest, 3. Mixed forest, 4. Bamboo, 5. Teak, 6. Grassland, 7. Podu/Blank, 8. Settlement, 9.Water.

5 Assessing Limits of Classification Accuracy Attainable through ML Method in Remote Sensing 103 overestimated in all the classification approaches and error is high with classification using total GIS vector overlay [approach (a)] for training sets, and the error further increases when GIS overlay is used for training sets as well as for assigning a priori probability at zeroth iteration stage. Table 3 provides the range for the observed minimum and maximum gray values, mean and standard deviation (SD) for each class, using entire GIS vector layer, in green, red and near-ir bands which have been used in the classifications. In all the three bands, bamboo has overlap with mixed forest and dense forest (with lesser degree in red band) while with grassland it has no overlap in red band, and with semi-evergreen forest it has no overlap in red and green bands. Thus, bamboo gets overestimated in all the methods. However, the intermixing among these four major subclasses of the forest (dense, semi-evergreen, mixed and bamboo) looks to be the possible reason for high error observed in case of area estimates under bamboo class. In case of mixed and dense forests, one finds high overlapping in gray values only in red band while separability on the lower end exists in green and near IR bands. With mixed forest, the separability for dense forest exists on the lower end side for all the three bands. Thus, there looks to be some opportunity to enlarge the training sets to provide additional care for variance and improve the classification which formed the basis of approach (d) for improving the classification obtained under approach (c). How to improve the classification accuracy with limited training sets (as done in real life situation) as in approach (c) was the germination point for approach (d). Studying the statistics of training sets as well as the achieved classification accuracies under approach (c), it was observed that there was scope to give additional training sets in mixed forest and dense forest classes to improve their representative aspect. The standard deviation for mixed forest in near-ir band under approach (a), (c) and (d) was 7.03, 3.06 and 8.67, respectively while for dense forest the respective values were 6.9, 5.5 and 7.4. Giving of additional training sets for dense and mixed forests brought dramatic change in percentage area for dense and mixed forest classes as seen in Table 1(d) in comparison with Table 1(c). Grass was getting set into dry condition and thus was much separable with different type of forests in red band and at the lower edge in green band. Except for semievergreen forest, the other forest classes had extended range in near-ir for the upper side, as compared to that for grassland. The settlements are located within the forest zones. Thus, purity of training set only for built-up area was not attainable. The non-gis overlay based approach (d), wherein ambiguities between dense and mixed forests were handled by giving additional training sets for these classes, provided near matching results with GIS overlay (Table 1 d ) for settlement. Probably taking of full area signatures in approaches (a) and (b) for themes occupying small areas resulted in mix up with forest classes except for water where contrast would be very high in near-ir band with reference to forest background (Table 3). In non-gis overlay method based approach (c), the equal a priori probabilities are assigned to each class to start with and the scope to improve with iterations is again governed by the area which got classified for the given class at the first instant. Thus, the error which initially creeps in continues to influence the subsequent iterations too. By assigning a priori probabilities using GIS overlay in the first iteration, better theme-wise classifications can be achieved even in situations with endlap/overlap in spectral signatures in contextually similar classes (comparing Tables 1 b and d ) except for classes having very less aerial extent (bamboo, settlement). Teak is clearly separable in green band with reference to dense, semi-evergreen, mixed and bamboo forests. Teak is clearly separable with grassland and podu/blank in red Table 3: Spectral characteristics of different classes as per GIS vector layer Class Ø Band Æ Green Band Red Band near-ir Band Min Max Mean SD Min Max Mean SD Min Max Mean SD Dense Forest Semi-evergreen Mixed Forest Bamboo Teak Grassland Podu/Blank Settlement Water

6 104 R.K. Gupta et al. band. This enables better classification of teak in approaches (a) and (b) even though the fraction percentage area under teak is 5.63%. Podu/blank with similar percentage area (5.12%) is not getting classified correctly in approaches (a), (b) and (d) while the assessment of percentage area under approach (c) is comparatively better. The reflectance in green as well as in near-ir bands for podu/blank is comparable with many forest classes. Further, very narrow range and mean value comparable to forest classes, with comparatively less SD value in near-ir band for podu/blank communicates presence of single variety of vegetation (say plantation). Podu/blank is separable from other forest classes in red band (not so with grassland). This looks to be the cause for high error in podu/blank even in approaches (a) and (b) even though percentage area under podu/blank is as good as that for teak. Semi-evergreen forest was considerably underestimated in classified outputs under approaches (a) and (b) while opposite was the case for bamboo. There is considerable overlap in the spectral signatures for bamboo and semi-evergreen forest. It seems that the area under semi-evergreen category is getting into other classes with significant shift into bamboo. Stratified Random Sample-based Accuracy Assessments Confusion matrices for approaches (a) through (d) were developed for the zeroth to last iteration. However, for reason of brevity, the results of last iteration are presented in Tables 4(a) through 4(d) for approaches (a) through (d), respectively. Using confusion matrix, errors of commission and omission are worked out. For providing clarity towards computation of producer s, user s and overall accuracies (Tables 5), numerical figures under confusion matrix 4(b) have been used. Once the rows for all the classes are filled up, the diagonal elements of the confusion matrix relate to producer s accuracy for the given class. Producer s accuracies are worked out by dividing the diagonal element pertaining to class with total number of random samples corresponding to the row. The ratio of the value in the diagonal element to the sum of all the values in the column gives the User s accuracy for the class corresponding to the column. Table 5 gives the classwise details of these accuracies for the confusion matrix at Table 4(b). The overall accuracy is worked out by taking the ratio of the sum of all the diagonal elements to the total number of random samples (pixels) used. The ratio of the sum of values in the row other than the diagonal elements (with the exclusion of total) to the total under the row gives error of commission for the class in the given row. The ratio of the sum of the values in a given column excluding the diagonal element (with the exclusion of total) to the total under the column gives the error of omission. Table 6 gives percentage error of Commission and Omission, and Table 7 gives the percentage Producer s and User s accuracy for all the iterations under classification approaches (a) through (d). Table 8 gives the details of overall accuracies attained during zeroth to third/ fourth/fifth iterations under classification approaches (a), (b), (c) and (d). In approach (a), the whole area under the class, in GIS vector overlay, is used for giving the training sets and thereafter (during iteration 1) it uses the area under each class in the classified image to decide the a priori probabilities for subsequent iteration. In approach (b), a priori probability based on fractional area under the class in GIS vector overlay is added to approach (a) at first classification itself. Approach (d) differs from approach (c) for including few more training sets in mixed and dense forest classes to have the variance for these in a more representative manner; the classification in both (c) and (d) is based on user-given training sets. The KHAT (k) statistic (Congalton, 1991) is computed using k = r r  ii  i+ + i i= 1 i= 1 r 2 N -Â( xi+ - x+ i) i= 1 N x - ( x - x ) (1) where r is the number of rows in the matrix, x ii is the number of observations in row i and column i, x i+ and x +i are the marginal totals of row i and column i, respectively; and N is the total number of observations. Using this, kappa-coefficient was computed under all the classification approaches for all the iterations and these are given in Table 9. It is seen that overall classification accuracy could reach 87.5% under approach (a) while it could reach only 83.5% under approach (b) whereas it was expected to be better than 87.5%. In stratified random sampling, the number of samples used and their chance placement in the given class puts the limit in assigning interpretation value to the statistical values worked out for accuracies. This becomes further critical when the classes are distributed in the image in scattered manner (Figure 2). Such detailed comparison could be undertaken only when the number of sample random pixels used is comparable with reference to total pixel population in the image.

7 Table 4: Confusion matrices for approaches (a) through (d) for the respective final iteration together with figures for overall accuracy for classification (a) (b) Theme DF SE MF BM TK GL PO ST WT Total DF SE MF BM TK GL PO ST WT Total DF SE MF BM TK GL PO ST WT Correctly classified pixels: 175; Overall accuracy = 87.5% Correctly classified pixels: 167; Overall accuracy = 83.50% (c) Theme DF SE MF BM TK GL PO ST WT Total DF SE MF BM TK GL PO ST WT Total DF SE MF BM TK GL PO ST WT Correctly classified = 146; Overall accuracy = % Correctly classified pixels= 146; Overall accuracy = % [DF: dense forest, SE: semi-evergreen forest, MF: mixed forest, BM: bamboo, TK: teak, GL: grassland, PO: podu/blank, ST: settlement, WT: water] (d) Assessing Limits of Classification Accuracy Attainable through ML Method in Remote Sensing 105

8 106 R.K. Gupta et al. Similar logic will hold good for the analysis of kappa coefficients. It is seen that, in general, the overall classification accuracy in user-defined training sets domain remains around 73% under iterations-based methodology and is around 65% in simple ML classification. The limiting figures for achieving overall accuracies look to be in 80 to 90% range even under ideal inputs [approaches (a) and (b)] for ML classification method and this brings out the limitation of ML method. Table 5: Producer s and User s accuracies for approach (b) derived from table 4(b) Class Name Reference Classified Pixels found Producer s User s Total Total Correct Accuracy % Accuracy % Dense Forest Semi-evergreen Mixed Forest Bamboo Teak Grassland Podu/Blank Settlement Water Correctly classified pixels: 167; Overall accuracy = 83.50% Class Name Æ Approach Ø Table 6: Percentage error of Omission and Commission for the last iteration for approaches (a) through (d) Type of Error DF SE MF BM TK GL PO ST WT Approach (a) Commission Omission Approach (b) Commission Omission Approach (c) Commission Omission Approach (d) Commission Omission [DF: dense forest, SE: semi-evergreen forest, MF: mixed forest, BM: bamboo, TK: teak, GL: grassland, PO: podu/blank, ST: settlement, WT: water] Table 7: Producer s and User s accuracies for the last iteration for approaches (a) through (d) derived from table 4 Class Name Approach (a) Approach (b) Approach (c ) Approach (d) P.A. U.A. P.A. U.A. P.A. U.A. P.A. U.A. % % % % % % % % Dense Forest Semi-evergreen Mixed Forest Bamboo Teak Grassland Podu/Blank Settlement Water [P.A.: Producer s Accuracy and U.A.: User s Accuracy]

9 Assessing Limits of Classification Accuracy Attainable through ML Method in Remote Sensing 107 Table 8: Overall percentage classification accuracies for different iterations under classification approaches (a) through (d) Iteration # Æ IT 0 IT 1 IT 2 IT 3 IT 4 IT 5 Classification % % % % % % Approach Id Ø a b N.A. N.A. c N.A. d N.A. N.A. = Not Applicable. Table 9: Overall kappa coefficient under different iterations for classification approaches (a) through (d) Iteration # Æ IT 0 IT 1 IT 2 IT 3 IT 4 IT 5 Classification % % % % % % Approach Id Ø a b N.A. N.A. c N.A. d N.A. N.A. = Not Applicable. Conclusions It is found that GIS overlay-based classifications gave good matching with the fraction area in GIS vector overlay for dense forest, mixed forest, teak, grassland and water classes; the zeroth iteration itself gave the matching value where GIS vector overlay was used in giving training sets (using full area under the class) as well as in giving a priori probability. Here, spectral overlapping was responsible for the mix-up between semi-evergreen forest and bamboo while podu/blank had plantations with high and narrow range signature in near- IR band which caused larger deviation in fraction area under the theme in the classified outputs as compared to GIS vector overlay. Settlement and water had small fractional area in the image and, thus, were susceptible to error but due to high spectral contrast the classified area under water matched well with values in GIS vector overlay. In non-gis based approach the results were not satisfactory under approach (c) and partially satisfactory under approach (d). Unlike in GIS-overlay approaches based area estimation (for each class), the iterations in non-gis overlay were causing adverseness. The percentage overall classification accuracy was in 83.5 to 87.5 range for GIS overlay based approaches, the corresponding figure for non-gis overlay based approaches was about 73%. Improvement per iteration in approach (b) was less as compared to that for approach (a) from second iteration onwards. The reason for this needs further examination. Acknowledgements Authors are thankful to Director, National Remote Sensing Agency, Hyderabad, India for encouragement during this research work and for the permission to publish this work. Authors are also indebted to Mr. P. Sudhakar for secretarial support. This paper was presented as a poster under the title Improving Accuracy of Image Classification using GIS in International Astronautical Congress Symposium on Earth Observation System and Business under ref. No. IAC-02-B.P.04, at Houston, USA, October, References Bruzzone, L. and S.B. Serpico (2000). A technique for feature selection in multiclass problem. Intl. J. Rem Sens., 21(3):

10 108 R.K. Gupta et al. Casals-Carrasco, P., Kubo, S. and B. Babu Madhvan (2000). Application of spectral mixture analysis for terrain evaluation studies. Intl. J. Rem. Sens., 21(16): Chuvieco, E. and R.G. Congalton (1988). Using cluster analysis to improve the selection of training statistics in classifying remotely sensed data. Photogramm. Engg. Rem. Sens., 54: Congalton, R.G. (1991). A review of assessing the accuracy of classifications of remotely sensed data. Rem. Sens. Environ., 37: Duda, T. and M. Canty (2002). Unsupervised classification of satellite imagery: Choosing a good algorithm. Intl. J. Rem.Sens., 23(11): Erol, H. (2000). A practical method for constructing the mixture model for a spectral class. Intl. J. Rem. Sens., 21(4): Janssen, L., Jaarsma, M. and E. van der Linden (1990). Integrating Topographic data with remote sensing for land cover classification. Photogramm. Engg. Rem. Sens., 56: