Evaluation of Classification Procedures for Estimating Wheat Acreage in Kansas

Transcription

1 Purdue University Purdue e-pubs LARS Sympsia Labratry fr Applicatins f Remte Sensing Evaluatin f Classificatin Prcedures fr Estimating Wheat Acreage in Kansas L. M. Flres D. T. Register Fllw this and additinal wrks at: Flres, L. M. and Register, D. T., "Evaluatin f Classificatin Prcedures fr Estimating Wheat Acreage in Kansas" (1976). LARS Sympsia. Paper This dcument has been made available thrugh Purdue e-pubs, a service f the Purdue University Libraries. Please cntact epubs@purdue.edu fr additinal infrmatin.

2 Reprinted frm Sympsium n Machine Prcessing f Remtely Sensed Data June 29 - July 1, 1976 The Labratry fr Applicatins f Remte Sensing Purdue University West Lafayette Indiana IEEE Catalg N. 76CH113-1 MPRSD Cpyright 1976 IEEE The Institute f Electrical and Electrnics Engineers, Inc. Cpyright 24 IEEE. This material is prvided with permissin f the IEEE. Such permissin f the IEEE des nt in any way imply IEEE endrsement f any f the prducts r services f the Purdue Research FundatinUniversity. Internal r persnal use f this material is permitted. Hwever, permissin t reprintrepublish this material fr advertising r prmtinal purpses r fr creating new cllective wrks fr resale r redistributin must be btained frm the IEEE by writing t pubs-permissins@ieee.rg. By chsing t view this dcument, yu agree t all prvisins f the cpyright laws prtecting it.

3 EALUATION OF CLASSIFICATION PROCEDURES FOR ESTIMATING WHEAT ACREAGE IN KANSAS* L. M. Flres and D. T. Register Lckheed Electrnics Cmpany, Inc.Aerspace Systems Divisin, Hustn, Texas I. ABSTRACT This reprt presents the results f experiments which were perfrmed t evaluate prcedures fr estimating wheat acreage in intensive test sites (ITS's) in Kansas. An analystinterpreter (AI) selected and labeled fields frm Landsat-l satellite imagery. Statistics were generated fr each selected ITS, and the imagery was classified using a maximum likelihd classifier. arius cmpnents f the classificatin prcess were tested. II. INTRODUCTION This experiment was designed t prvide sme insight int the factrs affecting the classificatin and mensuratin prcesses used fr crp identificatin. Specifically, the effects f the fllwing factrs n the acreage estimate andr the prbability f crrect classificatin (PCC) were investigated. a. The number f training fields selected by the AI fr the varius classes b. AI labeling errrs and pssible bias in the AI's selectin f training fields c. The number f training fields selected (AI r randm selectin) d. The methd f feature selectin (n feature selectin r ne in which the withut-replacement methd f feature selectin was used) * The material fr this paper was develped under Cntract NAS fr te Earth Observatic,ls Divisin, Science and Applicatins Directrate, Lyndn B. Jhnsn Space Center, Natinal Aernautics and Space Administratin, Hustn, Texas. The experiment encmpassed the analysis f five Kansas ITS's where grund truth was available. Three f the five ITS's (in,, and Cunties) had 4.8- by 4.8-kilmeter (3- by 3-mile, 9-square-mile, r 23-square-kilmeter) grund-truth areas; the ther tw (Mrtn and Finney Cunties) had 8- by 9.6-kilmeter (5- by 6-mile, 3-square-mile, r 77- square-kilmeter) grund-truth areas. Each ITS was interpreted by five AI's wrking independently f ne anther, thus prviding each site with five sets f training fields. The data cntained 16 channels btained by registering fur passes f the Landsat-l which crrespnded t the fur bilgical windws; i.e., emergence, drmancy, maximum grwth, and harvest. The verall experiment was divided int three designs. The purpse f the first design was t study the effects f sites, AI's, and feature selectin n classificatin. The variability in prprtin estimatin and fractin f training fields labeled wheat were als studied. The secnd design was t measure the extent prprtin estimatin was degraded by the mislabeling f training fields. The intent. f the third design was t cmpare the AI's methd f selecting training fields with that f a stratified randm selectin. The scpe fr each is as fllws. 1. The first design called fr selecting tw 4.8- by 4.8-kilmeter grund-truth areas frm each f the 8- by 9.6-kilmeter grund-truth ITS's in Finney and Mrtn Cunties. Classificatin was perfrmed n each 4.8- by 4.8-kilmeter area separately using the sets f training fields prvided by the five AI's. Thus, tw estimates f prprtin accuracy were available fr the Finney and Mrtn Cunty ITS's. Classificatin was als perfrmed n the,, and Cunty ITS's. 2. Only the Mrtn and Finney Cunty ITS's were used in design 2. This called 4B-24

4 fr classifying each f the 8- by 9.6- kilmeter grund-truth areas using the five sets f training fields prvided fr each site by the AI's and then crrecting the training field sets fr labeling and bundary errrs and reclassifying the ITS's. 3. The third design was used t cmpare the classificatin results frm the crrected fields f design 2 with the results f five classificatins fr each ITS using stratified randmly selected sets f training fields as described in the fllwing paragraph. Fr the third design, five sets f training fields were selected frm each f the 8- by 9.6-kilmeter grund-truth areas f the Mrtn and Finney Cunty ITS's. An unbiased selectin prcedure was USGd, which is defined here t be ne in which training fields are selected t represent each majr subclass. T place all pssible training fields int subclasses, the wheatfields in the grund-truth area were clustered with three as the maximum number f clusters. Using the majrity rule (based. n the number f pixels assigned t each cluster fr that field), each wheatfield was labeled as belnging t ne f the three subclasses. All the fields belnging t the secnd majr crp (crn fr Finney and fallw fr Mrtn Cunty) were clustered with tw as the maximum number f clusters. Again, by majrity rule, these fields were divided int tw subclasses. All the ther crps were represented by ne subclass. Fur wheatfields were selected randmly frm the grund-truth wheatfields f each subclass f wheat. Three fields were randmly selected frm the grund-truth fields f each subclass f the ecnd majr crp. Three fields (whenever available) were selected randmly t represent each f the ther crps. III. AI, SITE, AND FEATURE SELECTION EFFECTS The results presented in this sectin which pertain t design I cnsist f estimated wheat prprtins fr 4.8- by 4.8-kilmeter areas cmputed using AI training fields and iterative clustering with n feature selectin cnfiguratin. In this cnfiguratin, subclassing f training fields was accmplished by clustering the training fields f wheat and nnwheat separately using an iterative clustering algrithm. Classificatin was perfrmed using the resulting cluster statistics and the maximum likelihd classifier. Typically, there were between 7 and 9 clusters f wheat nd between 14 and 18 clusters f nnwheat. The estimated prprtins f wheat and nnwheatwere then cmputed by pixel cunting. The results were analyzed by subjecting them t varius statistical tests in an effrt t gain sme insight int the nature f the different factrs that might affect the accuracy f the estimate f the prprtin f wheat in a given area. The analysis f variance (ANOA) design was used in analyzing the results whenever applicable. Hwever, the results themselves suggested ther tests which prved t be f equal.r greater interest. Tables 1 thrugh 4 shw the results f the classificatins fr the prprtin errr (PE), p - P T, and the PCC when using ne cnfiguratin. Fr the n-threshld case, there were sites in which the PE's were, n the average, significantly erestimated and ther sites fr which they were significantly underestimated. Similarly, the training field sets selected by sme AI's prduced, n the average, significant verestimates while ther sets prduced significant underestimates. Ntably, the f all prprtin estimates ver all sites and AI training field sets were unbiased. Als, the relative perfrmance f the AI's changed frm site t site. Table 2 presents the results f applying a I-percent threshld. As mentined previusly, threshlded pixels were cnsidered t bennwheat thus, the bserved negative verall bias was t be expected. Hwever, the size f the bias (IS percentage pints) was larger than expected, and this was attributed t the large number f pixels being threshlded. The explanatin fr the large amunts f threshlding is either the incmpleteness f the training set when three r mre passes were used r the use f the wrng statistic (X 2 instead f F) fr threshld decisin making, r bth. Als, using a I-percent threshld, the site and the AI effects becme less prnunced and the interactin between them is n lnger significant. Tables 3 and 4 summarize the results when the PCC was used as the respnse,variable. In table 3, the PCC fr the nthreshld case shwed n significant difference frm site t site r frm AI t AI. This cntrasted with the results btained when the PE was used as a respnse variable in the ANOA. In the latter case, a site effect was bserved with a 99.S-percent cnfidence level and an AI effect with a 97.S-percent cnfidence level. It shuld be pinted ut hwever, that a site r an AI effect wuld be bserved if a cnsistent verestimate r underestimate f the wheat prprtin ccurred fr certain sites r AI's. One site r. AI culd have a cnsistent verestimate and anther site r AI a cnsistent underestimate, and each still,. l 4B-2S

5 culd have the same level f PCC. This wuld prduce a site effect in the ANOA fr PE and nt in the ANOA fr the PCC. Table 4, with I-percent threshlding, shws that sme sites had a significantly higher PCC than thers. There were n significant differences amng AI's and n interactin. ' Tables 5 thrugh 8 shw the results f running a secnd cnfiguratin; namely, the "iterative clustering withut replacement feature selectin" cnfiguratin. Tw values f the rati parameter f the withut-replacement feature-selectin prgram were used,.2 and.3. A cmparisn f the tables shws that little, if any, infrmatin was lst. The results f the ANOA are shwn in tables 9 and 1. Since threshlding is greatly dependent n the number f channels, it was expected that the feature-selectin effect wuld be statistically significant. Indeed, when threshlding was applied, it was fund t be significant at the 95-percent cnfidence level.. ARIABILITY IN THE PROPORTION ESTIMATION It was expected that if the PCC was lw the PE wuld exhibit large variance; whereas, if the PCC was high, a small variance fr PE wuld be expected. T determine if this was actually the case the PCC was divided int lw, medium, and high categries and the variance fr the PE cmputed fr each. The results presented belw are fr the all-channeln-threshld case PCC ariance,pe The decrease in PE variance with PCC can be seen in figure 1, which depicts a plt f PE versus PCC fr the all-channelnthreshld case. Figure 2 presents all 35 classificatin results and demnstrates that there was very little crrelatin between the wheat prprtin estimates and the grundtruth wheat prprtin. This implied that ther factrs influenced the prprtin estimatin prcess mre than the actual amunt f wheat in the segment. The size f training data may be ne f these factrs. Omitting all classificatins with less than 25 traini. fields reduced the square errr 43 percent. When the requirement was added that the fractin f the training fields labeled wheat be within.1 f the grund-truth wheat prprtin, the square errr was 52 percent lwer than that fr all classificatins. Fr thse classificatins in which at least 25 fields were used fr training and at least 1 f these were wheatfields, the square errr was reduced by a ntable 66 percent frm that f all classificatins. Thus, abut twthirds f the variability f the PE's was reduced by applying these minimum requirements. This suggests that the variability f PE's can be reduced significantly by applying sme simple minimum requirements when the phtinterpretatin f a segment is perfrmed. It shuld be nted that the abvementined criteria prvided an imprvement in the prprtin estimatin accuracy fr the five Kansas ITS's. It is expected that the criteria wuld be different fr ther sites under different cnditins.. EFFECT OF TRAINING FIELD LABELING AND BOUNDARY ERRORS Tables 11 and 12 shw the results btained frm design 2 by crrecting AI training field labels and bundaries fr the n-threshld and the I-percentthreshld cases, respectively. It can be seen that, with r withut threshlding, there was n significant imprvement in prprtin estimatin when the labels and bundaries were crrected with grund-truth infrmatin. T study imprvement a new variable was defined: where Ii p. imprvement fr the ith site estimated prprtin fr the ith site using uncrrected training fields estimated prprtin fr the ith site using crrected training fields grund-truth prprtin fr the ith site The imprvement is psitive if the PE frm an uncrrected training sample is, larger than the PE frm a crrected training sample and is negative fr the reverse situatin. A t-test was made fr the nthreshld and the I-percent-threshld cases t determine if the average imprvement was different frm zer. T determine if an inadequate amunt f training data was respnsible fr the' large variability f the PE's, the tw 4B-26

6 criteria which were arbitrarily suggested in sectin were used. The analysis f classificatins which met either criterin shwed an imprvement f 4.6 percentage pints fr the n-threshld case (table 13). Because f the much lwer variability when either criterin was met, this imprvement was statistically significant. With a I-percent threshld, crrecting the labels and bundaries seemed t impair the results, althugh this was nt significant (table 14). I. EFFECT OF AI BIAS IN TRAINING FIELD SELECTION Table 15 presents the PE's fr the n-threshld case f design 3. The PE's frm classificatins using AI-selected training fields (crrected fr labeling and bundary mistakes) shw a much larger variability than thse using a stratified randm selectin f training fields. The PE's fr bth types f training field selectin were psitively biased. This was fund t be significant fr the randmly selected fields but nt fr the AI-selected fields. A cause-and-effect relatinship is nt ensured because an effect is declared statistically significant. Factrs r variables which are highly crrelated with the effect culd prduce the significance. In this experiment, a lw number f training fields was typical fr the AI selectins. Therefre, an inadequate amunt f training data was f interest as a pssible significant factr in determining PE's. These cncepts were studied, and the results are shwn in figure 3. Figure 3 presents the relatinship f PE (with n threshld) t the ttal number f training fields. The results indicated that the primary surce f variability came frm classificatins using less than 25 training fields. With a I-percent threshld (table 16), the variability f the PE's was significantly larger with the AI-selected training fields than with the randmly selected fields. As expected, the PE's were negatively biased fr bth types f selectin. Ue FRACTION OF TRAINING FIELDS LABELED AS WHEAT Frm the nature f the classificatin prcedures, it was expected that with a sufficiently.large training sample (i.e., ne in which all classes and subclasses were well represented) the wheat prprtin. estimate wuld depend nly n class and subclass signature Specifically, the prprtin estimate wuld be independent f the relative number f training fields representing each class r subclass. If a dependence were fund, then it culd be cncluded that either the size f the training sample was nt sufficiently large r the AI intentinally r unintentinally selected his training fields fr each class in prprtin t the relative abundance f that class. Additinally, in the case f insufficient training samples, the dependence f the prprtin estimate n the fractin qf training fields representing a class r subclass wuld be expected t increase with the number f channels used fr classificatin. Tw heuristic supprting arguments are presented. First, as the number f channels increases, a greater increase ccurs in the number f parameters t be estimated. Statistically, fr a given training sample size, the mre parameters t be estimated, the less precise their estimatin becmes. Secnd, fr a given pass, suppse n subclasses are present. When an additinal pass is added, each f the riginal n subclasses either remains the same r is split int ne r mre new subclasses. Therefre, the number f subclasses fr bth passes must be greater than r equal t the number f subclasses fr the first pass. The greater the number f subclasses, the larger the number f training fields that are required t represent thse subclasses. T determine if the estimated wheat prprtin depended upn the fractin f training fields labeled wheat, these tw quantities were pltted against each ther. Classificatin results frm three separate surces were used t generate these plts. A statistical analysis was perfrmed n each data set using simple linear regressin. Fr the sake f brevity the plts are nt presented, but the crrelatin cefficients are in table 17. It can be seen that, in all f three separate experiments, the estimated wheat prprtin is definitely crrelated with the fractin f training fields labeled a.s wheat. The crrelatin cefficients were cmpared t determine if the crrelatin becmes significantly larger as the number f channels increases. Only tw cmparisns culd be made; namely, classificatins 6 and 7 and classificatins 3 and 5 f table 17. The first cmparisn shwed that with the multi tempral classificatins the crrelatin cefficient was significantly higher at the 9-percent cnfidence level. The secnd shwed that the multitempral crrelatin cefficient was higher than the single-phase crrelatin cefficient at the 99-percent cnfidence level, which is very significant. 4B-27

7 Thus, it has been demnstrated in tw separate experiments that the crrelatin cefficient increased with the number f channels used fr classificatin. The fact that a significant crrelatin was fund indicates that (1) generally an inadequate training sample was available andr (2) the AI's selected training fields fr a class in prprtin t the relative abundance f that class. The fact that the crrelatin increased in accrdance with the number f channels and that recent unpublished research cnfirms that AI's did nt select numbers f training fields fr each class n the basis f class abundance supprts the cntentin that inadequate training samples were used in the experiments. Table 1. Estimated Wheat Prprtin Minus Grund-Truth Wheat Prprtin With N Threshlding (Design 1) 1 II III Mean I Mrtn II Finney III Mean III. SUMMARY The mre salient findings during the curse f this investigatin are: a. The variability f the prprtin estimatin frm AI-selected training fields was s large that it masked bth the labeling errr and the crrelatin with the grund-truth wheat prprtin. b. Evidence shwed that the variability in the prprtin estimatin culd be reduced substantially by increasing the training data. c. A threshld f 1 percent prduced a negative bias in prprtin estimatin. d. Indicatins were that classificatins using randmly selected fields were nt substantially better than thse frm the AI-selected and crrected training fields, using cmparable training data. This suggests that a s-called "AI bias" in field selectin may have a much smaller effect than was previusly anticipated. e. It was demnstrated that the estimated prprtin was definitely crrelate with the fractin f training fields labeled as wheat. This crrelatin became larger as the number f channels increased. Table 2. Estimated Wheat prprtin Minus Grund-Truth Wheat Prprtin With I-Percent Threshlding (Design 1) I II III Mean I Mrtn II Finney II! _ Mean Table 3. Prbability f Crrect Classificatin With N Threshlding (Design 1) 1 II III Mean I Mrtn I! Finney III Mean B-28

8 Table 4. Prbability f Crrect Classificatin with I-Percent Threshlding and Threshlded Pixels Cnsidered as Nnwheat (Design 1) I II II! I Mean I Mrtn II Finney III !!O Mean Table 7. Estimated Wheat Prprtin Minus Grund-Truth Wheat Prprtin With r =.3 and I-Percent Threshlding I I! III Mean I Mrtn I! Finney II! Mean Table 5. Estimated Wheat Prprtin Minus Grund-Truth Wheat Prprtin With r =.3 and N Threshlding I II III Mean I Mrtn II Finney III Mean Table 8. Estimated Wheat Prprtin Minus Grund-Truth Wheat Prprtin With r =.2 and I-Percent Threshlding I II III Mean I l Mrtn II Finney III Mean Table 6. Estimated Wheat Prprtin Minus Grund-Truth Wheat Prprtin With r =.2 and N Threshlding I I! III Mean I Mrtn I! Finney III Mean B-29

9 Table 9. Analysis f ariance fr the Prprtin Errr Befre Threshlding Surce Sum f Degrees f Mean F- squares freedm square statistic Mean (s) a Feature extractin (F) F x S AI a AI x S a AI x F AI x S x F Errr aindicates significance at the I-percent level. NOTE: As usual, when the respnse variable is a prprtin, the arcsine square rt transfrmatin is applied. Table 1. Analysis f ariance fr the Prprtin Errr After Threshlding Surce Sum f Degrees f Mean squares freedm square Mean (s) Feature extractin (F) F x S AI AI x S AI x F ,)21 AI x S x F Errr F- statistic a b b b a Table 11. Effect f Training-Field Labeling and Bundary Crrectin n the Estimate f Wheat Prprtin With N Threshlding (Design 2) Bundaries Bundaries Imprvement AI and labels and labels due t uncrrected crrected crrecting Mrtn Finney Mrtn Finney NOTES: Overall MSE ariance Standard deviatin The entries in the table refer t the PE (estimated prprtin minus true prprtin) with n threshlding. t.16 tr1r = f U =.,nt sgn1 1cant. Mean square errr is increased by 3.9 percent. F = 1.5. The variance f the PE's frm stratified, randmly selected fields is significantly lwer than frm AI-selected and grund-truth-crrected fields. aindicates significance at the I-percent level. bindicates significance at the 5-percent level. NOTE: As usual, when the respnse variable is a prprtin, the arcsine square rt transfrmatin is applied. 4B-3

10 Table 12. Effect f Training-Field Labeling and Bundary Crrectin n the Estimate f Wheat Prprtin With I-Percent Threshlding (Design 2) Bundaries Bundaries Imprvement and AI labels and labels due t unc2rrected, crrected, crrecting p - Pw P - Pw Mrtn Finney Mrtn Finney Overall & l MSE ariance Standard deviatin NOTE: The entries in the table refer t the PE (estimated prprtin minus true prprtin) with I-percent threshlding. There is n significant imprvement and n significant difference in variance. Table 14. Imprvement in Classificatin Accuracy Using I-Percent Threshlding (Design 2, see Sectin ) AI Bundaries and labels uncrrected Bundaries and labels crrected Imprvementl Mrtn II Mrtn ' Finney II Finney Finney NOTE: Average imprvement = Even when adequate training data are available, there is n imprvement in prprtin estimatin if I-percent threshlding is applied. Table 15. Cmparisn f AI-Selected Fields With Grund-Truth-Selected Fields With N Threshlding (Design 3) AI selected Stratified randm and GT crrected selected training training fields fields c ""... :.: ;., '" c Mrtn Table 13. Impr"ement in Classificatin Accuracy Using N Threshlding (Design 2, see Sectin ) AI Labels uncrrected Labels crrected Imprvement Mrtn II O.lll Mrtn Finney II NOTE: Finney Overall MSE ariance Standard deviatin The entries, are the estimated wheat prprtin minus the grund-truth wheat prprtin with n threshlding. Finney Finney NOTES: Crrecting the labeling and bundary errrs n the average imprves the accuracy f prprtin estimatin by apprximately 5 percentage pints when adequate training data are available. This represents a relative imprvement f 46 percent. Uncrrected square errr = Crrected square errr = Mean square errr is reduced by 65.7 percent. The significant average imprvement is B-31

11 able 16. Cmparisn f AI-Selected Fields With Grund-Truth-Selected Fields With l-percent Threshlding (Design 3) NOTE: AI selected Stratified randm and GT crrected selected training training fields fields " :E > " OJ Mrtn Finney Overall ll MSE ariance Standard deviatin The entries are the estimated wheat prprtin minus the grund-truth wheat prprtin with I-percent threshlding. Table 17. Crrelatin f the Prprtin Estimate With Fractin f Training Fields Labeled as Wheat fr Three Separate Experiments PE LEG[NO -.2 r.:rto' FINNEY D-ELLIS + SALINE -.3 ORIeE 3 Figure 1. ",,- '" '" '" '" '""- "- '" '" D- D- O D- '" D-, D-.. OO SOD 6 7 r:u:j 9 PCC DECREASE IN RANGE WITH pec Prprtin Errr ersus PCC (Design 1). Descriptin f the classificatin and training set Crrelatin cefficient Level f cnfidence, % 1. Twenty-five training-field sets btained fr the five ITS's by the five AI's (design 1) 2. Ten sets f training fields randmly selected frm stratified samples frm Finney and Mrtn Cunties (design 3) 3. Single-phase classificatins using cluster statistics fr the five ITS's in a separate experiment 4. Same as abve using class statistics 5. Same as abve fr multitempral classificatins (8 channels) 6. Classificatins f 54 segments using 4-channel data in a secnd separate experiment 7. Classificatins f 21 segments using a-channel data which were a subset f the 54 segments where 2 passes were available B-32

12 -c.. BOO.7 z 8.6 t- :::... :::....5 t- e:( LJ.J :I: 3:.4 LJ.J t-... t-.3 A :. ',.,A; AS A LEGEND l lrtoii FINNEY A ELLIS + SALINE RICE :I: t ::: => t- <!l I Z....4 S.3 ::: I <!l ) LJ.J ) ::: => I ::; I-.2 ::;:LJ.J ::: ZO LL.... LJ.J t- c g;z :::t-... :::. t x ;:j a c i I... 3: t Oe:( LJ.JLJ.J t- I e:( 3: ::;:... t ) LJ.J TRUE WHEAT PROPORTION (P w ) MEAN SQUARE ERROR = Figure 2. Estimated Wheat Prprtin ersus True Wheat Prprtin With N Threshlding (Design 1) TOTAL NUMBER OF TRAINING FIELDS LEGEND AI-SELECTED FIELDS CORRECTED, MORTON COUNTY AI-SELECTED FIELDS CORRECTED, FINNEY COUNTY + RANDOMlY SELECTED TRAINING FIELDS, FINNEY COUNTY 6. RANDOMLY SELECTED TRAINING FIELDS, MORTON COUNTY Figure 3. Prprtin Errr ersus Size f Training Data fr the Mrtn and Finney Cunty Test s (Design 3). 4B-33