Assessing Accuracy of Land-Cover Change Data Aggregated to a Fixed Spatial Support. Stephen V. Stehman. James D. Wickham

Transcription

1 Assessing Accuracy of Land-Cover Cange Data Aggregated to a Fixed Spatial Support Stepen V. Steman State University of New Yor College of Environmental Science and Forestry 320 Bray Hall, Syracuse, NY USA Telepone: svstema@syr.edu and James D. Wicam United States Environmental Protection Agency, Landscape Caracterization Branc E243-05, Researc Triangle Par, NC USA Telepone: Wicam.James@epamail.epa.gov Abstract A metod for assessing te accuracy of land-cover cange products is described tat evaluates net cange at a specified spatial support. Accuracy is quantified by te mean absolute deviation, a metric derived from te absolute value of te difference between te map cange in a land-cover class and te true cange in te land-cover class. Several sampling design options are described for collecting te reference data. Stratification is a ey design feature because of te desire to increase sample size in te rare ig cange areas of te map. A protocol for a priori evaluation of different sampling designs is described. Details of te accuracy assessment protocol are illustrated via a cange product derived from te 1992 and preliminary 2001 National Land-Cover Data (NLCD). 1. Introduction Land-cover cange data are valuable to describe patterns in te environment, and tey are often incorporated as driving or explanatory variables in environmental modeling. Applications of land-cover cange data may require spatially explicit information in te form of a map, or use te data aggregated to some spatial unit to investigate relationsips at a designated spatial support (ere we use Dungan et al.'s (2003) definition of support as a property of a variable related to analyzing or modeling data). Altoug te spatial support can be as small as a point, our focus ere is on larger areas ranging from 1 a to several undred square ilometers because environmental assessment and modeling studies often aggregate land-cover measured for individual pixels. Te spatial unit employed in te assessment may be a regular geometric sape suc as a square or circle, or an irregular polygon suc as a county, province, or watersed. For example, canges in NDVI over a twenty-year period were compiled by watersed for an environmental assessment of te mid- Atlantic states (Jones et al., 1997), because waterseds are a logical and intuitive unit on

2 wic to base environmental management decisions. Liewise, tessellation of a regular spatial unit (e.g., 3x3, 5x5 m cells) as been used to examine geograpic patterns of te impact of land-cover cange on nutrient dynamics and breeding bird abitats (Jones et al., 2001; Wicam et al., 2002). Te watersed- and tessellation-based assessments are similar in teir pilosopy in tat it is often necessary to summarize cange at te level of individual pixels to render tem meaningful for environmental assessments and modeling studies. Te objective of tis article is to describe a general sampling design and analysis strategy for assessing te accuracy of land-cover cange at spatial scales larger tan a single pixel or minimum mapping unit (mmu) tat treats small groups of adjacent pixels as omogeneous units. Use of larger assessment units (e.g., counties, waterseds) represents a fundamental departure from land-cover cange accuracy assessments based on single pixels or a mmu. Maintenance of omogeneity is not possible for larger assessment units, and ence crisp from and to land-cover labels cannot be applied. A county or a watersed does not cange from forest to urban omogeneously. Four major temes are addressed: metrics to quantify accuracy of cange, formulas for estimating tese accuracy metrics, sampling design, and metods for evaluating te anticipated design performance wen planning te assessment (i.e., a priori design evaluation). Te assessment strategy is illustrated via an example in wic te accuracy of cange data derived from te 1992 National Land-Cover Data (NLCD) of te United States (Vogelmann et al., 2001) and a preliminary version of te 2001 NLCD. Users may be tempted to simply overlay te two NLCD products pixel by pixel to derive cange. Tis use of NLCD data is discouraged because it liely will result in large amounts of erroneous mapped cange due to inexact spatial registration of te two products and oter problems associated wit post-classification cange detection. Altoug a per-pixel cange map is not an intended use of NLCD data, it is anticipated tat by spatially aggregating te NLCD data, te cange values will prove useful for applications employing te data at larger spatial supports. Te metodology we propose is different in objectives and implementation from te traditional approac to accuracy assessment of land-cover cange described in Biging et al. (1998) and Congalton and Green (1999). An example elps to igligt te differences. Suppose te original cange map consists of 30-m pixels. Te traditional approac to accuracy assessment is based on an error matrix and measures of accuracy derived from tis matrix. For example, te proportion of tose pixels mapped as canging from forest to urban tat in reality ave canged from forest to urban is te user's accuracy for forest to urban cange. Te cange accuracy error matrix is large even for a small number of land-cover classes because te error matrix sows all possible transitions (i.e., from-to canges) between classes, as well as a no cange category for eac land-cover class. Te assessment of net cange is designed to address different objectives from te traditional approac. First, te spatial support to wic te assessment is intended is specified (e.g., a 20 by 20 pixel bloc). Te support is dictated by te application(s) for wic te map will most liely be used. Accuracy is ten defined in terms of te map cange and te true cange measured on spatial units corresponding to tis spatial support. For example, to assess accuracy of forest cange, te mapped area of forest for bot time t1 and time t2 for eac spatial unit is measured, and net cange is te difference between tese values. Map net cange is ten compared to te true net cange of eac spatial unit. Similar comparisons are conducted for eac land-cover class. Tis approac evaluates cange aggregated to te predetermined spatial support, and compensating errors witin te assessment unit may offset eac oter (e.g., a misclassification of one pixel as forest-to-urban cange may be offset by a

3 misclassification of one pixel as urban-to-forest cange). In contrast, te traditional approac focuses on location-specific accuracy, addressing te question of weter eac individual pixel s cange status is mapped correctly. Assessing accuracy for aggregations of pixels is sometimes labeled "non-site-specific" accuracy, a label tat often connotes aggregation over te entire region mapped. Te assessment we propose is "non-site-specific" to a degree, but also captures some spatially explicit caracter of accuracy at te scale of te spatial unit defined by te support. Anoter fundamental difference in te approaces is te definition of cange assessed, net cange in our approac versus gross cange in te traditional approac. Net cange is cange at an aggregate level (Fuller, 1999, p. 336), for example te difference in te area of forest between time t1 and time t2. Net cange does not focus on te internal, locationspecific mecanisms of ow cange taes place. For example, a net loss of 10% forest could be te result of losing 10% of te forest area to urban land cover, losing 5% of te forest area to agriculture and 5% to urban, or losing 25% of te forest area to agriculture offset by a 15% gain in forest attributable to cange from agriculture to forest. Te traditional approac to accuracy assessment focuses on gross cange. Gross cange is defined at te individual pixel level, and addresses te specific land-cover transitions of cange. A pixel-by-pixel cange map represents gross cange. For example, gross cange would quantify te number of pixels canging from forest to agriculture. Te "from-to" cange error matrix advocated by Congalton and Green (1999) in te traditional approac to cange accuracy assessment reflects te focus on gross cange. 2. Accuracy Assessment Protocol Cange accuracy assessment requires te tree basic components common to accuracy assessment of a single point in time. Tese components are te response design for caracterizing te ground or reference condition, te sampling design, and te analysis (estimation formulas). Eac of tese components will be described for te aggregated cange accuracy assessment. 2.1 Response design Te response design includes te protocols for determining te true cange on te ground, as well as metods for defining agreement between te true cange and te map cange. Reference data" is used to describe te cange data caracterizing te ground condition. Tese reference data liely also contain errors, but tey are assumed to represent a more accurate caracterization of reality tan te map itself. Te response design requires coosing te spatial unit upon wic te assessment will be based. Te assessment unit may be a fixed-area plot suc as a square, rectangle, or circle, or an irregular unit suc as a watersed or county. Tese units sould partition te region assessed, and tey sould retain teir identity for bot dates (i.e., te same set of units is applicable to bot time t1 and time t2). Te coice of spatial unit obviously strongly influences te assessment, and te decision is complicated by te lieliood tat users may be interested in different spatial supports. Our example analyses based on te NLCD cange data employ a 20x20 pixel (36 a) assessment unit. Tis unit was cosen because it is large enoug to diminis some of te effect on estimates of net cange attributable to misregistration wen overlaying te two NLCD products, it is a manageable size for obtaining reference data, and it is of sufficient size to be relevant for applications employing te cange data. For eac assessment unit, te area or percent of area occupied by eac land-cover class is obtained from te 1992 and 2001

4 NLCD. Net cange for eac land-cover class is ten te difference between te 2001 and 1992 values. 2.2 Quantifying accuracy Te descriptive results of te accuracy assessment are organized by reporting domains" of cange. Tese domains are defined by te magnitude and direction of cange for eac landcover type. A reporting domain is a subset of te full region, and it is defined to enance te interpretive value of te description. A domain consists of all spatial units in te region tat meet te defining conditions of te domain, and a spatial unit may belong to more tan one reporting domain. For example, one set of reporting domains for forest cange could be forest loss of 15% or more (> 15% cange), 7.5 to 15% forest loss (-7.5% to 15% cange), 2.5% to 7.5% loss, 2.5% loss to 2.5% gain (-2.5% to 2.5%), 2.5% to 7.5% gain in forest, 7.5% to 15% gain, and greater tan 15% gain in forest. Te reporting domains defined in tis example set include very ig cange domains, and tus target a scenario in wic map ig cange values are anticipated. Reporting domains can be defined or revised after te data are collected, and it is possible tat te domains defined for one land-cover type differ from te domains of anoter land-cover type. It is important to distinguis reporting domains from strata employed in te sampling design. Reporting domains do not impact ow te sample is selected, wereas strata directly influence te sampling units cosen. Strata constitute a fixed structure of te sampling design, wereas reporting domains are a caracteristic of te analysis. A sampling unit can belong to several reporting domains, but must belong to exactly one stratum. Te primary descriptor of cange accuracy empasized in tis article is te mean absolute deviation, MAD, computed for eac reporting domain. Te notation required is as follows. Let m u denote te net cange value derived from te map for sampling unit (bloc) u, and r u denote te net cange value derived from te reference data, wit te difference d u =r u -m u. MAD is te average of d u for all units in te reporting domain witin te region. For example, for te forest class, MAD is computed for eac reporting domain to quantify average absolute disagreement between te map net cange and reference net cange of forest. 2.3 Sampling design In practice, te region mapped will need to be sampled and estimates of te accuracy metrics derived from te sample data. Te ultimate sampling unit is te spatial unit (support) defined for te evaluation (e.g., a 20x20 pixel bloc in te NLCD example). For tose reporting domains tat are of ig interest but represent a relatively rare condition, stratification may be implemented to ensure MAD estimates are acceptably precise for tese domains. In te NLCD cange assessment, te rare domains of greatest interest are te ig cange domains (eiter gain or loss). Do tese areas of ig map cange truly represent ig cange on te ground? Te focus on ig cange strata creates a complication in te stratum assignment process. Some elements (spatial units) of te population may meet conditions for membersip in more tan one stratum. For example, if bot a ig loss forest stratum and a ig gain urban stratum exist, it is entirely possible tat some elements of te population will ave bot ig forest loss and ig urban gain, and tus are candidates for two different strata. A requirement of stratified sampling is tat eac element must belong to exactly one stratum, so te assignment protocol must guarantee tat tis requirement is met. Te protocol we invoe

5 is sequential, assigning eac element to a stratum via a pre-determined order for cecing if an element meets te conditions defining eac stratum. An example is provided in section 3.2. Once te strata ave been identified, te sample allocation to strata must be decided. Wen te objective is to obtain precise estimates for eac stratum, conventional guidelines for stratified sampling suggest allocating larger sample sizes to te more variable strata and to te more important strata. Te empasis on reporting domains to caracterize accuracy requires recognizing an additional dimension of te sample allocation decision. Because many of te reporting domains are not defined as strata, stratification may actually diminis precision of tese domain estimates relative to simple random sampling (SRS). Tat is, wile te stratified design will improve precision for tose domains targeted as strata, it may be detrimental to precision of estimates for oter domains. Tis feature is illustrated in te numerical examples provided later. Consequently, coosing strata and allocating sample size to strata must consider not only will tese coices improve precision for some domains, but also ow muc arm stratification creates for precision of oter reporting domains. 3. Example Assessment Using NLCD Cange Data We illustrate te accuracy assessment protocol via application to a cange product consisting of 10,000, 20x20 pixel blocs, wit eac pixel 30 m per side. Tese 10,000 blocs represent a simple random sample from a larger population of 180,000 suc blocs located in te mid- Atlantic region of te United States (Figure 1). Te map labels assigned to eac pixel in tis population are derived from te 1992 NLCD and a preliminary version of te 2001 NLCD. For some of te illustrative analyses, we also employ ypotetical reference data derived from te cange between te 2001 NLCD and 1989 National Oceanograpic and Atmosperic Administration (NOAA) Coastal Cange and Analysis Program (CCAP) data. Te NLCD (Vogelmann et al., 2001) and NOAA CCAP (Dobson et al., 1995) land-cover data sets were grouped into a simple classification sceme for te comparison of map and reference cange (Table 1). Te simple classification sceme approximates te Anderson et al. (1976) Level I detail. Grouping into a simpler classification sceme was done in part to reconcile te differences between CCAP and NLCD at te more detailed Level II classification, and because tere is little compelling reason for cange detection at Level II. Environmental managers and land use planners are less interested in cange witin different types of forest or urban, for example, tan canges between forest, agriculture, urban, and wetland. Forested wetlands in all tree data sets were grouped wit upland forest in te generalized sceme because previous NLCD accuracy assessments indicated tat reference data for wetland forests were nearly as liely to be labeled as upland as wetland (Yang et al., 2001; Steman et al., 2003). At a generalized level, a pixel labeled as wetland forest would be considered correct if te reference label was eiter te wetland or forest class. Te most significant disagreement between NOAA and NLCD generalized classes occurs wit te agricultural label. Te NOAA grassland class (detailed legend) includes agricultural pasturelands as well as golf courses and te grass infields surrounding airport runways. Te NLCD classification sceme attempts to differentiate between pasturelands and oter grasscovered land uses. As a result, some areas labeled agriculture in te NOAA data will be labeled urban in te NLCD data. Tese conceptual differences will be reflected quantitatively in te "map" versus "reference" comparisons. 3.1 Accuracy of net cange for te NLCD product

6 Table 2 displays te mean and median absolute deviations by reporting domain for forest, agriculture, and urban net cange derived from te NLCD (recall tat te 2001 NLCD and 1989 CCAP land-cover maps are used as te ypotetical reference data in tis example). Figure 1. Mid-Atlantic region of te United States used in te example net cange accuracy assessment. Te cross-atced region is te area from wic te 10,000 20x20 pixel blocs were selected.

7 Table 1. Grouping of Land-cover Classes NLCD 1992 Classes NLCD 2001 Classes NOAA 1989 Classes Detailed General Detailed General Detailed General Open water Water Open Water Water Hig intensity developed Urban Low-density residential Urban Developed Open Space Urban Hig intensity developed Urban Hig-density residential Urban Developed low intensity Urban Cultivated land Agriculture Commercial, Industrial, Urban Developed medium Urban Grassland Agriculture Transportation intensity Bare roc, sand, clay Barren Developed ig intensity Urban Deciduous Forest Forest Mining Urban Natural Barren Barren Evergreen Forest Forest Transitional 1 No data Deciduous Forest Forest Mixed Forest Forest Deciduous Forest Forest Evergreen Forest Forest Scrub/srub Forest Evergreen Forest Forest Mixed Forest Forest Palustrine forested Forest wetland Mixed Forest Forest Hay/Pasture Agriculture Palustrine scrub/srub Forest wetland Hay/Pasture Agriculture Row crops Agriculture Palustrine emergent Wetland wetland Row crops Agriculture Woody wetland Forest Estuarine emergent Wetland wetland Urban, recreational Urban Emergent Wetland Wetland Unconsolidated sore Barren grasses Woody wetland Forest Bare land Barren Emergent Wetland Wetland Water Water 1 Pixels labeled transitional in te NLCD 1992 data were converted to no data in all data sets. Table 2. Mean and Median Absolute Deviations by Domains of Forest, Agriculture, and Urban Cange for a Population of N=10,000 20x20 pixel blocs. Te domains are defined by te NLCD net cange for eac class, and M denotes te number of 20x20 pixel blocs in te domain. Forest Agriculture Urban Domain MAD Median M MAD Median M MAD Median M Domains (net cange): 1: >-15% (more tan 15% loss) 2: -15% to -7.5% 3: -7.5%to -2.5% 4: -2.5% to 2.5% 5: 2.5% to 7.5% 6: 7.5% to 15% 7: >15% (more tan 15% gain) Te results generally sow a pattern of increasing disagreement moving from te low cange to te ig cange domains. Te mean absolute deviation is generally iger tan te median absolute deviation, indicating a rigt sewed distribution of absolute deviations, as well as potential extreme ig values of d u. Te large values for mean and median absolute deviation suggest fairly substantial disagreements between te map and reference cange condition. For example, MAD is for forest domain 1 (forest loss exceeding 15%), indicating tat te reference cange differs (in absolute value) from te map cange by nearly 12% on average. Tis large disagreement is not surprising given te preliminary nature of

8 te 2001 NLCD product, and te fact tat errors in te NOAA CCAP 1989 land-cover data will also contribute to disagreement. Table 3 displays mean and median raw deviations (i.e., not absolute value). Te pattern of tese differences is also intuitively reasonable, wit large negative values occurring for domain 1 (ig loss), followed by a monotonic increase in te deviations, ultimately peaing at te large deviations of domain 7 (ig gain). Tat is, were te map displays large losses of a land-cover class, te reference data (on average) sow tat te loss was not as severe as te map indicates, tus te negative mean or median difference for te low domains. Te converse occurs wen map cange sows ig gains in a class. Table 3. Mean and Median Deviations by Domains of Forest, Agriculture, and Urban Cange for a Population of N=10,000 20x20 pixel blocs. Domains 1-7 are defined (see Table 2) by te NLCD net cange for eac class, and M denotes te number of 20x20 pixel blocs in te domain. Forest Agriculture Urban Domain Mean Median M Mean Median M Mean Median M Sampling design for NLCD cange example Te sampling design evaluated is a stratified random sample, wit te strata defined based on te map net cange. A sequential, one-stratum-at-a-time approac was adopted to assign blocs to strata to satisfy te requirement tat eac areal unit (20x20 pixel bloc) in te region must be assigned to exactly one stratum. At eac step, eac bloc is ceced to determine if it sould be assigned to te stratum of record at tat step. If te bloc meets te conditions defining te stratum, te bloc is placed in tat stratum and removed from consideration as a member of any subsequent stratum. Tose blocs not assigned at tis step of te sequence nor assigned at a previous step are passed on to te next step. Suppose seven strata are defined, wit te stratum assignment sequence as follows: 1) 15% forest loss, 2) 15% forest gain, 3) 15% urban loss, 4) 15% urban gain, 5) 15% agriculture loss, 6) 15% agriculture gain, and 7) all blocs not assigned in te first six steps (catc-all stratum consisting of low and moderate cange blocs). Te stratum assignment protocol is applied to all blocs in te mapped region (population). Te sequential approac represents one solution to te problem of ow to assign blocs to strata wen some blocs may satisfy conditions for membersip in more tan one stratum. For example, it is possible for a bloc to ave ig forest loss and ig urban gain (e.g., forest clearing for residential development). Depending on te order specified for stratum assignment, a bloc could be designated to te ig loss forest stratum (stratum 1) or te ig gain urban stratum (stratum 4). In te example sequence, suc a bloc would be assigned to stratum 1, te ig forest loss stratum. As long as eac bloc belongs to just one stratum, we are free to coose te assignment sequence. Numerous oter stratification options could be created. One option is to define strata based on a cross-classification of cange types, as for example, by defining a stratum as aving 15% loss in Forest and 15% gain in Urban. Alternatives to te sequential assignment

9 protocol could be envisioned to accommodate blocs tat meet conditions of more tan one stratum. For example, we could randomly assign te bloc to one stratum or te oter so tat alf te time suc a bloc is assigned to eac stratum. Te current sequential procedure assigns all suc blocs to te same stratum. 3.3 Estimation for te stratified design Te formulas and teory for estimating MAD for reporting domains are found in Cocran (1977, Sec. 5A.14) and Sarndal et al. (1992, p. 394). Bot texts recommend using a combined ratio estimator to estimate a domain mean wen te domain cuts across stratum boundaries, as is typically te case for te approac described. Recall tat d u is te difference between te map cange and reference cange. Te parameter (population value) for te mean absolute deviation for domain is ten d u /M, were U denotes te set of all M U 20x20 pixel blocs in te region. For bloc u of stratum, let y u = d u if bloc u is in domain, y u =0, oterwise, and let x u =1 if bloc u is in domain, and x u =0, oterwise. Note tat bot y u and x u are 0 for any bloc not in domain. MAD for domain can be expressed as H R = y = 1 U u H / x = 1 U u, were H is te number of strata. Te numerator of R is te population total of d u, and te denominator is simply M. Standard formulas for stratified sampling can readily be applied to estimate te numerator and denominator of tis ratio, H N y = Rˆ 1 = H. N x = 1 (1) Te estimated variance of Rˆ is H ˆ Mˆ N n N s y Rˆ 2 2 var( ) = (1 ) (1 )( sx 2Rˆ, +, sxy, = 1 R ) / n ), (2) were s 2 y, is te sample variance of y u in stratum, stratum, 2 s x, is te sample variance of x u in s xy, is te sample covariance between y u and x u in stratum, and Mˆ is te estimated number of blocs in te domain (i.e., te denominator of Rˆ ). All te statistics and variables in equations (1) and (2) are specific to domain, but te subscript is not used in most cases to simplify notation. For some domains, all values of y u and x u may be 0 for one or more strata because no sample blocs of domain are present for tat stratum. Tis does not create a problem because te contribution to Rˆ or to var( Rˆ ) is zero for suc strata. Lastly, if we replace d u wit d u, te formulas estimate te mean difference and te variance of te estimated mean difference (as opposed to mean absolute difference). 3.4 A priori evaluation of sampling design In planning te sampling design, it is invaluable to evaluate te potential performance of one or more candidate designs. Te primary information available at te planning stage is te map cange derived from te 1992 and 2001 NLCD products (i.e., te map cange product itself). Tis information can be used to define candidate cange strata, and to evaluate design performance. Te a priori evaluation of various design options focuses on precision

10 (standard errors) of te estimates specified by te accuracy assessment objectives. In te NLCD example, te estimation objectives focus on te reporting domains for eac of tree land-cover classes, forest, urban, and agriculture. Two analyses are proposed for te a priori evaluation of design performance. One is te expected sample size for eac reporting domain of eac land-cover type (i.e., te number of sample blocs expected to occur in te >15% forest loss domain, te number in te domain for less tan 2.5% gain or loss, etc.). Te expected sample size for a domain is computed by multiplying te proportion of te stratum occupied by tat domain times te sample size for tat stratum, and ten summing te results over all strata. For example, suppose we ave tree domains and two strata, and a stratified random sample of n =100 blocs from eac stratum will be selected. Te stratum proportions of domains A, B, and C are 0.6, 0.3, and 0.1 in stratum 1, and 0.45, 0.5, and 0.05 in stratum 2. Te anticipated sample sizes in eac domain resulting from tis design (contributions from strata 1 and 2 in parenteses) would be 105 (60 and 45) in domain A, 80 (30 and 50) in domain B, and 15 (10 and 5) in domain C. Because of te randomization built into te sampling protocol, tese are expected (average) sample sizes over all possible samples. Te outcome of an individual sample will vary from tese average numbers. Te second analysis is muc more detailed requiring construction of a ypotetical population of reference (i.e., true cange) data. Once tis population as been constructed, d u is nown for all blocs in te population, and te precision of te estimated MAD for eac domain can be calculated via H Rˆ ( ) = (1 M ) N (1 n N )( S y, + R S x, 2R S xy, = 1 V ) / n ), (3) were S 2 y, is te population variance of y u in stratum, in stratum, xy 2 S x, is te population variance of x u S, is te population covariance between y u and x u in stratum, and R is te true MAD for domain. V( Rˆ ) represents te variability of te estimator Rˆ over all possible samples tat could be selected from te population using te design under consideration. In practice, V( Rˆ ) is estimated by var( Rˆ ) using te sample of blocs actually selected for te assessment. At te planning stage, V( Rˆ ) is te relevant quantity because te sample as yet to be cosen. Because calculating V( Rˆ ) requires nowing d u for te population (census), te a priori design evaluation depends on constructing one or more ypotetical populations tat represent a good approximation to te true cange condition. Tat is, te goal is to create a population tat reflects te true magnitude and spatial pattern of te map errors to te degree tat te general findings of te a priori evaluation will reveal preferred design options. Te multi-objective caracter of accuracy assessment requires us to examine precision for te suite of domain estimates. No design option is liely universally best for all domain estimates, so coosing a design requires considering wic domain estimates are most important. To illustrate te approac, we use te population of 10,000 blocs described earlier for wic we ave ypotetical reference data. Five stratification options are evaluated. Tese include different numbers of strata (H=5, 6, or 7), different assignment sequences of blocs to strata, and different coices of te strata temselves. Te five options are listed below. Te % area

11 of gain or loss defining te strata is specified in parenteses for te first stratum and tis % applies to all strata for tat option. Te no cange stratum is always defined as net cange between 2.5% loss and 2.5% gain (-2.5% to 2.5%). Option A: 7 strata, wit te assignment sequence urban gain (>15%), urban loss, forest gain, forest loss, agriculture gain, agriculture loss, and no cange Option B: 6 strata, wit te assignment sequence forest gain (>15%), forest loss, urban gain, urban loss, agriculture loss, and no cange Option C: 6 strata, wit te assignment sequence urban gain (>15%), urban loss, forest gain, forest loss, agriculture loss, and no cange Option D: 5 strata, wit te assignment sequence urban gain (>15%), urban loss, forest gain, agriculture loss, and no cange Option E: 5 strata, wit te assignment sequence as in option D, but gain and loss are defined as >10% instead of >15%. Table 4 displays te anticipated sample sizes for te seven reporting domains for two of te stratified options. As a crude guideline, assume tat a sample size of blocs would produce adequate precision for a domain estimate. Because te forest domains are all relatively common in te region, stratification does not produce a mared difference in te distribution of te sample among te forest domains (Table 4). Te main advantage of stratification for te forest domain estimates accrues to domains 6 and 7, were te sample size is increased from 22 and 13 to over 40 in bot domains. Te estimates for te urban domains benefit most from te stratified options. Te no cange urban domain (domain 4) dominates in te region, so SRS will result in 75% of te sample being no urban cange blocs. Because bot options A and D ave urban domains 1 and 7 as te first two strata in te sequential selection, te design ensures a sample size of 50 in bot domains. Te stratified options do not increase te sample size above 30 in domains 2, 3, 5, and 6, so te stratified options primarily benefit only te rarest urban domains. Te agriculture domains are affected similarly, wit te rarest domain (domain 1) benefiting from its identification as one of te strata, but oter domains not gaining muc in sample size relative to SRS. In general, note tat tose domains not identified as strata ave smaller expected sample sizes tan would result from SRS. Tis is te crux of te caracteristic tat te stratified designs may ave poorer precision for tese domains tan SRS. Table 4. Expected sample sizes by domain for forest, agriculture, and urban for stratified sampling options A (StrA) and D (StrD), and simple random sampling (SRS). Total sample size is 400 for all tree designs. Stratified option A as a sample size of n =50 per stratum except for te low cange stratum wic as 100, and stratified option D also as a sample size of n =50 per stratum wit 200 samples in te low cange stratum. Option A as 7 strata, and option D as 5 strata. Forest Agriculture Urban Domain StrA StrD SRS StrA StrD SRS StrA StrD SRS

12 Te precision resulting from eac stratification option for te seven reporting domains of eac land-cover class is sown in Table 5. Precision for SRS is used as te baseline for comparison, and te sample size is maintained at n=400 for all designs, bot stratified and SRS. Negative values in Table 5 indicate a reporting domain for wic te stratified design improves precision relative to SRS, and positive values represent domains in wic SRS produces better estimates. Te results of Table 5 sould be examined for two ey features: ow muc does stratification improve precision for te important ig cange domains (i.e., large negative values for domains 1 and 7), and ow muc worse tan SRS are te stratified estimates for tose domains not defined as strata (i.e., large positive values for domains 2 troug 6)? Stratification improves precision for tose reporting domains identified as strata, but typically results in precision poorer tan SRS for te remaining domain estimates. Tis is not an unexpected result. Because te strata are constructed to focus sampling effort on te ig cange areas, tese strata are not necessarily a good coice for all domain estimates. Most domain estimates combine sample data from several strata, but tese strata may not be internally omogeneous for all domain variables. Internal omogeneity is te condition conducive to precise estimation for stratified sampling. Te design implications of tese results are tat strata sould be cosen to focus effort on only te most important domain estimates, and te stratification cosen sould not result in variances muc iger tan would be acieved by SRS for tose domains not identified as strata. Te urban results provide a good illustration of te evaluation process. All five stratification options ave two ig urban cange strata, one for ig loss and one for ig gain, representing domains 1 and 7. Because tese two domains are identified as strata, te stratified options are better tan SRS (note te negative numbers) except for one case, domain 7 for Option B. Option B's sequence differs from te oter four in tat te urban stratum assignments are made after blocs are assigned to te ig cange forest strata. Some of te ig urban gain blocs (urban domain 7) are liely assigned to te ig forest loss stratum, and tis produces sligtly poorer precision for urban domain 7 tan te oter four stratification options. Te results for forest domain 1 provide anoter interesting outcome. Relative to SRS, stratification does not improve precision for tis domain because it is a large domain. For example, under option B, we ensure tat exactly 50 sample blocs from forest domain 1 are selected. Because tis domain contains approximately 16% of te blocs in te population, on average SRS of 400 blocs will produce 64 sample blocs from tis domain. Te larger expected SRS sample size (n=64) results in a more precise estimate tan te stratified options (n=50). Terefore, a domain tat includes a large proportion of te region's blocs (i.e., a common domain) will not require identification as a stratum because adequate sample size for suc a domain will be produced by SRS. Anoter important conclusion derived from tis a priori evaluation is tat te sample size in te No Cange stratum (i.e., less tan 2.5% gain or loss) sould be large. Tis is a standard recommendation wen many estimates are required, suc as te multiple domain estimates employed to describe accuracy of cange. Increasing te sample size in te large No Cange stratum is tantamount to creating a large simple random sample. SRS is not tailored to enance precision for any particular domain estimate, but in contrast to stratification, it does not diminis precision for tose domains not identified as strata.

13 Table 5. Comparison of precision of domain estimators for various stratified sampling options. All designs ave n=400 20x20 pixel sample blocs selected from te N=10,000 blocs comprising te population. Te sample sizes are 50 per stratum, except for te no cange stratum tat as all remaining samples needed to mae up te total of 400. Te seven reporting domains are te same as in Table 2, and te 'All' row represents te estimates for te entire region. Te column M sows te number of blocs in te domain. Te SRS column is te standard error of te domain estimator under simple random sampling. All oter columns display te difference between te standard error of SRS and te stratified option, wit positive values indicating tat SRS is more precise for tat domain. a) Forest Domains Stratified Sampling Option Dom M MAD A B C D E SRS All b) Agriculture Domains Stratified Sampling Option Dom M MAD A B C D E SRS All c) Urban Domains Stratified Sampling Option Dom M MAD A B C D E SRS All Discussion Several components of te net cange accuracy assessment protocol merit additional development. Te metods described in tis article are based on a single spatial support, for example te 20x20 pixel bloc in te NLCD analyses. But applications of cange products will undoubtedly span a range of spatial supports. Using te protocol outlined, it is possible to assess accuracy for spatial support smaller tan tat cosen to determine te original

14 spatial unit of te assessment. Tat is, in our example NLCD assessment, it would be possible to evaluate a spatial support smaller tan te 36 a unit used as te sampling unit. Eac 20x20 pixel bloc could be partitioned into 4 10x10 pixel blocs, 16 5x5 pixel blocs, or even 100 2x2 pixel blocs. Te data analysis for te smaller support sizes requires onestage cluster sampling formulas treating te smaller units (e.g., te 10x10 pixel blocs) witin te larger original sampling units as secondary sampling units. Te sampling design is not tailored to control precision of te estimates for tese smaller spatial supports. Consequently, we sould not expect te assessment to be as good as it would be ad te design been created wit tis spatial support as te focus. Anoter important question related to spatial support is ow large of a spatial unit can be reasonably assessed. For example, if applications commonly employ a spatial support of 5 m by 5 m, is it possible to obtain quality reference data for units of tis size? Interpreting cange over suc large areas may be problematic. One possibility we are exploring is te use of two-stage cluster sampling. Rater tan interpret an entire sampling unit, we would employ a probability subsample of smaller secondary sampling units witin eac large unit to estimate net cange of te larger unit. Multistage sampling may also accommodate te desire to assess multiple spatial supports. Te sampling design we ave described is based on a fixed partition of te region into units corresponding to te spatial support cosen. Tis approac as two unsatisfying features. It requires eliminating small portions on te boundary of te study region to maintain complete 20x20 pixel blocs. Tis problem becomes more severe for larger support sizes. Secondly, te assessment is dependent on te partition selected (e.g., sifting te tessellation a small amount will cange te results). Altoug it is liely tat te differences resulting from suc a sift would be small, ideally te assessment would be immune to te tessellation s origin. Employing a floating spatial unit would resolve tese problems to some extent. In tis approac, individual pixels would first be selected, and ten a 20x20 bloc derived from tis pixel becomes te sampling unit (e.g., use te sample pixel as te upper left corner of te 20x20 pixel bloc). Te spatial units would tus not be fixed by a partition of te region, but rater would float witin te region depending on te pixels selected. How to stratify te region wen using a floating plot requires investigation. Two-pase sampling for stratification is a possible solution. A large first-pase sample would be selected and eac spatial unit in tis sample would be assigned to its appropriate stratum. A stratified subsample of te first-pase sample would ten be cosen and te response design applied to tis second-pase sample. Te sampling designs we ave proposed for te aggregated accuracy assessment of cange can be applied to simultaneously produce a traditional, site-specific assessment of gross cange. Te traditional assessment entails a muc more data-intensive response design protocol to obtain te individual pixel gross cange, but te sampling design protocols establised for te net cange assessment would be applicable to te traditional gross cange accuracy analysis. To attempt suc a dual-purpose assessment would exacerbate a major difficulty confronting accuracy assessments of any ind - accommodating te multiple objectives desired of te assessment. Adequately accomplising many objectives requires implementing a more complex sampling design to acieve tese goals wit little added cost, or eeping te design simple, tereby requiring greater cost to acieve equivalent precision to te more complex design.

15 5. Summary A sampling design and analysis strategy for assessing te accuracy of mapped net cange was developed. Tis strategy recognizes tat many applications of te cange product will aggregate te data to some spatial support to investigate or model relationsips between net cange and various oter penomena. In common wit one-point-in-time accuracy assessments, cange accuracy assessments sould be based on a probability sampling design to provide a rigorous statistical foundation. Several modifications from te traditional approac to accuracy assessment were developed. Rater tan base te analysis on te massive cange/no cange error matrix, te mean absolute deviation (MAD) between map and true cange serves as te basis for describing accuracy. MAD is computed for several user-defined reporting domains for eac land-cover class of interest, and te flexibility to tailor reporting domains to a particular user's needs is an attractive feature. As a consequence of employing MAD as te primary descriptive accuracy metric, te options for defining strata differ from te traditional use of te map land cover classes as strata for a one-point-in-time assessment, or strata based on te cange and no cange map areas for a cange accuracy assessment. Te ey information for stratification is still garnered from te cange product itself, but te strata definitions are complicated by te fact tat te spatial units (i.e., elements of te population) may meet conditions for membersip in several strata. Lastly, we developed metods for evaluating te anticipated performance of te sampling design at te planning stage. Tis a priori evaluation provides quantitative information as well as qualitative insigts on te relative merits of different designs, tus allowing a more informed coice of wic strata and sample allocations are liely to be most effective. General guidelines derived from our a priori evaluation of an NLCD cange product were to use a small number of strata and to allocate a large sample size to te large no cange (-2.5% to 2.5%) stratum. References Anderson, J. F., Hardy, E. E., Roac, J. T., and Witmer, R. E., 1976, A land use and land cover classification system for use wit remote sensor data, U.S. Geological Survey Professional Paper 964, U.S. Geological Survey, Wasington, DC, 28 pp. Biging, G. S., Colby, D. R., and Congalton, R. G., 1998, Sampling systems for cange detection accuracy assessment, Remote Sensing Cange Detection: Environmental Monitoring Metods and Applications, R. S. Lunetta and C. D. Elvidge (editors) (Ann Arbor Press, Celsea, Micigan), pp Brown, D. G., Pijanowsi, B. C., and Du, J. D., 2000, Modeling te relationsips between land use and land cover on private lands in te Upper Midwest, USA. Journal of Environmental Management, 59, Cocran, W. G., 1977, Sampling Tecniques, 3rd ed. (New Yor: Jon Wiley & Sons). Congalton, R. G., and Green, K., 1999, Assessing te Accuracy of Remotely Sensed Data: Principles and Practices (CRC Press, Boca Raton, FL). Dobson, J. E., Brigt, E. A., Ferguson, R. L., Field, D. W., Wood, L. L., Haddad, K. D., Iredale, H., Jensen, J. R., Klemas, V. V., Ort, R. J., and Tomas, J. P., 1995, NOAA Coastal Cange Analysis Program (C-CAP): guidance for regional implementation, NOAA Tecnical Report NMFS 123 (U.S. Department of Commerce, Seattle, Wasington).

16 Dungan, J. L., Perry, J. N., Dale, M. R. T., Legendre, P., Citron-Pousy, S., Fortin, M.-J., Jaomulsa, A., Miriti, M., and Rosenberg, M. S., 2002, A balanced view of scale in spatial statistical analysis, Ecograpy, 25, Fuller, W. A., 1999, Environmental surveys over time. Journal of Agricultural, Biological, and Environmental Statistics, 4, Jones, K. B., Neale, A. C., Wade, T. G., Wicam, J. D., Cross, C. L., Edmonds, C. M., Loveland, T. R., Nas, M. S., Riitters, K. H., and Smit, E. R., 2001, Te consequences of landscape cange on ecological resources: an assessment of te United States mid-atlantic region, Ecosystem Healt, 7, Jones, K.B., Riitters, K.H., Wicam, J.D., Tanersley, R.D., O Neill, R.V., Caloud, D.C., Smit, E.R., and Neale, A.C., 1997, An Ecological Assessment of te U.S. Mid-Atlantic Region. EPA/600/R-97/130, Office of Researc and Development, U.S. Environmental Protection Agency, Wasington, DC. Sarndal, C. E., Swensson, B., and Wretman, J., 1992, Model-Assisted Survey Sampling (New Yor: Springer-Verlag, New Yor). Steman, S. V., Wicam, J. D., Smit, J. H., and Yang, L., 2003, Tematic accuracy of te 1992 National Land-Cover Data (NLCD) for te eastern United States: statistical metodology and regional results. Remote Sensing of Environment (in press). Vogelmann, J. E., Howard, S. M., Yang, L., Larson, C. R., Wylie, B. K., and Van Driel, N., 2001, Completion of te 1990s National Land Cover Data set for te conterminous United States from Landsat Tematic Mapper data and ancillary data sources. Potogrammetric Engineering and Remote Sensing, 67, Wicam, J. D., O Neill, R. V., Riitters, K. H., Smit, E. R., Wade, T. G., and Jones, K. B., 2002, Geograpic targeting of increases in nutrient export due to future urbanization. Ecological Applications, 12, Yang, L., Steman, S. V., Smit, J. H., and Wicam, J. D., 2001, Sort Communication: Tematic accuracy of MRLC land-cover for te eastern United States. Remote Sensing of Environment, 76,