Learning based super-resolution land cover mapping

Transcription

1 earning based super-resolution land cover mapping Feng ing, Yiang Zang, Giles M. Foody IEEE Fellow, Xiaodong Xiuua Zang, Siming Fang, Wenbo Yun Du is work was supported in part by te National Basic Researc Program (973 Program) of Cina under Grant No. 2013cb733205, and in part by Natural Science Foundation of Hubei Province for Distinguised Young Scolars under Grant No. 2013CFA031. F. ing, Y. Zang, X. and Y. Du are wit te Key aboratory of Monitoring and Estimate for Environment and Disaster of Hubei Province, Institute of Geodesy and Geopysics, Cinese Academy of Sciences, Wuan , Cina ( G. M. Foody is wit te Scool of Geograpy, University of Nottingam, University Park, Nottingam NG7 2RD, UK X. Zang is wit te Wuan Institute of ecnology, Wuan , Cina S. Fang is wit te Scool of Public Administration, Cina University of Geosciences, Wuan , Cina. W. i is wit te Hefei Institute of ecnology Innovation, Cinese Academy of Sciences, Hefe , Cina. * Corresponding autor: lingf@wigg.ac.cn

2 Abstract: Super-resolution mapping (SRM) is a tecnique for generating a fine spatial resolution land cover map from coarse spatial resolution fraction images estimated by soft classification. e prior model used to describe te fine spatial resolution land cover pattern is a key issue in SRM. Here, a novel learning based SRM algoritm, wose prior model is learned from oter available fine spatial resolution land cover maps, is proposed. e approac is based on te assumption tat te spatial arrangement of te land cover components for mixed pixel patces wit similar fractions is often similar. e proposed SRM algoritm produces a learning database tat includes a large number of patc pairs for wic tere is a fine and coarse spatial resolution representation for te same area. From te learning database, patc pairs tat ave similar coarse spatial resolution patces as tose in input fraction images are selected. Fine spatial resolution patces in tese selected patc pairs are ten used to estimate te latent fine spatial resolution land cover map, by solving an optimization problem. e approac is illustrated by comparison against state-of-te-art SRM metods using land cover map subsets generated from te USA s National and Cover Database. Results sow tat te proposed SRM algoritm better maintains te spatial pattern of land covers for a range of different landscapes. e proposed SRM algoritm as te igest overall accuracy and Kappa values in all tese SRM algoritms, by using te entire maps in te accuracy assessment. Index erms:super-resolution mapping; earning database; Patc pairs; Neigboring patces

3 I. Introduction Super-resolution mapping (SRM), wic is also referred to sub-pixel mapping, is a metod to generate fine spatial resolution land cover maps from coarse spatial resolution remote sensing images. SRM can be viewed as te post-processing of soft classification to furter address te mixed pixel problem tat is common in coarse spatial resolution images [1, 2]. In general, soft classification estimates te fraction images tat illustrate te area percentage cover of land cover classes witin coarse spatial resolution mixed pixels. e fraction images may be input to a SRM analysis to predict te spatial locations of te land cover class at a fine spatial resolution. e output of te SRM analysis is a ard classification land cover map, wic as a finer spatial resolution tan tat of input fraction images. At present, SRM as become a promising metod to reduce te mixed pixel problem tat is widely encountered wit coarse spatial resolution images, and as been successfully used in many applications, suc as mapping waterlines [3-5], lakes [6], urban buildings [7], urban trees [8], forests [9], as well as in ground control point refinement [10] and te calculation of landscape pattern indices [11]. A large number of SRM algoritms ave been proposed [12-27]. Generally, SRM is an ill-posed problem [28], and te prior information about te spatial pattern of different land cover classes at te fine spatial resolution scale needs be known before SRM is performed. erefore, in order to estimate te latent fine spatial resolution land cover map from input coarse spatial resolution fraction images, one of te key issues is te definition of te prior model. e latter as been described from different perspectives. In te simplest case, wen only te coarse spatial resolution fraction images are available, te prior model is often based on te spatial dependence principle, wic aims to make te fine spatial resolution land cover map ave te maximal spatial dependence [1]. In practice, te spatial dependence of a certain fine spatial resolution pixel can be calculated by comparing it wit its neigboring fine spatial resolution pixels [14], fractions of its neigboring coarse spatial resolution pixels [29], or bot of tem [30]. Moreover, anisotropic land cover dependences ave also been

4 proposed to describe te spatial land cover pattern more precisely [31], especially for some special land cover classes [7, 32]. e spatial dependence model is a popular model for use in SRM due to its simplicity and absence of requirements for additional information about te spatial pattern of te land cover. However, te model can be inappropriate for areas wit complex land cover patterns. e use of te wrong prior model in a SRM analysis can result in an inappropriate and inaccurate land cover representation, possibly even worse tan tat of a standard ard classification of te coarse resolution image in some cases [6, 33]. A promising approac to improve te effectiveness of te prior model is troug te incorporation of additional information on te land cover to inform te SRM analysis. Various approaces ave been used. An intuitive additional dataset to use in a SRM analysis is anoter kind of fine spatial resolution images, suc as a pancromatic band image [34-37] or a fine spatial resolution syntetic aperture radar (SAR) image [38]. Fine spatial resolution digital elevation model [4, 5] and ligt detection and ranging (IDAR) data [39], and vector datasets [40] ave also been successfully used to refine SRM analyses. Multiple sub-pixel sifted coarse spatial resolution images wic are used as an alternation datasets to provide additional information about te spatial land cover pattern [41] for SRM, and tis metod as been furter developed [6, 42-45]. A istorical fine spatial resolution land cover map can also be used to elp te SRM analysis [9, 46, 47]. Adding site-specific additional datasets can increase te accuracy of SRM, owever, tis kind of approac is often limited because te additional dataset sould cover te same area, in its entirety, as te input fraction images, and tis may often not be te case wit te additional data only available for part of te region being mapped. In addition to te aforementioned approaces, te spatial land cover pattern can also be learned from te training image, wic is often a fine spatial resolution land cover map tat as a similar spatial land cover pattern to te objective fine spatial resolution land cover map. In tis situation, SRM predicts te fine spatial resolution land cover map from input coarse spatial resolution fraction images wit te aid of te prior model

5 tat is learned from tese training fine spatial resolution land cover maps. One kind of learning based SRM algoritm uses a model to describe te fine spatial resolution land cover pattern, and te parameters of te prior model are learned from te training maps [48-50]. Central to tis kind of approac is te selection of te prior model. Presently, a popular approac is to use a semi-variogram based approac in wic te SRM aims to produce a fine spatial resolution land cover map tat as te same spatial pattern as tat represented by te semi-variogram fitted to te available fine spatial resolution land cover map. A critical drawback of tis category of SRM algoritms is, owever, tat te geo-statistical metodologies are constructed based on te assumption of spatial stationary, and tey are limited for complex land cover patterns wic are often non-stationary. Anoter kind of learning based SRM algoritms directly learns te relationsip between coarse spatial resolution fraction images and te fine spatial resolution land cover maps witout a predefined model [51-54]. e basic assumption of tis metod is tat te fine spatial resolution spatial land cover pattern is similar for a mixed pixel patc, wic is basically a block of coarse resolution pixels, wit similar land cover class fractional composition. Several algoritms ave been proposed to learn te relationsip, including te back-propagation (BP) neural networks [51-54], and te support vector regression algoritms [55]. e SRM algoritms belonging to tis category often include two steps. In te first step, a fine spatial resolution image is estimated for eac land cover class using te learned relationsip. In te second step, fine spatial resolution pixel labels are assigned wit te maximum a posteriori principle using all of tese estimated fine spatial resolution images and input fraction images as constraints. is kind of learning based SRM algoritm are special cases of te interpolation-based SRM algoritm [56]; only te interpolation step is performed by te learning based metods. As a result, te limitation of te interpolation-based SRM algoritm, notably salt-and-pepper and linear artifacts, is unavoidable for tis category of learning based SRM algoritm. In tis paper, we propose a novel SRM algoritm, wic is also based on te assumption tat coarse spatial

6 resolution mixed pixel patces wit similar fractions ave a similar fine spatial resolution land cover pattern. Different to oter learning based SRM algoritms, te novelty of te proposed learning based SRM algoritm lies in te way te coarse spatial resolution and fine spatial resolution patces are used witin te analysis. e proposed metod does not need te additional label assignment step and patc outliers are effectively addressed during te analysis, avoiding commonly encountered error sources in oter two-step learning based SRM algoritms. e remainder of tis paper is organized as follows. Section II details te proposed learning based SRM algoritm. Section III validates te performance of te proposed algoritm troug several experiments. Section IV discusses some issues about te proposed algoritm and Section V concludes tis paper. II. Metods A. Problem description Suppose tat te original coarse spatial resolution remotely sensed image as M N pixels and te number of land cover classes in te wole image is C. It is assumed tat te fraction images F for all classes ave been estimated by soft classification. e SRM analysis aims to generate a fine spatial resolution land cover map H using F as input. By setting te zoom factor to be z, eac coarse spatial resolution pixel is divided into z z fine spatial resolution pixels. All fine spatial resolution pixels are considered to be pure pixels and eac one sould be assigned to a single land cover class. e resulting fine spatial resolution land cover map tus contains ( z M) ( z N) pixels, wose labels are defined to a unique class of C. In tis paper, it is assumed tat fraction images F are exactly estimated witout error. In eac coarse spatial resolution pixel V, te number of fine spatial resolution pixels Qc ( V ) assigned to te class c (1,2,, C) is computed according to te equation: Q V f V z (1) 2 c ( ) round( F, c ( ) ) were ffc, ( V ) is te fractional abundance of te class c witin te area represented by te coarse spatial

7 resolution pixel V in te fraction images F, round( x ) returns te value of te closest integer to x. In tis situation, te objective of te SRM analysis is to arrange tese fine spatial resolution pixels witin eac coarse spatial resolution pixel to make te fine spatial resolution land cover map onor a pre-defined spatial pattern model. For te proposed learning-based SRM algoritm, it is assumed tat tere are a set of fine spatial resolution land cover maps available. ese fine spatial resolution maps could be istorical land cover maps or land cover maps derived from fine spatial resolution remote sensing images. ese available fine spatial resolution land cover maps are used to provide information about te spatial land cover patterns at te fine spatial resolution. en, assuming tat a mixed pixel patc wit similar fractions as similar spatial land cover pattern, te SRM analysis may yield a fine spatial resolution map. B. e fine and coarse spatial resolution patc pair For te land cover class c, te spatial land cover pattern is represented by te patc pair [ x, y ], in wic y c is a coarse spatial resolution patc and its corresponding fine spatial resolution patc is c c x c. Here, square patces of size p are used wit a coarse spatial resolution patc including p p coarse spatial resolution pixels. e coarse spatial resolution patc is represented by te vector y [ f (1), f (2),, f ( p p)], were c c c c fc ( V ) is te fraction value of te coarse spatial resolution pixel V of te class c. e corresponding fine spatial resolution patc includes z p z p fine spatial resolution pixels, and is represented by te vector x [ I (1), I (2),, I ( z p z p)], were I () v is an indicator number sowing weter a fine spatial c c c c c resolution pixel v belongs to te class c, and is defined as: 1 if fine spatial resolution pixel vlabelled wit te land cover class c Ic () v 0 oterwise (2) us, x c represents te spatial distribution of fine spatial resolution pixels of te class c.

8 Fig. 1. A patc pair example, were te zoom factor is 4 and te coarse spatial resolution patc size is 3. (a) is a fine spatial resolution land cover map including pixels of two land cover classes, 1 (black) and 2 (wite); (b) is te corresponding fine resolution patc of te class 1, were te number 1 indicates tat te fine resolution pixel belongs to te class 1, and 0 indicates tat te fine resolution pixel belongs to oter classes; (c) is te corresponding coarse resolution patc, were te number witin pixels is te area percentages of te class 1 in eac coarse spatial resolution pixel. Fig. 1 sows a patc pair example wit p 3 and z 4. Fig. 1(a) is a fine spatial resolution land cover map including two classes. For te land cover class 1, as sown in black in Fig. 1(a), te fine and coarse spatial resolution land cover patces are sown in Fig. 1(b) and Fig. 1(c), respectively. e fine spatial resolution patc is an indicator map [Fig. 1(b)], were a label 1 means tat te fine pixel belongs to te class 1, and 0 means tat te fine spatial resolution pixel belongs to oter classes. erefore, te indicator map x e first line e 5t line e 12t line [0,0,...,0,...,0,0,0,1,1,1,1,1,0,0,0,0,...,0,0,...,0] sows te spatial pattern of te class 1. e coarse patc is te fraction image of te class, were te value represents te area percentages witin eac coarse pixel, and is represented as y1 [0,,0,,1,,0,, ]. ogeter [ x1, y 1] is a patc pair for te class Once a large number of patc pairs available, te SRM problem can ten be solved by using a pattern matcing metod. Given a coarse spatial resolution patc in te input fraction images, te patc pairs tat ave similar coarse spatial resolution patces are selected from tose available patc pairs. ese selected patc pairs are called as neigboring patc pairs, because if all patc pairs are listed in order according to te fraction values, tey are located in neigboring sites. It is noted tat tese neigboring patc pairs all include a coarse spatial resolution patc and a fine spatial resolution patc. Because coarse spatial resolution patces wit similar

9 fraction values often ave similar spatial land cover patterns, te latent fine spatial resolution patc for te coarse spatial resolution patc in te input fraction images sould be similar wit te fine spatial resolution patc included in te neigboring patc pairs. us, te SRM seeks to make te spatial land cover pattern of te resultant fine spatial resolution land cover map matc tose of neigboring patc pairs. In general, in te proposed learning based SRM algoritm, te fine and coarse spatial resolution patc pairs are first extracted from available fine spatial resolution land cover maps to produce a learning database. Similar patc pairs are ten found for eac coarse spatial resolution patc in input fraction images. Finally, tese available patc pairs in te learning database are used to reconstruct te final fine spatial resolution land cover map. All tese steps are described in detail as follows. C. Generating te training database Finding te relationsip between te fine spatial resolution land cover maps and coarse spatial resolution fraction images, wic is represented by patc pairs in te training database, is one of te key issues of te proposed learning based SRM algoritm. Here, te patc pairs are generated class by class. Given a set of fine spatial resolution land cover maps, a corresponding set of coarse spatial resolution patces for eac land cover class can be produced from tem. An example fine spatial resolution land cover map wic includes tree land cover classes, as sown in Fig. 2(a), is used to illustrate te training database generation procedure. Before te training database is generated, te zoom factor z and te coarse spatial resolution patc size p are set. In tis example, z is set to be 4, and p is set to be 3. A fine spatial resolution patc ten includes z p z p fine spatial resolution pixels (equals to in tis example). In order to generate a patc pair, a fine spatial resolution land cover map wit te size of z p z p is first exacted. Generally, by moving a fixed window containing z p z p rows and columns witin te original land cover map, various fine spatial resolution maps can be extracted.

10 ese extracted fine spatial resolution maps can be overlapped, tus te fine spatial resolution land cover pattern included in te original land cover map is fully exploited. For an extracted fine spatial resolution map, as sown in Fig. 2(b), one fine spatial resolution patc, tat is, one indicator map was ten generated for eac land cover class, as sown in Fig. 2(c)-(e). For eac fine spatial resolution patc, te corresponding coarse spatial resolution patc consists of fraction values of all coarse spatial resolution pixels. Eac coarse spatial resolution pixel corresponds to z z fine spatial resolution pixels, and te fraction value f ( V ), wic is te percentage of te c fine spatial resolution pixels assigned to te class c in te coarse spatial resolution pixel V, is calculated as: f ( V ) I ( v) z c 2 c (3) v V Fig. 2. An example of te training database generation procedure, were te zoom factor is 4 and te coarse resolution patc size is 3. (a) is te original fine resolution land cover map; (b) is one fine resolution land cover map including fine resolution pixels extracted from te original fine resolution land cover map. (c), (d) and (e) are fine resolution patces of different land cover classes, wic are generated from te extracted fine resolution land cover map. e number 1 indicates tat te fine resolution pixel belongs to te class, and 0 indicates tat te fine resolution pixel belongs to oter classes; (f), (g) and () are corresponding coarse resolution patces generated from (c), (d) and (e), were te number witin pixels means te area percentages of different class in eac coarse resolution pixel. One fine resolution patc and one coarse resolution patc form a patc pair, including (c) and (f), (d) and (g), and (e) and () for tree different land cover classes.

11 Eac extracted fine spatial resolution map (Fig. 2) can generate one patc pair for eac land cover class. Supposed tat K different fine spatial resolution maps are extracted from te original fine spatial resolution land cover map, K patc pairs can be generated for eac class. ose patc pairs comprise te training database, were te fine spatial resolution patces are represented as, { i K X c x, c} i 1 and te coarse spatial resolution patces are represented as, { i K Y c y, c} i 1 for class c. D. Searcing neigboring patc pairs Once te training database as been built, it is used to provide land cover information to estimate te latent fine spatial resolution land cover map wit input coarse spatial resolution fraction images. For eac coarse spatial resolution patc in te input fraction images, neigboring training patc pairs tat ave similar coarse spatial resolution patc are searced from te training database. Since te training database is generated class by class, te searc procedure is also performed class by class. et, { i R YF c yf, c} i 1 be coarse spatial resolution i patces in te input fraction images F for class c. For te it coarse spatial resolution patc y,, its Fc neigboring training patc pairs are cosen according to te following criterion:, 1, 2 f ( y i,, i j, ) ( i, ( ) i j F c y c ff c V f, c ( V )) p p (4) were i f ( y, y ) is te difference of fraction values between coarse spatial resolution patc y, in te i j F, c, c Fc fraction image and te jt corresponding patc i y in te training database. f, ( V ) is te fraction value of j c, Fc i te class c of te coarse spatial resolution pixel V in y Fc,, and j fc, ( V ) is te fraction value of te class c of te corresponding coarse spatial resolution pixel V in y j c,. e more similar te coarse spatial resolution patces are, te lower te value of f. e tresold is te tolerable fraction difference between two patces. If te value of i j f ( yf, c, y, c ) is no more tan tat of, y is tus considered te similar patc of j c, y i Fc,, and te patc pair tat includes y is te neigboring training patc pair. e value of is an j c,

12 important parameter for searcing neigboring patc pairs. If it is too large, te neigboring patc pairs are too different wit tat of te input fraction images to provide accurate land cover information. On contrast, if is too small, only few neigboring patc pairs can be found, leading to insufficient land cover information. e effect of te value of will be assessed in our later experiments. e k-dimensional (K-D) tree algoritm is applied to find neigboring training patc pairs in te present work, due to its efficient and successful application in te image super-resolution field [57]. e K-D tree organizes coarse spatial resolution patces in te training database off-line to enable a fast searc by defining a binary tree of tresolds, wic are cosen optimally so as to expedite te searc. Moreover, te K-D tree relies on a special ig dimensional data structure and tus te neigbors searcing step can be speeded up significantly. More detailed information about te K-D tree can be found in references [57, 58]. For eac coarse spatial resolution patc in te input fraction images F, te neigboring coarse spatial resolution patces and teir corresponding fine spatial resolution patces are searced from te training database using te K-D tree algoritm. A simple example for a two class situation given in Fig. 3 is used to illustrate te neigboring training patc pairs searcing procedure. In tis example, te coarse spatial resolution patc size p is set to be 3, and one coarse spatial resolution patc ten includes 3 3 coarse spatial resolution pixels. For a coarse spatial resolution pixel in te fraction image, all nine coarse spatial resolution patces tat include tis coarse spatial resolution pixel are extracted, by scanning te entire fraction images F. For eac coarse spatial i resolution patc y Fc,, we searc te neigboring coarse spatial resolution patces from te training database. e neigboring training patc pairs {, } i j j k x, c y, c j 1, were i k is te number of te searced patces, are ten used to estimate te latent fine spatial resolution land cover map, as described below.

13 Fig. 3. An example of te fine resolution land cover map estimation procedure, were te coarse resolution patc size is 3. wo land cover classes are included. For one pixel, all relative coarse resolution patces are extracted for te input fraction images for eac class. e neigboring patc pairs are selected from te training database for eac exacted coarse resolution patc. All selected patc pairs are ten used to estimate te resultant fine resolution land cover map, wic sows te distribution of te classes and is still in correct proportion in te input fraction images (9/16 and 7/16, respectively). E. Estimating te fine spatial resolution land cover map e fundamental assumption of te proposed learning based SRM algoritm is tat coarse spatial resolution patces wit similar class fraction values ave similar fine spatial resolution land cover patterns. In order to estimate te latent fine spatial resolution land cover map, te proposed SRM algoritm aims to make te fine spatial resolution patces in te latent fine spatial resolution land cover map similar wit te fine spatial

14 resolution patces identified from te selected neigboring training patc pairs. erefore, te objective of te SRM is to obtain a minimal difference between te fine spatial resolution patc in te estimated fine spatial resolution land cover map and corresponding fine spatial resolution patces in te neigboring training patc pairs. During te estimation process, all coarse spatial resolution pixels in te input fraction images are andled simultaneously, and SRM can be addressed by using te following minimization optimization model: C M N k i µ j j i Min ( H) w ( x, c ) E( x, c, xµ ) (5) Hc, c 1 i 1 j 1 1 E( x, x ) ( I ( v) I ( v)) (6) z p z p j i j i 2, c µ H, c, c Hµ, c z p z p v 1 w ( x ) 1 f ( y, y ) 1 f ( y, y ) (7) j j i j i, c, c Hc µ,, c F, c Subject to C Iµ, ( v) 1, (8) Hc c 1 2 Iµ ( v) f, µ ( V ) z H c H, c for all V F v V. (9) were µ H is te fine spatial resolution land cover map tat we aim to estimate. Setting te coarse spatial i resolution patc size to be odd witout loss of generality, y, is te coarse spatial resolution patc in wic te Fc i i t coarse spatial resolution pixel is located in te patc center in te fraction images F. Note tat y, is te i same as y µ, because te values in F are preserved in H µ i as (1). x µ, is te corresponding fine spatial Hc Hc Fc i resolution patc of y µ, in H µ for te class c. Hc { x } i are fine spatial resolution patces corresponding to j k, c j 1 te coarse spatial resolution patces { y } i j i in te selected neigboring training patc pairs. E( x, x µ ) is j k, c j 1 c, Hc, i te difference between two fine spatial resolution patces, x µ, and Hc x j c,, and is computed as (6), were i i Iµ, () v is te indictor of te fine spatial resolution pixel v witin x Hc Hc µ,, and I, () v is te indicator of te fine j c spatial resolution pixel v witin x j c,, respectively. w ( x ) is te weigt value assigned to te fine spatial j, c resolution patc x j c,, and is computed in te coarse spatial resolution scale by comparing te fraction difference between two corresponding coarse spatial resolution patces as (7). e larger te fraction difference, lower te

15 weigt value. Moreover, Equation (8) ensures tat eac fine spatial resolution pixel is assigned one and only one land cover class, and Equation (9) is te area constraint provided by te input coarse spatial resolution fractions for all coarse spatial resolution pixels. e minimization problem as a large solution space, wic increases wit te number of neigboring fine spatial resolution patces, land cover classes, and te image size. o solve te problem witin a sort computational time, tis work used a simulated annealing algoritm to find te solution. A power-law annealing scedule is used in te simulated annealing algoritm [59], were te temperature em n at iteration n is modified according to em n em (10) n 1 were (0,1) controls te decrease rate of temperature em n. In te initialization step, te fine spatial resolution pixels of eac class witin eac coarse spatial resolution pixel are randomly labeled according to te input coarse spatial resolution fractions. In tis situation, te constraints in (8) and (9) are naturally satisfied. In eac iteration, two fine spatial resolution pixels wit different land cover labels are randomly selected in eac coarse spatial resolution pixel. e values are ten calculated by using equation (5) according to te current fine spatial resolution land cover map configuration. If swapping tese two fine spatial resolution pixels decreases te object function value in equation (5), tese two fine spatial resolution pixels are swapped. Oterwise, te swap can only be accepted wit a small probability according to te current temperature in equation (10). e algoritm stops wen te previously fixed number of iterations is acieved. F. Patc outlier rejection e above-proposed minimization optimization model sould be generally considered as a simple fine spatial resolution pixel averaging metod. is model as limitations, notably tat te derived fine spatial

16 resolution map may be disrupted because of te problem of patc outliers. In general, a fine spatial resolution patc only corresponds to one coarse spatial resolution patc, and teir relationsip is sown in equation (3). But on te contrary, given a coarse spatial resolution patc, many different fine spatial resolution patces can correspond to it, making SRM be an underestimated inversion problem [28]. Fig. 4. An example of patc outliers in selected neigboring patc pairs. (a) is a patc pair for one land cover class. e fine resolution patc includes pixels and te coarse resolution patc includes 3 3 pixels. e fine resolution pixel filled by grey color indicates tat it belongs to te class. e number witin pixels means te area percentages of class in eac coarse resolution pixel of te class. (b) to (j) are searced neigboring patc pairs of (a) using te tolerable fraction difference value. e fraction values in (b) to (j) are similar wit te fraction values in (a), owever, only fine spatial resolution patces in (b) and (c) are similar wit tat in (a). Fine spatial resolution patces in (g)-(j) are muc different wit tat in (a). Fig. 4 presents an example of a patc outlier. In Fig. 4(a), a patc pair is taken from an experiment reported below. is patc pair includes a coarse spatial resolution patc of size 3 3, and a fine spatial resolution patc of size e numbers witin coarse spatial resolution pixels mean te fraction values of te class. Grey fine spatial resolution pixels represent te pixels belonging to tis class, and wite fine spatial resolution pixels

17 represent te pixels belonging to oter classes. Searcing in a training database including 120,000 patc pairs by using te coarse spatial resolution patc in Fig. 4(a) as te reference, te resultant 9 neigboring patc pairs are simultaneously sown in Figs. 4(b)-(j). As te neigboring patc pairs are searced by comparing te fraction different between coarse spatial resolution patces, coarse spatial resolution patces in tese neigboring patc pairs are all muc similar to te coarse spatial resolution patc in te reference patc pair. Wen comparing te fine spatial resolution patces, it is noticed tat only fine spatial resolution patces in Fig. 4(b) and Fig. 4 (c) are very similar wit tose in Fig. 4(a). Fine spatial resolution patces in Figs. 4(g)-(j) are very different wit tose in Fig. 4(a), and are ten not suitable to be applied to estimate te fine spatial resolution patc. erefore, in order to improve te performance of te learning based SRM algoritm, only te patc pairs wic ave similar fine spatial resolution patces wit te reference sould be applied to estimate te latent fine spatial resolution land cover map, and oter patc pairs sould be considered as outliers. For te fine resolution patc x witin te neigboring training patc pairs, it is defined as te outlier j c, patc according to te following criterion: 1 f ( x, x ) ( I ( v) I ( v)) (11) z p z p j i j i 2, c H, c, c H, c z p z p v 1 were f ( x, x ) is te difference between two fine spatial resolution patces, and is computed as same as j i, c H, c i E( x, x ) is equation (6). x, is te fine spatial resolution patc in te latent fine spatial resolution land j i, c H, c Hc cover map, and x is te fine resolution patc witin te neigboring training patc pairs. e more similar j c, te fine resolution patces are, te lower te value of j i f ( x, c, xh, c ). e tresold is te tolerable difference between two fine spatial resolution patces. If te value of j i f ( x, c, xh, c ) is no less tan tat of, te fine spatial resolution patc x is considered as an outlier patc. j c, In te object function (5), w ( x ) is used to give te weigt values. However, tis weigt value is j, c computed by comparing coarse spatial resolution patces and is ten not sufficient to distinguis outliers from

18 te selected neigboring patc pairs. In order to consider te outlier patces, an additional patc weigt w ( x ) tat is computed at te fine spatial resolution is added in te object function of te proposed SRM H j, c algoritm as: C M N k i µ j j j i Min ( H) w ( x, c ) wh ( x, c ) E( x, c, xµ ) (12) Hc, c 1 i 1 j 1 j i j 0 If f ( x, c, xh, c ), te patc is an outlier wh ( x, c ) 1 Oterwise, te patc is not an outlier (13) In general, if te patc x is an outlier, te weigt value j c, w ( x ) is set to be zero, meaning tat tis H j, c fine spatial resolution patc no longer used during te estimation procedure. Oterwise, te weigt value w ( x ) is set to be one, if te patc is not te outlier. H j, c Fig. 5. e flowcart of te iterative patc outlier andling procedure. Setting an appropriate value of is important if patc outliers are to be addressed effectively. In te present work, an iterative metod is used. e main iteration procedure is sown in Figure 5. At first, a fine spatial resolution land cover map is estimated according to te minimization optimization model (5), using all searced neigboring learning patc pairs. Because all neigboring patc pairs are searced by using a tolerant fraction value as (4), it is expected tat te estimated fine spatial resolution land cover map is similar wit

19 te latent fine spatial resolution land cover map. en, we begin to refine te estimated fine spatial resolution land cover map, by decreasing te value of step by step, from te maximal tresold Max to te minimal tresold Min. Wit eac value of, a new fine spatial resolution land cover map is estimated according to (12). At te beginning, wit a large value, only a relatively small amount of neigboring patces, wose fine spatial resolution patces are markedly different to te current estimated fine spatial resolution land cover map, are considered as outlier. Witout tese patc outliers, a more accurate fine spatial resolution land cover map is expected to be estimated. e value of decreases iteratively and more patces are considered as outliers. en, te estimated fine spatial resolution land cover map is expected to be more accurate. At te end, te iteration converges to a stable solution until te minimal value Min is reaced, and te estimated fine spatial resolution land cover map is considered as te result of SRM. G. e proposed algoritm According to te aforementioned principles, we summarize te proposed learning based SRM algoritm in Algoritm 1. In brief, a fine spatial resolution land cover map is first randomly generated using input coarse spatial resolution fraction images in te initialization step. Meanwile, te patc pairs in te training database are constructed according to te zoom factor and te coarse spatial resolution patc size. e input coarse spatial resolution fraction images are ten scanned and neigboring patc pairs are searced by te K-D tree algoritm, for all coarse spatial resolution patces in fraction images class by class. Using tese searced neigboring patc pairs, te initial fine spatial resolution land cover map is estimated by using te simulated annealing algoritm. Outliers in tese neigboring patc pairs are ten found and te fine spatial resolution land cover map is re-estimated, by canging te tresold value. Once reaces Min, te iteration is finised, te estimated fine spatial resolution land cover map is te result of te learning based SRM algoritm.

20 Algoritm I Objective: Estimate fine spatial resolution land cover map H Input: Coarse spatial resolution fraction images F, zoom factor z, land cover class number C, coarse spatial resolution patc size p, coarse spatial resolution fraction tresold, te maximal and minimal fine spatial resolution tresolds Max and Min, fine spatial resolution tresold cange value d, parameters of te simulated annealing algoritm: em 0,, and Ite. 1. Initialization: 1) For eac coarse spatial resolution pixel, calculate te number of fine spatial resolution pixels for eac class as (1); 2) Randomly set class label for all fine spatial resolution pixels using te number as te constraints; 2. raining database generation 1) Extract fine spatial resolution patces from available land cover maps; 2) Estimate te corresponding coarse spatial resolution patc for eac fine spatial resolution patc and generate patc pairs in te training database; 3. Similar Patc Finding 1) Build te K-D tree for all coarse spatial resolution patces in te training database; 2) Scan F, and extract all coarse spatial resolution patces; 3) Searc neigboring patc pairs from te training database for all coarse spatial resolution patces in F. 4. Generating initial ig resolution land cover map 1) Reconstruct µ H by minimizing te objective function in (5) using te simulated annealing algoritm. 5. Iterative patc outlier rejection 1) Set 2) Do { Max [1] Calculate te weigt values using current H µ and, according to (13); [2] Reconstruct µ H according to (12); [3] d ; } Until Min Result: Output te fine spatial resolution land cover map µ H. III. EXAMPE e National and Cover Database 2001 (NCD 2001) tat sows te land cover for te conterminous USA was used to test te proposed learning based SRM algoritm. NCD 2001 is a raster-based 16-class land

21 cover map over all 50 US states and Puerto Rico at spatial resolution of 30 m, and is primarily generated from te unsupervised classification of andsat Enanced ematic Mapper Plus circa 2001 satellite dataset [60]. o simplify te experiment, te original 16 classes of te NCD images were converted into simple class sceme wic includes four general land cover classes: water, urban, forest and agriculture. Fig. 6. All 12 subsets of NCD land cover maps used to generate te training database. Eac subset as pixels and four land cover classes. Fig. 7. Four subset NCD land cover maps (wit pixels) used to test te proposed algoritm.

22 We select twelve subsets of NCD maps (eac contains pixels) as sown in Fig. 6 to generate te coarse and fine spatial resolution patc pairs, wic formed te training database. A furter four subsets of NCD maps (eac contains pixels; Fig.7), were used to assess te proposed SRM algoritm. ese twelve subsets are called as training maps and te four subsets are called as test maps. e locations of tese four test maps are different to tose of te twelve training maps used to construct te training database. For eac of te four test maps, we generate syntetic coarse fraction images by linear averaging te fine spatial resolution pixel number witin te coarse spatial resolution pixel according to different zoom factors. Using simulated fraction images as input, as well as te constructed training database, te proposed SRM algoritm is applied to estimate a fine spatial resolution land cover map. In order to assess te accuracy of te proposed learning based SRM algoritm, by using te test land cover maps as te reference, te overall accuracy (OA) and Kappa coefficient are used to evaluate te accuracy of te estimated land cover maps, by comparing te entire estimated map wit te corresponding reference map. e proposed learning based SRM algoritm was also compared wit te pixel-based ard classification metod (HC) and several popular SRM algoritms including te pixel swapping algoritm based on simulated annealing (PS) [61], te sub-pixel/pixel attraction algoritm (SPA) [29], te bilinear interpolation based algoritm (BI) [56] and te BP neural network based algoritm (BP) [53]. It is noted tat te input of PS, SPA and BI algoritms includes only coarse spatial resolution fraction images, because all tese SRM algoritms describe te land cover distribution using te spatial dependence principle. By contrast, te input of BP algoritm includes not only te coarse spatial resolution fraction images, but also fine spatial resolution land cover maps tat are used to learn prior land cover information. In te experiments, all twelve training maps are used in te BP algoritm.

23 Fig. 8. Simulated fraction image generated from te test map I as sown in Fig. 7. e top row presents te fraction images at z 5, and te bottom row presents te fraction images at z 8. e test map I as sown in Fig. 7(a), was first used to assess te impact of parameters in te proposed SRM algoritm. Syntetic fraction images, as sown in Fig. 8, were simulated from te fine spatial resolution test map I. wo zoom factors, z 5 and z 8, were applied. e fine and coarse spatial resolution patc pairs in te training database were generated from all twelve training maps as sown in Fig. 6 wit zoom factors of 5 and 8, respectively. Wen te proposed learning based SRM algoritm is performed, te resultant fine spatial resolution land cover map is affected by parameters used in te algoritm. According to our experiments, te coarse patc size p was set to be 3, because a large p value makes te spatial structure of a coarse patc too complex to find enoug similar image patces. Max was set to 1, meaning tat no fine spatial resolution neigboring patces are considered as outliers at te beginning. d was set to 0.05, in order to decrease te value of gradually. Moreover, to furter assess te impact of parameter values on te performance of te proposed metod, tree most important parameters including te number of patc pairs in te training database, te coarse spatial resolution fraction difference tresold value and te minimal fine spatial resolution difference tresold value Min are discussed in te following sections, respectively.

24 Fig. 9. Kappa values of te resultant fine resolution land cover maps of te proposed learning based SRM algoritm wit different numbers of training patc pairs. (a) is te result at z 5 ; (b) is te result at z 8. 1) Impact of te number of patc pairs in te training database: o assess te impact of te number of patc pairs in te training database, te analysis was repeated 8 times over te range of 4 4 [1 10,15 10 ] wit an interval of Fig. 9 indicates te Kappa values of resultant fine spatial resolution land cover maps produced by te learning based SRM algoritm wit different numbers of patc pairs in te training database, at zoom factors of 5 and 8, respectively. Wen te number is less tan , te Kappa values of te results at z 5 and z 8 are bot at a low level. is is because te use of only a few patc pairs cannot provide enoug information about te spatial land cover patterns. Wit te increment of te number, bot Kappa values at z 5 and z 8 increase rapidly until te number reaces to about , and ten te Kappa values maintain stable, as sown in Figs. 9 (a) and (b). It is also noticed tat te increment of Kappa values at z 5 is more rapid tan tat at z 8, wen te number is in te range of 4 4 [3 10, 9 10 ]. e reason is tat more patc pairs are needed in order to decrease te uncertainty caused by a large zoom factor. According to te experiment, a number of patc pairs larger tan is reasonable. As more patc pairs may increase te calculation burden, te number of patc pairs in te training database is set to be 2) Impact of te tresold : e tresold value in our latter experiments. is a pre-defined parameter tat represents te tolerable fraction difference between two coarse spatial resolution patces. is parameter guarantees tat te searced coarse spatial resolution patc wic as a fraction difference f larger tan cannot be accepted as te candidate image patc. Fig. 10 sows te Kappa values of te resultant fine spatial resolution land cover

25 maps produced by te proposed learning-based SRM algoritm wit different values. Wen is in te range of [0.02, 0.14] at z 5 and in te range of [0.02,0.12] at z 8, te Kappa values increase wit te increment of te value of. is is because tat te number of candidate patc pairs is too small to provide enoug land cover pattern information for SRM, if is set to be a too low value. For z 5, as sown in Fig. 10(a), if te value of is larger tan 0.14, te Kappa values are kept at a stable level. For z 8, owever, te Kappa values begin to decrease until a stable value, if is larger tan In general, wit a larger value of, more patc pairs wit larger fraction errors are considered as candidate patces, wic indeed generally provide erroneous information and make te estimated fine spatial resolution land cover map differ significantly from te latent map. Setting too large a value of may degrade te estimated fine spatial resolution map. However, te result is only sligtly different because of te additional patc outlier rejection procedure. Fig. 10. Kappa values of te resultant fine resolution land cover maps of te proposed learning based SRM algoritm wit different values of coarse resolution tresold F. (a) is te result at z 5 ; (b) is te result at z 8. Fig. 11. Kappa values of te resultant fine resolution land cover maps of te proposed learning based SRM algoritm wit different minimal fine resolution tresold Min. (a) is te result at z 5 ; (b) is te result at z 8. 3) Impact of te minimal fine spatial resolution tresold of Min : e value of Min is denoted as te minimal tresold value, wic is used to distinguis te outlier patces from all neigboring patces in te

26 estimation procedure. o assess te impact of te Min value, te learning based SRM algoritm was applied wit different values of Min in te range of [1.0, 0.1] wit an interval of e Kappa values of te resultant fine spatial resolution land cover maps produced by te proposed learning based SRM algoritm at z 5 and z 8 are sown in Fig. 11. Wit te decrement of te values of Min, te corresponding Kappa values of te resultant fine spatial resolution land cover maps increase, because more coarse spatial resolution patces are considered as outlier patces and not used in te estimation procedure. Wen Min is larger tan 0.2 at z 5 and 0.3 at z 8, owever, te Kappa values begin to decrease as te value of decreases. e reason is tat most neigboring patces are considered as outliers if te value of is too low, and te remaining neigboring patces can not provide enoug information for SRM. Fig. 12. Resultant fine resolution land cover maps produced by te proposed learning based SRM algoritm. (a) is te result witout patc outlier andling procedure and (b) is te result using patc outlier andling procedure at z 5. (c) is te result witout patc outlier andling procedure and (d) is te result using patc outlier andling procedure at z 8. Using te patc outlier andling procedure reduces te salt-and-pepper artifacts as sown in te enlarged part A, and te linear discontinuities as sown in te enlarged part B. A visual comparison, as sown in Fig. 12, is used to furter assess te impact of te value of Min. For z 5, as sown in Fig. 12(a), many salt-and-pepper artifacts appear and te spatial continuities of some linear Min features are interrupted wen 1.0, wic means tat all patces are accepted and no outliers exist. In Min contrast, wit patc outlier rejection (wen 0.2 ), most of te salt-and-pepper artifacts are eliminated and

27 te spatial continuities are well maintained, as sown in Fig. 12(b). e impact of Min on te resultant fine spatial resolution land cover map is more obvious wen z 8. Many salt-and-pepper artifacts and linear discontinuities tat appear in te fine spatial resolution land cover map, as sown in Fig. 12(c), are eliminated by te outlier rejection, as sown in Fig. 12(d). Fig. 13. Resultant land cover maps generated by different metods wit z 5 and z 8 for te test map I. HC produces jagged boundaries (suc as te area A); PS produces isolated patces (suc as te area B); SPA and BI produce linear artifacts (suc as te area C); and BP produces salt-and-pepper artifacts and isolated patces (suc as te area D). 4) Comparison wit oter metods: According to aforementioned discussion about parameters used in te proposed learning based SRM metod, te optimal parameter values were used to produce te resultant fine spatial resolution land cover maps. In particular, te number of patc pairs in te learning database is , te value of f is 0.12, and te value of Min is 0.2 for z 5 and 0.3 for z 8, respectively. By using te same aforementioned simulated coarse fraction images used for te proposed learning based SRM metod as input, te resultant land cover maps of HC, PS, SPA, BI, BP and te proposed learning based SRM are all sown in Fig. 13. Additionally, te Kappa and OA values of all tese maps produced by different metods are sown in able. I.

28 able I. Kappa and Overall Accuracy (OA) values of te resultant land cover maps produced by different metods at z 5 and z 8 for te test map I. HC PS SPA BI BP Proposed z=5 Kappa OA z=8 Kappa OA In general, for all SRM algoritms, te zoom factor plays an important role on te accuracy of result. e key point of SRM is to determine te class labels of te fine spatial resolution pixels witin te coarse spatial resolution pixel. For a given zoom factor z, tere will be z z fine spatial resolution pixels witin te coarse spatial resolution pixel to be estimated. Wit te increase of z, te number of fine spatial resolution pixels would be increased exponentially, and more fine spatial resolution pixels witin te coarse spatial resolution pixel need to be estimated. erefore, te uncertainty of te estimation would be expected to increase, and te performance of te algoritm would decrease. For te results of HC, as sown in Fig. 13 and Fig. 13, te land cover boundaries are jugged and many spatial details are missed. e Kappa and OA values of te results of HC, as sown in able. I, stay at te lowest level. is is because tat HC is based on te pixel scale, and does not consider te spatial distribution of classes at sub-pixel scale. By contrast, more land cover details at te sub-pixel scale are maintained by SRM including PS, SPA, BI, BP and te proposed learning-based SRM metod. Visual comparison of te fine spatial resolution maps obtained from te various SRM analyses igligted differences in te way tey represented te land cover. For te results of PS, many land cover features are mapped as isolated rounded patces, and te spatial continuities are interrupted. is sortcoming becomes more serious wit te increment of te zoom factor.

29 Wen z 8, te Kappa and OA values of PS are and , and are even lower tan tose of HC because te uncertainty of te fine spatial resolution pixel distributions in te resultant fine spatial resolution land cover map generated by PS is serious wen te zoom factor is large. e fine spatial resolution land cover maps produced by SPA and BI visually differed from tat obtained wit PS wit more spatial detail are maintained. e Kappa and OA values of fine spatial resolution land cover maps produced by SPA and BI are also iger tan tose of PS. However, numerous linear artifacts are found near te land cover boundaries in te obtained fine spatial resolution land cover maps produced by SPA and BI, and te linear artifacts become more serious wit te increment of te zoom factor. e fine spatial resolution land cover maps produced by PS, SPA and BI are, terefore, less tan ideal. is is because tese SRM metods just apply te spatial dependence assumption to describe te land cover pattern, and tis assumption will often be too simple to provide enoug information for SRM in areas wit complex land cover patterns. Compared wit te results of PS, SPA and BI, te linear artifacts in te results of BP are eliminated and te spatial continuities are maintained to some extent. e Kappa and OA values of te resultant fine spatial resolution land cover maps produced by BP at zoom factor of 5 and 8 are bot iger tan tose of PS, SPA and BI. is improvement arises from te use of additional information about te spatial land cover pattern tat is learned from te training database in te SRM procedure. However, in te resultant fine spatial resolution land cover maps produced by BP, many salt-and-pepper artifacts are found and many linear land cover features are also mapped as isolated small-sized patces. Moreover, wit te increment of te zoom factor, te geometric integrity of features is more difficult to be maintained by te BP based SRM metod. e sortcoming of te BP based SRM metod is mainly caused by its two-step procedure and te impact of outliers during te learning procedure. e resultant fine spatial resolution land cover maps produced by te proposed learning-based SRM metod

30 are more similar to te reference (Fig. 7) at bot zoom factors tan te maps produced from te oter SRM metods. Isolated land cover patces, jagged sapes and linear artifacts, wic are common in te resultant fine spatial resolution land cover maps produced by aforementioned metods, are effectively eliminated by te proposed learning based SRM metod. More spatial details, especially te linear features and te spatial land cover continuities are maintained. e Kappa and OA values of te resultant fine spatial resolution land cover maps produced by te proposed learning based SRM metod are all te igest at bot zoom factors.. Bot te visual comparison and accuracy analysis indicate tat te proposed learning-based SRM metod is superior to te SRM algoritms used for comparison. e test maps II to IV, as sown in Figs. 7(b) to (d), were used to furter validate te performance of te proposed learning-based SRM metod. As wit te analyses focused on test map I, wit te zoom factors of 5 and 8, te tree fine spatial resolution test maps were degraded to produce te simulated coarse spatial resolution fraction images. ese simulated coarse spatial resolution fraction images are ten used as input to te SRM metods in order to produce te resultant fine spatial resolution land cover maps. e same parameter values used in te experiment of te test map I were applied for test maps II to IV. e resultant land cover maps produced by HC, PS, SPA, BI, BP and te proposed learning based SRM algoritm at zoom factors of 5 and 8 are all sown in Fig. 14 and Fig. 15, respectively. By using te original fine spatial resolution land cover maps as te reference, te accuracy analysis of tree land cover maps at z 5 and z 8 are sown in able II. A similar trend as te experiment of te test map I is found by visually comparing te resultant fine spatial resolution land cover maps produced by different metods. e resultant land cover maps produced by HC ave jagged boundaries and many spatial details are missed. By contrast, te fine spatial resolution land cover maps produced by SRM metods ave smoot boundaries and more spatial details are maintained. e fine spatial resolution land cover maps produced by PS are locally smoot and many linear land cover features are mapped

31 into individual round patces for all tree test maps. e spatial land cover pattern of te results produced by SPA and BI are muc improved, owever, numerous irregular linear artifacts exist near te boundaries. Moreover, more linear artifacts appear in te SPA and BI results of te test map III tan tose of te test map II and IV, due to te complex land cover features of te test map III, especially te urban class. e BP based SRM metod can reconstruct more spatial details, and eliminate linear artifacts in te results of SPA and BI to some extent, due to additional spatial land cover information is learned from te extra fine spatial resolution land cover maps. However, many salt-and-pepper artifacts and isolated small-sized patces are still unavoidable due to te errors caused by te two-step process. By contrast, te fine spatial resolution land cover maps produced by te proposed learning-based SRM metod, as sown in Fig. 14 and Fig. 15 at z 5 and z 8, are more similar to te reference fine spatial resolution land cover maps as sown in Fig. 7. e fine spatial resolution land cover maps produced by te proposed algoritm are smoot. Salt-and-pepper artifacts, isolated patces and irregular linear artifacts tat appear in te results of some of te oter SRM analyses are mostly eliminated. For all of te tree testing land cover maps at bot zoom factors of 5 and 8, more spatial detail, especially te linear features, are well maintained in te results of te proposed learning based SRM algoritm.

32 Fig. 14. Resultant land cover maps generated by different metods for test maps II-IV wit z 5. HC produces jagged boundaries (suc as te area A); PS produces isolated patces (suc as te area B); SPA and BI produce linear artifacts (suc as te area C); and BP produces salt-and-pepper artifacts and isolated patces (suc as te area D). Fig. 15. Resultant land cover maps generated by different metods for test maps II-IV wit z 8. HC produces jagged boundaries (suc as te area A); PS produces isolated patces (suc as te area B); SPA and BI produce linear artifacts (suc as te area C); and BP produces salt-and-pepper artifacts and isolated patces (suc as te area D). able II. Kappa and Overall Accuracy (OA) values of te resultant land cover maps produced by different metods at z 5 and z 8 for test maps II-IV. HC PS SPA BI BP Proposed est map II z=5 z=8 Kappa OA Kappa OA est map III z=5 Kappa OA z=8 Kappa