Usability of the SLICES land-use database

Usability of the SLICES land-use database Olli Jaakkola * & Ville Helminen** * Finnish Geodetic Institute, P.O.Box 15 (Geodeetinrinne 2), FIN-02431 MASALA, FINLAND, e-mail: olli.jaakkola@fgi.fi ** Finnish Environment Institute, P.O. Box 140, 00251 HELSINKI Abstract. The research describes a usability of the SLICES land-use database. The land-use database is briefly depicted. Major properties of the quality assessment method, the used stratified random sampling method, the error matrix and derived statistical summary measures, and the possible error sources in assessment, are illustrated. The major results of different output products, main land-use classes, test areas, and data acquisition dates, are summarised. Finally, as a conclusion from the quality assessment we may state that probably many of the usability requirements are meet, although some additional research may still be needed. Introduction The quality assessment and description of spatial databases is an increasingly salient aspect of the use of geographical information systems. Data of unknown quality cause many kinds of inconvenience for the user, and evidently this disadvantage should be avoided. Often only the information concerning errors and their properties helps the user to judge whether the database suits his/her particular needs, although subsequent investigation does not correct any errors. The objective of quality assessment of the SLICES land-use database is to test and describe the accuracy of geographical objects and classes in order to help users evaluate how the database meets their requirements for spatial analysis. The accuracy assessment report provides an overview for the common user, who then should be able, according to testing methods used and calculated quality descriptors, to evaluate whether SLICES land-use classification is usable to his/her needs. Usability in general can be considered as a quality or state of being usable, concerning how well the data fit to be used. It is a relative quality in relation to user needs or suitability to a certain task. Usability is users point of view to the data, in most cases it can be regarded as a quality applied with a view to the particular application. Usability engineering is a process whereby the usability of a product (e.g. database) is specified quantitatively (Faulkner 2000). It can be studied by looking over the available databases and their common properties. The most elementary view to usability looks for the minimum requirements of database, i.e. for the things that prevent the use of database. Common known problems that prevent the use of certain database are related to the timeliness, content, scale, transferability, format, coverage or accuracy (variation in accuracy) of the database. In this study we are directly dealing with the accuracy and variation in accuracy as a preventing property of using certain database, although via the metadata layers of source data and data acquisition date also the timeliness and content are taken into account. Then the question arises how to test the usability of land cover database. As already noted, the quality of database is the main objective in this research. A rather conventional view to the quality assessment has been taken to the sampling design and to the compilation of accuracy assessment results. Classification accuracy is taken as a degree to which the derived land-use classification agrees with reality (Smits et al. 1999). In other words, it is assumed that the ground or reference data used are themselves an accurate representation of reality, although in fact they are just another approximation of reality. Thus, the accuracy assessments are actually only a measure of the degree of agreement or correspondence to the ground data (Foody 2000). Sample size was thought of providing a representative sample when the size exceeds the normal distribution assumptions limit (central limit theorem) minimum of 30 samples per test area (Shaw & Wheeler 1985). This size was not achieved with simple random sampling, and therefore a stratified random sampling approach was taken. Despite of this approach still some classes in individual testing sites do not reach the limit. SLICES land-use database The SLICES project was initiated in 1997 as a cooperative project founded by the Ministry of Agriculture and Forestry and headed and processed by the National Land Survey of Finland. The project team consisted of the owner organizations of 6 different input data sources. The role of the Finnish Geodetic Institute has been to act as a quality assessment body. The name SLICES (Separated Land Use / Land Cover Information System) states the basic structure of the used classification system, it consists of disparate slices of GIS data elements describing the various aspects of land related information. The aim of the SLICES project was to establish a system for producing suitable data slices on Finnish land-use, land cover, soil and land-use restriction conditions. 1

In 1998 compilation of test data for the land-use database began, and production of the entire database was finished by the end of 2000. During the years of 2000 and 2001, an accuracy assessment study on the quality of SLICES land-use database was conducted. The SLICES land-use database was built by combining various types of present land-use/cover information databases into one unified database of land-use information encompassing the whole of Finland (see SLICES project homepage). The foundations for land-use database consist of 10 different input sources, having 6 different owner organisations. The input databases have different data types of point, line, area and raster, and various levels of scale ranging from 1:5 000 to 1:50 000 (Table 1). Database/description Owner organisation Type of data Scale of data FLPIS (Finnish Land Parcel Ministry of Agriculture and Identification System) Forestry Area 1:5000 Topographic database National Land Survey Area 1:10 000 Road database National Land Survey Line 1:20 000 Topographic map in raster CD: fields and base features National Land Survey Raster 1:20 000 National Forest Inventory Finnish Forest Research Institute Raster - Water area database Finnish Environment Institute Area 1:20 000 Nature conservation areas Finnish Environment Institute Area 1:20 000 Building and housing register Population Register Centre Point - Power lines FINGRID Oyj Line 1:50 000 CORINE land cover digitalisation Finnish Environment Institute Area 1:10 000 Table 1. The main SLICES land-use input database sources. The making of a unified database has taken place in several phases. Firstly, the original data sources of point and line, were transformed to areas by using the attribute information for guiding the buffering function. After that all data sources were transformed to common raster format, and to the same Finnish YKJ (Transverse Mercator) coordinate system. Secondly, all data sources were brought together according to the predefined piling order. In the piling the order is chosen so that in principle each pixels class is determined by the most accurate and newest data source. This improves the positional accuracy and timeliness of the combined database, although it produces overlapping classes and slivers of unknown data. However, the highest class in piling order, roads, was kept a unified road network. Thirdly, after combination and piling, the single raster layer database was generalised in order to remove slivers of unknown data and very small patches of land-use. Generalisation reduces the amount of heterogeneity brought by the various input data sources, and the result is a unified scale dataset, which has common defined properties. The aim of the generalisation is to produce an end product suitable for various kinds of geographical analysis, merely not a graphical map output. In overall, the main objective is to remove patches of land us smaller than the defined minimum mapping unit (MMU), and at the same time fill the sliver gaps of unknown data. The result is a common generalised land-use database having a MMU of 0.25 hectares. Together with the database itself a metadata layer is provided as a separate input data source layer, and as a data acquisition date layer providing information for each pixel. The SLICES land-use classification consists of several output products: the main product is a 10-m generalised database, and other products are 25-m generalised database, 10-m ungeneralised database, and a generalised vector database. The SLICES land-use database has various types of uses. The main users at the moment have very different tasks to solve with the land-use data: it is used for statistical area summaries in various areas (mainly for municipality and NUTS statistics from the whole of Finland); it is used for pixel based summaries as a background knowledge for various environmental analysis (land-use in 250 meter pixels); it is used for calculating the link site positions for cellular phone networks; it is used for tracking the most suitable path for military movements; and it is used for as a background knowledge in areal planning and in road planning. 2

Testing procedure The first part of the usability research was an accuracy assessment performed by picking sampling points from the areas and comparing each sampling point to the corresponding correct class in the reference data. The correct class was interpreted either from digital orthophotographs of 1:55 000 scale, topographic database of 1:10 000 scale, or by field verification using GPS as a helping device. The SLICES land-use database to be tested consists of raster databases of 10-m ungeneralised, 10-m generalised or 25-m generalised. The classification consists of 8 main classes: A = residential and leisure areas, B = business, administrative, and industrial areas, C = supporting activity areas, D = rock and soil extraction areas, E = agricultural land, F = forestry land, G = other land, and H = water areas. The accuracy assessment of SLICES land-use classification is based on stratified random sampling, where the stratification is carried out according to the classes surface area share. The classes are divided into 3 categories according to their surface area share, and inverse proportion of share is adjusted to the classes weight. The first category consisted of classes having a more than 10 % share with weight 1; the second category, classes having a less than 10 % but greater than 1 % share with weight 5; and the third category, classes having a less than 1 % share with weight 25. Stratification was performed separately for each test area. Nevertheless, it is difficult to obtain a representative sample from each of the 48 classes, although the overall matrix from all areas or matrices only with 8 main classes do give satisfactory results (Figure 1). There were a total of 12 testing areas, with an area of 10 km x 10 km, from which fewer than 6000 samples were taken (Helminen et al. 2001). Accuracy assessment is based on the principle of the error, or confusion, matrix. From this matrix overall accuracy of the entire classification, interpretation (user s) and object (producer s) accuracies for the land-use classes, and surface area correspondence was calculated. From the error matrix the confidence intervals, and the kappa statistic for each measure were also calculated. Finally, a surface area corrected error matrix was calculated to give each element in the error matrix new values according to proportional share and surface area based on corrective measures. The result is an error matrix in which the proportions of sample points are similar to that of the surface area of each class. Sources of error in testing The use of orthophotographs is considered as a cost-effective way to collect reference ground data. However, there are several sources of error in accuracy assessment done from orthophotographs or field checking. One obvious source of error in assessment is the time difference between the land-use database (nominal dating: summer 1999), the reference material (orthophotographs dated from 1997 to 1999), and field checking (dated back to summer/autumn 2000). Other uncertainties related to the using and interpreting orthophotographs include definitions of land-use classes, heterogeneity of sample units, consistency in photo interpretation, and sample location (Yang et al. 2000). H 12% A 15% F 29% G 4% B 3% D 1% C 15% A B C D E F G H E 21% Shares of sampling points in SLICES land-use accuracy assessment 3

G 4% H 7% A B 3% 0% C 3% D 0% E 23% A B C D E F G H F 60% Shares of main classes in SLICES land-use, 10-m generalised database Figure 1. Shares of classes in the SLICES database and in sampled points. For class definitions see chapter 3. Several classes were especially difficult to interpret. The main classes themselves were easily separated from the orthophotographs. A hard-to-interpret class is residential areas, since the separation of different housing areas from business, industrial, or leisure areas is not an easy task. Another problem is the separation between agricultural fields and unused agricultural land, since an area may be classified as unused if no EU support has been applied for within a particular year. The separation between productive forestland and forestland of low productivity is also sometimes difficult, since the line between them is by definition drawn with reference to forest land productivity, which varies not only between patches, but also within a single patch. Various types of errors have been found in the SLICES land-use classification. These errors are not evenly distributed among all classes but are concentrated between certain classes only. Not all errors have the same significance, since it is a more serious mistake to confuse real forest with fields than with forestland of low productivity. Errors in position, especially errors caused by the wrong position of road segments are the most significant. For the other classes, wrong positions were not as easy to detect. The second important type of error is that caused by the generalization of land-use patches. Many of the small patches less than MMU of 0.25 hectares, are covered over by the dominant class in this area, and thus are generalized out from the database. On the other hand, generalization also removes data with no class label, which partly covers real information about land-use. Data class age appears to have little or no influence on the errors found, which may be due to the fact that older classes change less (Figure 2) 4

Number of sampling points 2000 1800 1600 1400 1200 1000 Erroneous points Correct points 800 600 400 200 0 63 67 73 77 78 84 87 88 89 90 91 92 93 94 95 96 97 98 99 Year of data acquisition Results Figure 2. Number of correct and erroneous sampling points for the SLICES 10-m generalised database according to the year of data acquisition. The result of the research is an accuracy estimate of the entire land-use data and a set of statistical accuracy measures. The accuracy of SLICES land-use classification according to the main classes gives a picture of the error types found in the database. According to the main classes, 10-m generalised land-use database have about 8-14 % better accuracy than the 25-m generalised database (Figure 3). In the following, the accuracies are given to the main output product 10-m generalised database, although Table 3 gives the accuracies also for the 25-m generalised database. Main class A (residential and leisure areas) has an object accuracy (OA) of 91 %, and interpretation accuracy (IA) of 88 % (Table 2). The main error types are the positional errors of individual buildings or totally missing buildings, especially summer houses having no coordinates. Main class B (business, administrative and industrial areas) has an OA of 88 %, and IA of 90 %. Major errors arise from larger industrial areas, which occupy too small an area. Also, some parking areas are missing from the database. Built-up areas are generally under-delineated in the database due to missing buildings and other deficiencies. 5

100% 90% 80% 70% 60% 50% 40% 30% 20% 10 meter generalised, all 48 land-use classes 10 meter generalised, 8 main land-use classes 10 meter ungeneralised, 8 main land-use classes 25 meter generalised, 8 main land-use classes 10 meter ungeralised, all 48 land-use classes 25 meter generalised, all 48 land-use classes 10% 0% Test areas Figure 3. The differences between overall accuracies of the SLICES land-use output products in various test areas. Main class C (supporting activity areas) has an OA of 94 %, and IA of 91 %. The main class consists largely of roads and road segments. The major error type is the shifting of roads, which fortunately is often no greater than 10 meters. A common error occurs here when a road forms a small loop, the area within it is generalized as road. Conversely, market squares and parking lots for example, are missing and often represented in the database as forestland. Main class D (rock and soil extraction areas) has an OA of 95 %, and IA of 99 %, although some targets are likely to be missing from the database. Main class E (agricultural land) has an OA of 95 %, and IA of 92 %. The most important error type is the division between used agricultural land and unused land which may be only temporarily unused at the time of inspection. Other confusing classes include forestland and residential buildings; these often limit the borders of agricultural land. Main class F (forestry land) has an OA of 90 %, and IA of 95 %. Forestry land covers the largest portion of total land area, namely 60 % of tested areas. The most difficult aspect of forestry land inspection was establishing class limits between productive forestland, forestland of low productivity, and wasteland. Both low-productivity forestland and wasteland often exist in scattered individual pixels, forming small clumps, which are difficult or impossible to delimit from their surroundings. Main class G (other land, consisting mainly of wasteland) has an OA of 82 %, and IA of 91 %. Main class H (water areas) has excellent accuracies: OA is 98 % and IA 95 %. Some forestry and agricultural land has been classified as water due to positional error near water lines. Comparing various test areas reveals some of the differences between land-use areas, and also differences between their qualities. Overall accuracy varies from a low of 88.2 % to a high of 92,5 % for the 10-m generalised database, so generally the differences were rather small. However, for the main class A the differences are significant. Also were found the differences between the qualities of various input data sources. 6

Tables 2 and 3. SLICES land-use. Accuracies of generalised database with pixel size of 10 meters, and generalised database with pixel size of 25 meters, classification according to main classes. Conclusions We may conclude that SLICES land-use classification fulfills quality expectations. It is probably in most cases accurate enough for the geographical analysis task, however this needs to be studied more thoroughly by giving examples of usable and non-usable geographical analysis. Nevertheless, the accuracy assessment defines the maximum obtainable quality for the used geographical analysis. From the accuracy assessment gives a possibility for calculating the uncertainty associated with quantitative conclusion drawn from the land-use database. The usability is increased by taking into account the metadata layers of data acquisition date, and of input data source information together with the quality evaluation report. The various data acquisition dates for the source data deteriorate to certain extent the usability especially for the change detection, although the spatial heterogeneity of the database is diminished during the data generalization process. The differences between different scale data products are great, i.e. the databases of 10 m vs. 25 m, the scale of output data affects how detailed and accurate the classification may be. Generally, the differences between various testing sites are not large, however for some classes are. Regarding the accuracy assessment, we may improve quality testing by taking more fine-graded, stratified, random samples using more stratified categories, and we could ensure that more samples be taken for classes having a small proportional area. Also, the time errors could be limited if simultaneous timing could be employed for the land-use database and the inspected ground truth. Finally, we could also evaluate neighboring pixels in order to identify positional errors in the database. The second part of the research, the evaluation of the usability of SLICES land-use database to the certain geographical analysis, is still in its initial phase. The usability evaluation continues with the assessment of SLICES database as a source for selected quantitative geographical analyses. Especially the defects of present land-use database are depicted, and the solutions for overcoming the defects are searched. 7

References Faulkner, X., 2000. Usability engineering. Palgrave, Houndmills, Basingstoke, Hampshire. 244 p. Foody, G.M., 2000. Accuracy of thematic maps derived from remote sensing. Proceedings Accuracy 2000, Amsterdam, pp. 217-224. Helminen, V., Jaakkola, O. & T. Sarjakoski, 2001. SLICES maankäyttöluokituksen laadun tarkastus (In English: Accuracy assessment of SLICES land-use database). Geodeettisen laitoksen tiedote, (23), 57 s. Shaw, G. & D. Wheeler, 1985. Statistical techniques in geographical analysis. John Wiley & Sons, Chichester. 364 p. SLICES project, Homepage of the SLICES project (In Finnish), accessed 19 December 2001, http://www.slices.nls.fi/ Smits, P.C., S.G. Dellepiane & R.A. Schowengerdt, 1999. Quality assessment of image classification algorithms for land cover mapping: a review and proposal for cost-based approach. International Journal of Remote Sensing, 20, pp. 1461-1486. Yang, L., S.V. Stehman, J. Wickham, S. Jonathan & N.J. VanDriel, 2000. Thematic validation of land cover data of the Eastern United States using aerial photography: feasibility and challenges. Proceedings Accuracy 2000, Amsterdam, pp. 747-754. 8