Estimation or stochastic simulation in soil science?

Transcription

1 Estimation or stochastic simulation in soil science? Castrignanò A., Lopez N., Prudenzano M., Steduto P. in Zdruli P. (ed.), Steduto P. (ed.), Kapur S. (ed.). 7. International meeting on Soils with Mediterranean Type of Climate (selected papers) Bari : CIHEAM Options Méditerranéennes : Série A. Séminaires Méditerranéens; n pages Article available on line / Article disponible en ligne à l adresse : To cite this article / Pour citer cet article Castrignanò A., Lopez N., Prudenzano M., Steduto P. Estimation or stochastic simulation in soil science?. In : Zdruli P. (ed.), Steduto P. (ed.), Kapur S. (ed.). 7. International meeting on Soils with Mediterranean Type of Climate (selected papers). Bari : CIHEAM, p (Options Méditerranéennes : Série A. Séminaires Méditerranéens; n. 50)

2 ESTIMATION OR STOCHASTIC SIMULATION IN SOIL SCIENCE? 1 1 ANNAMARIA CASTRIGNANÒ, NICOLA LOPEZ, 1 2 MARCELLA PRUDENZANO AND PASQUALE STEDUTO 1-Istituto Sperimentale Agronomico, via C. Ulpiani 5, Bari, Italy; annamaria.castrignano@tin.it 2-Istituto Agronomico Mediterraneo, Valenzano - Bari, Italy; steduto@iamb.it Introduction Soil texture is a basic soil property that is rather easy and relatively cheap to measure and is generally available in most soil databases included in modern Geographic Information Systems (GIS). It is then often used in pedotransfer functions for predicting more difficult to measure and more expensive properties, such as soil hydraulic conductivity (Peterson et al., 1968; Mc Keague et al., 1982). In many cases, its spatial variation is used as an input in complex non-linear models to investigate the impact of a given scenario, such as a particular remediation process or land use policy, in environmental decision-making. Therefore, it is essential to make a correct evaluation of these non-linear predictions as related to the spatial variability and uncertainty of soil texture. A map of kriging estimates of texture represents the best estimates in the least-squares sense, because the local error variance is minimum, but not necessarily the best of all. In fact, this map it is not designed to reproduce spatial patterns of extreme values, as in the presence of sandy lens in a clay matrix, or in restricted zones of high permeability, or rich in a given metal. In addition, kriged maps are well known to be smoothed in their spatial variation and therefore fail to reflect the proportion of extreme values (both high and low) as observed on real data. In land planning or irrigation scheduling, the detection of occurrence patterns of extreme values of permeability or porosity may be more critical than local accurate estimation of mean values. In such cases, we need maps, which honour patterns of spatial continuity and provide an assessment of joint spatial uncertainty. Another drawback of estimation is that smoothing is not uniform, but it depends on the local data configuration, being minimal close to the data. Consequently, a kriging map appears more variable in densely sampled areas than in sparsely sampled ones, which can produce artefact structures. An alternative approach to kriging estimation consists in generating multiple realisations of the spatial distribution of soil texture, providing a visual and quantitative measure of spatial uncertainty. This approach has become popular in soil science only in the last decade (Castrignanò et al., 2001) and is known as stochastic simulation. It does not aim at minimising local estimation variance, but rather at reproducing statistics, such as the sample histogram and the semivariogram model, besides honouring sample data (Goovaerts, 1997a; Deutsch and Journel, 1998).

3 Estimation or Stochastic Simulation in Soil Science? A simulated map preserves the correlation structure, whereas kriging smoothes, i.e. loses variance, and thus looks more realistic. However, there is a price to be paid, i.e, a doubled prediction error variance. Nevertheless, it can be tolerated because spatial uncertainty analysis, rather than optimal prediction, is the aim of simulation (Olea and Pawlowsky, 1996; Goovaerts, 1997b, 1998). If alternative realisations of the spatial distribution of texture are sent as input in complex flow simulation models, the processing of each input realisation yields a unique value for each response, for example a unique value for soil water content. The distribution (histogram) of the different responses corresponding to the input values of texture provides a measure of water content uncertainty, resulting from our imperfect knowledge of reality. This piece of information is fundamental in evaluating the risks consequent on some strategic decisions or in analysing the propagation of errors through GIS (Heuvelink, 1998). Since soil texture is a composite variable, defined by the relative proportions of clay, silt and sand in the soil, simulation of texture requires at least two of the said variables. This paper presents an application of the geostatistical methods of kriging estimation and stochastic simulation for modeling spatial patterns of clay proportion in an area of southern Italy. The specific objectives of the study were: (i) to describe and assess spatial variability of clay content and (ii) to compare the performances of the two approaches by using a validation data set and standard statistical criteria. Experimental Design In order to evaluate the theoretical performance of the two methods -estimation and simulation- we worked on a real hard data set from an area of the Apulia region (southern Italy) in the land surrounding the city of Foggia (14.93 E, N; E, N). The data set included several physical, chemical and hydraulic properties, measured in soil samples collected from top soil (0-40 cm) in 1,153 locations during different soil surveys. The data were measured in different laboratories and the homogeneity of variance was checked for the interaction data source x landscape unit, which the survey site belongs to. In this paper only the results related to the clay proportion will be reported. Estimation and Simulation Maps The data were submitted to two types of processing: estimation, through ordinary kriging, and stochastic simulation, through conditional sequential gaussian technique, at the 30,312 nodes of a regular 500mx500m grid. From the variogram map, two directions were identified showing the minimum (NW) and maximum (NE) spatial continuity, thus an anysotropic model was fitted to the corresponding directional variograms. The goodness of fitting was tested by cross-validation. Since data from several surveys resulted to be of different qualities, kriging with measurement error was applied. This technique consists in a slight modification of the theory which makes it possible to take into account variable measurement errors at data points, provided the error variances have been computed separately for each sub-population, as described above. 168

4 Soil Genesis, Classification and Cartography Since gaussian simulation requires a normality assumption, the raw variables were preliminarily transformed into a normal distribution and, afterwards, the simulation results were back-transformed into the raw distribution. Such a transformation was realised by the gaussian anamorphosis, which is a mathematical function transforming a variable with a gaussian distribution into another one with any distribution. This function was written as a polynomial expansion of Hermite Polynomials stopped to a given order. The same directions of zonal anisotropy identified for estimation were used in calculating the experimental variogram of the gaussian variable, which a theoretical directional model was fitted to. Conditional sequential gaussian procedure enables to perform several simulations of the clay variable in the gaussian assumption and in the conditions of second order stationarity. All simulations replicate the variogram model given as input and are constrained to adhere to all the actual information (conditional simulation). This method consists in defining a regularly spaced grid, covering the region of interest (in our case the same used for kriging estimation), and a random path through the grid, such that each node is visited in sequence only once. At each unsampled location, the expected value is simulated according to a conditional gaussian distribution, taking into account all the values previously generated for all the points and using ordinary kriging. In practice, the simulation domain has too many target points and the procedure cannot be applied directly. Therefore, the simulation requires a selection of the already generated values inside a neighbourhood, centred on the target grid node and whose extension is defined by its radius in grid meshes. Within this neighbourhood, the respective maximum counts for data and for already simulated grid nodes are previously specified. The search is performed by increasing the distance through the variogram function, which ensures that the data most closely correlated with the target point are taken first. Due to the assumption of gaussian spatial behaviour, the form of the cumulative probability distribution function of the variable is known and therefore the probability of obtaining any possible value can be computed. A random value is then drawn from the cumulative probability distribution function and the corresponding value of the variable is assumed as one reasonable simulated value at that point in space. The simulated value is added to the conditioning data set and the process is repeated at the next node along the random path through the grid. In this way, simulated values are conditioned both to the original (measured) data and to any nearby previously simulated values, which ensures spatial continuity is introduced into the simulation model as required by the variogram. The only difference among the several individual simulations is the random-number seed, which initiates the simulation process and affects the random drawing of a value from the conditional probability distribution function used to generate each simulated value. As a consequence, most likely, the same location will be assigned different values in different simulations, producing maps variable in their specific appearance. 169

5 Estimation or Stochastic Simulation in Soil Science? However, these alternative images of the reality are virtually identical in their statistical character, because they reproduce, by construction, the sample data values at the original data locations and their univariate statistics. In the conditional simulations, several values of the variable, also called realisations, can be generated for one grid node. The application of post-processing simulations allows performing statistical calculations on the results of simulations, by comparing a given pixel throughout the different realisations. Two main types of results can be produced: (i) the statistical maps, used to compute different statistical values, in particular mean value and standard deviation of the realisations of the target variable at each grid-node, and (ii) the iso-cutoff maps, representing the probability of the target variable to be above a given cutoff, and that is realised by counting the values (realisations) above a given cutoff for each grid node and then normalising such count to the total number of realisations. Comparison between the two methods In order to evaluate the theoretical performance of the two methods, we worked on a separate data set, including 132 samples, randomly drawn from the original data set and not used to compute the estimated or simulated values of the clay variable. To quantify map quality and to determine the experimental error, the differences between each data point and: (a) the kriged value, (b) the simulated values, corresponding to three individual simulations, and (c) the mean value of the simulations, were calculated at each validation location. Since the validation points did not always coincide with the predicted grid locations, the nearest grid node was compared with the data point, if it was less than 100 m away. The two methods were compared on the basis of 1) the pattern of the clay map they produce; 2) the prediction rate; 3) the error distribution, including mean error (ME) and root mean squared error (RMSE), and 4) the spatial distributions of the errors. The visual comparison of the clay maps gives a first indication of the ability of the methods to reproduce the expected spatial pattern. The prediction rate was computed as the proportions of validation points which were best predicted by the method in terms of minimum absolute difference between predicted value and observed value, which is a way to translate the visual impressions into synthetic objective values. The analysis of the error distributions allows us to discriminate the methods on the basis of their capacity to predict the clay content with accuracy and precision. Bias is measured by ME, whereas precision is defined by the mean squared error (MSE) that is the square of RMSE, therefore an accurate and precise map should have no bias and the lowest possible RMSE. Finally, the spatial distributions of the errors reveal the ability of each method to extract the complete spatial structure. The spatial distribution was assessed by the variogram of the errors. A nugget effect variogram with a small sill is a sign that the structural variability has been well reproduced. An indicator (N ) of the map quality in this sense is determined by the r 170

6 Soil Genesis, Classification and Cartography computation of the contribution (%) of the nugget effect to the total structure (sill), as: N = C / (C + C ) r Where: C is the nugget effect, C the partial sill and C + C the total sill. Completely non correlated error values will have a N equal to 1. r Results The histogram and the descriptive statistics of the 1,153 sample data of clay proportion (%) are reported in Fig. 1a and Table 1, respectively. The overall sample distribution looks slightly positively skewed and the raw data do not follow a normal distribution, as it also 2 2 results from test: experimental (40.12) greater than the tabled value (27.59) for p < Figure 1b shows the histogram of the transformed raw data, by using an expansion in terms of 30 Hermite polynomials, which looks symmetric with a minimum of 3.20 and a maximum of The correctness of the transformation is also checked from the basic statistics of the new gaussian variable with a mean value of and a variance of (a) (b) Figure 1. Histograms of the raw clay data (a) and the normal transformed data (b). Table 1. Basic statistics of the raw clay data and the normal transformed data. Variable Count Min Max Mean Std. dev Variance Skewness Kurtosis Clay (%) 1, Gaussian variable 1, * The experimental variograms for the two main directions of variation and their fitted models are shown in Figure 2 for (a) the original clay values and for (b) the normal transformed values. The directional variograms for both the variables show clear 171

7 Estimation or Stochastic Simulation in Soil Science? anisotropic features of zonal type. Two particular models for each variable were used, each consisting of the nugget effect and two anisotropic structures: (i) two spherical functions for the clay variable and (ii) a cubic function and a spherical one for the gaussian variable. The characteristic parameters of the two variogram models are reported in Table 2, from which it results that the ranges along the two main directions of variation are approximately equal for the two variables; only the nugget effects and the sills are significantly different. This means that the normal transformation applied to the original data has not altered the spatial character of the data. The clay proportion shows a clear anisotropic spatial distribution, characterised by greater continuity and larger variance along the NE direction. The results of cross-validation, shown in Table 3, were quite satisfactory for both the variables, especially for the normal transformed data, because the statistics used, i.e. mean of the raw estimation errors and variance of the standardised errors, were close to 0 and 1, respectively. (a) (b) Figure 2. Experimental directional variograms relative to the NE (D1) and NW (D2) directions, with the fitted models (M1, M2) for (a) the clay variable and (b) the gaussian variable. Table 2. Estimated parameters of the directional variogram models of the clay variable and the gaussian variable. Variable Nugget effect Clay Direction Range (km) Sill NE NW Gaussian variable NE NW

8 Soil Genesis, Classification and Cartography Table 3. Results of the cross-validation test performed for clay data and normal transformed data. Clay Gaussian variable Mean Variance Error Std. Error Error Std. Error Figure 3 shows the maps of clay proportion, produced by (a) kriging and by (b) stochastic simulation, using the same information (data and variogram model). Quite clearly, the kriging map looks too smooth as compared with the simulation map; the contours are weakly defined and only major structures are identified. On the contrary, the simulation map incorporates any information about the sample distribution and succeeds in identifying even the smallest structures. The above example gives a clear insight into the relationship between estimation and simulation, which can be presented as two optimisation problems differing in their optimisation criteria: minimisation of a local expected loss (error variance) for estimation and reproduction of global statistics (semivariogram, histogram) for simulation. The goal of the interpolation algorithm is to provide the best, hence unique, local estimate of the clay proportion, without any specific regard to the resulting spatial statistics of the estimates taken together. In simulation, the reproduction of global features and statistics prevails on local accuracy. From what the above, it follows that the statistics for kriged values and simulated values are quite different, as shown in Table 4. Table 4. Basic statistics of the kriged values and the simulated values. METHOD Count Minimum Maximum Mean Std. Dev. Kriging 30, Simulation 38, The standard deviation of ordinary kriging was about half the one of simulation and also smaller than the one of the sample data (Table 1). Furthermore, the smoothing effect of kriging is evident in the significant underestimation of the clay values. Therefore, simulation application would be preferred to kriging in soil science whenever one wants to preserve soil variation (Srivastava, 1996), e.g. in drawing the boundaries of very fine-textured zones which can be critical for flow performance in a reservoir or for slope stability in an open pit mine. 173

9 Estimation or Stochastic Simulation in Soil Science? (a) (b) Figure 3. (a) Kriging map and (b) simulation map of clay fraction. The second major aspect in which simulation differs from kriging is that the latter provides a representation of clay proportion where local accuracy prevails, whereas the former provides several alternative global representations aimed at reproducing the main patterns of spatial continuity. In Fig. 4, the same clay data were used to generate three stochastic images that look similar but not identical and their differences provide a measurement of spatial uncertainty. 174

10 Soil Genesis, Classification and Cartography Figure 4. Three samples of realisations of clay fraction obtained using the sequential gaussian simulation algorithm. 175

11 Estimation or Stochastic Simulation in Soil Science? Therefore, unless a gaussian model for errors is assumed, kriging produces only an incomplete measure of local accuracy, without any appreciation of joint uncertainty when several locations are considered together. On the contrary, simulations are designed specifically to provide such measures of uncertainty, which are precisely given by the differences between the alternative simulated values at each grid node. To assess such uncertainty, one hundred simulations of the clay values were generated on the same grid used for interpolation and then the mean and the standard deviation of the 100 realisations at each node were calculated, producing two statistical maps shown in Fig. 5a and 5b, respectively. (a) (b) Figure 5. (a) Mean map and (b) standard deviation map obtained through stochastic simulation. 176

12 Soil Genesis, Classification and Cartography The comparison of the mean map with the one obtained by kriging (Fig. 3a) confirms the differences between the two approaches. Though the smoothing effect, typical in kriging, shows to be reduced also in the mean map, here the spatial fluctuations are less emphasised than in the stochastic images (Fig. 4a, b, c). The standard deviation map represents a way of assessing uncertainty and depends on sample density and variogram model, as well as on the actual sample value. It must then be preferred to kriging variance whenever the original variable is non-gaussian. Another way to jointly process the simulated images is to produce the probability map by calculating the proportions of realisations exceeding a given threshold at each grid node. Such a map is shown in Fig. 6, where the grey-scale values represent the probability of exceeding the clay limit of 40%, deemed a critical value because more finely textured soils might affect soil solution flow and tillage unfavourably. An overall look at the five maps (Fig. 3a, 4a, 4b, 4c, 5a) gives us an idea of the ability of the methods to reproduce spatial pattern, as previously described. Figure 6. Probability map of exceeding the clay proportion threshold of 40%. However, the analysis of the error distribution and the comparison of the variograms of the errors deserve a deeper insight. Five types of maps were compared: the kriged map (KG), the three realisations of stochastic simulation (S1, S2, S3), and the mean map (SMEAN). Table 5 summarises the results of the prediction rate for each type. As expected, kriging produces the best results, with 24% of the validation points predicted with the smallest absolute error. Simulation application generally causes a drop in the prediction rate, even if with wide differentiation among the realisations. 177

13 Estimation or Stochastic Simulation in Soil Science? Table 5. Prediction rate. Maptype Prediction rate (%) KG 24 S1 21 S2 20 S3 16 SMEAN 19 Figure 7 gives the distributions of the errors and Table 6 summarises the ME and RMSE for the five types of maps. Not surprisingly, kriging reveals its capacity to produce an accurate local estimation of clay proportion, with the narrowest error distribution and the highest mode (Fig. 7a), which means the highest probability to produce errors close to zero and the smallest frequency of gross errors. Table 6. Values of ME and RSME computed for each map type. Maptype ME RMSE KG S S S SMEAN (a) (b) (c) (d) (e) Figure 7. Error distributions for the different map types: (a) KG, (b) S, (c) S, (d) S, and (e) SMEAN. 178

14 Soil Genesis, Classification and Cartography This is also translated into the RMSE value, which is the smallest among all the map types. The mean map shows a quite similar behaviour, with a smaller absolute value of ME but a slightly larger RMSE. The ME values for all map types are generally close to zero, with the exception of S3 which reveals an important bias. Moreover, quite unlike the kriging map, S3 map shows an error distribution (Fig. 7d) with the lowest mode and the widest range, as confirmed also by the high RMSE value. Once again, though a simulation algorithm aims at drawing realisations reflecting the same statistics modeled from the sample data, large differences are evident among the different stochastic images. The variograms of the errors for the five types of maps are shown in Fig. 8. A theoretical model with nugget effect and one spherical model was fitted to each experimental variogram, except for S1, which was pure nugget effect. The estimated model parameters are reported in Table 7, from which the relative contribution of the nugget effect to the total structure (N r) was calculated and reported in Table 8. S3 and partly S2 produce a clear spatial structure with a spherical behaviour, revealing spatial correlation in the errors. KG performs better than each individual realisation, except for S1, and than SMEAN, showing a smaller total sill and a greater contribution of the nugget effect in the variogram structure (Table 8); the simulation variance was generally twice the kriging variance (Table 7). Both Figure 8 and Table 8 show that the variogram structures of S1, KG and partly SMEAN are strongly influenced by the nugget effect, which means most of all the spatial information has been adequately used in the prediction algorithm. Table 7. Estimated parameters of the variogram models of errors for each map type. Maptype Nugget effect Range (km) Sill KG S S S SMEAN Table 8. Relative contribution of the nugget effect in the variogram structure (N ) of each map type. r Map type Nr KG 0.95 S S S SMEAN

15 Estimation or Stochastic Simulation in Soil Science? (a) (b) (c) (d) (e) Figure 8. Variograms of errors for (a) KG, (b) S, (c) S, (d) S, and (e) SMEAN

16 Soil Genesis, Classification and Cartography Discussion and Conclusions The main objective of this work was to evaluate a set of spatial approaches on the basis of their ability to predict clay fraction using a validation set of soil data. The best appropriate method would be the one which produces the least biased error distribution (smallest ME in absolute value), the most accurate predictions (smallest RMSE) and the least spatially correlated errors (variogram with only a nugget structure and the smallest sill). Additionally, it should also produce a clay map similar to reference maps (if available) or match the expert knowledge on local soil type. Considering all these criteria, the kriging method clearly stands out from the others, because it simultaneously shows an ME value not far from zero, the smallest RMSE and one of the least structured errors variograms. Therefore, compared with stochastic simulation, we can conclude that kriging produces the most accurate spatial prediction of the clay fraction. On the other hand, the estimated clay map looks too smooth and fails in identifying the smallest structures. An important crucial issue is the wide spectrum of variability observed among the different simulated realisations. At this point, an inevitable question arises: Interpolation or Simulation? Stochastic simulation was originally developed not with the aim of minimising but providing measures of spatial uncertainty. Indeed, stochastic simulation algorithms have proven to be much more versatile than traditional interpolation algorithms in reproducing the global spatial variability of data and in accounting for the presence of outliners. In as much as a simulated realisation honours the data deemed important, it can be used as an interpolated map and preferred to the kriged map in all those applications where reproduction of spatial features is more important than local accuracy. However, the significant differences among the alternative simulated images raise a crucial problem in soil science, because practitioners generally tend to retain only one realisation, i.e. to use stochastic simulation as an improved interpolation algorithm. In the worst case, the first realisation is retained without any post-processing aimed at reproducing the sample histogram, since simulation theory guarantees matching only on an average (expected value) over a large number of realisations. Another alternative could be to draw several realisations and retain only one of them, or very few, but the criteria to select it/them are many and are essentially based on subjective appreciation. Furthermore, the number of realisations to be drawn depends on how many images are sufficient to model the uncertainty of the investigated spatial process. In conclusion, we can say that estimation and simulation are two alternative approaches to spatial prediction with quite different objectives: kriging is aimed at producing the most locally accurate estimates; simulation at reproducing global spatial statistics and assessing spatial uncertainty. It is up to the soil scientist to decide which is the most appropriate to his case study. 181

17 Estimation or Stochastic Simulation in Soil Science? References Castrignanò A., Bragato G. and Prudenzano M., Valutazione probabilistica th dell'errore nel dichiarare contaminata una falda idrica. Proceedings of 5 ASITA National Conference, vol.1, pp Deutsch, C.V. and Journel A.G., GSLIB: Geostatistical software library and user's Guide. Oxford University Press, New York, NY. 340 pp. Goovaerts, P., 1997a. Geostatistics for Natural Resources Evaluation. Oxford University Press, New York, p.483. Goovaerts, P., 1997b. Kriging vs. stochastic simulation for risk analysis in soil contamination. In: Soares, A., Gòmez-Hernàndez, J., Froidevaux, R.,(Eds.), GeoENV I Geostatistics for Environmental Applications. Kluwer Academic Publishing, Dordrecht, pp Goovaerts, P., Accounting for estimation optimality criteria in simulated annealing. Math.Geol.30, Heuvelink, G.B.M., Error Propagation in Environmental Modelling with GIS. Research Monographs in GIS Series, London, 127pp. Mc Keague, J.A., Wang, C. and Topp, G.C., Estimating saturated hydraulic conductivity from soil morphology. Soil Sci. Soc. Am. J., 46, Olea, R.A. and Pawlowsky V., Compensating for estimation smoothing in kriging. Math.Geol. 28, Peterson, G.W., Cunningham, R.L. and Matelski, R.P., Moisture characteristics of Pensylvania soils: I. Moisture retention as related to texture. Soil Sci. Soc. Am. Proc., 32, Srivastava, M.R., An overview of stochastic spatial simulation. In: Mowrer, H.T., Czaplewsky, R.L., Hamre, R.H.(Eds.), Spatial Accuracy Assessment in Natural Resources and Environmental Sciences: Second International Symposium: U.S. Department of Agriculture, Forest Service, Fort Collins, pp.13-22, General Technical Report RM-GTR