Network requirements for sensor accuracy and precision: a case study to assess atmospheric variability in simple terrain

Transcription

1 INTERNATIONAL JOURNAL OF CLIMATOLOGY Int. J. Climatol. 28: (2008) Published online 7 August 2007 in Wiley InterScience ( Network requirements for sensor accuracy and precision: a case study to assess atmospheric variability in simple terrain Melissa J. Melvin, Arthur I. Zygielbaum, Denise Gutzmer, Scott Rentschler, Jeremy Bower and Kenneth G. Hubbard* School of Natural Resources, University of Nebraska-Lincoln, Lincoln, NE , USA ABSTRACT: This simulation-based study is an effort to develop design requirements for a hypothetical network of instruments to measure average atmospheric variability over a relatively simple terrain. A dataset of air and soil temperature, relative humidity, wind speed, and soil moisture from the vicinity of Lincoln, Nebraska, was used to simulate atmospheric daily measurements (max and min temperature as well as infra-red surface temperature (grass cover), relative humidity, wind speed and soil moisture) at various distances from the site. An analysis was performed to determine the instrumentation requirements needed to distinguish natural atmospheric variability from input instrumental error at the 0.05 significance level. Both accuracy and precision requirements were examined to determine the affect of systematic and random errors, respectively. It was found that sensor accuracy (small systematic bias relates to high accuracy) does not affect the ability to determine the atmospheric spatial variability. However, sensor precision (small random error equates to high precision) was found to affect the ability to determine the spatial variability of the atmospheric variable in question; commercial sensors are currently available that meet or exceed the precision requirements for all except one of the variables examined. Copyright 2007 Royal Meteorological Society KEY WORDS climate networks; atmospheric monitoring; network design; precision; accuracy; sensors Received 28 July 2006; Revised 18 April 2007; Accepted 21 April Introduction In the past, studies have focused on network design for a number of purposes. Accuracy of precipitation networks has been a focus (Huff, 1970; Silverman and Koshio Rogers, 1981; Krstanovic and Singh, 1992; Tsintikidis et al., 2002; Shepherd et al., 2004). Other studies have considered design of networks for specific purposes like studying urban effects on weather (Changnon, 1975), studying acid precipitation (Finkelstein, 1984), detecting fire danger (Fujioka, 1986), and examining forecast skill in light of initial data resolution (Warner et al., 1989). More recently, scientists have been interested in the design characteristicsrequired to study long-term climatic change (Janis et al., 2004; Vose and Menne, 2004; Vose, 2005). This study proposes a hypothetical network and examines the influence of sensor precision and accuracy on the ability to discover unambiguously the atmospheric spatial variability over a local, simple terrain flat terrain over a short distance with uniform surface cover. The following analyses were performed to determine the level of accuracy and precision required for specific instruments to meet the stated purpose of the network. In this paper, the term accuracy refers to the systematic error or bias * Correspondence to: Kenneth G. Hubbard, 711 Hardin Hall, University of Nebraska-Lincoln, Lincoln, NE , USA. khubbard@unl.edu of the sensor measurements and a low systematic error equates to a high accuracy. The term precision refers to the random errors associated with sensor measurements and low random errors equate to high precision. While the reported results fit only the stated purpose of the hypothetical network, the methods herein can be adapted to error analysis for other types of networks. A search of the literature indicates that our study is unique in the approach to separating the effects of precision and accuracy in network design. To characterize atmospheric variability over a local, simple terrain, a variogram is taken here to mean the relationship between the root mean square error (RMSE) for paired stations and the distance between the station pairs. The simulated variables reported in this article include maximum and minimum air and surface temperatures (grass cover), relative humidity, wind speed, and soil moisture (10 cm depth). Simulated sensors with specific systematic and random errors are used to determine to what degree such errors inhibit the ability to produce an accurate variogram. Precipitation is a necessary component of climate networks. However, after careful consideration it was decided not to include precipitation in this analysis. Preliminary work on the variogram for precipitation (Hubbard, 2001) shows evidence that most of the variogram rise occurs within the first few kilometers (below the scale represented here) separation between stations Copyright 2007 Royal Meteorological Society

2 268 M. J. MELVIN ET AL. and therefore we do not have a good basis for inclusion of precipitation in this study. The objectives of this exercise are to (1) determine atmospheric spatial variability using RMSE variograms for five specific variables, (2) establish the minimum sensor performance characteristics required to effectively measure atmospheric spatial variability, and (3) determine if sensors exist which meet or exceed the necessary performance characteristics. 2. Data and methods The maximum and minimum air temperature (T max and T min ), relative humidity, wind speed, and soil moisture data are from the Mead Turf Farm, near Lincoln, NE for the period 1 January 2001 through 13 November Maximum and minimum surface temperature data from an infrared temperature sensor (IRT max and IRT min ) was taken at the climate reference network (CRN) station Lincoln 8 ENE (Prairie Pines) near Lincoln, Nebraska, from 15 January 2002 to 29 November Measurements were based on a 24-h period ending at midnight. These observations will be referred to as the observed data. This observed data (X i0,where0 denotes the original station and i denotes day number from the beginning time) were used to generate a set of hypothetical station data (X ij ) free of measurement error. X ij = X i,j 1 + δ ij F j. (1) X ij represents the data value for the j th simulated station at time i, δ ij is a random number between 1 and +1, and F j is the spatial variability factor. For this analysis we assume there is a statistically meaningful relationship between the distance separating station X ij and X i,j 1 and the magnitude of F. This assumption is supported by previous studies using automated weather data network data (Hubbard, 2001). The variogram rise from these studies was used as a guide to creating the true variograms in this study. A Monte Carlo simulation was conducted to represent six stations separated by 10 km along a 50-km transect. Using trial and error, that value of the spatial variability factor, F, which resulted in a variogram rise for the data (X ij ) similar to that found by Hubbard (2001). This method of generating data ensures that the correlation decreases in a realistic manner as the distance between the simulated stations increases. Once F was determined, the resulting variogram was assumed to be the true variogram and was used for comparison with variograms resulting from the simulations below. Systematic and random measurement or instrument error was introduced to create simulated measurements using Equation (2): X s ij = X ij + j + δ ij G j. (2) Where j represents a user selectable systematic error and G j represents the random error scaling or precision factor for the j th station. The goal, in terms of these data, is to find the maximum instrument measurement error that can be tolerated before the simulated measurement variogram (for X s ij ) can be distinguished from the true variogram. Statistical comparison of the two variograms was conducted over a range of measurement errors to find the point at which the two variograms could be distinguished, one from the other, as described below. A variogram indicates the variability between data taken at a point and the data taken at various distances from the point. The indicator of variability here is the RMSE between the variable measured at a base location and the variable measured at a distance from the base location. The separation distance is plotted on the x-axis In our simulation, data taken daily over 1413 days at a base location, were used in 100 simulation runs. For each simulation, RMSEs between the base station data and the simulated measurements at stations 10, 20, 30, 40, and 50 km distances were determined. An average of the 100 resulting RMSE sets provided the five RMSE values plotted on the variogram. Averaging was used to reduce the uncertainty in estimating the RMSE values. Figure 1 shows one such set of variograms. The data points are best fit with a power curve. In order to determine whether the slope of the two curves are significantly different from each other statistically, the traces were made linear by taking the log of the RMSE points. The resulting data, shown in Figure 2, indicates the quality of the linearization. Figure 1. The true variogram and a simulated variogram for relative humidity with G = 2%. This figure is available in colour online at

3 NETWORK REQUIREMENTS FOR SENSOR ACCURACY AND PRECISION 269 Figure 2. Linearized variograms for relative humidity. This figure is available in colour online at To determine whether the two lines are statistically different, an analysis of covariance (ANCOVA) was performed. The resulting F -statistic and associated P - value were adjusted by varying the error scaling, j and G j. The specified probability of incorrectly finding the two variograms indistinguishable, α, was established at A range of errors was examined by simulating variograms for various j and G j values. In each simulation, the P -value for the F -statistic representing the two variograms was examined. When the P -value was just above 0.05, the precision was noted and this was taken as the limiting precision for a potential sensor because a higher precision would mean the variograms would not be distinguishable and a lower precision would mean the variograms are distinguishable. The weather station network Monte Carlo simulation was implemented using a Microsoft Excel workbook augmented with visual basic for a-pplications (VBA) software. The Workbook includes three worksheets. The first sheet contains the observed data taken from the Mead Turf Farm and CRN sites described above. The two other sheets mechanise the simulation based on Gaussian distribution of random variables. The main worksheet contains parameter entry cells allowing the user to tailor the simulation (i.e. input values of F,, and G from Equations (1) and (2)). After selecting the data type (T max, T min, etc.), the user initiates a simulation. An intermediate spreadsheet is used to generate simulated atmospheric variables at each of the stations (Equation (1)). Random and systematic instrument errors are then imposed upon the simulated variables to generate the set of simulated measurements (Equation (2)). One hundred iterations of simulated instrument measurements are taken for each set of simulated data. Variograms are produced representing the atmospheric variable and instrument measurement variability over the six-station network. The ANCOVA is automatically computed upon completion of the simulation yielding the P -value needed to ascertain whether the two variograms are distinguishable. 3. Results The above methods were followed for each particular data type. In analysing the simulated data, it was discovered that there was no effect on the variogram when systematic error (bias) was introduced into the simulated data. The bias introduced did cause a correctable offset but this does not affect RMSE values and therefore does not impact the ability to distinguish atmospheric variability. Each data type will be discussed in turn. The accumulated set of data is summarized in Table I along with the variogram rise over 50 km. While Figure 1 showed the variograms obtained for Relative Humidity, Figure 3 contains examples for T max,irt max, wind speed, and soil moisture Maximum and minimum air temperature The statistical properties of the observed data are: an average T max (T min ) of 17.3 C (4.1 C) with a standard deviation of C (11.11 C). An analysis of Nebraska weather station data shows that the variogram rise in RMSE over 50 km for the simulated network should be about 0.4 C (0.3 C) for T max (T min ) (Table I). For both T max and T min, setting the spatial variability factor (F ) to 1.3 C produced the desired change in RMSE with distance. Various values of the spatial variability factor (F ) were used to determine at what point the true variogram was distinguishable from the new variogram at the α = 0.05 level. The RMSE versus distance shown in Figure 3 is an example of the variograms produced during the simulation. For T max (T min ), an instrument precision of 0.6 C (0.7 C) was determined to be adequate to distinguish atmospheric variability in the presence of instrument error (Table I). Several manufacturers currently design temperature sensors with precisions between 0.2 and 0.7 C, which meet the determined criteria. Some manufacturers use the terms accuracy and precision interchangeably, which is reflected in sensor specifications since only accuracy is typically reported. However, the authors believe that a manufacture s use of the word accuracy is actually describing precision, as defined in this paper Maximum and minimum surface temperature The observed values of IRT max (IRT min ) were noted to have an average value of 23.6 C (23.6 C) and a standard deviation of 13.8 C (10.6 C). Observed variogram data indicates that the rise of the RMSE in this simulation

4 270 M. J. MELVIN ET AL. Figure 3. Variograms for (a) maximum temperature (T max ), (b) maximum IR temperature (IRT max ), (c) wind speed, and (d) soil moisture. This figure is available in colour online at should be roughly 0.6 C (0.4 C) for IRT max (IRT min ) (Table I). Setting the spatial variability factor (F ) to 2.0 C (1.3 C) for IRT max (IRT min ) achieved the required variogram rise (Table I). To explicitly measure atmospheric variability, an instrument precision of 1.0 C (0.65 C) is necessary for IRT max (IRT min ). In a realistic setting, each network site would only be equipped with one instrument, such as an infrared thermometer (IRT), to measure IRT max and IRT min. Therefore, it is suggested that the instrument precision be 0.65 C or better. However, the best commercially available IRTs identified for this research only had precisions of 1.0 C Relative humidity The relative humidity (RH) for the observed data-set averaged 70.4% with a standard deviation of 13.6%. On the basis of the Nebraska weather network data, the

5 NETWORK REQUIREMENTS FOR SENSOR ACCURACY AND PRECISION 271 Table I. Summary of simulation results and variogram rise over 50 km based on Automated Weather Data Network information (Hubbard, 2001). Variable Variogram rise in RMSE change per 50 km Spatial variability factor (F - Equation (1)) Slopes P -value Precision Instrument availability T max (T min ) 0.4 C (0.3 C) 1.3 C (1.3 C) (0.050) 0.6 C (0.7 C) Yes (Yes) IRT max (IRT min ) 0.6 C (0.4 C) 2.0 C (1.3 C) (0.068) 1.0 C (0.65 C) Yes (No) RH 1.2% 4% % Yes Wind speed 0.08 m/s 0.26 m/s m/s Yes Soil moisture Yes variogram RMSE rise over 50 km is about 1.2% RH (Table I). Setting the spatial variability factor (F )to4% achieved a RMSE rise of approximately 1.2% (Table I). At the α = 0.05 level, the precision of the RH instrument must be 2.0% or better to unambiguously determine the spatial variability. While expensive, RH sensors capable of 1.5 2% accuracy over the 0 100% RH range are available. Therefore, a network of RH sensors capable of distinguishing atmospheric variability is feasible Wind speed The observed wind speed data has a mean of 3.2 m/s and a standard deviation of 1.3 m/s. The variogram RMSE should increase by 0.08 m/s over a 50-km distance, the rise seen in the Nebraska weather network data (Table I). Setting the spatial variability factor (F ) to 0.26 m/s produces the desired slope (Table I). A wind sensor precision of 0.14 m/s or better is required to ensure that true variogram and derived variogram are not significantly different at the α = 0.05 level. Instruments with 0.14 m/s, or better, precision are available commercially Soil moisture The observed volumetric soil moisture data measured at a depth of 10 cm yielded an average soil moisture content of 0.26, with a standard deviation of The desired variogram RMSE rise over 50 km should be approximately 0.014, which is achieved by setting the spatial variability factor (F )to0.06(tablei). The precision of the soil moisture sensor must be 0.03 for the variogram to be unambiguously determined at the α = 0.05 level. Instruments with the required precision are commercially available. 4. Summary and conclusions In this study, a simulated network of instrumentation was used to test for accuracy and precision requirements to measure atmospheric variability over local, simple terrain. Observations from a Nebraska site were used to develop a simulated dataset. Once created, varying degrees of random and systematic error were introduced. It was determined that introducing systematic error had no impact on the simulated variogram. Instrument precision was the most important factor in determining whether atmospheric variability can be reliably measured. While systematic error affected the accuracy at a site, the RMSE between sites is unchanged and for the stated purpose of the network, systematic errors are not problematic. Results from statistical investigation indicate that atmospheric variability can be distinguished from instrumental error with sufficiently precise instrumentation. A search of currently available sensors indicates that maximum and minimum air temperatures, maximum surface temperature, RH, wind speed, and soil moisture sensors are commercially available that meet or exceed the precision requirements found in this study. Furthermore, while this investigation is specific to a network with the goal of measuring atmospheric variability, the general approach in this study may be used to specify instrument requirements for other networks. The main steps are to clearly state the purpose of the network, define potential outcomes in a quantitative manner and select the appropriate statistical indices to characterize the minimum uncertainty in the network, simulate network data based on knowledge gained from existing networks, and simulate the influence of precision and accuracy on the potential outcome. Expansion of this work into complex terrain may require inclusion of the terrain effects on the correlation structure of the atmosphere. A network might require a different station density at higher and lower elevations within the complex terrain depending on what level of uncertainty (systematic or random) is acceptable to the purpose of the network. One element of future efforts is the quantification of the variogram for precipitation. This will require data from a high-density precipitation network from which the variogram rise can be determined. The simulation of neighboring stations would need to be undertaken at spatial scales down to 1 km or less. Acknowledgements The authors thank Dr Ann Parkhurst, professor in the statistics department at the University of Nebraska- Lincoln, for her statistics support and expertise during this research. References Changnon SA Operations of mesoscale networks, illustrated by Metromex. Bulletin of the American Meteorological Society 56(9):

6 272 M. J. MELVIN ET AL. Finkelstein PL The spatial analysis of acid precipitation data. Journal of Applied Meteorology 23(1): Fujioka FM A method for designing a fire weather network. Journal of Atmospheric and Oceanic Technology 3(3): Hubbard KG Station density and areal coverage of networks. In Automated Weather Stations for Applications in Agriculture and Water Resources Management, WMO/TD No World Meteorological Organization, University of Nebraska: Lincoln, NE; Huff FA Sampling errors in measurement of mean precipitation. Journal of Applied Meteorology 9(1): Janis MJ, Hubbard KG, Redmond KT Station density strategy for monitoring long-term climatic change in the contiguous United States. Journal of Climate 17(1): Krstanovic PF, Singh VP Evaluation of rainfall networks using entropy: I. Theoretical development. Water Resources Management 6(4): Shepherd JM, Taylor OO, Garza C A dynamic GIS-Multicriteria technique for siting the NASA-Clark Atlanta urban rain gauge network. Journal of Atmospheric and Oceanic Technology 21(9): Silverman BA, Koshio Rogers L On the sampling variance of raingauge networks. Journal of Applied Meteorology 20(12): Tsintikidis D, Georgakakos KP, Sperfslage JA, Smith DE, Carpenter TM Precipitation uncertainty and raingague network design within Folsom Lake Watershed. Journal of Hydrologic Engineering 7(2): Vose RS Reference station networks for monitoring climatic change in the conterminous United States. Journal of Climate 18(24): Vose RS, Menne MJ A method to determine station density requirements for climate observing networks. Journal of Climate 17(15): Warner TT, Key LE, Lario AM Sensitivity of mesoscale-model forecast skill to some initial-data characteristics, data density, data position, analysis procedure and measurement error. Monthly Weather Review 117(6):