An evaluation of three diagnostic wind models (CALMET, MCSCIPUF, and SWIFT) with wind data from the Dipole Pride 26 field experiments

Transcription

1 Meteorol. Appl. 12, (2005) doi: /s An evaluation of three diagnostic wind models (CALMET, MCSCIPUF, and SWIFT) with wind data from the Dipole Pride 26 field experiments Robert M. Cox 1, John Sontowski 2 & Catherine M. Dougherty 2 1 Science Applications International Corporation, Science Drive, Orlando, FL 32826, USA 2 Science Applications International Corporation, Suite 400, 1150 First Avenue, King of Prussia, PA 19408, USA This paper describes the evaluation of three diagnostic wind models by direct comparison with wind field data. The models are the California Meteorological Model (CALMET), the Mass Consistent model (MCSCIPUF) associated with the Second Order Closure Integrated Puff (SCIPUFF) transport/dispersion model, and the Stationary Wind Field and Turbulence (SWIFT) model. The evaluation follows previous works by Chang, Franzese & Hanna, who compared the same three models, and by Bradley & Mazzola who evaluated SWIFT coupled with SCIPUFF. As with SWIFT, MCSCIPUF is incorporated in the Hazard Prediction and Assessment Capability (HPAC), while CALMET is linked with the California Puff model (CALPUFF), another transport and dispersion model. The Dipole Pride 26 (DP26) experiments, performed at the US Department of Energy (DOE) Nevada Test Site, are used as the source of the wind data. They provide a comprehensive set of meteorological data with wide-ranging atmospheric stability conditions over a complex terrain. Model calculations were compared with measured data in two phases. The first phase uses complete sets of data from eight locations (the 8M phase) as model inputs, and thus tests the ability of the models to reproduce input conditions. In the second phase, five of the measured wind sites are withheld from input, and instead used for validation of model calculations (the 3M phase). In the first phase, the errors were found (with some exceptions) to be quite small. In the second phase, mean absolute errors were found to be of the order of 1 ms 1 and 30, with only small differences among models in terms of performance. 1. Introduction In recent years the requirements for modelling atmospheric transport and diffusion of hazardous materials have become much more demanding and complex (Sontowski et al. 1995, Cox et al and Cox et al. 2000). The ability to model three-dimensional (3D) wind fields consistent with complex terrain is now generally considered an essential element of most mesoscale or regional transport/diffusion models. Further, since the release of hazardous materials often involves emergency situations, rapid response and therefore high-speed computation is also required. An ideal methodology for providing timely 3D wind fields that are consistent with complex terrain is the diagnostic or mass consistent modelling approach. This approach constructs wind fields fitted to available meteorological data and terrain, while also being constrained to satisfy the governing equation for conservation of mass. The work presented here deals with the evaluation of selected mass consistent models for suitability of use within a fully comprehensive hazardous material transport and diffusion capability. The U.S. Defense Threat Reduction Agency (DTRA) has developed such a capability called the Hazard Prediction and Assessment Capability (HPAC). It is fully described in The HPAC User s Guide (1999) and Byers et al. (1995). HPAC provides source characterisation, including a wide range of source materials and release scenarios, extensive databases and processing for meteorological, terrain, and land use data, wind field and transport and diffusion models, as well as population effects models. Applications can involve a wide variety of meteorological conditions, data types and completeness, spatial distributions and timing options. Within this context, the incorporated wind field model must be highly efficient computationally, and flexible in terms of input data acceptance and processing. Of paramount importance, 329

2 Robert M. Cox, John Sontowski & Catherine M. Dougherty Table 1. Overview of CALMET, MCSCIPUF and SWIFT model configurations. CALMET MCSCIPUF SWIFT Terrain database USGS DTED DTED Land use database USGS DTED DTED Grid system UTM UTM UTM Horizontal resolution 1 km 1 km 1 km Vertical coordinate system Z = z z g z = D z h z = H z z g H z g = Hs D h = Ds Vertical levels Height of first layer 10 m exactly 50 m 10 m nominally Vertical domain Height 2650 m 10,927 m 6200 m Slop Flow Yes No No Similarity theory for surface wind Yes Yes Yes Stability Effect based on input Effect based on input Effect based on input temperature input temperature input temperature input Total number of surface stations available (8M cases used all 8; 3M cases used only 3) Total number of upper-air stations available however, is the ability of the wind model to provide sufficiently accurate wind field constructions. Therefore, the key objective of this study was the evaluation of specific mass consistent wind field models against data collected from a DTRA field experiment. Three models (CALMET, MCSCIPUF, and SWIFT) have been selected for the evaluation. The evaluation follows previous works by Chang et al. (1999), who also compared three models (CALPUFF, HPAC and VLSTRAC), and by Bradley & Mazzola (1999), who evaluated SWIFT/SCIPUFF. Both of these studies relied on the use of field data from the DTRA Dipole Pride 26 (DP26) experiments. Emphasis in these studies was on determining the ability of the models to provide accurate concentration/dosage fields as a result of the complete process of material release, transport by wind and turbulent diffusion in the atmosphere. By contrast, the emphasis here is strictly on evaluating a single element of that process, i.e. the ability of the models to construct valid wind fields. The use of the DP26 data was motivated first by the nature and the quality of the data, and also by a desire to build upon the previously cited studies. The DP26 experiments were conducted over a complex terrain in Nevada, and provide a complete and detailed set of data, including material source specifications, meteorological observations and concentration/dosage measurements. These were collected with a relatively dense network of concentration sensors and meteorological stations, making the data well suited for validation of the models in this study. Details of the experimental data, particularly the meteorological data that were used in the current work, will be discussed more fully below, following a brief section describing the mass consistent 330 models being evaluated. Subsequent sections will describe the procedures for evaluation of the models, results of the model evaluations and the conclusions. 2. Mass consistent model descriptions As noted above, three mass consistent wind models (CALMET, MCSCIPUF and SWIFT), appropriate for complex terrain, were selected for evaluation. Table 1 provides a brief overview of some of the model configuration and model implementation information. If an item is not listed, the model default was used. The defaults are found in the technical notes or papers listed in more detailed model descriptions below CALMET CALMET is a diagnostic meteorological model that generates mass consistent wind fields over complex terrain. It was originally developed for the California Air Resources Board and later enhanced for the US Environmental Protection Agency (EPA). Details of the CALMET modelling approach are provided by Scire et al. (1998). In addition to a divergence minimisation procedure for satisfying mass consistency, CALMET includes parameterisations for slope flows, kinematic terrain effects and terrain blocking effects. It also includes a micrometeorological model distinguishing over-land and over-water boundary layers. The CALMET wind fields are generated to fit hourly meteorological observations and/or use gridded wind data from prognostic meteorological models such as the Penn State/NCAR Mesoscale Model with four-dimensional data

3 An evaluation of three diagnostic wind models assimilation. Operation of the model involves a twostep approach. In the first step, the parameterisations for slope flow, terrain blocking and kinematic effects of terrain are used to adjust an initial-guess wind field. The step 1 procedure includes the imposition of mass consistency in conjunction with the procedure for kinematic effects of terrain. The CALMET mass consistent requirement is imposed by adjusting the two-dimensional/horizontal components of wind, with the vertical component being held fixed. The step 1 wind field is modified in the second step by an objective analysis procedure using the input observational or the gridded prognostic wind field data. In this step, the step 1 wind field is adjusted towards the input wind values by using an inverse distance squared interpolation procedure to weight the adjustment most heavily in the neighbourhood of the input data locations. Mass consistency is again imposed as part of step 2. Options exist within CALMET which allow the step 1 procedure to be bypassed. This allows the final wind field of CALMET to be based solely on the objective analysis of step 2, with the possible use of an externally generated wind field, e.g. a prognostic model output, as the initial step 2 field. Operationally, the CALMET model is structured to interface with the CALPUFF non-steady state Gaussian puff transport and dispersion model (Scire 1998). It also includes preprocessors for raw meteorological data that are designed to accept the US National Climatic Data Center (NCDC) file formats. Alternatively, meteorological data can be entered in user-prepared free formatted files. Terrain elevation and land use categories are entered as gridded inputs. Related geophysical parameters, e.g. surface roughness and albedo, are parameterised in terms of land use category, or can be input independently. Categories and parameterised values are based on US Geological Survey land use classification systems. Horizontal gridding for geophysical inputs as well as for computation by CALMET includes the Universal Transverse Mercator (UTM) grid commonly used in meteorological modelling. Since the other models being tested also use UTM coordinates, a common UTM set of coordinates can be set up for evaluation of the models. However, because each of the models has its own vertical coordinate system, and because this affects evaluation of the differences between calculated and measured winds, a brief outline of the vertical grid is included with the description of each model. The CALMET vertical grid is a terrainfollowing system defined by the following equation: Z = z z g (1) Each grid level is thereby represented by a constant value of Z and maintains a constant height above the average ground elevation (z g ) over the grid square. This is most convenient in comparing calculations to surface data, as the first CALMET grid level and surface measurement heights are both at 10 m above ground. The top grid level for the CALMET computational domain used here sets the vertical extent to 2.65 km above ground MCSCIPUF MCSCIPUF is a mass consistent wind model developed specifically for application, and coupling, with the Second Order Closure Integrated Puff (SCIPUFF) transport and diffusion puff model (Sykes et al. 1984, 1986, 1988, 1993, 1998). In this form MCSCIPUF is used as an integral part of the HPAC. The mathematical basis of MCSCIPUF, in terms of governing equations, is basically the same as that of the SWIFT mass consistent model, which is briefly described in the following subsection. Differences in the models exist primarily in the areas of interpolation procedures for wind field initialisation, and the numerical solution schemes. Regarding the coordinate system, MCSCIPUF uses a terrain-following system that is also very much like that of SWIFT, differing only by a constant factor in the vertical coordinate. The MCSCIPUF vertical coordinate (z ) is defined as: z = D z h = Ds (2) D h where z, h, and D represent heights of the grid point of interest, the local terrain and the top of the computational domain, respectively. All of the heights are referenced to the minimum terrain height within the computational domain. As the top of the domain is at constant height, it is noted that the factor D is a constant. On the other hand, s is a variable ranging from 0 at the ground to 1 at the top of the domain, independently of horizontal location. Grid levels are thus defined by constant values of the variable s, or equivalently z. For the computations here, the height D is 10 km, with the terrain elevations ranging from 927 to 2,236 m. The MCSCIPUF domain therefore extends to an altitude of km. The first grid level in MCSCIPUF is selected to be nominally, or in an average sense, at the typical surface measurement height of 50 m SWIFT The SWIFT model is a variational mass consistent wind field model, and was derived from the MINERVE wind model that was originally developed in the 1980s by the French Electricity Board (Geai 1987). Like the other models, SWIFT is formulated to fit weather data, from either observations or gridded model outputs, in a complex terrain environment, while also satisfying the principle of conservation of mass. The observed wind data are fit in a least squares sense, using the variational methodology. The data can include any number of surface stations and upper air profiles. An initial gridded wind field is constructed from the observation data by interpolation. Various interpolation 331

4 Robert M. Cox, John Sontowski & Catherine M. Dougherty procedures are included as options that can be selected based on the nature/distribution of the input data points. Adjustments are then made to the initial 3D interpolated wind field vectors so as to satisfy conservation of mass in a way that also minimises an integral function of the differences between the initial and adjusted fields. To account for the effects of atmospheric stability, the relative amount of adjustment to vertical and horizontal wind components is controlled by specification of an adjustment coefficient, α (the Gauss precision modulus). The adjustment coefficient can be spatially uniform or a three-dimensional function of position; it can be user specified or internally calculated in terms of stability-related factors such as input temperature profiles or Pasquill-Gifford stability class. Although SWIFT and MCSCIPUF incorporate stability effects by the same mathematical formulation, parameterisations of the adjustment coefficient, α, are different. Like CALMET and MCSCIPUF, the SWIFT/ MINERVE equations are also formulated in terms of a terrain following coordinate system. The vertical coordinate (z ) in the SWIFT system is defined as: 332 z = H z z g = Hs (3) H z g where z g represents ground elevation and H is the altitude of the top of the computational domain and s is the relative height above the terrain. Whereas the MCSCIPUF heights were all referenced to the minimum terrain height, h, the SWIFT heights, z, z g and H are referenced to sea level. As a result, the MCSCIPUF and SWIFT vertical coordinates differ from each other by the constant multiplicative factor H/D. The vertical grid points are non-uniformly distributed, ranging from 0 to 1 in s, so as to provide enhanced resolution in lower boundary layer regions where profiles change most rapidly, and reduced resolution in upper regions where change is generally more moderate. Various functional forms are provided as options for the distribution. With each grid level represented by a constant value of s, or z, it can be seen that height above ground of any grid level will vary with horizontal location depending on the variation of ground elevation. This is intended to provide a smooth transition from the variable ground height to a constant, horizontal surface at altitude H for the top of the computational domain. This is in contrast to CALMET, where each grid level, including the top, is at a constant height above ground. The first grid level in SWIFT is selected to be nominally, or in an average sense, at the typical surface measurement height of 10 m. As can be seen from the coordinate definition, the height above ground, (z z g ), for any grid level, will be greatest in the lowest valley, and least above the highest peak. For the grid used, and the evaluation performed here, the first SWIFT grid level at the surface measurement stations varied from 10.6 to 11.5 m above ground. Thus, all grid heights were within 1.5 m of the measurement height, and therefore required minimal adjustment or interpolation in the vertical direction for comparison with the data. The constant altitude H for the top of the SWIFT domain was at 6.2 km. 3. Data selection A key element of the Dipole Pride 26 (DP26) experiments was the collection of a comprehensive set of meteorological data, with a primary objective of providing data relevant to transport and diffusion of hazardous releases into the atmosphere. This data set, with wind observations collected at multiple times from a welldistributed network of stations, serves as an excellent database for the evaluation of wind models in a complex terrain environment. An overview of these data is presented here. Biltoft (1998) provides a complete and detailed description of the experiments and resulting data. Other descriptions are given by Bradley et al. (1999) and by Chang et al. (1999). The DP26 field experiments were conducted during November 1996 at the Yucca Flat range (37 N, 116 W), a basin measuring 30 km north to south and12 km west to east. The basin is approximately 1200 m above mean sea level (MSL), with surrounding mountains ranging from 1500 to 1800 m above MSL (Figure 1) Description of meteorological data The data for the DP26 tests were collected from a continuously operating network of surface MEteorological DAta (MEDA) stations along with a variety of intermittent upper air measurement systems at several sites. Data collected at the MEDA stations include temperature, pressure and humidity measured at a height of 2 m above ground, plus wind speed and direction measured at a height of 10 m. Figure 1 shows the eight stations operating in the test area. Measurement frequency at MEDA stations was every 15 minutes, with wind speed and direction being averaged over 5-minute periods ending at the reported time. Upper air wind speeds and directions were sampled with a radiosonde system and with pilot balloons (PIBALS). The location of the radiosonde profiles, as indicated by the launch site of the balloon-borne instrumentation, was within 100 m of the M6 station, and was called YFW. PIBALS on the other hand were launched hourly at a site called UCC within 100 m of M6, and at a site 1 km north of M16 called BJY. Radiosonde and PIBAL measurements near station M6 were not concurrently taken. A third site was occasionally used. It is identified as CSE and was located approximately 6 km southeast of the M28 MEDA station. Completing the upper air measurements, sonic anemometer/thermometer systems were employed near the BJY and YFW sites to measure mean and fluctuating wind components, and thereby determine turbulence intensities and fluxes (heat and momentum). The

5 An evaluation of three diagnostic wind models Figure 1. Terrain and MEDA surface meteorological stations associated with DTRA Dipole Pride 26 experiments at Nevada Test Site/Yucca Flat. MEDA sites are indicated by a square with the S. The site number is given next to the square. The coordinate system is UTM and is in zone 11 (WGS 84). winds from the lowest level of the sounding were found to be consistent with those of the surrounding surface stations. The vertical levels were represented by interpolation between the observed value and the desired level. It was decided to study those times that were used in previous studies (Chang et al and Bradley et al. 1999) to build upon the previous work. It was necessary to limit model inputs for upper air to radiosondes because CALMET cannot operate without temperatures and PIBAL data do not include temperature profiles. Thus, 12 time periods, corresponding to the times when radiosonde data were available during the month of November, were selected. The times were primarily in the early morning or mid afternoon. It was assumed that the radiosonde data are representative of upper air conditions for at least a two-hour period including the hour before and the hour after the designated time of measurement. This allows for 36 five-minute time periods (MEDA averaging time) for model evaluation. Sensitivities to the time and amount of observations (both surface and upper air) are discussed by Cox et al. (2000) Quality assurance of meteorological data To assess the quality of the meteorological data, many plots were generated and examined for all of the data considered for potential use. These included plots of wind speed profiles for the radiosonde, and horizontal wind vectors for the MEDA surface data. The radiosonde profiles were plotted along with the PIBAL profiles that were closest to the radiosondes in time and location. The lower levels of the profile, i.e. the first few hundred metres above ground were of particular concern in evaluating the radiosonde data. Wind speed measurements at these levels have been found to be questionable by Chang et al. (1999), who mentioned occasional erratic movements of the balloon during its initial ascent. In screening the profile data, it was decided that all radiosonde data below 100 m above ground should be rejected. This judgement is qualitatively consistent with the work of Chang et al. (1999), where all data below 500 m were rejected. For one time, on 14 November, it was our judgement that radiosonde winds clearly were unrealistically high and were rejected up to 500 m. No MEDA surface wind measurements were found to be obviously erroneous. 333

6 Robert M. Cox, John Sontowski & Catherine M. Dougherty Table 2a. SWIFT predictions and observations for all 35 time periods for the 3M study at MEDA station 1. Sp and So are the wind speed predicted and observed, Dp and Do are the wind direction predicted and observed, ERR is the predicted minus observed error, and ABS ERR is the absolute value of the ERR. Wind speed (ms 1 ) Wind direction (deg) Time (PST) MMDDHH Sp So ERR ABS ERR Dp Do ABS ERR The final total of 35 hourly time periods was selected because at 1600 UTC on 14 November, MEDA 1 was missing a wind measurement. 4. Evaluation procedures For each of the three models, two sets of evaluations were run for each of the 35 hourly time periods. The first set of evaluations included surface measurement inputs from all eight MEDA stations, and was intended to test the model s ability to produce a wind field consistent with the inputs. These sets of evaluations are referred to as the 8M evaluations. In the second set of evaluations, data from five surface stations were withheld from model simulation and used instead for validation of model produced wind fields. The three stations retained for model input were MEDA stations 2, 3, and 28. These sets of evaluations are referred to as the 3M 334 evaluations. Both sets of evaluations used the same radiosonde profiles. The calculations were performed with MCSCIPUF and SWIFT as part of HPAC. CALMET, on the other hand, was run in a stand-alone mode. As none of the codes provides grid points coincident with the measurement locations, computer programs were written to find grid points nearest to measurement stations, to perform horizontal interpolation (bilinear), and to carry out a vertical adjustment according to surface layer similarity theory as employed internally by the individual models. In the case of CALMET, the vertical adjustment was not required since the first vertical grid level is at the measurement height of 10 m. SWIFT on the other hand required a minimal vertical adjustment, while MCSCIPUF, with a nominal first grid level height of 50 m, required the most adjustment.

7 An evaluation of three diagnostic wind models Table 2b. SWIFT model statistics for 3M study at MEDA station 1. Sp and So are the wind speed predicted and observed, Dp and Do are the wind direction predicted and observed, ERR is the predicted minus observed error, and ABS ERR is the absolute value of the ERR. N is the number of cases, N < 0 is the number of cases less than zero, Q1 and Q3 are the first quartile and third quartile value, IQR is the interquartile range, RANGE is the total range, STDEV is the standard deviation, RMSE is the root mean square error, NMSE is the normalised mean square error, FRAC BIAS is the fractional bias, CORR is the correlation coefficient, N < or = 1,20 is the number of cases less than or equal to 1 ms 1 or 20 o,and% < or = 1,20 is the percentage of cases less than or equal to 1 ms 1 or 20 o. Wind speed (ms 1 ) Wind direction (deg) Stats. Sp So ERR ABS ERR Dp Do ABS ERR N N < MIN ERR Q MEDIAN Q MAX ERR RANGE IQR MEAN STDEV RMSE NMSE FRAC BIAS CORR N < or = 1, % < or = 1, The computational domain is in UTM zone 11, with coordinates for the southwest corner at km (easting), km (northing), and a grid extending with equal east west and south north grid spacing of km. Digital Terrain Elevation Data (DTED) from the US National Geospatial-Intelligence Agency (NGA) were used for MCSCIPUF and SWIFT. CALMET was developed for use with the US Geological Survey (USGS) terrain elevation data. Land use categories and associated surface parameters were obtained from a different database for CALMET (USGS), and for MCSCIPUF and SWIFT (ORNL/Oak Ridge National Laboratory). The reason for the different databases for the models is that CALMET was developed for a civil agency (EPA), whereas, MCSCIPUF and SWIFT were developed for US Department of Defense uses. The databases used by the models are the ones that are used for those applications. Tables 2a and 2b are examples of the computational results and statistical evaluations that were carried out. The models were run with results reported hourly using 5-minute average observational data valid at either the hour or half-hour. The SWIFT model using three MEDA (3M) stations as inputs is used in this example. Table 2a shows the results of SWIFT calculations at MEDA station 1 compared with the observations for the 35 time periods. Calculated, or predicted, wind speed and direction are listed as S p and D p respectively, with the observed speed and direction listed as S o and D o.the listed wind speed errors ( ERR ) are predicted minus observed values, so that positive values represent overprediction. ABS ERR represents the absolute values of the errors. Given the corresponding predicted and observed wind values, selected statistical performance measures were calculated as shown in Table 2b. The first entry in the table (N) indicates the number of members in the statistical sample. Statistical distributions of the errors and absolute errors are quantified by the minimum error, first quartile value (Q1), median, third quartile (Q3), and the maximum error. The total range (RANGE) and the interquartile range (IQR), as well as standard deviation (STDEV) characterise the spreads in the distributions. Mean values (MEAN) are included for errors as well as for predicted and observed winds. Also included is the root mean square error (RMSE). Following Hanna et al. (1993), the fractional bias (FB) and the normalised mean square error (NMSE) were calculated. FB = ( S o S p )/(0.5( S o + S p )) (4) NMSE = (S o S p ) 2 /( S o Sp ) (5) The correlation coefficient (CORR) between calculated and observed wind speeds was calculated as: CORR = (S p S p )(S o S o )/(σ(s p )σ(s o )) (6) where σ indicates the standard deviation. Finally, the number and percentage of the sample cases for 335

8 Robert M. Cox, John Sontowski & Catherine M. Dougherty which absolute errors were below specified threshold values were determined. For the sample considered in Table 2b, the absolute error threshold is 1 ms 1 for wind speed and 20 for wind direction. Comparisons between calculations and observations, as displayed in Tables 2a and 2b, were made on a stationby-station basis for each model. In addition, the same comparisons and statistical evaluations were made on a collective basis over all station and hours. For the 3M sets of calculations, this resulted in a combined sample size for each model of 175 trials, (35 time periods 5 MEDA stations), while the 8M calculations provided samples of 280 trials (35 time periods 8 MEDA stations). 5. Model evaluation results 5.1. Ability of models to preserve input winds (no data withheld) The 8M evaluation was intended to measure the models faithfulness to the input data. This fitting of input data is a primary feature of mass consistent models (Finardi et al., Ratto et al. 1994). Considering wind speed, all three models tend to underestimate speeds at the measurement/input locations and produced larger errors for stable conditions than for non-stable. Of the three models, CALMET underestimated most frequently (76% of the time), while MCSCIPUF and SWIFT both underestimated 66% of the time. The mean values of the absolute errors were found to be 0.26 ms 1,1.01ms 1 and 0.30 ms 1 for CALMET, MCSCIPUF and SWIFT, respectively. As shown, CALMET and SWIFT were found to perform extremely well with nearly equal performance measures. Both models showed mean absolute errors of less than 1 ms 1 nearly 100% of the time, and correlation coefficients of 0.98 with absolute errors that are less than 10% of the mean observed wind speed of 3.93 ms 1.The MCSCIPUF absolute errors are about 25% of the mean observed wind speed. Examination of the number and percentage of times for which calculation errors were less than 1 ms 1 further demonstrate this difference in the models. While CALMET and SWIFT were below this threshold nearly all of the time (277 and 273 of 280 samples for CALMET and SWIFT, respectively), MCSCIPUF was below the threshold only 60% of the time. This relative difference by MCSCIPUF is similarly reflected in the wind direction statistics. Wind direction errors were less than 10 87% of the time for CALMET, and 83% of the time for SWIFT. The MCSCIPUF absolute errors were larger than those of CALMET and SWIFT. While the MCSCIPUF median and mean errors were approximately three times larger and 50% larger, respectively, than those of CALMET and SWIFT, they were comparable or smaller than the median and mean errors reported by Ross et al. (1988). Examination 336 MEAN ERRORS - (ms -1 ) WIND SPEED - MEAN ABSOLUTE ERROR (AT DATA WITHHELD DP26 MEDA STATIONS) ALL STABLE NONSTABLE TIMES CALMET MCSCIPUF SWIFT Figure 2. Mean values of absolute errors in wind speed at datawithheld stations (MEDA 1, 6, 9, 10, 16) for the three models and the three stability groups. of errors at the initialisation and interpolation steps of the MCSCIPUF and SWIFT modelling procedures reveals that the MCSCIPUF error might be reduced by modification of the interpolation scheme. This possible reason for these differences will be discussed further in the following sections Model validations at five stations with data withheld The results of the 3M calculations are examined here. The same statistics were gathered here as above, however, only tabular results for CALMET (Table 3) will be shown as an example. The other models (MCSCIPUF and SWIFT) results were almost identical to CALMET. Similar to the results at input stations (8M evaluations), wind speeds at the five non-input/validation stations are underestimated, with all models showing negative median and mean errors of about 10% of observed mean wind speeds. Consistently, all models yield fractional biases of 0.1. Mean values of absolute errors are displayed in the form of bar charts in Figures 2 and 3 for wind speed and MEAN ERRORS - (DEG) WIND DIRECTION - MEAN ABSOLUTE ERROR (AT DATA WITHHELD DP26 MEDA STATIONS) ALL STABLE NONSTABLE TIMES CALMET MCSCIPUF SWIFT Figure 3. Mean values of wind direction absolute errors at data-withheld stations (MEDA 1, 6, 9, 10, 16) for the three models and for the three stability groupings.

9 An evaluation of three diagnostic wind models Table 3. CALMET model statistics for five non-input MEDA stations for 3M runs. Sp and So are the wind speed predicted and observed, Dp and Do are the wind direction predicted and observed, ERR is the predicted minus observed error, and ABS ERR is the absolute value of the ERR. N is the number of cases, N < 0 is the number of cases less than zero, Q1 and Q3 are the first quartile and third quartile value, IQR is the interquartile range, RANGE is the total range, STDEV is the standard deviation, RMSE is the root mean square error, NMSE is the normalised mean square error, FRAC BIAS is the fractional bias, CORR is the correlation coefficient, N < or = 1,20 is the number of cases less than or equal to 1 ms 1 or 20 o,and% < or = 1,20 is the percentage of cases less than or equal to 1 ms 1 or 20 o. Wind speed (ms 1 ) Wind direction (deg) Stats. (CALMET) Sp So ERR ABS ERR Dp Do ABS ERR N N < MIN ERR Q MEDIAN Q MAX ERR RANGE IQR MEAN STDEV RMSE NMSE FRAC. BIAS CORR N < or = 1, % < or = 1, direction, respectively. The average mean absolute error for all models is about 1.5 ms 1 for speed and 30 for direction. For the 8M tests, the differences between models are a maximum of only 0.03 ms 1 in wind speed, and 1 in direction. Near equivalence of the model performances over the full sample is particularly emphasised by examination of the RMSE values, which, though sensitive to large excursions, were found to be 1.7 ms 1 for all models. For stable conditions, mean errors are larger, but still less than 2 ms 1, while smaller errors closer to 1 ms 1 are obtained for non-stable conditions. Comparing the relative performances for stable vs. nonstable conditions, the 3M validations here show the same behaviour as in the previous tests at all (8M) input stations. To explore this further, the correlation coefficients are shown in Figure 4. Negative correlations are found to occur for stable conditions. This behaviour was observed in scatter diagrams plotted for stable conditions (Figure 5), for CALMET. Each of the models displays essentially the same behaviour. All show a tendency towards smaller calculated speeds for higher observed values, as indicated by the negative correlations. It is believed this is a result of the light and variable nature of the winds caused by the morning radiation inversion, and especially the strong spatial variability indicated by the observations at different stations. This behaviour can be caused by several factors such as the wind speeds being smaller at the three input stations relative to the other five stations. Or, the wind CORRELATION COEFFICIENT ALL STABLE NONSTABLE TIMES CALMET MCSCIPUF SWIFT Figure 4. Correlation coefficients between calculated and observed wind speed at data withheld stations (3M study). direction at the input stations relative to the other five is nearly against each other. We suggest that the low wind speeds may actually over-emphasise the model errors, particularly in wind direction due to the high variability in the data. Examining the surface and radiosonde data does show the variability and thus these models do not perform as well in cases of lighter wind speeds. Other studies by Cox et al. (1998 and 2000) suggest that the models more effectively model both speed and direction in strong dynamically forced conditions. What this study shows is the difficulty that these models have in weakly forced conditions. 337

10 Robert M. Cox, John Sontowski & Catherine M. Dougherty PREDICTED (ms -1 ) OBSERVED (ms -1 ) Figure 5. Scatter plot for CALMET calculated vs. observed wind speeds at data withheld stations (3M study) during stable conditions. For an indication of the spatial variability, the standard deviations of the observed wind speeds were evaluated at each individual time, for the eight surface stations. Values were found to be significantly higher at stable times than at non-stable times. The average standard deviation was found to be 1.71 ms 1 for the stable times vs ms 1 for non-stable times. Given the generally lighter winds during stable times, the difference appears even more significant considering standard deviations normalised by the mean of the wind speeds (over the eight stations) at the individual times. Average values for the normalised standard deviations were found to be 0.60 and 0.27 for the stable and non-stable times, respectively. This variability in the observed conditions is believed to be primarily responsible for the poor performance at the stable times. Further, it does not seem likely that the performance would have been better with a selection of three different stations for input, as no three would seem capable of characterising the variability. Under such highly variable conditions during stable periods it would appear that a more dense observation network would be required if small-scale details are to be captured by mass consistent models. The length scale of the input stations for this study is of the order of 4 km. In weakly forced events, the input station scale needs to be denser otherwise the models will be unable to resolve the details in the wind field. This is further shown given generally light winds and the relatively poor performance during stable times, as indicated by the normalised mean square error (NMSE) and the FB. The NMSE for wind speed is about 0.2 for all stabilities, but is about 0.8 for stable conditions and 0.1 for unstable conditions. The FB for wind speed is about 0.1 for all stabilities, but is about 0.3 for stable conditions and about 0.06 for non-stable conditions. Within a given stability class, the difference 338 in NMSE or FB among the three models is relatively small. It should be noted that statistics were calculated on a station-by-station basis, in order to reveal any possible systematic difficulties at particular stations. Some minor differences were found with MEDA stations 9 and 10 showing about 20% larger absolute error in wind speed and about 5 to 10 larger absolute errors in wind direction. Station 1 gives the smallest absolute error in speed (about 1 ms 1 ), while station 16 has the smallest absolute error (about 18 ) in wind direction. The M9 and M10 stations are close to the mountains, whereas M1 and M16 are relatively far away from the mountains. The different models are consistent in showing similar station-to-station variation in errors. Most notable in the analysis of mean errors is the performance of MCSCIPUF. Contrary to its slightly poorer ability to reproduce winds at input locations (8M study), MCSCIPUF performance at non-input locations (the 3M study) has been found to be close to that of the other models. Reasons behind the difference in performance by MCSCIPUF will be explored further in section Overall performance of models On the basis of the work presented here, it is not possible to recommend any one of the three mass consistent models over the other. If anything, the results for each model helps to validate the others, as all performed similarly. It is believed that the general level of errors obtained for the models are sufficiently low to justify use in many wind-modelling applications. This result is especially true for quick response calculations where time does not permit use of finer resolution prognostic mesoscale meteorological models. The performance measures found here can be compared to those obtained in a somewhat similar study by Ross et al. (1988), and to those described in a review by Ratto et al. (1994). As was the case here, Ross et al. (1988) evaluated both the ability of a mass consistent wind model to preserve input data, and the performance at non-input stations. Regarding the preservation of input data, errors were found to have median values of 0.3 ms 1, and mean values of 0.4 ms 1, i.e. slightly under-predicting as for the models examined here. This compares with median values of 0.11, 0.35 and 0.13 ms 1, and mean values of 0.17, 0.27 and 0.18 ms 1 for CALMET, MCSCIPUF and SWIFT respectively. In the validation of Ross for non-input stations, the median and mean errors were 1.3 and 1.4 ms 1. In comparison, CALMET, MCSCIPUF and SWIFT were found to have median validation errors of 0.40, 0.39 and 0.28 ms 1, and mean values of 0.43, 0.37 and 0.35 ms 1, respectively. Thus, relative to the level of errors, all of the models examined here are performing relatively well.

11 An evaluation of three diagnostic wind models Table 4. Summary of mean errors for interpolation vs. mass consistent adjustment steps, for 8M and 3M studies for MCSCIPUF and SWIFT models. INTERPOLATION MC ADJUSTMENT ADJUST-INTERP. MODEL 8M 3M 8M 3M 8M 3M MCSCIUPF SWIFT Model initialisations and interpolation performance The slightly poorer performance of MCSCIPUF in preserving input data for the cases involving inputs from all eight MEDA stations was of concern as it was not consistent with the better performance of MCSCIPUF at non-input stations. It was decided to evaluate errors for both MCSCIPUF and SWIFT at the interpolation or initialisation step of each model, i.e. before adjustment for mass consistency. Although the slightly larger MCSCIPUF errors were originally revealed in the 8M evaluations, the interpolations were also examined for the 3M calculations. The specific times/cases selected for examination included the first nine times for the 8M cases, and the first 18 times for the 3M cases. In both cases, the first nine times were selected because they all represent stable conditions, and were therefore expected to be computationally demanding. The second nine times involve non-stable conditions and were included to increase the sample size in the 3M cases, since these have only three input stations for evaluation, and also to reveal possible performance differences between stable and non-stable conditions. Sample sizes resulting from these selections total 72 error samples for the 8M cases (8 input stations 9 times), and 54 samples for the 3M cases (3 input stations 18 times). For each of these samples both the interpolation step and the adjusted mass consistent step were evaluated for the two models and compared with the input data. Results are summarised in Table 4. The evaluations show first of all that, for both models, the errors after mass consistent adjustment are greater than for the interpolated/initialised wind field. Finardi et al. (1993) observed this same behaviour in comparing US EPA wind tunnel data with the interpolated and adjusted wind fields calculated by the MATHEW and MINERVE mass consistent models. (MATHEW is the mass consistent model used by Lawrence Livermore National Laboratory, while MINERVE is the model from which SWIFT was derived.) Although the error increases due to adjustment are somewhat greater for MCSCIPUF than for SWIFT, they are essentially of the same order of magnitude. On the other hand, SWIFT was seen to start with substantially smaller errors (factor of 8 20 smaller) as indicated by the interpolation results. The difference between MCSCIPUF and SWIFT errors at the interpolation step appear to be the major contributor to the error differences after adjustment. It does appear possible, therefore, that MCSCIPUF performance could be improved by modification of its interpolation procedure. As a point of interest, and as might be intuitively expected, it is noted from the results in Table 4 that both interpolation and adjustment errors tend to increase as the number of input stations is increased. Finally, no systematic differences in performance, for either the interpolation or the adjustment, were observed between the stable and non-stable conditions examined. Again it should be noted that these conclusions are based on limited sampling and are therefore preliminary Vertical wind speeds Vertical wind components are also important to transport analysis. Unfortunately, observations are not available for vertical winds. For qualitative assessment, vertical wind speeds have been extracted from the output files for the three models. For this purpose, the time of 0600 hours on 12 November was selected as an example, though it is realised that a single time is not sufficient to draw definitive conclusions. Both the 3M and the 8M calculations were examined. We found that CALMET produced unrealistically large vertical wind speeds of several metres per second at the top of its computational domain (2650 m AGL). In the general area above MEDA stations 6, 10 and 28, maximum vertical winds of 6.6 ms 1 were noted for the 3M calculation, and 5.3 ms 1 for the 8M calculations. At about the same height above ground, both MCSCIPUF and SWIFT were found to produce more realistic vertical wind speeds of a few cm 1, or approximately two orders of magnitude lower than CALMET. The discrepancy in vertical winds is believed to be a consequence of the differences in the divergence minimization process employed by the models. While this is essentially a three-dimensional process in MCSCIPUF and SWIFT, the CALMET procedure adjusts only the horizontal components of wind, while holding the vertical component fixed. Another source of large vertical wind speeds at the domain top for CALMET is that the model vertically extrapolates surface wind observations based on similarity theories. It is expected that, without some modification, such winds calculated by CALMET would cause a puff model such as SCIPUFF to advect a large amount of pollutant mass up through 339

12 Robert M. Cox, John Sontowski & Catherine M. Dougherty the top of the computational domain. Along with this is the concern that surface concentrations will be underpredicted, while upper air concentrations could be overpredicted. It should be noted that the CALPUFF transport and dispersion model, which is linked with CALMET, does not use vertical wind speeds in its calculations (Scire et al. 1998). 6. Conclusions The primary objective was to assess the performance of three diagnostic wind models (CALMET, MCSCIPUF, and SWIFT) by direct comparison against wind field data. The source of the data was the Dipole Pride 26 (DP26) series of experiments, carried out at the Nevada Test Site during November A common subset of meteorological data was extracted from the DP26 database, after screening for quality assurance, and used as input to the models. These were then exercised over the same computational domain, with identical grid resolution. Evaluation of the models involved testing of the ability of the models to reproduce wind fields, as well as the validation of performance against wind observations that were not used as input data. The work presented here was built upon previous evaluations accomplished by Chang et al. (1999) and Bradley & Mazzola (1999). For the part of the study involving model faithfulness to the input data (the M8 evaluation), CALMET and SWIFT were found to perform extremely well with nearly equal performance measures. Both models showed mean absolute errors of less than 1 ms 1 for nearly 100% of the time, and correlation coefficients of Wind direction errors were less than 10 for 87% of the time for CALMET, and 83% of the time for SWIFT. The models displayed a slight tendency towards underestimating wind speed and producing larger errors for stable conditions than for non-stable. The MCSCIPUF absolute errors were larger than those of CALMET and SWIFT. While the MCSCIPUF median and mean errors were approximately three times larger and 50% larger, respectively, than those of CALMET and SWIFT, they were comparable or smaller than the median and mean errors reported by Ross et al. (1988). Examination of errors at the initialisation and interpolation steps of the MCSCIPUF and SWIFT modelling procedures reveals that the MCSCIPUF error might be reduced by modification of the interpolation. Based on the evaluations, the models appear to have nearly equal ability to produce valid horizontal wind fields, given three surface stations (the data withheld part of the study) and an upper air profile as input data. Few differences were found in the performance of the models. The magnitudes of the wind speed errors were small compared to the average wind speeds of about 3 4ms 1. For example, the normalised mean square errors were 0.2 for all models. Mean wind speed absolute errors 340 were essentially the same for all three models, about 1.3 ms 1, with an error of less than 1 ms 1 occurring more than 50% of the time for all models. Mean absolute errors in wind direction were about 30 with errors of less than 20 for 42% of the time. Another finding was the relative performance of the models for stable vs. non-stable atmospheric conditions. All models performed best during non-stable times, as would be expected when more mixing is present. For the light and variable conditions associated with the stable cases, station-to-station comparisons showed relatively large variations in space, in wind speed and wind direction. These variations could not be captured with a limited number of surface meteorological stations. Given a limited number of stations, such variations might be better simulated with refined grid resolution or with improved physics within the models. In the grid resolution solution, it is assumed that the spatial variations are terrain induced, while in the improved physics solution it is assumed that physical effects not incorporated in the models generate the variations. For these cases, subsequent transport/dispersion calculations would correctly predict that material releases would not travel far from their source. Acknowledgements The work described in this paper was supported by the US Defense Threat Reduction Agency under Contract No. DTRA C The authors are indebted to Lt. Col. Thomas Smith, Mr Ron Meris and Major Brian Beitler of DTRA, who have been the DTRA technical monitors. This particular work also benefited from generous guidance from Mr Joseph Chang of TRW for CALMET, and Dr Ian Sykes and Mr Douglas Henn of the Titan Corporation for MCSCIPUF. In regard to the SWIFT model, the expertise and longterm collaboration of Dr Jacques Moussafir of ARIA Technologies, starting with the original adaptation of SWIFT for the DTRA, has been invaluable. Finally, our appreciation is extended to Mr Scott Bradley of Logicon Advanced Technology for his help in the acquisition and understanding of the data from the Dipole Pride 26 field experiments. References Biltoft, C. A. (1998) Dipole Pride 26: Phase II of Defense Special Weapons Agency Transport and Dispersion Model Validation, Dugway Proving Ground Report DPG-FR , Dugway, UT, 37 pp. Bradley, S. & Mazzola, T. (1999) HPAC 3.1 Predictions Compared to Dipole Pride 26 Sampler Data, Defense Threat Reduction Agency, Alexandria, VA, 33 pp. Byers, M. E., Hodge, J. K. & Cox, R. M. (1995) Hazard Prediction and Assessment Capability (HPAC). Proc. Fifth Topical Meeting on Emergency Preparedness and Response, , American Nuclear Society, Savannah, GA.