Hurricane Track Prediction Using a Statistical Ensemble of Numerical Models

Transcription

1 VOLUME 131 MONTHLY WEATHER REVIEW MAY 2003 Hurricane Track Prediction Using a Statistical Ensemble of Numerical Models HARRY C. WEBER Meteorological Institute, University of Munich, Munich, Germany (Manuscript received 8 October 2001, in final form 30 August 2002) ABSTRACT A new statistical ensemble prediction system for tropical cyclone tracks is presented. The system is based on a statistical analysis of the annual performance of numerical track prediction models, assuming that their position errors are systematic and depend on storm structure, location, and motion. For a given tropical cyclone advisory and given model forecasts of a particular storm at any base date and time, the statistical analysis of the model performances in the year preceding the base date and time can be used to produce a track prediction and geographical maps of strike probability distributions at all prediction times. The statistical ensemble prediction system was developed using tropical cyclone advisories, model predictions, and best-track positions of all Atlantic hurricane seasons between 1996 and Track predictions were carried out in each individual year between 1997 and The 24-, 48-, and 72-h mean position errors, averaged over the whole time period , were found to be 120, 215, and 296 km, respectively, and showed positive skill (negative relative error) of about 20% relative to all high quality numerical models and approximately equal skill relative to the consensus model of Goerss at all prediction times. Equivalent experiments with an operational ensemble, consisting only of models that were available at the issuing times of official forecasts, resulted in corresponding mean position errors of 128, 238, and 336 km and positive skill of about 5% 15% versus the official forecasts. A major characteristic of the new track prediction system lies in the automatic production of geographical strike probability maps. Mean diameters of the 66.7% strike probability regions of 274, 535, and 749 km (304, 682, and 1033 km in the case of the operational ensemble) at 24-, 48-, and 72-h prediction time and good agreement between the observed percentages of storm positions inside regions of given strike probabilities with the corresponding predicted percentages, document the potential of the new system with regard to operational tropical cyclone track prediction. 1. Introduction During the last decade, the quality of numerical track predictions has improved so much that they have become nearly indispensible in current operational tropical cyclone (TC) track prediction. However, the large number of available model forecasts, 1 in combination with their spread of future storm positions, represents a dilemma for operational forecasters: on the basis of their professional experience alone, they have to anticipate whether a particular model prediction should be included in or excluded from the process of producing an operational forecast. Even the prediction quality of highly valued numerical models such as the regional model of the Geophysical Fluid Dynamics Laboratory (GFDL; Kurihara et al. 1993, 1995, 1998; Bender et al. 1993) 1 Between 1996 and 2000, track predictions were carried out by over 80 operational and experimental numerical models. Corresponding author address: Harry C. Weber, Meteorological Institute, University of Munich, Theresienstr. 37, Munich, Germany. harry@meteo.physik.uni-muenchen.de or the global model of the U.K. Met Office (UKMO; Heming and Radford 1998) may vary considerably from season to season 2 or fail unpredictably under particular circumstances, with negative effects on the quality of official predictions. In an effort to improve the assessment of the expected model performance during the daily routine of operational track forecasting, the present study focuses inter alia on a statistical evaluation of the performance of numerical models and their combination in a forecast ensemble, assuming that their position errors can be regarded as being systematic with regard to the structure, location, and motion of tropical cyclones. 3 The use of forecast ensembles has a long tradition in numerical weather prediction. Evidence so far indicates that the statistical evaluation of a large number of forecasts, produced by using either a single numerical model 2 For example, the 72-h mean position errors (sampled at base dates/ times 0000 and 1200 UTC) of the GFDL model in 1996, 1997, 1998, 1999, and 2000 were 330, 332, 398, 352, and 333 km, and those of the UKMO model were 413, 425, 368, 328, and 401 km, respectively. 3 This assumption includes the one that updates and modifications of numerical models have only a moderate effect on their performance characteristics American Meteorological Society 749

2 750 MONTHLY WEATHER REVIEW VOLUME 131 started from different initial conditions or a set of independent numerical models, yields in principle a prediction that is superior in quality to the forecasts of individual numerical models. The generation of prediction ensembles can be divided into the two major categories addressed below. The first category follows a method suggested by Leith (1974). A single numerical model is initialized with a set of different initial conditions and integrated in time for each member of the set. The generation of initial states of numerical models includes methods such as singular vector decomposition (Molteni et al. 1996), the breeding method (Toth and Kalnay 1997), or Monte Carlo methods (Mullen and Baumhefner 1994). In a few cases, these methods were applied also to the track prediction of tropical cyclones, for example by Krishnamurti et al. (1997) and Zhang and Krishnamurti (1999) using The Florida State University (FSU) global spectral model, by Aberson et al. (1998) using the GFDL model, and by Puri et al. (2001) using the global ensemble prediction system of the European Centre for Medium- Range Forecasts. Although all of the above studies document some success in providing improved track guidance in comparison with the current operational models, no long-term evaluation of single-model ensemble predictions of tropical cyclones has been carried out to date for a general assessment of their operational value. In the second category, the statistical ensemble is not generated by multiple forecasts of a single numerical model but by forecasts of a set of numerical models. Applications of this approach to the prediction of tropical cyclones have been published, for example, by Leslie and Fraedrich (1990), Mundell and Rupp (1995), Goerss (2000), and Krishnamurti et al. (2000). Especially the latter two studies have shown that, on average, an ensemble of forecast models is capable of providing significantly better track guidance than individual models. Goerss combined three major numerical models, the GFDL model, the UKMO model, and the U.S. Navy Operational Global Atmospheric Prediction System (NOGAPS; Hogan and Rosmond 1991; Goerss and Jeffries 1994) into a so-called consensus model (CONS) by simply averaging the predicted positions of the three models. The consensus model was tested during the Atlantic hurricane seasons on 166 cases of storms with wind speeds greater than 34 kt for base date/time intervals of at least 30 h. With a mean 72-h position error of 266 km, it performed much better than each individual ensemble member (GFDL, 364 km; UKMO, 348 km; NOGAPS, 383 km). Based on these results, a number of consensus models are currently applied with success at the Joint Typhoon Warning Center [using a set of five dynamical prediction models and subsets thereof; cf. Elsberry and Carr (2000)] and the National Hurricane Center (NHC; using two sets of three and four dynamical models, respectively). Taking a different approach than that of Goerss, Krishnamurti et al. (2000) used an ensemble of two global models and one spectral model of the FSU, the GFDL model, the UKMO model, and NOGAPS to predict the track and intensity of storms during the 1998 Atlantic hurricane season. During a training period (including the model forecasts of all storms except the one to be predicted), all available forecasts of the ensemble members were subjected to a linear multiple regression relative to best track information for the derivation of statistical weights of the performance of each ensemble member. In the forecast period the weighted individual forecasts of all ensemble members were used to produce track and intensity predictions. Krishnamurti et al. justified this cross-validation approach by the major modifications made to some of the models after With mean position errors of about 125, 190, and 260 km at 24-, 48-, and 72-h prediction times, respectively, the average track guidance was found to be significantly better than that of each individual model and the official NHC forecast. The prototype of a statistical ensemble prediction system (named STEPS) presented in this study belongs to the second of the categories addressed above. Similar to the approach taken by Krishnamurti et al. (2000), STEPS uses statistical weights for tropical cyclone track prediction. However, in contrast to Krishnamurti et al. and irrespective of effects of model modifications, the weights are computed by a statistical evaluation of the forecast performance of the numerical models of the ensemble as functions of parameters of storm structure, location, and motion in the year prior to the forecast period. In addition to the prediction of storm positions, the weights of STEPS allow also the automatic production of a map of the geographical strike probability distribution. 2. Datasets The datasets used in the present study cover the Atlantic hurricane seasons and consist of two parts: TC advisories issued by the National Centers for Environmental Prediction (NCEP) and predicted positions of all existing numerical models and the NHC (official forecasts) at 6-hourly intervals, that is, at base times 0000, 0600, 1200, and 1800 UTC. Verification of all predicted storm positions was carried out using manually revised and corrected 4 best-track positions downloaded from the web site com. Position errors were computed with spherical geometry. At any given base date and time t i (i 1,...,I; I represents the total number of available base dates/times during a season), the NCEP TC advisories contain information on the structure, location, and motion of a storm in question. In particular, they provide estimates 4 Obvious errors of the best-track positions such as data format errors or unrealistically large distances between successive prediction times were replaced by no data available indicators.

3 MAY 2003 WEBER 751 of the storm positions (latitudes, c, and longitudes, c ) at t i, t i 12, and t i 24 h, the radius of maximum wind speed r m, the radius of the outermost closed isobar r o, the central surface pressure p c, and the surface pressure p o at r o at t i, and the maximum wind speed m, the translation direction c d, and the translation speed c at t i and t i 12 h. In addition, estimates of the radii of 35- and 50-kt winds in each storm quadrant are provided, as is an indicator of the vertical extent of the storm in question (S for a shallow, M for a medium, and D for a deep system). At all t i, the dataset containing the model predictions lists the predicted geographical positions and intensities of a given storm at the prediction times t k t i 12, t i 24, t i 36, t i 48, and t i 72 h. Between 1996 and 2000, track predictions of more than 80 experimental and operational numerical models existed. With three exceptions, all numerical models have been used for the prototype of STEPS (henceforth referred to as the complete or original ensemble), including experimental models, if they existed in at least two consecutive hurricane seasons and if their total number of annual predictions corresponded approximately with that of the major models. The exceptions are the semioperational VICBAR model (DeMaria et al. 1992, acronym stands for Vic Ooyama s barotropic model), the Limited area barotropic track (LBAR) model (Horsfall et al. 1997), and NCEP s Aviation Model (AVN; Lord 1993; Surgi et al. 1998); the reasons for their exclusion are addressed in section 4. Numerical models occurring for the first time during a given hurricane season are excluded from the ensemble by definition. Among the more prominent ensemble members of the complete version of STEPS are variants of the GFDL model 5 [e.g., the U.S. Navy version; Rennick (1999)], the UKMO model, and NOGAPS as well as NCEP s Medium-Range Forecast (MRF) model 6 and Eta Model, the Beta and Advection Models (BAM; Marks 1992), the Climatology Persistance (CLIPER) model (Neumann 1972), and NHC s statistical regression model A90E and extrapolation model (XTRP). Furthermore, the consensus model of Goerss (2000, cf. section 1) was included and treated as a regular operational model, if forecasts of each of its ensemble members (the GFDL model, NOGAPS, and the UKMO model) existed at a given base date/time. The model prediction datasets contained also the official NHC track predictions (OFCL), but they were not used in the STEPS ensemble. For comparisons of the predictions of the NHC and the consensus model with STEPS, the size of the original ensemble was reduced to those of the above numerical models whose predictions were available at the deadlines for the issuing of official prognoses (henceforth 5 Note that for storms 11 (Eleven) through 16 (Lenny) 1999, no track predictions of the GFDL model existed in the datasets. 6 Note that the MRF and AVN predictions differ significantly from each other in the model forecast datasets used in the present study. referred to as the operational ensemble). It should be noted that in this version of STEPS, the AVN model was kept in the ensemble, mainly to increase the relatively small number of ensemble members. 3. Model description The STEPS model consists of two parts, defined as the initialization period and the forecast period, which are discussed in detail in the following two subsections. The general concept of STEPS can be summarized as follows: during the initialization period, the mean performance of each available numerical model at all prediction times is statistically analyzed on the assumption that the position errors of each model are systematic and depend on the storm parameters provided by TC advisories for a particular storm. A set of statistical weights (referred to as quality indices) in the form of a linear combination of expected position errors and their standard deviations are computed as functions of all possible values of the relevant storm parameters and stored in a database. To minimize possible changes of model performance characteristics that may occur in response to their modification during the period of interest, the initialization period covers only the hurricane season prior to the forecast period. Tests showed that an extension of the initialization period to more than one season preceding the forecast period degrades the prediction quality of STEPS. For any given storm at any base date/time during the forecast period, the expected performance quality of individual ensemble members is defined by their quality indices, computed for the currently valid storm parameters. Application of the weights to the predicted positions of all ensemble members results in a track forecast in the form of a geographical map, showing statistical confidence regions of different strike probability. a. The initialization period A large number of statistical numerical sensitivity experiments with STEPS showed that not all storm parameters provided by the operational TC advisories are suitable for a statistical evaluation of model performance. Some parameters such as p c and m or p o and r o were found to be mutually dependent, while the use of other parameters such as t i [expressing a possible seasonal dependence of model performance; cf. DeMaria et al. (1992)], c d,or c had no or, in the case of the latter two parameters, even negative impacts on the average performance of STEPS. For this reason, an approximately optimum subset of all available storm parameters X j (j 1,...,6),consisting of c, r m, r o, m, c, and the 12- hourly change of maximum wind speed m m (t i ) m (t i 12 h) was derived in the experiments. Each X j is divided into N bins (n 1,...,N), using the minimum and maximum value of each X j in the initialization period as lower and upper boundaries. Sensitivity tests produced

4 752 MONTHLY WEATHER REVIEW VOLUME 131 an optimum bin number of N 20, with sizes of about 2, 10 km, 25 km, 3 m s 1,1ms 1,and2ms 1, respectively, for each of the X j listed above. Smaller/ larger values of N resulted in too weak/strong amplitude variations of the statistical quantities discussed below, with negative effects on the mean performance of STEPS. The following procedure is illustrated by Fig. 1, showing the relevant statistical quantities related to the performance of the GFDL model in the 1998 hurricane season as functions of the X j. First, all existing position errors e lik of each available model l (l 1,...,L) at all base times t i and prediction times t k of the initialization period are computed (mathematical definitions of all statistical quantities are given in the appendix). From the e lik, the most probable position error M LIk (defined as the maximum of the density distribution of all e lik ; note that capital letters L and I in the indices denote a summation over all indices) of all models at each t k can be derived. The M LIk, listed in Table 1, reflect the mean performance of all models during the hurricane seasons examined. They are used to compute nondimensional position errors q li e lik /M LIk, which to a first approximation can be regarded as being independent 7 of the t k and are shown as dots in all panels of Fig. 1. In consequence of their independence of t k, the q li can be sorted into the N bins of each X j. In each bin n of all X j, the mean nondimensional position error Q ljn, the standard deviation ljn, and the most probable position error M ljn are computed from the q li and represent quasicontinuous functions of the bin midpoints x jn. Note that the ljn are computed relative to a zero mean nondimensional position error, assuming that the expected position error of each model is optimal and zero. They form the basis for the strike probability maps produced in the forecast period (see below). Note furthermore that M ljn is computed in the same way as M LIk and reflects the maximum of the density distribution of all q li in each bin as shown by the contours in all panels of Fig. 1. The quality indices of model l as a function of the x jn of all X j are defined as QI ljn (Q ljn ljn M ljn )/ 3 and shown by the bold curves in the panels of Fig. 1. They describe the sensitivity of the predictions of a model as functions of all possible values of a given storm parameter. Smaller (larger) magnitudes of QI ljn represent a superior (inferior) expected model performance in the forecast period. For example, the QI ljn of the GFDL model document a strong sensitivity with respect to m, m, and r m, with an expected better performance in cases of stronger (Fig. 1d), intensifying (Fig. 1e) storms with smaller eye sizes (Fig. 1b), while 7 A more differentiated statistical analysis, based on nondimensional position errors that depend on the t k, produced better track guidance at all t k in many cases, but with the unwanted side effect of often large and unrealistic jumps in the predicted positions at adjacent prediction times. Note also that the independence of the q li of the t k made it necessary to exclude VICBAR, LBAR, and the AVN model from the original STEPS ensemble, for reasons explained in section 4. the dependence of the expected model performance on the variation of the latitude (Fig. 1a), the storm size (Fig. 1c), and the translation velocity (Fig. 1f) is relatively weak. The linear combination of all three statistical quantities contributing to the QI ljn ensures that the quality indices are not only a reflection of the mean model performance during the initialization period, but also of the robustness of the prediction quality as expressed by the standard deviation. Outside the valid range of the x jn,qi ljn, Q ljn, ljn, and M ljn are set equal to the nearest corresponding value inside the valid range of the x jn. The quantities M LIk, x jn, ljn, and QI ljn are stored and represent the database for the forecast period. b. The forecast period The procedure discussed in this section is illustrated by the three forecast examples shown in Fig. 2. The left panels of Fig. 2 give some impressions of the ensemble sizes and the spread of the track predictions of the ensemble members, while the right panels show the corresponding STEPS (S) track predictions and superpositions 8 of the 66.7 (inner) and 99.9% (outer contours centered on S) strike probability regions of all prediction times. At any given base date and time t i during the forecast period, STEPS chooses all models available both in the database and the dataset containing the model predictions at t i, but all model forecasts with unrealistic 12-h position differences of more than 1000 km (corresponding to c 23 m s 1 ) are eliminated from the ensemble. Once the ensemble is complete, the information contained in the database and the current TC advisory are evaluated such that the currently valid quality indices QI lji and standard deviations lji are computed at t i by rational interpolation (Späth 1990) of the functions QI ljn and ljn at the current values x ji of all X j. The QI ljn and ljn are averaged over all X j to give mean values QI li and li for each ensemble member and the minimum and maximum values min max QIi, QIi of all models are de- termined. The predicted storm positions and geographical strike probability maps of STEPS are produced as follows: first, quality weights w li are computed by w li 0.5 [1 min max min cos( s)] with s (QI li QIi )/( QIi QIi ). The functions w li and s ensure that the model expected to perform best at t i obtains the maximum weight. The li are multiplied with the M LIk to give the mean standard deviations lik in kilometers of model l at prediction time t k. A preliminary strike probability map at t k is produced by the computation of a geographical Gaussian distribution function of the form L 2 ik li lik l 1 F (, ) w exp( 0.5z ), 8 In an operational context, separate maps would have to be produced for individual prediction times.

5 MAY 2003 WEBER 753 FIG. 1. Nondimensional position errors q li (dots), their normalized density distribution (contour interval 0.2), and quality indices QI ljn (bold curves) of the GFDL model in the 1998 Atlantic hurricane season as functions of (a) the latitude (YC) in, (b) the radius of maximum wind speed (RM) in km, (c) the radius of the outermost closed isobar (ROCI) in km, (d) the maximum wind speed (VM) in m s 1, (e) the 12-hourly change in maximum wind speed (DVM) in m s 1, and (f) the translation velocity (C) in m s 1.

6 754 MONTHLY WEATHER REVIEW VOLUME 131 TABLE 1. Mean annual most probable position errors M LIk in km of all models of the STEPS ensemble at all prediction times during each of the Atlantic hurricane seasons The values in parentheses have been computed using the operational ensemble of models. Year 24 h 48 h 72 h (123) 83 (146) 135 (136) 98 (95) 70 (130) 224 (248) 194 (357) 252 (320) 178 (178) 206 (138) 431 (253) 286 (681) 362 (362) 212 (338) 328 (373) where z lik r/ lik and r is the great circle distance of a geographical location (, ) from the position predicted by model l. The absolute maximum of F ik represents the predicted storm center position of STEPS (marked by S in the right panels of Fig. 2). It is located by application of a downhill method (Bach 1969; cf. also Weber and Smith 1993) in combination with birational interpolation (Späth 1991) of the matrix F ik. The construction of the final geographical strike probability map (as shown in the right panels of Fig. 2), that is, the correlation of the isolines in the normalized two-dimensional preliminary distribution F ik with different strike probability values, is achieved by a statistical evaluation of the position error distribution of STEPS in the preliminary strike probability maps F ik.inan attempt to simulate a statistical evaluation of the climatological (long term) performance of STEPS, the evaluation is carried out using a cross-validation approach similar to that of Krishnamurti et al. (2000, p. 4209) for all hurricane seasons between 1997 and 2000 excluding the forecast period (the current season). In this context it should be noted that due to the robustness of the performance of STEPS (cf. section 4), the correlations obtained by the evaluation based on the crossvalidation approach do not differ significantly from those obtained by an evaluation of all hurricane seasons between 1997 and 2000 including the forecast period. Two important remarks regarding the procedure described above are required. First, the use of standard deviations for the identification of the regions of high strike probability assumes a Gaussian position error distribution for each model, which is only a low-order approximation. As can be seen by the contours drawn in in Fig. 1, the real position error distributions correspond with one-sided, skew distribution functions. However, tests showed that the use of the real error distributions complicates the spatial structure of the strike probability maps F ik by the occurrence of local maxima embedded in comparably small-sized regions of high strike probability. The consequence is that a satisfactory interpretation of the F ik and location of a predicted position is almost impossible. The second remark is that in response to the spread of the model predictions and their relative weighting, more than one region with a strike probability higher than 66.7% may exist in the F ik. The algorithm designed to locate the maximum strike probability in the F ik produces only a single predicted storm position, which may not necessarily represent the most probable predicted position and affects the position error statistics of STEPS presented in section 4. However, all secondary maxima of significant magnitude are visible in the F ik and can be included in an interpretation of the tracks predicted by STEPS. 4. Forecast results The focus of this section is on presenting a statistical analysis of the forecast performance of STEPS during all Atlantic hurricane seasons and on comparing the results of the original version of STEPS with some of the major and/or best-performing numerical models in this period of time, that is, the AVN model, CLIPER, the CONS model, the GFDL model, LBAR, NOGAPS, and the UKMO model. The official forecasts of the NHC and an operational version of CONS 9 are compared with the operational version of STEPS. In contrast to the STEPS initialization period, where all available base times (i.e., 0000, 0600, 1200, and 1800 UTC) have been used for the generation of the database, track predictions during the forecast period are restricted to base times 0000 and 1200 UTC, in response to tests of STEPS on serial correlation of successive track predictions (Neumann et al. 1977). Application of the procedure described in Aberson and DeMaria (1994, their p and appendix B) to the original STEPS predictions between 1997 and 2000 showed that successive track forecasts became uncorrelated for separation times of 15.4, 16.6, and 16.9 h (mean 16.3 h) at 24-, 48-, and 72-h prediction time, respectively. The corresponding values of the operational ensemble were 15.2, 16.4, and 17.0 h (mean 16.2 h). As these values do not differ much from 12 h, they justify the choice of base times during the forecast period. In this context it should also be noted that all statistics, results, and comparisons presented in this section refer exclusively to the base time pair 0000 and 1200 UTC. Based on sensitivity experiments, the VICBAR, LBAR, and the AVN models were excluded from the original ensemble. It was found that often, especially in cases of relatively large position errors at the 48- and 72-h prediction times, the predicted positions of STEPS were located in the immediate vicinity of those of VIC- BAR, LBAR, and the AVN model. The first two models normally produce excellent track guidance at early prediction times (e.g., at 12 or 24 h), while their prediction quality degrades for longer forecast periods. For the reasons given in section 3a, the STEPS initialization procedure does not include a differentiated statistical evaluation of model performance at individual predic- 9 In equivalence to the operational version of STEPS, the ensemble of this consensus model consists of versions of the GFDL and UKMO models and NOGAPS that were available at the issuing times of the official forecasts.

7 MAY 2003 WEBER 755 FIG. 2. Selected forecast examples of the operational version of STEPS: (left panels) the predicted positions of all ensemble members at 12, 24, 36, 48, and 72 h. The letters represent the individual ensemble members, e.g., the AVN model (a); the deep, medium, and shallow BAM model (c, b, B); CLIPER (C); the Eta Model (e); the GFDL model (g); NOGAPS (n); the UKMO model (t); and the extrapolation model (X). (right panels) The corresponding positions of STEPS (S) and the NHC (O). The inner and outer contours encircling S define superpositions of the 66.7% and 99.9% strike probability regions at all prediction times, respectively. Storms shown are (a), (b) Hurricane Bonnie at 0000 UTC 25 Aug 1998 and (c), (d) at 1200 UTC 26 Aug (e), (f) Hurricane Georges at 1200 UTC 19 Sep Continents are outlined in solid contours and observed positions are marked by the hurricane symbol.

8 756 MONTHLY WEATHER REVIEW VOLUME 131 TABLE 2. Annual and total ( ) MPE and SD of the complete and (in parentheses) the operational version of STEPS in km and the number of cases (Cases) at 24-, 48-, and 72-h prediction times. Measure 24 h 48 h 72 h MPE SD Cases (129) 130 (142) 116 (126) 108 (113) 120 (128) 81 (101) 85 (89) 76 (85) 70 (74) 78 (86) (215) 221 (247) 221 (242) 205 (228) 215 (238) 106 (117) 156 (191) 133 (153) 138 (148) 140 (163) (330) 283 (326) 289 (319) 316 (375) 296 (336) 130 (134) 223 (238) 190 (204) 172 (232) 194 (219) tion times. Hence, the excellent performance of VIC- BAR and LBAR at early prediction times results in an overestimation of their expected performance quality relative to the other ensemble members at later prediction times and therefore degrades the prediction quality of the corresponding forecasts of STEPS. The frequent proximity of the predicted positions of the AVN model, VICBAR, and LBAR at later times, which motivated the exclusion of the AVN model from the model ensemble, may be a consequence of the fact that the global analyses and predictions of the AVN model form initial and boundary conditions for VICBAR and LBAR. However, in the operational ensemble the AVN model (but not VICBAR and LBAR) was retained for reasons given earlier. a. General performance of STEPS Table 2 shows the annual and total ( ) mean position errors (MPE; for the definitions of all statistical quantities see the appendix) and their standard deviations (SD) at all prediction times. The relatively large number of cases, shown in the third group of rows of Table 2 (e.g., 332 cases at 72 h between 1997 and 2000) provides some confidence in the statistical significance of the results presented here. In contrast to all major numerical models (cf. footnote 2), the year-by-year prediction quality of STEPS varies very little (e.g., from 108 to 130, 205 to 221, and 283 to 319 km in the case of the complete ensemble at 24-, 48-, and 72-h prediction time) and indicates the consistency of track predictions of STEPS and its ability to compensate for possible year-by-year quality variations of the individual ensemble members. With the exception of the year 2000, this result is also valid for the operational version of STEPS (Table 2, values in brackets). The mean position errors of the complete ensemble are with 120, 215, and 296 km at 24-, 48-, and 72-h prediction time only moderately smaller than the corresponding values of the operational ensemble (128, 238, and 336 km). On average, the standard deviations are comparable to those of all numerical models providing superior track guidance, with magnitudes of about two-thirds of the corresponding magnitudes of the position errors. A further indicator of the performance quality of STEPS is the low percentage of extremely large position errors relative to the total number of forecasts: at 24-, 48-, and 72-h prediction times, the position errors of the complete (operational) STEPS exceed 200, 400, and 600 km only in 12% (17%), 11% (13%), and 6% (11%) of all cases between 1997 and The corresponding values of the GFDL model are 20%, 14%, and 10% and of the UKMO model are 21%, 16%, and 13% and should be compared with the results of the original ensemble, while those of the official NHC forecast (18%, 16%, and 14%, respectively) should be seen in relation to the values of the operational subset given in parentheses above. In fact, out of a total of h track forecasts between 1997 and 2000, only 2 cases of position errors greater than 1000 km occurred in the original ensemble (for comparison: GFDL, 17 cases; UKMO, 7 cases). In the same period of time, the operational STEPS produced 6 cases of 72-h position errors over 1000 km (NHC, 28 cases). The most important feature of STEPS is its automatic prediction of a map of the geographical strike probability distribution. However, it is very difficult to relate these to center position predictions of numerical models. The predicted areas of high strike probability should rather be compared with the corresponding warnings as issued by, for example, the NHC. As such information was not available for the present study, Table 3 attempts to provide an impression of the prediction quality of STEPS by showing the annual and total ( ) mean diameters (D) of the regions of strike probabilities higher than 66.7%, 77.8%, and 88.9%, estimated as described in section 3b, in relation to the percentage of corresponding observed storm positions inside these regions (%). Together with the standard deviations of the position errors listed in Table 2, this may provide an adequate basis for an assessment of the quality of the strike probability maps of STEPS. The diameters of the regions of different strike probabilities depend, to a degree, on the spread of the track predictions of the ensemble members and the number of the ensemble members. However, comparison of the mean annual most-probable position errors of all ensemble members, listed in Table 1, with the annual mean diameters of the regions of different strike probabilities of Table 3 shows a much stronger dependence of the diameters of the forecast period on the mean performance of all models during the initialization period (at present the preceding year). This is a consequence of the use of standard deviations ljn of the initialization period for the construction of the strike probability maps F ik in the forecast period. For example, the relatively

9 MAY 2003 WEBER 757 TABLE 3. Annual and total ( ) mean diameters (D) in km of the 66.7%, 77.8%, and 88.9% strike probability regions of the original and (in parentheses) the operational version of STEPS and annual and total ( ) percentage of observed storm positions inside the 66.7%, 77.8%, and 88.9% strike probability regions relative to the total number of cases (cf. Table 2) at 24-, 48-, and 72-h prediction times. Year Strike probability D 24 h % D 48 h % D 72 h % % 77.8% 88.9% % 77.8% 88.9% % 77.8% 88.9% % 77.8% 88.9% % 77.8% 88.9% 338 (303) 403 (375) 524 (480) 197 (283) 235 (342) 298 (427) 359 (375) 431 (453) 549 (588) 244 (240) 295 (297) 385 (391) 274 (304) 329 (370) 422 (475) 73 (59) 84 (80) 95 (89) 43 (63) 56 (71) 69 (81) 87 (83) 91 (91) 96 (96) 65 (64) 79 (74) 95 (94) 66 (69) 76 (79) 87 (90) 628 (613) 751 (758) 975 (972) 448 (686) 535 (827) 676 (1030) 674 (872) 807 (1051) 1029 (1362) 439 (449) 530 (554) 692 (729) 535 (682) 641 (827) 821 (1057) 83 (69) 91 (89) 97 (100) 62 (79) 75 (87) 82 (94) 81 (87) 89 (95) 96 (99) 60 (57) 74 (72) 89 (90) 69 (75) 81 (86) 90 (95) 836 (646) 1002 (799) 1303 (1026) 669 (1295) 798 (1562) 1007 (1949) 965 (1009) 1157 (1217) 1476 (1579) 539 (832) 650 (1027) 851 (1349) 749 (1033) 898 (1254) 1149 (1604) 72 (55) 90 (79) 100 (86) 73 (92) 79 (97) 86 (98) 86 (84) 94 (91) 96 (96) 40 (70) 61 (81) 78 (95) 69 (81) 81 (89) 89 (95) large 1997 and 1999 annual mean diameters of the 66.7% strike probability regions at 72 h of 836 and 965 km are correlated with the moderate average prediction quality of all ensemble members in the years 1996 and 1998 (431 and 362 km, respectively), while the smaller diameters of 669 and 539 km in 1998 and 2000 are reflected in the excellent average prediction quality in 1997 and 1999 (286 and 212 km). The total mean diameters of the 66.7% (77.8%, 88.9%) strike probability regions (cf. Table 2) at 24-, 48-, and 72-h prediction time are 274 (329, 422), 535 (641, 821), and 749 (898, 1149) km for the case of the complete ensemble, respectively. The corresponding values of the operational subset (values in parentheses) are larger than those of the original ensemble. However, this feature is expected and mostly a result of the poorer mean annual performance of the operational ensemble members and the fundamental dependence between initialization and forecast period. Furthermore, the smaller number of operational ensemble members allows for greater influence of each individual model, which may have an additional effect on the size of the diameters of the different strike probability regions. The total mean percentages of observed storm positions inside the 66.7%, 77.8%, and 88.9% strike probability regions (right columns of Table 3) of the original ensemble generally agree very well with the expected percentages. At 24-h prediction time, the percentages are slightly lower, while at 48 and 72 h, the percentages are slightly higher than the predicted values of 66.7%, 77.8%, and 88.9%. The relatively low percentage of storms inside the 66.7% strike probability region of only 40% at 72-h prediction time in the year 2000 (cf. Table 3) was caused by an underestimate of the expected performance of the GFDL model in 2000 relative to that of the UKMO model on the basis of the performance of both models in 1999, in strong contrast to their true performance in the year 2000 (cf. footnote 2, but see also Tables 7 and 8); in combination with the generally excellent average performance of the numerical models during the 1999 hurricane season, this resulted in too small diameters and consequently in a relatively low percentage of storms inside the 66.7% region. In the case of the operational version of STEPS (the values in parentheses in Table 3), the observed percentages of storms inside the 66.7%, 77.8%, and 88.9% strike probability regions are also in good agreement with the expected percentages at 24 h, while at later prediction times they are generally too large, indicating a too conservative prediction of the sizes of the 66.7%, 77.8%, and 88.9% strike probability regions. The reason for the latter is essentially the same as that responsible for the size of the diameters of the different strike probability zones discussed in the last paragraph. With regard to an operational application of STEPS, current work focuses inter alia on a replacement of the present algorithm for the prediction of the geographical strike probability distribution by one that provides more realistic (i.e., less conservative) sizes. Of course, variations in the percentage of storms inside defined regions of strike probability and diameter sizes are also expected to decrease if the consistency of the year-by-year performance of the available numerical models is improved. b. Comparison of STEPS with selected models and the NHC official forecasts The results of homogeneous comparisons of the original version of STEPS with the major operational models and the operational subset of STEPS with the official NHC forecasts and the operational variant of CONS for the whole period and for each individual season between 1997 and 2000 are summarized in Ta-

10 758 MONTHLY WEATHER REVIEW VOLUME 131 TABLE 4. Homogeneous comparisons of the MPE and SD of STEPS (middle columns) in km with the corresponding values of selected numerical models (left columns) and the official NHC forecasts at 24-, 48-, and 72-h prediction time, averaged over all Atlantic hurricane seasons For the comparison with the official NHC forecasts and an operational version of the consensus model (the second entries in the CONS rows), the operational ensemble was used. The right columns show the number of cases of the homogeneous comparisons. Asterisks (*) represent mean position errors of STEPS that were found to be significantly different (at the 95% level of a paired Student s t test) from those of the corresponding model or the official forecast. Measure Model Model STEPS 24 h no. Model STEPS 48 h no. Model STEPS 72 h no. MPE SD AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS * 120* 115* 119* 118* 128* 120* * * 215* 203* 214* 206* 236* 214* * 296* 284* 293* 280* 335* 297* bles 4 8. Furthermore, Figs. 3 and 4 show the corresponding mean relative skill (as defined in the appendix) and the percentage of superior performance of STEPS in comparison with the models listed above (except CLI- PER) and the official forecast. The numerical models used for the comparison with STEPS in this paper were chosen for the following reasons: a comparison with CLIPER can be seen as a benchmark test for the performance of STEPS; LBAR represents the best barotropic numerical model in operational use, with usually an excellent prediction quality at 12- and 24-h prediction time; and finally, the GFDL model, the UKMO model, NOGAPS, and the AVN model provided the best average track guidance either in an individual year (NO- GAPS in 1997 with 286 km; the UKMO model in 1998 and 1999 with 368 and 328 km, respectively; and the AVN and GFDL models in 2000, with 334 and 333 km) or during the whole period (the GFDL model with 362 km). Tables 4 8 show also the results of a paired Student s t test (Press et al. 1986, p. 467), carried out to determine the statistical significance of the differences between the mean position errors of STEPS and those of the numerical models and the official forecast. For the null hypothesis of the t test, it was assumed that the compared mean position errors are equal. Statistical significance was tested at the 95% level (marked by asterisks in Tables 4 8). Table 4 shows that the mean performance of STEPS, averaged over all seasons between 1997 and 2000, is significantly better than that of each of the major numerical models and the official forecast at all prediction times. The superior mean performance is also documented in Figs. 3e and 4e, showing the mean relative skill and the mean percentage of better performance of STEPS versus the models and the NHC. At all prediction times, STEPS shows positive skill (negative relative error) of 15% 20% versus the GFDL and the UKMO models, 25% 45% versus the AVN model and NO- GAPS, and values increasing from 10% at 12- to 40% at 72-h prediction time relative to the LBAR model. The positive skill of the operational STEPS relative to the official forecast is approximately 5% at 12 h and increases to 15% at 72-h prediction time. On average, STEPS provides better track guidance than each individual model and the NHC forecast in over 55% of all cases at all prediction times (Fig. 4e). As the performance of STEPS is associated with the variable prediction quality of all ensemble members with regard to different storm strengths, sizes, locations, and motions, the above results reflect also the high standard of the majority of the numerical models currently in use. In contrast to the results discussed above, the differences in mean position errors in the year 1997 were found to be statistical significant to the 95% level at all prediction times only in the case of the AVN model and CLIPER (Table 5). However, the relatively small number of track predictions in comparison with the other hurricane seasons (e.g., only 29 cases at 72 h) limits the meaningfulness of the 1997 statistics. This is expressed also by Figs. 3a and 4a, showing larger variations in relative skill and better performance of STEPS versus the numerical models and the official forecast over 72 h than in the other seasons. However, the 1997 standard deviations of STEPS are considerably smaller than those of most numerical models and similar to those of other years, documenting the relative consistency of

11 MAY 2003 WEBER 759 TABLE 5. As in Table 4 but for the 1997 Atlantic hurricane season. Measure Model Model STEPS 24 h no. Model STEPS 48 h no. Model STEPS 72 h no. MPE SD AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS * 126* * * * 205* * * 319* * 393* its performance. As shown by Table 6, the 1998 Atlantic hurricane season was a rather difficult season with regard to operational track prediction. The mean 72-h position error of the best numerical model in this season, the UKMO model, was 368 km, significantly larger than the corresponding mean position errors of the best models in the other seasons. In contrast to the relatively poor performance of all numerical models and the NHC in the 1998 hurricane season, the mean position errors and (in most cases) the standard deviations of STEPS were found to be significantly smaller than those of the models. Relative to the official NHC predictions, only the 72-h mean position error was found to be significantly better. After 36-h prediction time, STEPS showed positive skill (Fig. 3b) of more than 10% relative to all numerical models and provided better track guidance (Fig. 4b) in more than 55% of all base dates and times. The results of STEPS in the 1999 hurricane season (Table 7 and Figs. 3c and 4c) are similar to those of 1998: again, the mean position errors and standard deviations of STEPS were found to be significantly smaller than those of all models at all prediction times and the official forecast at 72 h, resulting in comparable values of skill and percentages of better performance relative to the preceding hurricane season. However, as a consequence of the excellent track guidance of the UKMO model (over a period of 72 h) and the official forecasts (over a period of 48 h), the corresponding positive relative skill of STEPS was found to be less than in 1998, varying between 8% and 15% relative to the numerical models and between 0% and 15% relative to the NHC. In the 2000 hurricane season (Table 8), STEPS again produced significantly smaller mean position errors than the NHC and all models except the AVN model (334 TABLE 6. As in Table 4 but for the 1998 Atlantic hurricane season. Measure Model Model STEPS 24 h no. Model STEPS 48 h no. Model STEPS 72 h no. MPE SD AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS * 130* 127* 130* 131* * * 220* 212* 220* 219* * * 282* 276* 282* 264* 327* 286* 249*

12 760 MONTHLY WEATHER REVIEW VOLUME 131 TABLE 7. As in Table 4 but for the 1999 Atlantic hurricane season. Measure Model Model STEPS 24 h no. Model STEPS 48 h no. Model STEPS 72 h no. MPE SD AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS * 116* 99* 115* 115* * * 221* 186* 217* 207* * * 289* 241* 280* 261* 319* 285* km at 72 h) and the GFDL model (333 km at 72 h). The positive skill of STEPS relative to these two models (Fig. 3d) decreases from 30% and 20% at 12- to only 5% at 72-h prediction time, respectively (cf. also Fig. 4d). As stated earlier, the main reason for the small difference in skill at 72 h was the underestimate of the expected performance of the GFDL model in 2000 relative to the UKMO model on the basis of the statistical evaluation of their performance during the initialization period 1999 (cf. section 3). The AVN model was excluded from the original ensemble for the reasons given above, but its effect on the track prediction of STEPS during the hurricane season of 2000 would presumably have been relatively small in response to its rather poor prediction quality in It can be speculated that the use of a more complex initialization period (e.g., in form of a moving window of a given number of storms prior to the storm in question) may have led to an inclusion of the predictions of the AVN model in the later season of 2000 and therewith to an improvement of the performance of STEPS. In summary, the season-by-season results obtained with STEPS confirm the results of the statistical analysis of all seasons discussed earlier and provide confidence in the robustness and consistency of the quality of the new statistical ensemble prediction system. A postevaluation of all Atlantic hurricane seasons between 1997 and 2000 showed that, apart from the relatively small number of forecasts (cf. Table 4), the track prediction quality of the consensus model (Goerss 2000) was far superior to all numerical models and the official track predictions issued by the NHC. However, the inclusion of this model in the original STEPS ensemble had only a minor positive effect on its prediction quality, TABLE 8. As in Table 4 but for the 2000 Atlantic hurricane season. Measure Model Model STEPS 24 h no. Model STEPS 48 h no. Model STEPS 72 h no. MPE SD AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS AVN CLIP GFDL LBAR NGPS OFCL UKMO CONS * 108* 108* 108* 99* 112* 109* * * * 189* 224* 206* * * * 306* 369* 311*