MM5 Ensemble Mean Precipitation Forecasts in the Taiwan Area for Three Early Summer Convective (Mei-Yu) Seasons

Transcription

1 AUGUST 2004 CHIEN AND JOU 735 MM5 Ensemble Mean Precipitation Forecasts in the Taiwan Area for Three Early Summer Convective (Mei-Yu) Seasons FANG-CHING CHIEN Department of Earth Sciences, National Taiwan Normal University, Taipei, Taiwan BEN JONG-DAO JOU Department of Atmospheric Sciences, National Taiwan University, Taipei, Taiwan (Manuscript received 8 June 2003, in final form 9 April 2004) ABSTRACT This study presents precipitation verification of individual members and ensemble means in the Taiwan area for a real-time mesoscale ensemble prediction system, during the 2000, 2001, and 2002 early summer convective (mei-yu) seasons. The ensemble system, using the fifth-generation Pennsylvania State University NCAR Mesoscale Model (MM5) as a forecast model, consists of six members, each run with a different combination of moisture physics schemes. Precipitation forecasts within the 15-km domain were verified against the observational data from the 342 rain gauge stations on the island. In general the model that utilized the Grell cumulus parameterization scheme (CPS) and the Reisner I microphysics scheme (the GR model) had the best forecast skill among the six members. This physics combination is, therefore, recommended for MM5 rainfall simulations in the Taiwan area during the mei-yu season. The Kain Fritsch CPS dominated the rainfall process and generally underforecast rainfall at high rainfall thresholds. The Betts Miller CPS overforecast rainfall, especially at high thresholds. The equitable threat scores of the ensemble mean were not the highest, but were, in general, above the average among all members. Several other methods were examined for determining an ensemble mean (or weighted mean) rainfall forecast. WT1, which calculated rainfall by giving each member weightings determined by model performance of the member in rainfall forecast of the A period (0 12 h), generally outperformed the ensemble mean and every single member in the B period (12 24 h). This advantage did not extend to the C period (24 36 h), because the relation of model performance between the C and the A periods became weaker. WT2, in which weightings were determined according to the performance of each member in rainfall forecasts of the preceding year, performed slightly worse than WT1 in the B period, while it did better than WT1 in the C period. Another method that utilized the multiple linear regression technique to calculate weightings also showed positive impact on improving the rainfall forecast at medium to heavy precipitation thresholds. Unfortunately, its weightings appeared to be inadequate for another year s rainfall forecasts. The probability matching method helped reduce the bias problem inherent in the ensemble mean. 1. Introduction An accurate precipitation forecast is one of the most challenging tasks in meteorology. Since the advancement of numerical weather prediction in the mid-1950s the overall forecast accuracy for temperature, pressure, and wind has shown steady improvement (e.g., Landis 1994; Kalnay et al. 1990; Shuman 1989). However, progress has been slow in the skill of precipitation forecasting (Olson et al. 1995). The complex physical processes for producing rainfall, many of which are not yet fully understood, have made it difficult for a numerical model to predict precipitation with high accuracy. Com- Corresponding author address: Fang-Ching Chien, Department of Earth Sciences, National Taiwan Normal University, 88, Section 4, Ting-Chou Road, Taipei 116, Taiwan. jfj@cc.ntnu.edu.tw pared with other seasons, quantitative precipitation forecasts (QPFs) in a warm season are even more difficult, owing to the convective nature of the rainfall that is not perfectly parameterized in the model. One approach that is popularly used to reduce such model uncertainties and to increase the accuracy of QPFs is utilizing an ensemble of model forecasts. The concept of ensemble forecasting originated from the work of Lorenz (1963) in which the author first pointed out that a single deterministic forecast is sensitive to small uncertainties in the initial condition. An ensemble forecast is, thus, needed to predict as completely as possible the probability of future weather conditions (Epstein 1969; Leith 1974). There have been many methods proposed for creating members in an ensemble forecast by adding perturbations to the initial condition, for example, the breeding of growing modes 2004 American Meteorological Society

2 736 WEATHER AND FORECASTING VOLUME 19 (Toth and Kalnay 1993), singular vectors (Buizza and Palmer 1995), and the Monte Carlo technique (Mullen and Baumhefner 1988; Du et al. 1997). If the model is perfect, the ensemble mean could be the best estimate of the true state of the atmosphere. However, unless a maximum dispersion between ensemble members could be reached, a general limitation of these approaches is the requirement of a large set of forecasts. It is, therefore, computationally expensive to produce a beneficial ensemble run. The model, as a matter of fact, is never perfect. Therefore, error could also come from imperfect representation of atmospheric processes in the model. For example, Kain and Fritsch (1992) show that the simulation result of a squall line is highly dependent on the trigger function, an on-and-off switch that is used in the model to determine where and when the parameterized convection is activated. A similar conclusion is also provided by Stensrud and Fritsch (1994a,b). Wang and Seaman (1997) find that different convective parameterization schemes produce different precipitation results, and no one scheme always outperforms the others. Therefore, it is considered to be beneficial to include perturbed model physics in a mesoscale ensemble system (Stensrud et al. 2000; Houtekamer and Mitchell 1998; Andersson et al. 1998). Since the early 1990s, ensemble forecasting has been extensively used in operations at many forecast centers in the world, such as the European Centre for Medium- Range Weather Forecasts (ECMWF; Molteni et al. 1996), the National Centers for Environmental Prediction (NCEP; Tracton and Kalnay 1993; Stensrud et al. 1999), the Canadian Meteorological Centre (CMC; Houtekamer et al. 1996), and the Japan Meteorological Agency (JMA). In recent years, there have been many studies discussing the performance of an ensemble rainfall forecast (e.g., Du et al. 1997; Ebert 2001; Zhang and Krishnamurti 1997), with most of them showing that the average of rainfall forecasts from ensemble members provides more accurate results than a rainfall forecast from a single ensemble member. There has also been great interest in using ensembles to create probabilistic rain forecasts (e.g., Buizza et al. 1999; Ebert 2001; Mullen and Buizza 2001). In this study, we only examine the performance of ensemble mean rainfall forecasts by using various averaging methods. Taiwan is one of the regions around the world that is frequently affected by heavy precipitation. Its complex and high-altitude terrain, which consists of steep mountain ranges such as the Central Mountain Range (CMR), with peaks exceeding 3000 m (Figs. 1 and 2), is often a major factor in extreme rainfall events. During a mei-yu (plum rain) season, usually occurring from mid-may to mid-june, quasi-stationary fronts (the socalled mei-yu fronts) frequently form over southeastern China and move to the Taiwan area, resulting in extended periods of precipitation. Since the Taiwan Area Mesoscale Experiment (TAMEX; Kuo and Chen 1990), FIG. 1. (a) Domain configuration of the real-time ensemble MM5 system. The horizontal resolution is 45 km for domain 1 and 15 km for domain 2. Sounding stations are also shown. (b) Terrain height of domain 2, with a contour interval of 200 m. many studies have been performed to improve the understanding of weather systems during a mei-yu season (e.g., Chen and Hui 1990; Ray et al. 1991; Chen and Liang 1992; Lin et al. 1992; Li et al. 1997). From a numerical weather prediction (NWP) perspective, there are several models currently used in Taiwan for predicting and studying weather systems. The Central Weather Bureau (CWB) operates a NWP system that includes a global spectral model (Liou et al. 1997), and a regional model (Jeng et al. 1991). In addition, the Civil Aeronautics Administration (CAA) of Taiwan, the CWB, and the National Center for Atmospheric Research (NCAR) have jointly developed a real-time mesoscale forecasting system, based on the fifth-generation

3 AUGUST 2004 CHIEN AND JOU 737 Pennsylvania State University (PSU) NCAR Mesoscale Model (MM5; Chien et al. 2002). Despite these observational and numerical efforts, the forecast skill of QPFs in Taiwan during a warm season, like the mei-yu season, is still in general very low (Chien et al. 2002). Such a forecast deficiency is partly due to the aforementioned numerical problems that are shared by many other regions over the world, and partly owing to the unique complex and steep terrain in Taiwan that further complicates the numerical processes by introducing many small-scale phenomena. Yang et al. (2000) studied the impact of model precipitation physics on rainfall prediction in the Taiwan area for a heavy rainfall event of the 1998 mei-yu season, and found that precipitation prediction was quite sensitive to the choice of cumulus parameterization schemes. In view of the need for a mesoscale ensemble of precipitation forecasts, a group of researchers in Taiwan has, since 2000, jointly run an MM5 ensemble system in real time during each mei-yu season. The system consists of six members, each run by a university or a government institute. The participants include the National Taiwan University, the National Central University, the National Taiwan Normal University, the Chinese Culture University, the CWB, and the CAA. The purpose of this paper is to examine the performance of this modeling system in terms of precipitation forecasts during the mei-yu seasons of 2000, 2001, and Through comparisons based on simulated data of these three seasons, we attempted to find the best physics combination of MM5 suitable for simulations in the Taiwan area during a mei-yu season. Furthermore, several methods, in addition to the ensemble mean, were examined in order to identify a better way to improve a deterministic forecast made from ensemble information. FIG. 2. The locations of the 342 rain gauge stations in Taiwan. Terrain height (shading) starts at 200 m, with an interval of 200 m. 2. Configuration of the real-time MM5 ensemble system The model configuration of each ensemble member included two domains with 45- and 15-km horizontal grid spacing (see Fig. 1). The forecasted precipitation data used for verification were obtained from the nested domain. Twenty-three sigma levels 1 were used in the vertical, with maximum resolution located in the planetary boundary layer. Mesoscale model predictions could be more accurate if more than 23 vertical layers were used. This relatively coarse resolution, however, was chosen because we wanted to save time in model simulations and file transferring among participating institutes during the real-time experiments 2. The model ran twice a day at 0000 and 1200 UTC, with each forecast extended to 36 h. The initial (first-guess field) and lateral boundary conditions were supplied by the objective analyses and forecasts of the CWB global spectral model. The initial data were first interpolated to the MM5 grid, and then an objective analysis procedure, based on successive correction (i.e., the Cressman scheme), was used to incorporate upper-air and surface observations. Such analyses were performed on the 45- km grid, which were then interpolated to the 15-km grid for model initialization. No further objective analysis was performed on the 15-km grid. All members of the ensemble system use the Medium- Range Forecast model (MRF) PBL parameterization (Hong and Pan 1996) to represent planetary boundary layer processes, including surface fluxes of heat, moisture, and momentum. The only difference between the members is found in the hydrological processes. The grid-resolvable explicit moisture scheme [or microphysics scheme (MS)] and the subgrid-scale convective 1 The sigma levels are as follows: 0.995, 0.985, 0.97, 0.945, 0.91, 0.87, 0.825, 0.775, 0.725, 0.675, 0.625, 0.575, 0.525, 0.475, 0.425, 0.375, 0.325, 0.275, 0.225, 0.175, 0.125, 0.075, The vertical coordinate was defined as (p p t )/(p s p t ), where p was pressure, p s was surface pressure, and p t was a constant pressure at the top of the model (100 hpa). 2 We compared model runs with different numbers of vertical levels and found that the difference between the runs with 23 layers and those with a higher vertical resolution is negligible for the purposes of this study.

4 738 WEATHER AND FORECASTING VOLUME 19 TABLE 1. The ensemble members and their hydrological settings. No. Member CPS MS KS* KG KR GR BR AR Kain Fritsch Kain Fritsch Kain Fritsch Grell Betts Miller Anthes Kuo Simple ice Goddard Reisner I Reisner I Reisner I Reisner I * Member acronym drawn from the combination of the first letters of CPS and MS scheme used. parameterization scheme (CPS) used by each member are listed in Table 1. The first three members (KS, KG, and KR) apply the same CPS (the Kain Fritsch scheme), but different MSs, namely, the simple ice scheme, the Goddard scheme, and the Reisner I microphysics scheme, respectively (see Grell et al for detailed descriptions of each scheme). Comparisons among these three members identified the impact of the different MSs on the rainfall forecast. The last four members (KR, GR, BR, and AR) utilized the same MS (the Reisner I scheme) but different CPSs, including the Kain Fritsch scheme, the Grell scheme, the Betts Miller scheme, and the Anthes Kuo scheme, respectively. This group of models served for comparing the performance among the four CPSs studied. Precipitation forecasts of the ensemble mean (hereafter, MEAN) were computed by simply averaging the rainfall forecasts from the six ensemble members. This was done in real time during the three mei-yu seasons, with the result providing additional information for the forecasters in the CWB. In this study, five other methods of creating the ensemble mean (or weighted mean) precipitation forecast were examined for the 2001 and 2002 mei-yu seasons. The purpose was to search for a better scheme in determining ensemble mean rainfall forecasts, based on existing simulated rainfall data, and to improve the operational rainfall forecast. The first method (WT1) calculated the ensemble-weighted mean rainfall by assigning different weightings for each member. For each single run, the weightings at each grid point were determined according to the performance of the member on rainfall forecasts of the A period (0 12 h). The best performing member obtained twice the weighting of the second, the second best received twice the weighting of the third, and so on. Each weighting was then normalized such that the summation of the weightings was one. These weightings were then used to compute the ensemble-weighted mean rainfall forecasts for the B (12 24 h) and C (24 36 h) periods of each simulation. The second method (WT2) was similar to WT1, but it determined the weightings for the B and C periods according to the performance of each member on rainfall forecasts during the corresponding forecast periods in the preceding year. As for the third method, the multiple linear regression technique (MLR1) was used to calculate weightings at each grid point. The weightings were determined by the best fit between simulated rainfall of the ensemble members and the observed rainfall, using the least squares method. They were then applied at each grid point to the same simulated data to obtain an ensemble-weighted mean precipitation forecast. Because it was not possible to have such weightings ready during the course of the rainfall forecast, MLR1 provided limited benefits to the real-time rainfall forecast. Another ensemble-weighted mean forecast (MLR2) was, therefore, created by applying the weightings from the multiple linear regression technique of the previous year. The last approach, probability matching, was adopted from Ebert (2001). The method (PM) first ranked the rain rates in MEAN from greatest to smallest, with the location of each value stored along with its rank. It then pooled the forecast rain rates of all six members, ranked them in order of greatest to smallest, and assigned every sixth value to the grid point of MEAN with a corresponding order in rain rate. This method, which assumed that MEAN gave a better spatial representation, while the extracted data from all members provided greater accuracy, could help reduce the underforecast problem typical of ensemble means at high thresholds and improve the event hit rate (Ebert 2001). 3. Rainfall observation The observational data used for precipitation verification were obtained from 342 rain gauge stations 3, spread geographically over the island of Taiwan (see Fig. 2 for locations). Generally speaking, the station density was relatively homogeneous over most of Taiwan, including the western slope of the CMR. However, the rain gauge stations over the ridgeline and eastern portions of the CMR were sparse and inhomogeneous, which might induce error. Figure 3 presents the observed 12-h accumulated precipitation amounts averaged for all stations, from 11 May to 22 June for the years 2000, 2001, and In general, there was an obvious common feature among the three mei-yu seasons: accumulated rainfall tended to be larger during the daytime ( UTC, LST) than during the nighttime ( UTC, LST) hours. This was because more convection could be generated by thermal effects during the daytime than the nighttime. The nighttime precipitation amounts became substantial only when major rainfall events, usually associated with mei-yu fronts, occurred. In 2000, rainfall events happened predominantly after 5 June, with the heaviest precipitation observed during June. In contrast, there was heavy precipitation in May for the 2001 mei-yu season; specifically, the primary events occurred during May and May. Another major rainfall event happened approximately on 14 June. The total 3 The stations used the tipping-bucket rain gauge for rainfall measurement, with a resolution of 0.5 mm.

5 AUGUST 2004 CHIEN AND JOU 739 FIG. 3. Twelve-hour accumulated precipitation averaged for the 342 rain gauge stations in Taiwan. Ending times are from 1200 UTC 11 May to 0000 UTC 22 Jun at a 12-h interval for 2000, 2001, and The bars in black denote rainfall accumulated from 0000 to 1200 UTC (daytime, LST), while those in gray represent rainfall accumulated from 1200 to 0000 UTC (nighttime, LST). rainfall over the island during the 2002 mei-yu season was below average, similar to that of Moderate amounts of precipitation were observed from mid-may to mid-june, with the heaviest rainfall during the daytime of 31 May. Figure 4 presents islandwide daytime and nighttime rainfall distributions accumulated for the entire mei-yu season of 2000, 2001, and All three seasons showed that there was more precipitation in daytime than in nighttime, as presented in Fig. 3. Furthermore, daytime precipitation appeared to take place mostly along the mountain slopes, while nighttime rainfall was more evenly distributed in the east west direction. This suggests that the mountain range played an important role in enhancing the daytime precipitation, but had relatively small influence on the nighttime precipitation. Such results are similar to those of the 1998 mei-yu season (Chien et al. 2002), and appear to be common for mei-yu seasons in the Taiwan area. Comparisons among these seasons showed the most rainfall in the 2001 mei-yu season, with the precipitation primarily occurring in the south. In the 2000 and 2002 mei-yu seasons, rainfall amounts were small and appeared lower than normal, resulting in mild droughts in Taiwan during those summers. 4. Precipitation verification a. The verification method In this study we verified precipitation forecasts of the nested domain (15 km) on the model grids for two fore-

6 740 WEATHER AND FORECASTING VOLUME 19 both subtracted by the expected number of hits 4 in a random forecast (R): H R ETS, (1) F O H R where H was the number of hits, and F and O were the numbers of samples in which the precipitation amounts were greater than the specified threshold in forecast and observation, respectively; and the random forecast R FO/N, where N was the total number of points being verified. The thresholds used in this study included 0.3, 2.5, 5, 10, 15, 25, 35, and 50 mm of precipitation. The bias scores were calculated by B F/O. For some unique events, a huge and meaningless bias could occur at high thresholds as a result of large F but very small O. They are excluded in the calculation, which happened more frequently when the season was relatively dry. b. The precipitation forecasts in 2000 FIG. 4. Observed 12-h rainfall (mm) accumulated from 11 May to 22 Jun for 2000, 2001, and 2002 (from left to right columns, respectively). (top) Daytime period ( UTC) and (bottom) nighttime period ( UTC). cast periods, including and h. Because the rain gauge stations were irregularly distributed around the grid points over the island, the observed 12-h accumulated rainfall data had to be processed before verification. Therefore, in order to provide rainfall observations at a resolution compatible with the model, data from the rain gauges located within a 15-km square, centered at each grid point, were averaged to represent the observed rainfall amount at that grid point. This value was then compared with the model precipitation amount at the same grid point, over the same time period. The result registered as a sample for verification. Because there were 141 grid points over the land area of Taiwan in the 15-km domain, each MM5 run would create, at most, 141 samples for a specified time interval. In some cases, fewer samples were obtained because of missing observational data or low rain gauge density over a certain region of the domain (particularly over high mountains and over eastern Taiwan). We used an equitable threat score (ETS; see Schaefer 1990) and bias score to verify the precipitation forecast of the ensemble system. The definition of ETS was the same as the standard threat score except that the numerator (H) and the denominator (F O H) were Figure 5 shows ETSs and bias scores computed from precipitation forecasts of all members and MEAN, as a function of rain threshold, for the forecast periods of B and C in the 2000 mei-yu season. It was found that for the B period the GR model in general had the best forecast skill among the six members, especially at medium to high thresholds, while the AR model performed the worst (Fig. 5a). The ETSs of MEAN were at about the 4th or 5th place for low thresholds ( 10 mm), and were almost the highest or the second highest for higher thresholds. The bias scores of the GR model appeared to be close to one for most thresholds (Fig. 5b). MEAN slightly overforecast rainfall at low thresholds, and predicted nearly the same precipitation occurrence as observed at higher thresholds. Performing quite uniquely, the BR model overforecast precipitation, especially at high thresholds. As for the C period, the top two members were the GR and the BR models (Fig. 5c). Because the BR model overforecast rainfall at medium to high thresholds (Fig. 5d), the GR model, with bias scores all around unity, was generally the best model among the six members. This, along with the result of the B period, suggested that the combination of the Grell CPS and the Reisner I MS was strongly recommended, among the six members, for MM5 rainfall simulations in the Taiwan area during a mei-yu season. The ETSs of MEAN were around the third place. Owing to the numerical averaging characteristic, MEAN usually overforecast precipitation at low thresholds and underforecast rainfall at high thresholds. 4 A hit was represented by both the observed and forecasted rainfall amounts greater than a specified threshold.

7 AUGUST 2004 CHIEN AND JOU 741 FIG. 5. The (left) ETS and (right) bias score vs various thresholds for (top) and (bottom) h precipitation forecasts, during the 2000 mei-yu season. Curves 1 6 denote scores for the six ensemble members: KS, KG, KR, GR, BR, and AR, respectively, while curve 7 represents scores for MEAN. c. The precipitation forecasts in 2001 In the 2001 mei-yu season, the verification showed similar features in terms of relative performance among members as those in the 2000 mei-yu season, although rainfall intensities of the two seasons were different as discussed in section 3. During the B period, the GR model had the highest ETSs among all members with bias scores close to one at nearly all thresholds (Figs. 6a,b). The BR model overpredicted rainfall at high thresholds (except at the highest threshold). The ETSs of MEAN were about the second highest among all the participants at the mm thresholds, and dropped to the third or the fourth highest at higher thresholds. As for the C period, among the six members the ETSs of the GR model were approximately the third highest at low thresholds ( 10 mm) and about the second highest at higher thresholds (Fig. 6c). The highest ETSs for rainfall thresholds higher than 10 mm belonged to the BR model. However, because its corresponding bias scores showed a tendency toward overprediction, the BR model might not have been the best-performing model overall (Fig. 6d). MEAN had the highest ETSs at low thresholds, and was only behind the BR model at nearly all high thresholds. d. The precipitation forecasts in 2002 Compared to the results of 2000 and 2001, the precipitation forecasts of the 2002 mei-yu season exhibited generally the lowest skill (Fig. 7). Even so, there were still some similarities among the 3 yr in terms of relative performance among members. First of all, Fig. 7a shows that during the B period the GR model performed the best at low thresholds and was approximately the second best at high thresholds. The best model at medium to high rain thresholds (15 50 mm) appeared to be the AR model, which, however, was generally the worst model in the previous years. The ETSs of the AR model were probably more consistent with the past two seasons at low thresholds (0.3 5 mm), where it remained one of the worst models. It was, thus, clear that there was possibility for a model to perform differently between different mei-yu seasons. This is consistent with Wang and Seaman (1997) and Gallus and Segal (2001), in which

8 742 WEATHER AND FORECASTING VOLUME 19 FIG. 6. Same as Fig. 5, except for the 2001 mei-yu season. they found that no single model configuration could do consistently the best for warm-season precipitation. It explains why we needed to create ensemble rainfall predictions instead of making forecasts from a single model. The ETSs of MEAN in this 12-h period did show its advantage over single models. The bias scores (Fig. 7b) exhibited a similar pattern as that of the 2000 meiyu season (Fig. 5b), except that rainfall was more underforecast at high thresholds. During the C period, the GR model performed quite well in comparison with other members, while the AR model had very low accuracy at nearly all thresholds (Fig. 7c). MEAN was approximately around the average among all members at the thresholds from 0.3 to 10 mm, and was above the average at higher thresholds. By any means, the ETSs were overall very low for this period in the 2002 mei-yu season. The bias scores for the C period (Fig. 7d) were similar to those of the B period. e. Comparisons among years Comparisons among the three mei-yu seasons (see Figs. 5 7) show that the ETSs in the B period of the 2000 season were unusually high at high thresholds (25 and 35 mm) almost for all members. This is different from the typical ETS distribution, as shown in other figures, that usually exhibited the decline of ETSs with increasing thresholds. During the 2000 mei-yu season, almost all heavy rainfall occurred in one event that lasted from 12 to 13 June (see Fig. 3). The models appeared to do relatively well for this event on predicting heavy rainfall in the B period, but not in the C period. As a result, the ETSs in the B period were exceptionally high at those higher thresholds. For the same reason, the bias scores of MEAN in the B period also showed an atypical pattern at the high thresholds, where the bias scores were slightly higher than one. It is also found that such a pattern happened only for this particular forecast period of the season. As seen from other figures, MEAN tended to have high bias scores at low thresholds and low bias scores at high thresholds, similar to Ebert (2001). It is noted that the bias scores at high thresholds were, in general, high in the 2000 and 2002 mei-yu seasons when the seasons were relatively dry, while they were low in the 2001 mei-yu season when the season was wet. This suggests that the models appeared to predict too much heavy rainfall during a relatively dry season.

9 AUGUST 2004 CHIEN AND JOU 743 FIG. 7. Same as Fig. 5, except for the 2002 mei-yu season. During a season when many heavy rainfall events occurred, however, the models tended to underpredict the precipitation. f. Comparisons among members Comparisons among the KS, KG, and KR models (see Figs. 5 7), which all used the Kain Fritsch CPS, show that they had approximately the same level of accuracy in terms of rainfall forecasts. In order to examine the spread of their rainfall forecasts, we arbitrarily picked three out of the six ensemble members and calculated their spread ratios at different rain thresholds (Wandishin et al. 2001). The result shows that the spreads of the KS, KG, and KR models were the smallest among all of the twenty possible combinations. In addition, the convective rainfall components of these models, as evident from the simulated rainfall data, were mostly larger than the grid-resolvable components. These together imply that the Kain Fritsch CPS dominated the rainfall process, with the contribution from the MSs being greatly limited. Comparisons between the KR and the GR models, which used different CPSs, further denote that the Kain Fritsch CPS generally underforecast rainfall for high thresholds, while the Grell CPS overall did not experience the same problem. The Kain Fritsch CPS, thus, appeared to produce most but not enough of the rainfall, resulting in underestimated total rainfall for heavy rain events. The ETSs of the BR model were not too bad overall; however, its bias scores were in general greater than one, especially at medium to high thresholds. This overforecast characteristic will prevent us from using it in an MM5 simulation in the Taiwan area during a mei-yu season. In this model, the Betts Miller CPS was used. It has a known problem of producing heavy spurious precipitation over warm water, and was greatly modified by Janjic (1994) for the NCEP Eta Model. Gallus and Segal (2001) used the modified scheme (the so-called Betts Miller Janjic CPS) to simulate convective rainfall systems and found that overprediction tended to occur at small thresholds, but not at the highest threshold. As a result, there were differences on bias scores at high thresholds between the two schemes, which reflect that the Betts Miller Janjic scheme is improved from the original Betts Miller scheme with heavy rain amounts. At low and medium thresholds, however, both schemes appear to predict too much rainfall, because they tend to create areas of rain-

10 744 WEATHER AND FORECASTING VOLUME 19 fall that are too large and too light in intensity. As for the AR model, it usually had the lowest ETSs. This is related to the fact that the Anthes Kuo CPS appeared to prevent grid-resolved precipitation, which is evident from the rainfall data that showed the ratios of convective to grid-resolved precipitation being around 80% 90%. From the above discussions, it is concluded that the GR model was the one that most consistently outperformed other members during all three mei-yu seasons. In order to further examine statistically the overall performance of the GR model, we combined the verified samples of the 2000, 2001, and 2002 mei-yu seasons, and computed a new set of ETSs and bias scores for the six members. The differences of ETSs between the GR model and each other member were then computed for the B and C periods. In addition, it is of interest to know how significant the differences are in a statistical point of view. We, thus, followed the resampling test method of Hamill (1999) to examine the significance level of each ETS difference. The ETS differences that reach a significance level of 90% were indicated by boldface, and those with a significance level of 95% were denoted by italic in Table 2. Furthermore, as discussed in Mason (1989), high ETSs are often obtained by having overly high bias scores. Therefore, in order to get a fair comparison between different model runs, we recomputed ETSs of the five other members by adjusting their bias scores to those of the GR model (see Hamill 1999) and obtained another set of ETS differences shown in parenthesis. As seen in Table 2, the ETSs of the GR model averaged for the three mei-yu seasons were above 0.2 for thresholds lower than (or equal to) 10 mm, and decreased toward higher thresholds in the B period. Those of the C period were, in general, about lower than in the B period. As shown in the ETS differences, it is clear that the GR model outperformed all five other models in the B period. Most of the differences at the 2.5-, 5.0-, and 10.0-mm thresholds reach the 95% significance level, while at other thresholds some of the differences reach the 90% significance level. With the bias adjustment technique, some of the differences became smaller, but many of them still reach at least the 90% significance level. As for the C period, the GR model still mostly outperformed other models, except for the BR model at thresholds higher than (or equal to) 10.0 mm. Because the BR model had a tendency of overprediction at large thresholds, the negative differences of ETS became smaller or even turned into positive values after their bias scores were adjusted to those of the GR model. 5. Ensemble mean and weighted mean precipitation forecasts As shown earlier, rainfall forecasts from MEAN had the advantage of performing well consistently at all TABLE 2. ETSs of the GR model and ETS differences between the GR model and each other model for eight thresholds (mm) and two forecast periods, averaged over the three mei-yu seasons of 2000, 2001, and Those resulting from bias adjustment to the GR model are shown in parentheses. Boldface indicates that the difference reaches a statistically significant level of 90%, the italic indicates significance at the 95% level. Threshold (0.106) (0.029) (0.020) (0.020) (0.017) (0.062) (0.142) (0.014) (0.017) (0.017) (0.037) (0.048) (0.168) (0.006) (0.015) (0.008) (0.040) (0.041) (0.180) (0.012) (0.021) (0.028) (0.034) (0.036) (0.208) (0.034) (0.031) (0.043) (0.036) (0.050) (0.237) (0.039) (0.034) (0.034) (0.044) (0.059) (0.244) (0.038) (0.041) (0.032) (0.047) (0.066) (0.226) (0.005) (0.013) (0.012) (0.028) (0.044) B period (12 24 h) GR GR KS GR KG GR KR GR BR GR AR (0.061) (0.035) (0.040) (0.029) ( 0.025) (0.039) (0.097) (0.019) (0.045) (0.039) ( 0.011) (0.043) (0.121) (0.018) (0.048) (0.034) ( 0.018) (0.041) (0.153) (0.033) (0.042) (0.040) (0.006) (0.056) (0.174) (0.044) (0.035) (0.045) (0.007) (0.063) (0.200) (0.039) (0.042) (0.034) (0.012) (0.069) (0.206) (0.024) (0.029) (0.020) (0.004) (0.059) (0.201) (0.000) ( 0.010) ( 0.005) (0.016) (0.038) C period (24 36 h) GR GR KS GR KG GR KR GR BR GR AR

11 AUGUST 2004 CHIEN AND JOU 745 FIG. 8. Same as Fig. 5, except for different ensemble mean and weighted mean forecasts for the 2001 mei-yu season. Shaded boxes denote the range of scores among members at the specified thresholds. Curve 1 represents scores for MEAN, which is the same as curve 7 in Fig. 6. Curves 2 6 are scores for WT1, WT2, MLR1, MLR2, and PM, respectively. thresholds, compared with those of the individual members. Its ETSs, however, were not the highest, but in general ranged only above the average among all participants. An immediate question arises: how can one produce an ensemble mean or weighted mean precipitation forecast to outperform forecasts from every single member, based on the existing rainfall forecasts? To answer this, we used the methods described in section 2 to produce ensemble mean (or weighted mean) precipitation forecasts for the 2001 and the 2002 mei-yu seasons. a. The 2001 mei-yu season Figure 8 presents the scores of the six ensemble mean (or weighted mean) rainfall forecasts (MEAN, WT1, WT2, MLR1, MLR2, and PM) and the score ranges of the ensemble members for the 2001 mei-yu season. During the B period, WT1 considerably outperformed MEAN at all thresholds (Fig. 8a). In addition, its ETSs were higher than each single member, except for the smallest threshold (0.3 mm). The bias scores of WT1 were close to those of MEAN at low thresholds and were better than MEAN (still lower than 1, though) at higher thresholds, such as 35 and 50 mm (Fig. 8b). WT2 generally performed worse than WT1, but it still outperformed MEAN, especially at medium and high thresholds. As for MLR1, the ETSs were the lowest among these six methods at low thresholds, and were the highest at thresholds higher than 10 mm. It was, thus, concluded that the multiple linear regression technique could improve the rainfall forecast when medium to heavy precipitation occurred. In contrast to MLR1, MLR2 exhibited the lowest ETSs at all thresholds, with all of them approximately in the lower bound of all ensemble members. Among all of the methods PM had the best bias scores (closest to 1), though it still had the underforecast problem at high thresholds. The ETSs of PM were mostly lower than those of MEAN, except at the lowest and the highest two thresholds. During the C period, the ETSs of WT1 were close to those of WT2 at low thresholds (Fig. 8c), and were slightly lower than WT2 at high thresholds (e.g., 35 and 50 mm). Because WT1 used information from the A

12 746 WEATHER AND FORECASTING VOLUME 19 period to determine weighting, it could obtain high ETSs during the B period because there was a close relation between periods A and B in terms of model performance. However, when the integration was longer, like in the C period, the relation of model performance with the A period became weaker. The ETSs of WT1 in this 12-h period were, therefore, not as impressive as in the B period. In contrast, WT2, which used weightings obtained from the 2000 mei-yu season, performed more consistently between periods B and C. As a result, WT2 was slightly better than WT1 during the C period, especially at high thresholds. Comparisons show that these two weighting methods outperformed MEAN, but only at thresholds higher than 10 mm. This is related to the fact that they predicted more rainfall than MEAN at low thresholds where overforecasting already occurred, but obtained a much better bias than MEAN at high thresholds (Fig. 8d). MLR1 had the highest ETSs among all members and all ensemble methods. It performed exceptionally well at high thresholds such as 25, 35, and 50 mm. The reason that the multiple linear regression technique helped create better rainfall predictions at medium to high thresholds was at least partly because the method allowed the summation of weightings of the six members to exceed one in some situations. 5 Because bias scores at those thresholds were still low, it implied that even larger weightings might be used. We did not do that, however, because it would result in a serious overforecast problem at low thresholds. Similar to those in the B period, the ETSs of MLR2 for the C period were also very low. Although the multiple linear regression method performed well, as shown in MLR1, its weightings appeared to be inadequate for another year s ensemble-weighted mean rainfall forecasts. It is, thus, concluded that the multiple linear regression technique used in this study was not ready for a real-time mesoscale rainfall forecast. Last, similar to that in the B period, PM tended to correct the bias problem of MEAN, and had better ETSs only at high thresholds. b. The 2002 mei-yu season In the 2002 mei-yu season, the ETSs of WT1 and WT2 were close to those of MEAN for nearly all thresholds during the B period (Fig. 9a). Because MEAN for this period already had very good scores compared with all ensemble members (also see Fig. 7a), the two methods appeared to barely perform better than MEAN. They had only a slight advantage at the thresholds of 10 and 15 mm. Although the ETSs were not much different between the two methods, it is still seen that WT1 performed slightly better than WT2 for the B period, similar 5 The weightings were obtained from the best fit between simulated rainfall of members and the observed rainfall, using the least squares method. In the case of underforecasting by all members, for example, the summation of weightings could exceed one. to that of the 2001 mei-yu season. The bias scores of WT1 and WT2 show that they overforecast rainfall at low thresholds, and underforecast precipitation at high thresholds (Fig. 9b). Similar to the result of the 2001 mei-yu season, MLR1 had the highest ETSs at all thresholds except at the 0.3-mm rainfall, while MLR2 had the lowest ETSs. This again indicates that applying weightings obtained from the multiple linear regression technique during the same season could increase the forecast skill of an ensemble-weighted mean forecast, but the weightings were not suitable for rainfall forecasts of other years. At low and high thresholds PM appeared to have better bias scores than MEAN. As for the C period, WT1 obtained higher ETSs than MEAN at most of the thresholds (Fig. 9c). The ETSs of WT2 were even higher. This, along with the results of the B period, again suggested that WT1 performed better than WT2 in the B period because the performance of the model was closely related between periods A and B. The improvement of WT1 over MEAN in the C period, however, decreased because of the longer integration, while that of WT2 maintained relatively well or even better. The performances of MLR1 and MLR2 were both similar to those in the B period. The ETSs of MLR1 were around the average among all participants at low thresholds from 0.3 to 5 mm, and they became the highest at higher thresholds. However, the improvement of MLR1 from other methods was not as impressive as in the B period. MLR2, once again, obtained ETSs that were approximately in the lower bound of all members. As for the bias scores, PM was the method that most consistently had scores close to one for all thresholds (Fig. 9d). 6. Discussion and summary This study presented precipitation verification in the Taiwan area for a real-time mesoscale ensemble prediction system, during the 2000, 2001, and 2002 meiyu seasons. This MM5 ensemble system consisted of six members, each run with a different combination of moisture physics schemes and maintained by a different university or government institution in Taiwan. Precipitation forecasts of the 15-km domain from all members were collected for verification against the observational data obtained from the 342 rain gauge stations on the island. There were a few common features among the three mei-yu seasons. First, accumulated rainfall tended to be greater during the daytime ( UTC) than during the nighttime ( UTC) hours. This was because of increased convection generated by daytime thermal effects. Furthermore, daytime precipitation appeared to take place mostly along the mountain slopes, while the nighttime rainfall was more evenly distributed in the east west direction. This suggests that the mountain range played an important role in enhancing the daytime precipitation, but had a relatively small influ-

13 AUGUST 2004 CHIEN AND JOU 747 FIG. 9. Same as Fig. 8, except for the 2002 mei-yu season. ence on the nighttime precipitation. Such results were similar to those of the 1998 mei-yu season (Chien et al. 2002), and appeared to be the common characteristics for a mei-yu season in the Taiwan area. Comparisons between the three seasons also showed that the 2001 mei-yu season had the most rainfall, with the precipitation mostly occurring in the south. In the 2000 and 2002 mei-yu seasons, rainfall amounts were small and appeared to be lower than normal. Although rainfall intensities and characteristics of the three seasons were different, there were some similarities in terms of relative performance between members. First of all, it was found that, in general, the GR model had the best forecast skill among the six members, while the AR model performed the worst. The combination of the Grell CPS and the Reisner I MS is, therefore, recommended, among the six members, for MM5 rainfall simulations in the Taiwan area during a mei-yu season. The ETSs of the ensemble mean were around or above the average and, in some cases, were the highest among all members. The ensemble mean usually overforecast precipitation at low thresholds and underforecast rainfall at high thresholds, because of the numerical averaging characteristics. Comparisons also indicate differences among the three mei-yu seasons. The ETSs in the B period of the 2000 mei-yu season were unusually high at high thresholds almost for all members. This is because it was relatively dry during the 2000 season, and almost all heavy rainfall occurred in one single event. The models appeared to do really well for this event on predicting heavy rainfall in the B period, but not in the C period. As a result, the ETSs in the B period were exceptionally high for heavy rainfall. It is also noted that the bias scores at high thresholds were in general high in the 2000 and 2002 mei-yu seasons when the seasons were relatively dry, while they were low in the 2001 mei-yu season when the season was wet. This suggests that the models appeared to predict too much heavy rainfall during a relatively dry season. During a wet season with many heavy rainfall events, however, the models tended to underpredict the precipitation. Comparisons among the members that all used the Kain Fritsch CPS show that this CPS dominated the rainfall process, with the contribution from the MSs being greatly limited. The KR model tended to underforecast rainfall because the Kain Fritsch CPS produced most, but not enough, of the rainfall. In the GR model

14 748 WEATHER AND FORECASTING VOLUME 19 the grid-resolved component became larger and it predicted better rainfall than the KR model. The bias scores of the BR model were, in general, greater than one, especially at medium to high thresholds. The version of the Betts Miller CPS that we used is different from the Betts Miller Janjic scheme that is currently used in the NCEP Eta Model. Unlike our results, the Betts Miller Janjic scheme tended to overforecast rainfall at low thresholds, but not at high thresholds. Such differences at high thresholds between these two schemes reflect that the Betts Miller Janjic scheme is improved from the original Betts Miller scheme with heavy rain amounts. At low and medium thresholds, however, both schemes appear to predict too much rainfall, because they tend to create areas of rainfall that are too large and too light in intensity. As for the AR model, it usually had the lowest ETSs because the Anthes Kuo CPS prevented grid-resolved precipitation. Because the ETSs of the ensemble mean were not usually the best, but instead ranged only above the average among all ensemble members, we intended to search for a better method of producing the ensemble mean rainfall forecast. WT1, which calculated rainfall by giving each member different weightings determined by model performance of each member in rainfall forecasts of the A period, generally outperformed the ensemble mean and all single members in the B period. For the same period, WT2, in which the weighting was determined according to the performance of each member in rainfall forecasts in the preceding year, generally performed worse than WT1; but it still outperformed the ensemble mean. The scenario was reversed in the C period, during which time the ETSs of WT1 were, in general, slightly lower than those of WT2. Because WT1 used information from the A period to determine the weighting, it could obtain high ETSs during the B period, because there was a close relationship between periods A and B in terms of model performance. However, when the integration was longer, like in the C period, the relation with the A period became weaker. The ETSs of WT1 in the C period were, thus, not as impressive compared to the ensemble mean as in the B period. They became lower than those of WT2. WT2 used weightings obtained from the previous mei-yu season and, therefore, avoided the problem present in WT1. As for MLR1, which utilized the multiple linear regression technique to calculate weightings at each grid point, the ETSs, except at low thresholds, were remarkably high compared with all other methods. The multiple linear regression technique could, therefore, improve the ensemble-weighted mean rainfall forecast when medium to heavy precipitation occurred. This is because the method allowed the summation of weightings of the six members to exceed one in some situations, which helped to predict better heavy rainfall than WT1 and WT2. Unfortunately, the ETSs of MLR2, which used weightings from the multiple linear regression method of the previous year and were, thus, meaningful for forecasting purposes, were predominantly very low. Therefore, although the method could help increase the accuracy of an ensemble-weighted mean rainfall forecast as shown in MLR1, weightings based on an earlier year appeared to be inadequate for the following year s rainfall forecasts. This was different from the result of Krishnamurti et al. (2001) who found that the multiple linear regression method is useful for a real-time ensemble-weighted mean rainfall forecast in a global scale after a period of training. It should be noted that Krishnamurti et al. (2001) used many meteorological parameters besides rainfall to compute the weightings, while we only considered precipitation in the regression process. MLR2, therefore, requires more studies in order to determine whether it is useful for real-time mesoscale ensemble-weighted mean rainfall forecasts. Combining rainfall data of multiple years (maybe at least 5 yr) to obtain a set of weightings may help and will be the first approach that we take in the future. Finally, we used PM, a method proposed by Ebert (2001), to produce the ensemble mean forecast, and found that the method had better bias scores than the ensemble mean. Its ETSs, however, were mostly lower than those of the ensemble mean, except at low and high thresholds. We conclude that WT1 and WT2 were the two methods that could be used, besides the ensemble mean, for creating a real-time deterministic ensemble rainfall forecast. In a practical sense, it should be straightforward to operate WT2 in a real-time mode, as long as the physics of each member is fixed among different years. WT1, however, requires a tight time schedule for making a real-time rainfall forecast. After the simulation is finished, the simulated rainfall of the A period from each member is compared with the observational precipitation to obtain weightings, and then the ensembleweighted mean rainfall forecasts of the B and the C periods are computed by applying these weightings. For an example of a simulation initialized at 0000 UTC, the MM5 simulations could usually start at as early as 0600 UTC, and probably finish at 0900 UTC. 6 Because the observed rainfall of the A period would be available shortly after 1200 UTC, owing to the fast transmission of data from rain gauge stations, the ensemble-weighted mean rainfall forecasts of the B and C periods could be obtained and broadcasted to the community by 1300 UTC. Finally, we would like to point out that there are many other possible physics combinations by just choosing from the CPSs and MSs available in the MM5. If other physics processes are considered, for instance, the PBL parameterization scheme, there could be an enormous number of possibilities. As for the number of members to be included in an ensemble system, it would strongly 6 This, of course, depends on the platform on which the model runs. For current computers, the estimated time given here should belong to the lowest limit.