A simple approach for combining seasonal forecasts for southern Africa

Transcription

1 Meteorol. Appl. 10, (2003) DOI: /S A simple approach for combining seasonal forecasts for southern Africa E. Klopper & W. A. Landman Meteorology Group, Department of Geography, Geoinformatics and Meteorology, University of Pretoria, Pretoria 0002, South Africa klopper@intekom.co.za Currently the South African Weather Service (SAWS) compiles seasonal rainfall and temperature outlooks by combining output from various models in a subjective manner. A group of scientists discusses the forecasts from a variety of models and then decides, from these discussions, what the strongest climate signal is. A map is drawn to represent the gut feeling from the specialists based on the results from the various models. Although the models used are all scientifically sound, the compilation of the outlook remains subjective. In order to compile a single probability forecast out of the many different model outputs an objective method has to be implemented. This paper proposes such a method. The results were tested during the December January February season over a retroactive period from 1991/92 to 1999/2000. Results show that a combination of different models consistently delivers a more skilful forecast than any individual model on its own. 1. Introduction Considerable progress has been made since the early 1990s to develop models that are able to predict the seasonal behaviour of the southern African climate (e.g. Cane et al. 1994; Hastenrath et al. 1995; Barnston et al. 1996; Mason et al. 1996, 1999; Mason 1998; Jury et al. 1999; Landman & Mason 1999; Landman & Goddard 2002). Currently a number of different operational models produce output that could potentially benefit the user. The end-user could therefore easily be confused by the vast amount of information available, especially in cases where different models produce contradicting forecasts. Users have repeatedly expressed problems with the confidence that can be attached to such forecasts. One way to overcome these problems is to try to develop more objective ways of consolidating the various model forecasts. The output from such methods should be verified in an operational environment to demonstrate their potential usefulness. By definition, forecast verification is the process of determining the quality and value of forecasts. Although a wide variety of forecast methods exists, all verification methods involve the comparison between matched pairs of forecasts and observations. Forecast verification involves investigation of the statistical properties of the joint distribution of forecasts and observations (Murphy & Winkler 1987). Ideally the association between the forecast and corresponding observation should be reasonably strong, and the nature and strength of this association are reflected in their joint distribution. Objective evaluation of forecast quality is undertaken for a variety of reasons. Brier & Allen (1951) categorised these as serving administrative, scientific (or diagnostic) and economic purposes. Administrative use pertains to the ongoing monitoring of operational forecasts in order to examine trends of forecast performance through time. Analysis of verification statistics can help to assess specific strengths and weaknesses of forecasters or forecasting systems. Ultimately the justification for any forecasting enterprise is that it supports better decisionmaking, and the usefulness of forecasts in this regard depends on the error characteristics of the forecast. Thus, to derive full value from forecasts and enable estimation of the economic value, this information is needed. The aim of this paper is to set up a system by which model output can be combined objectively to produce a single forecast for the end-user. First, the different operational forecast models used to predict seasonal rainfall for southern Africa are described. A system is proposed to produce a combined forecast from output of these models. Results from this method are then evaluated over an independent test period from 1991/92 to 1999/2000 for the austral summer months (December January February) of southern Africa. 2. Consolidation of forecasts The combination of two or more skilful but independent predictions of the same event will yield a prediction that is on average more accurate than any of them 319

2 E. Klopper & W. A. Landman 9 northern Namibia/ western Botswana south-western Cape 1 8 western Interior 7 central Interior 2 south coast 6 northeastern Interior 3 5 Transkei Lowveld 4 Kwa Zulu Natal coast Figure 1. Location map of the nine homogeneous rainfall regions over southern Africa. Countries shaded grey are not included. taken individually (Leith 1974; Thompson 1977). In order to combine output from various models, some preconditions should be stipulated. The output from different models should be compatible in the sense that it covers the same regions and time periods. These preconditions are described first, followed by discussion of the models that are operational at the South African Weather Service (SAWS) and have similar output features that meet these preconditions Preconditions Homogeneous rainfall regions Regional rainfall indices for December January February (DJF) rainfall totals from stations over South Africa, Lesotho, Swaziland, Namibia and Botswana have been calculated for a 43-year period (1957/58 to 1999/2000). The raw rainfall data were obtained from the South African Weather Service and the National Meteorological Services of the other individual countries (Lesotho, Botswana and Namibia). The nine homogeneous regions depicted in Figure 1 were defined from an analysis of the indices (Mason 1998; Landman & Mason 1999). The regions are defined as: (1) southwestern Cape; (2) south coast; (3) Transkei; (4) KwaZulu-Natal coast; (5) Lowveld; (6) northeastern interior; (7) central interior; (8) western interior; (9) northern Namibia/western Botswana. A complete set of rainfall indices is available for all three-month seasons for these nine regions Forecast period 320 Operational forecasts are made for any three consecutive months with lead times of zero to six months. Predictions are for three equi-probable categories (below-normal, near-normal and abovenormal). This study will focus on the DJF period that has been identified as the main rainfall months for most of the summer rainfall region (regions 3 to 9) (Tyson & Preston-Whyte 2000) Seasonal prediction models Three operational models are used in this study. They all fulfil the two preconditions specified. The models are a linear statistical model (canonical correlation analysis) (Landman & Mason 1999), a non-linear statistical model (quadratic discriminant analysis) (Mason 1998) and output from a Model Output Statistics (MOS) recalibrated atmospheric general circulation model (Landman & Goddard 2002) Canonical correlation analysis model Canonical correlation analysis (CCA) is a statistical technique often used as a forecasting tool (Barnett & Preisendorfer 1987; Graham et al. 1987a, 1987b; Barnston & Ropelewski 1992; Barnston 1994; Chu & He 1994; Barnston & Smith 1996; Shabber & Barnston 1996; Landman & Klopper 1998). The model that is used operationally is constructed to relate globalscale sea-surface temperatures (SST) to regional rainfall indices for the nine homogeneous regions depicted in Figure 1. The predictor field consists of globalscale mean SSTs for four consecutive, non-overlapping three-month periods (for example, JFM, AMJ, JAS and OND). SSTs are used in this way as predictor in order to capture evolutionary and steady-state features included in the internal memory of the climate system (Palmer & Anderson 1994). In addition, there is a significant association between the ocean and atmosphere affecting southern Africa (Nicholson & Entekhabi 1987; Walker 1990; Jury & Pathack 1991; Mason et al. 1994; Mason 1995; Mason & Jury 1997; Rocha & Simmonds 1997; Reason & Lutjeharms 1998; Reason 1999). The predictand field consists of three-month regional rainfall indices. Optimal CCA equations are constructed for a cross-validation period of several decades Quadratic discriminant analysis model A quadratic discriminant analysis (QDA) model was constructed to produce three-month rainfall forecasts for southern Africa (Mason 1998). The model relates rainfall over the predefined homogeneous regions of southern Africa to principal components of sea-surface temperature in the Indian, South Atlantic and Pacific Oceans. The general problem addressed by discriminant analysis is to assign an individual variable to a category, based on the values of the vector of independent variables. Once the discriminant functions are defined from the data for the training period, a new vector of observations on the independent variables can be submitted. In practice, the model will then define

3 Combining seasonal forecasts for southern Africa the probabilities of three-month rainfall totals falling within each of the three categories, given the sea-surface temperature fields averaged over the previous two months. The model is validated using perfect prognosis (Wilks 1995) and jack-knife validation approaches (Michaelsen 1987) Atmospheric general circulation model The atmospheric general circulation model (AGCM) used in this study is the ECHAM 3.6 AGCM (Deutsches Klimarechenzentrum 1992). The ECHAM climate model was developed from the European Centre for Medium-range Weather Forecasting (ECMWF) model with changes for climate simulations by a comprehensive parametrisation package developed at Hamburg. The data incorporated in this study are from the ECHAM 3.6 version of the model run in hindcast mode for hindcasts, the prescribed SSTs are obtained by persisting the previous month s SST anomalies through the forecast period. A set of CCA equations were designed to relate simulated large-scale circulation patterns to regional rainfall using a model output statistics (MOS) approach (Wilks 1995; Landman & Goddard 2002) Forecast format The type of forecast plays a major role in determining the way in which it should be evaluated. In the main, two types of forecasts are possible when seasonal forecasts are considered, namely deterministic and probabilistic. Deterministic means that the forecast consists of an unqualified statement that only one of a set of possible events will occur. It contains no expression of uncertainty, in contrast to probabilistic forecasts (Wilks 1995). In many cases deterministic forecasts are derived by converting the underlying probabilistic forecasts to the categorical format. This inevitably results in the loss of information, to the detriment of the forecast user. Fundamentally the conversion from probabilistic to deterministic forecasts requires the selection of a threshold probability (Wilks 1995). The proper threshold depends on the user and the particular decision problems to which that user will apply the information. Since different decision problems will require different thresholds, this conversion amounts to the forecaster making the decisions for users without knowing the particulars of the problem. Necessarily then, the conversion from a probability to a deterministic forecast is arbitrary. Seasonal climate is inherently probabilistic. Therefore, probability forecasts are more informative and potentially more useful than either unmodified deterministic forecasts (precipitation vs. no precipitation) or deterministic forecasts with qualitative indications of uncertainty (chance of precipitation). Unlike deterministic forecasts, probability forecasts provide a quantitative measure of uncertainty that is less ambiguous than qualitative indications. Moreover, in the modelling of decision-making problems under uncertainty, probability forecasts not only represent uncertainty but are used in the calculation of various measures of interest such as payoffs and expected utilities. Therefore, two principal reasons exist for formulating and expressing forecasts in probabilistic terms. First, uncertainties are inherent in the forecasting process. This implies that deterministic forecasts will seldom accurately reflect the true state of knowledge of the forecasting system (whether this is an objective model or human forecaster). Secondly, in the absence of perfect forecasts, users require estimates of the likelihood of occurrence of the relevant events in order to make optimal decisions in uncertain situations. The value of probabilistic forecasts generally is equal to or greater than the value of deterministic forecasts for all users of such forecasts (Murphy 1977; Krzysztofowicz 1983). In this study probabilistic forecasts will thus be compiled and evaluated Method of consolidation of probabilistic forecasts In order to compile a single consolidated forecast out of the many different model outputs available, an objective method has to be implemented. The method proposed here first of all constructs retroactive model runs for a period (10 years) from each individual model. Observed rainfall categories and forecast categories were obtained for the CCA, QDA and ECHAM 3.6 MOS models for DJF 1981/82 to 1999/2000 for a 0-lead (i.e. the forecast directly follows the observed SST fields in this case the latest SST data is for November and the forecast is made for the DJF season). From this, contingency tables are constructed to represent the chance of a probability in favour or against a certain result or category. These odds or likelihood of occurrence as produced by each model will be used in the process to construct a single combined forecast Construction of the likelihood of occurrence To construct the likelihood of occurrence, the most recent 10 years of independent model forecasts are used. Using 10 years ensures that the model climate (i.e. the period prior to the retroactive period) is still adequate to train the statistical models, and that a variety of both ENSO and non-enso years are incorporated in compiling the likelihood of occurrence. The scheme in Figure 2 explains the time frames used to construct the likelihood of occurrence. Retroactive forecasts for the three individual models described earlier are obtained for a 10-year period, and used to predict rainfall probabilities for the next three summer 321

4 E. Klopper & W. A. Landman Figure 2. The years in blocks indicate the 10 years from which the odds are derived to predict probabilities for the three years on the right-hand side. Table 1. Contingency tables for DJF for homogeneous rainfall region 7 (central interior) calculated over a 10-year period (1987/88 to 1996/97) from the CCA, QDA and MOS models respectively. The table rows depict the forecasts, and the columns the observations. The values are percentages (%). CCA QDA MOS A N B A N B A N B A A A N N N B B B seasons. For example, the likelihood of occurrence are determined from the 1981/82 to 1990/91 retroactive forecasts and used to predict probabilities for the following three seasons (1991/92, 1992/93 and 1993/94). The last three seasons are then added while the first three years are removed from the retroactive set of forecasts, and a new set of likelihood of occurrence values are constructed to produce probabilities for the next three seasons. The sets are updated every three years to ensure the most recent data are used for the construction of the likelihood of occurrence values and to ensure that the forecasts remain equally probable. The likelihood of occurrence values are determined by counting the number of hits and misses per forecast category and compiling contingency tables. As an example, Table 1 depicts the contingencies for homogeneous region 7 (the central interior) for each of the three models over the 10-year period from 1987/88 to 1996/97. Using the likelihood of occurrence values, retroactive forecasts for the 1990s are constructed. The approach could explicitly be defined by: Pr{ ˆX = x} = 1 m Pr{X = x ˆX i = ˆx i } m i=1 where m is the number of models, ˆx i is the forecast of the ith model, and x is the forecast category of the combined forecast. An example illustrates the procedure that follows: Example: If for instance the following operational categorical forecasts for each model are made for region 7 for the 1997/98 DJF season: Model 1 CCA: Below-normal Model 2 QDA: Below-normal Model 3 MOS: Near-normal The corresponding likelihood of occurrence for each model is derived from Table 1: A N B CCA: QDA: MOS: By calculating the mean for each category, the result is a probabilistic forecast of A (46.7%), N (23.3%) and B (30%). Therefore, although none of the individual models in this example predicted above-normal rainfall, the combined (or consolidated) forecast gave the highest probability to the above-normal category. The eye-ball method currently used would not indicate the preferred category to be above-normal. By combining different models objectively, some of the biases of the models are being taken into account. Also, discrepancies between models are being catered for in most instances. This will become even more apparent when additional models are introduced to the process. Table 2 shows the individual and combined forecasts, as well as the observed categories for the retroactive Table 2. CCA, QDA, MOS and combined retroactive probability forecasts for 1991/92 to 1999/2000 for region 7. The observed category is indicated in the last column. A: above-normal; N: near-normal; B: below-normal. CCA QDA MOS Combined Season A N B A N B A N B A N B Observed 91/ B 92/ B 93/ A 94/ B 95/ A 96/ N 97/ N 98/ N 99/ A 322

5 Combining seasonal forecasts for southern Africa period 1991/92 to 1999/2000 for region 7 following the procedure illustrated above. Region 7 is shown because of its known strong association with the El Niño/Southern Oscillation phenomenon, which is a major contributor to seasonal forecast variability in southern Africa (Lindesay 1988; Ropelewski & Halpert 1987). Tables were constructed for all nine regions for all the seasons but are not shown here. This objectively calculated forecast can serve as a first guess to the operational forecast that might also include model output that is not for the regions specified here. The final outlook issued to the end-users will also be based on the forecaster s interpretation of the other models output. In the case of more models becoming available that are compatible in the way demonstrated here, the process will become more objective since fewer model forecasts would have to be subjectively included in the final forecast. This simple procedure is proposed for operational use at the South African Weather Service. 3. Forecast verification Having described the different operational forecast models and the combination of their output, these consolidated forecasts must be verified to test their validity and potential usefulness. An evaluation of the consolidated forecasts is subsequently described for the retroactive test period from 1991/92 to 1999/ Selection of verification measures Evaluation procedures depend on the nature of the predictand and on the type of forecast. Verification of probability forecasts is somewhat subtler than verification of deterministic forecasts. The latter have no expression of uncertainty; thus it is clear when an individual forecast is correct or not. Unless a probability forecast is either 0.0 or 1.0 the situation is less clearcut. For probability forecasts, a single forecast is neither right nor wrong, only useful or not useful. Either way, it is the joint distribution of forecasts and observations that contains the relevant information for forecast verification. Verification of probability forecasts traditionally has consisted largely of the computation of a few overall performance measures such as the Brier Score (Brier 1950) or scores based on other scoring rules, or through graphical analysis. These traditional practices are helpful in the evaluation of probability forecasts, but they clearly are deficient when the objective is either to identify the fundamental strengths and weaknesses in probability forecasts, or to provide modellers or forecasters with feedback as a basis for improving forecast performance. Moreover, the needs of users for information regarding the quality of probability forecasts are not met by overall performance measures. Having described the different operational forecast models and the combination of their output, the forecasts must be verified to test their validity and potential usefulness. Verification methods used in this study will follow the methods and guidelines used by Ward et al. (1998) to evaluate the forecasts produced at the Southern African Regional Climate Outlook Forum (SARCOF). The SARCOF forecasts are similar to those presented here in the sense that they are probabilistic, for three equi-probable categories, and the forecasts are made for relatively large homogeneous regions. The suite of verification methods used includes Hit Rate (HR), Heidke Skill scores and Linear Error in Probability Space scores (LEPS). The local significance of all these scores is calculated using a Monte Carlo approach (Livezey & Chen 1983; Wilks 1995). In addition, the Ranked Probability Score (RPS) (Wilks 2000) was also calculated. These measures are described below: Hit Rate The hit rate is the simplest of all verification methods (Ward & Folland 1991; Barnston 1992) and indicates how accurate a forecast is. If the forecast category agrees with the observed category, the forecast scores one point. Sometimes two categories were given equal probabilities (e.g. near-normal and below-normal) because in practice the forecast could not distinguish between the likelihood of two categories occurring. A half-hit can be appointed to such a forecast if one of the two categories was observed. The hit rate is simply the sum of points scored divided by the total number of forecasts Heidke Hit Skill score In a tercile forecast system, a large number of random forecasts will have an expected hit rate of 33.3%. The hit rate can be adjusted to a score that has a chance value of 0 (Heidke 1926). A score of +100% will indicate a set of perfect hits and a score of 100% a set with no hits Linear Error in Probability Space (LEPS) scores Another technique to evaluate independent forecasts is known as Linear Error in Probability Space scores (LEPS) (Potts et al. 1996). LEPS scoring is regarded as the preferred score for deterministic type of forecasts (Barnston 1992; Livezey 1995) since it aims to measure the error in a forecast according to the distance between the position of the forecast and the corresponding observation. This method operates in a similar way to the Heidke Hit Skill score, but penalises more heavily 323

6 E. Klopper & W. A. Landman Figure 3. Hit rate of three models and the combined forecasts per homogeneous region over the retroactive period 1991/92 to 1999/2000. a forecast that is two categories in error than a forecast that is one category in error Ranked Probability Skill score The ranked probability score (RPSS) measures the cumulative squared differences between forecast probabilities and the observed category vectors (Murphy 1971; Wilks 1995). This is an extension of the Brier score to describe multiple events. The error between the forecast probabilities of all three categories and the observed category are computed. In this case the ranked probability skill score (RPSS) is used, which is a skill score for a collection of RPS values relative to the RPS obtained from climatological probabilities. A RPSS of 1 will indicate a collection of perfect forecasts, while 0 indicates a set of perpetual forecasts Verification of retroactive forecasts Figure 4. Hit rate of three models and the combined forecasts over the retroactive period 1991/92 to 1999/2000. The number of hits per season is shown in Figure 4. The maximum number per season is again nine, this time referring to nine homogeneous regions. Analysis of the graph shows that the combined forecasts score better than at least two of the individual models. Overall, the combined forecast once again scored more hits than any of the other models during five out of the nine retroactive seasons. From this simple comparison it is already evident that the combination of the three different models produces improved forecasts. To evaluate the combined forecast over the nine-year retroactive period the evaluation methods describe in Section 2.4 are subsequently applied and discussed. The verification scores for the combined probability forecasts over the retroactive period are shown in Figure 5. The scores are for forecasts produced at the beginning of December for the DJF season, here referred As described earlier, three models are combined to obtain a consolidated forecast the CCA and nonlinear QDA statistical models, and a model output statistics (MOS) forecast system that uses output from the ECHAM 3.6 general circulation model. Retroactive consolidated seasonal forecasts for all nine homogeneous regions are evaluated over the period from 1991/92 to 1999/2000. A comparison between each individual model and the combined deterministic forecast is depicted in Figure 3. The number of hits (correctly forecast categories) per region over the retroactive period is calculated. The maximum number of hits per region is nine (nine years are evaluated). From the graph the combined forecast scores better than at least two of the other models per region. In some cases, such as region 9, the combined forecast are correct for six of the nine years, while each of the other models received four or fewer hits. In general, the combined forecast performs better than any individual model. 324 Figure 5. Skill scores for each of the nine homogeneous regions: (a) RPSS, (b) LEPS, (c) Hit Rate, and (d) Heidke Skill Scores. Hatched areas indicate significant scores at the 95% level for LEPS, Hit Rate and Heidke Skill Scores, and positive values for RPSS.

7 Combining seasonal forecasts for southern Africa analyses it can be concluded that the combined forecasts performed generally better than any of the individual forecasts. Improved skill was found for all the summer rainfall regions, except for the Lowveld (region 5). Figure 6. RPSS of the combined forecast and each individual model for nine regions. to as a 0-month lead-time. The CCA model produced forecast categories over the retroactive forecast period that are stable with increasing lead-time which suggests that for years associated with accurate categorised deterministic rainfall forecasts at the 0-month lead-time, the forecast categories produced at increasing lead-time did not vary. The hatched areas on the maps in Figure 5 indicate significant scores (or positive scores in the case of RPSS). Significant scores are found over the central parts of the country. The winter and all year rainfall areas (regions 1 and 2) have no skill during the mid-summer months. An area of concern is the Lowveld (region 5) where no significant skill scores are present during the retroactive test period. In Figure 6 a comparison is drawn between the RPSS scores of each individual model and the combined model for the nine regions. The CCA, MOS and combined forecasts compare well over the summer rainfall regions. Region 5 (Lowveld) received a low score from all the models. 4. Summary and conclusions A method has been proposed to combine output from three operational seasonal rainfall forecast models for southern Africa into a consolidated forecast. The need for such a combined forecast arises from the huge amount of information, sometimes contradicting, regarding the seasonal climate of the region that is available to the user. The method used here is a simple, unweighted average of the conditional probabilities of each model (Mason & Mimmack 2002), which are assumed to be equally probable. More comprehensive methods are needed to optimally combine model output (e.g. Rajagopalan et al. 2002). The three models presented here did not produce equally accurate forecasts during each of the DJF seasons, indicating the need to combine them into one consolidated forecast. Such a consolidated forecast has incorporated the strengths of the different forecast models involved and has subsequently improved on the forecast skill of each individual model. From the DJF total rainfall forecasts for southern Africa have been presented over a nine-year retroactive period starting in 1991/92. Three of the nine years evaluated are El Niño events, namely 1991/92, 1994/95 and 1997/98. The latter event is considered to be the warmest El Niño event on record. Also, La Niña events occurred in three years (1995/96, 1998/99 and 1999/2000). It was also during these years that most of the rainfall forecasts have been found to be accurate. The only exception is the 1998/99 La Niña where none of the evaluation scores is significant at the 95% level of confidence. The worst scores are found for a non-enso season, 1992/93, while the highest scores are for the 1999/2000 La Niña season. The proposed procedure is thus able to capture rainfall scenarios sufficiently during ENSO events. This evaluation shows the proposed objective method of consolidation results in skilful, thus potentially useful, forecasts for southern Africa over the retroactive test period. Even though forecasts associated with longer lead-times may have more economic value, the highest accuracy is expected for the shortest lead. In an operational environment forecasts with varying leadtimes are available for all three models. The results have indicated that even a simple combination of forecasts of the three independent models yield a prediction that is on average more accurate than any of the individual models on their own. In addition, the consolidated forecasts presented here could further aid those in the forecast office by eliminating a significant amount of human error associated with the subjective combination of forecasts from different models. The consolidated forecasts produced evaluation statistics that are acceptable and useable in the real world. Acknowledgements The International Research Institute for Climate Prediction supplied data from the ECHAM3.6 GCM, while observed data and output from the CCA and QDA models were obtained from the South African Weather Service. References Barnett, T. P. & Preisendorfer, R. W. (1987) Origins and levels of monthly and seasonal forecast skill for United States air temperature determined by canonical correlation analysis. Mon. Wea. Rev. 115: Barnston, A. G. (1992) Long-lead forecasts of time-mean US surface temperature using canonical correlation analysis. Proceedings of the 16th Annual Climate Diagnostic Workshop, October (1991), Los Angeles, California, pp

8 E. Klopper & W. A. Landman Barnston, A. G. (1994) Linear statistical short-term climate predictive skill in the Northern hemisphere. J. Climate 7: Barnston, A. G. & Ropelewski, C. F. (1992) Prediction of ENSO using canonical correlation analysis. J. Climate 5: Barnston, A. G. & Smith, T. M. (1996) Specification and prediction of global surface temperature and precipitation from global SST using CCA. J. Climate. 9: Barnston, A. G., Thiao, W. & Kumar, V. (1996) Long-lead forecasts of seasonal precipitation in Africa using CCA Wea. and Forecasting 11: Brier, G. W. (1950) Verification of forecasts expressed in terms of probability. Mon. Wea. Rev. 78: 1 3. Brier, G. W. & Allen, R. A. (1951) Verification of weather forecasts. In: Compendium of Meteorology, T. F. Malone (ed.), Boston: American Meteorological Society, pp Cane, M. A., Eshel, G. & Buckland, R. W. (1994) Forecasting Zimbabwian maize yield using eastern equatorial Pacific sea surface temperature. Nature 370: Chu, P.-S. & He, Y. (1994) Long-range prediction of Hawaiian winter rainfall using canonical correlation analysis. Int. J. Climatology 14: Deutches Klimarechenzentrum (DKRZ) (1992) The ECHAM3 Atmospheric General Circulation Model. Technical Report 6, Hamburg, Germany, 184 pp. Graham, N. E., Michaelsen, J. & Barnett, T. P. (1987a) An investigation of the El Niño-Southern Oscillation cycle with statistical models. 1. Predictor field characteristics. J. Geophys. Res. 92: Graham, N. E., Michaelsen, J. & Barnett, T. P. (1987b) An investigation of the El Niño-Southern Oscillation cycle with statistical models. 2 Model results J. Geophys. Res. 92: Hastenrath, S., Greichar, L. & van Heerden, J. (1995) Prediction of summer rainfall over South Africa. J. Climate 8: Heidke, P. (1926) Berechung des Erfolges und der Gute der Windstarkevorhersagen im strumwarnungsdienst. Geografisika Annaler 8: Jury, M. R., Mulenga, H. M. & Mason, S. J. (1999) Exploratory long-range models to estimate summer climate variability over southern Africa. J. Climate 12: Jury, M. R. & Pathack, B. M. R. (1991) A study of climate and weather variability over the tropical southwest Indian Ocean. Meteorol. Atmos. Physics 47: Krzysztofowicz, R. (1983) Why should a forecaster and decision maker use Bayes theorem? Water Resources Research 19: Landman, W. A. & Goddard, L. (2002) Statistical recalibration of GCM forecasts over southern Africa using model output statistics. J. Climate 15: Landman, W. A. & Klopper, E. (1998) 15-year simulation of the December to March rainfall season of the 1980s and 1990s using canonical correlation analysis (CCA). Water SA 24: Landman, W. A. & Mason, S. J. (1999) Operational longlead prediction of South African rainfall using canonical correlation analysis. Int. J. Climatol. 19: Leith, C. E. (1974) Theoretical skill of Monte Carlo forecasts. Mon. Wea. Rev. 6: Lindesay, J. A. (1988) South African rainfall: the Southern Hemisphere semi-annual cycle. J. Climatol. 8: Livezey, R. E. (1995) The evaluation of forecasts. In: Analysis of Climate Variability. Applications of Statistical Techniques, H. von Storch and A. Navarra (eds.), Springer-Verslag: Berlin, Ch. 10, pp Livezey, R. E. & Chen, W. Y. (1983) Statistical field significance and its determination by Monte Carlo techniques. Mon. Wea. Rev. 111: Mason, S. J. (1995) Sea-surface temperature South African rainfall associations, Int. J. Climatol. 15: Mason, S. J. (1998) Seasonal forecasting of South African rainfall using a non-linear discriminant analysis model. Int. J. Climatol. 18: Mason, S. J. & Jury, M. R. (1997) Climate variability and change over southern Africa: a reflection on underlying processes. Progress in Physical Geography 21: Mason, S. J. & Mimmack, G. M. (2002) Comparison of some statistical methods of probabilistic forecasting of ENSO. J. Climate 15: 8 29 Mason, S. J., Goddard, L., Graham, N. E., Yelaeva, E., Sun, L. & Arkin, P. A. (1999) The IRI seasonal climate prediction system and the1997/98elniño event. Bull. Amer. Meteorol. Soc. 80: Mason, S. J., Lindesay, J. A. & Tyson, P. D. (1994) Simulating drought in southern Africa using sea surface temperature variations. Water SA 20: Mason, S. J., Joubert, A. M., Cosijn, C. & Crimp, S. J. (1996) Review of seasonal forecasting techniques and their applicability to Southern Africa. Water SA 22: Michaelsen, J. (1987) Cross-validation in statistical forecast models. J. Climate Appl. Meteorol. 26: Murphy, A. H. (1977) The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation. Mon. Wea. Rev. 105: Murphy, A. H. & Winkler, R. L. (1987) A general framework for forecast verification. Mon. Wea. Rev. 115: Nicholson, S. E. & Entekhabi, D. (1987) Rainfall variability in equatorial and southern Africa: relationships with sea surface temperatures along the southwestern coast of Africa, Int. J. Climatol. 17: Palmer, T. N. and Anderson, D. L. T. (1994) The prospects of seasonal forecasting a review paper. Quart J. R. Met. Soc. 120: Potts, J. M., Folland, C. K., Jolliffe, I. T. & Sexton, D. (1996) Revised LEPS scores for assessing climate model simulations and long-range forecasts. J. Climate 9: Rajagopalan, B., Lall, U. & Zebiak, Z. E. (2002) Categorical climate forecasts through regularization and optimal combination of multiple GCM ensembles. Mon. Wea. Rev. 130: Reason, C. J. C. (1999) Interannual warm and cool events in the subtropical/mid-latitude south Indian Ocean region. Geophys. Res. Lett. 26: Reason, C. J. C. & Lutjeharms, J. R. E. (1998) Variability of the south Indian Ocean and implications for southern African rainfall. SA J. Science 94: Rocha, A. & Simmonds, I. (1997) Interannual variability of south-eastern African summer rainfall. Part I: Relationships with air sea interaction processes. Int. J. Climatol. 17: Ropelewski, C. F. & Halpert, M. S. (1987) Global and regional precipitation patterns associated with the El Niño/Southern Oscillation. Mon. Wea. Rev. 115:

9 Combining seasonal forecasts for southern Africa Shabber, A. & Barnston, A. G. (1996) Skill of seasonal climate forecasts in Canada using canonical correlation analysis. Mon. Wea. Rev. 124: Thompson, P. D. (1977) How to improve accuracy by combining independent forecasts.mon. Wea. Rev. 105: Tyson, P. D. & Preston-Whyte, R. A. (2000) The Weather and Climate of Southern Africa. Oxford University Press: Cape Town, 396 pp. Walker, N. D. (1990) Links between South African summer rainfall and temperature variability of the Agulhas and Benguela currents systems J. Geophys. Res. 95: Ward, M. N. & Folland, C. K. (1991) Prediction of seasonal rainfall in the north Nordeste of Brazil using eigenvectors of sea-surface temperature Int. J. Climatol. 11: Ward, M. N., Unganai, L. S. & Garanganga, B. J. (1998) Verification of the 1997/98 ENSARCOF seasonal rainfall outlook maps: issues, methodology and results. In: Second report of the ENRICH Southern Africa Regional Climate Outlook Forum to the European Commission, M. Harrison (ed.), September 1998, Meteorological Office, UK. Wilks, D. S. (1995): Statistical Methods in the Atmospheric Sciences, Academic Press: San Diego, 467 pp. Wilks, D. S. (2000) On interpretation of probabilistic climate forecasts. J. Climate 13: Winkler, R. L., Murphy, A. H. & Katz, R. W. (1983) The value of climate information: a decision-analystic approach. J. Climatol. 3: