Simple robust averages of forecasts: Some empirical results

Transcription

1 Available online at International Journal of Forecasting 24 (2008) Simple robust averages of forecasts: Some empirical results Victor Richmond R. Jose, Robert L. Winkler 1 The Fuqua School of Business, Duke University, Box 90120, Durham, NC , USA Abstract An extensive body of literature has shown that combining forecasts can improve forecast accuracy, and that a simple average of the forecasts (the mean) often does better than more complex combining schemes. The fact that the mean is sensitive to extreme values suggests that deleting such values or reducing their extremity might be worthwhile. We study the performance of two simple robust methods, trimmed and Winsorized means, which are easy to use and understand. For the data sets we consider, they provide forecasts which are slightly more accurate than the mean, and reduce the risk of high errors. Our results suggest that moderate trimming of 10 30% or Winsorizing of 15 45% of the forecasts can provide improved combined forecasts, with more trimming or Winsorizing being indicated when there is more variability among the individual forecasts. There are some differences in the performance of the trimmed and Winsorized means, but overall such differences are not large International Institute of Forecasters. Published by Elsevier B.V. All rights reserved. Keywords: Combining forecast; Robustness; Trimming; Winsorizing; M3 Competition 1. Introduction Research in the area of forecast aggregation has shown that combining forecasts generally leads to improvements in forecast accuracy. In addition, the evidence has shown that a simple average (the mean of the forecasts being combined) often provides better results than more complicated and sophisticated combining models (Clemen, 1989). The beauty of using simple averages is that they are easy to understand and implement, not requiring any estimation of weights or Corresponding author. Tel.: addresses: vrj@duke.edu (V.R.R. Jose), rwinkler@duke.edu (R.L. Winkler). 1 Tel.: ; fax: other parameters. This makes them robust, in that they are not sensitive to estimation errors, which can sometimes be substantial. For example, Winkler and Clemen (1992) consider weighted averages and show how relatively small estimation errors can lead to negative weights or weights much greater than one, and to weighted averages that are considerably outside the range of the individual forecasts being combined. It is well known that simple averages themselves are quite sensitive to extreme values, and forecasts can sometimes vary considerably. Because of this, some authors have suggested other more robust alternatives. The median, which is less sensitive to extreme values than the mean, has also been considered, with mixed results. The median performed better than the mean in Agnew (1985), worse in Stock and Watson (2004), and /$ - see front matter 2007 International Institute of Forecasters. Published by Elsevier B.V. All rights reserved. doi: /j.ijforecast

2 164 V.R.R. Jose, R.L. Winkler / International Journal of Forecasting 24 (2008) about the same in McNees (1992). Armstrong (2001, p. 422) suggests the use of the trimmed mean: Individual forecasts may have large errors because of miscalculations, errors in data, or misunderstandings. For this reason, it may be useful to throw out the high and low forecasts. I recommend the use of trimmed means when you have at least five forecasts. Forecasters have not studied the effects of trimming, so this principle is speculative. Stock and Watson (2004) used a single trimmed mean, with symmetric 5% trimming, and found that it performed about the same as the mean. In this paper, we investigate the empirical performance of the trimmed mean as a combining method more comprehensively, considering varying amounts of trimming. We also study the performance of another robust measure from statistics, the Winsorized mean, for varying amounts of Winsorizing. We primarily use the extensive data set of forecasts from the M3 Competition (Makridakis & Hibon, 2000), but also analyze some data from the Federal Reserve Bank of Philadelphia's Survey of Professional Forecasters (Croushore, 1993). In Section 2 we discuss the methods and data sets that are used. The results are presented in Section 3, and Section 4 contains a brief summary and discussion. 2. Methods and data Suppose that we are interested in forecasting Y, and we have n forecasts X 1,, X n of Y. These forecasts might come from forecasting models, from human forecasters, or from other sources. One class of widely used location estimators is the family of L-estimators (Huber, 2004), which stands for linear combinations of order statistics. This family includes the mean and median, as well as measures such as trimmed means and Winsorized means. If X (i) is the ith order statistic for X 1,, X n, then the trimmed and Winsorized means are defined as follows, where i is an integer with 0 ibn/2: Trimmed Mean ¼ TðiÞ ¼ 1 X n i X ðkþ ; n 2i k¼iþ1 Winsorized Mean ¼ W ðiþ ¼ 1 n½ ix ðiþ1þ þ Xn i k¼iþ1 X ðkþ þ ix ðn iþ Š : These measures involve taking the i smallest and i largest forecasts and either deleting them or setting them equal to the (i+1)th smallest and (i+1)th largest forecasts, respectively. They correspond to symmetric 100(2i/ n)% trimming or Winsorizing. The tradeoff in terms of i is that a smaller value of i means that more information is being used (fewer data points are being deleted or adjusted), but the mean is more likely to be overly influenced by an extreme value. At one limit (i=0) we have the mean, or simple average, which corresponds to no trimming or Winsorizing. At the other limit (i=greatest integerbn/2) is the median, which represents the maximum amount of trimming or Winsorizing. Note that trimming and Winsorizing make sense only if n 3, and are distinct from each other when n 5. The comparative advantage of these measures is that they can be more robust than the mean, yet not as limited as the median, which ignores a great deal of the information contained in the set of forecasts. A rough analogy from time series forecasting is the use of a moving average (the average of a subset of the past data), as opposed to the two extremes of the average of all of the past data or just the most recent observation. The primary data set used here, from the M3 Competition (Makridakis & Hibon, 2000), consists of forecasts of 3003 different time series with varying time horizons (6 years for the 645 yearly series, 8 quarters for the 756 quarterly series, 18 months for the 1428 monthly series, and 8 periods for the other series). We use only 22 of the 24 forecasting methods from this data set because the other two (AAM1 and AAM2) do not provide forecasts for all of the series. The M3 Competition and earlier versions of the M competition have been influential in the forecasting literature, showing, among other things, that (1) simpler models tend to perform better than more complex forecasting models, (2) different methods perform better with respect to different forecasting accuracy measures, and (3) combined forecasts (particularly simple averages) tend to perform better than individual forecasts. To avoid reliance on a single data set, we also analyze quarterly forecasts of the nominal gross domestic product (NGDP) from The Federal Reserve Bank of Philadelphia's Survey of Professional Forecasters, formerly the ASA/NBER Economic Outlook Survey

3 V.R.R. Jose, R.L. Winkler / International Journal of Forecasting 24 (2008) (Croushore, 1993). We consider 149 quarters, from the 4th quarter of 1968 to the 4th quarter of 2005, each with forecasts for the upcoming four quarters. Variants of this data source have often been used in studies of forecast aggregation methods (e.g., Clemen & Winkler, 1986; Graham, 1996; Zarnowitz, 1984). In evaluating the performance of the robust combining methods, we utilize the following forecast accuracy measures: (1) Symmetric Mean Absolute Percentage Error (smape), the mean of sape =100 (X Y)/[(X + Y)/2] for a forecast X of Y; (2) Symmetric Median Absolute Percentage Error (smedape), the median of sape; and (3) Percentage of times better than or equal to the simple average of sape (% Better). To provide measures of the risk of high errors, we also present the percentage of times the symmetric APE exceeds certain values (% sape exceeds). Finally, we study the relative performance of the robust combining methods as a function of the coefficient of variation of the forecasts from the individual methods. We base our evaluations on smape rather than the simple MAPE (which entails dividing by Y instead of (X+Y)/2), because we feel that smape is more stable and avoids some of the potential difficulties that can arise with MAPE (e.g., see Makridakis, 1993). The nature of our results is not changed if MAPE is used instead of smape. 3. Empirical results Using the entire set of n=22 individual methods, we study the performance of 20 different combining methods: the mean, the median, and the trimmed and Winsorized means for i =1,,9. Some overall summary statistics are provided in Table 1, and comprehensive lists of smape values for different forecasting horizons are provided in Tables 2 and 3.More detailed results are available in an online supplement at From Table 1, we can see that the differences among the combining methods are not large, but that the performances of the trimmed and Winsorized means follow a similar pattern for smape and smedape as i increases from i=0 (the mean) to i=10 (the median). Performance improves at first as i increases, then gets worse beyond some particular point. The net result of this pattern is that many of the trimmed and Winsorized means outperform the mean, which in turn outperforms the median, using these measures. For example, all but T(8), T(9), W(9) and the median do better than the mean according to smape. The trimmed means have lower smapes for i 3(9 27% trimming) than for larger values of i, whereas the Winsorized means do better for 2 i 5 (18 45% Winsorizing) than for other values of i. Comparing T(i) and W(i) for Table 1 Accuracy measures and measures of the risk of large percentage errors for combinations of the 22 individual methods from the M3 Competition i smape smedape % Better % sape exceeds 40% 60% 80% T(i) W(i) T(i) W(i) T(i) W(i) T(i) W(i) T(i) W(i) T(i) W(i) Combined forecasts Individual forecasts Min Ave Max

4 166 V.R.R. Jose, R.L. Winkler / International Journal of Forecasting 24 (2008) Table 2 Symmetric Mean Absolute Percentage Error (smape): All 3003 series using trimmed means Method Forecasting horizon Average of forecasting horizon Mean T(1) T(2) T(3) T(4) T(5) T(6) T(7) T(8) T(9) Median fixed values of i, shows that trimming generally does better for i=1 and Winsorizing does better for i 3. The measures smape and smedape give averages over all of the forecasts. On a forecast-by-forecast basis, the results for % Better indicate that the trimmed means for i 2 and the Winsorized means for 2 i 5 have smaller errors than the mean for slightly over 50% of the forecasts. For higher values of i, the mean is equal or better slightly more than 50% of the time. One argument for robust methods is that they can reduce the risk of large errors. From Table 1, we see that sape exceeds 40, 60, and 80 more frequently with the mean than with any of the trimmed or Winsorized means. This helps to explain why, even for higher values of i, most of the T(i)s and W(i)s have lower smapes, despite beating the mean for slightly under half of the individual forecasts. Tables 2 and 3 give smape values by horizon, and they show that the results discussed above hold for individual horizons as well as for the overall data set. The amounts of trimming or Winsorizing that yield better performances might be slightly larger for shorter horizons. However, the insensitivity of the measures to the amount of trimming or Winsorizing for a given horizon is striking. Similar results hold if we use MAPE instead of smape, and also for other measures, as can be seen from the online supplement. Of course, the reason for combining forecasts is the expectation that they will perform better than the individual forecasts being combined, and that is definitely the case here. The smapes for all of the combined forecasts in Table 1 are lower than the best of the smapes for the 22 individual methods, and the same is true for the smedapes. It is also true for the occurrence Table 3 Symmetric Mean Absolute Percentage Error (smape): All 3003 series using Winsorized means Method Forecasting horizon Average of forecasting horizon Mean W(1) W(2) W(3) W(4) W(5) W(6) W(7) W(8) W(9) Median

5 V.R.R. Jose, R.L. Winkler / International Journal of Forecasting 24 (2008) Table 4 Ranking of combining methods for low, medium, and high coefficients of variation of the forecasts being combined Rank Low Medium High Rank Low Medium High 1 Mean W(4) W(4) 11 W(6) T(4) W(7) 2 W(1) T(1) W(3) 12 T(5) T(5) T(6) 3 T(1) W(1) W(5) 13 W(7) W(7) W(8) 4 W(2) W(3) T(3) 14 T(6) T(6) T(7) 5 W(4) W(2) T(2) 15 T(7) W(8) W(1) 6 W(3) Mean T(4) 16 W(8) T(7) T(8) 7 T(2) W(5) W(2) 17 T(8) T(8) W(9) 8 W(5) T(2) W(6) 18 W(9) W(9) T(9) 9 T(3) T(3) T(5) 19 T(9) T(9) Median 10 T(4) W(6) T(1) 20 Median Median Mean of high errors; the upper-tail percentages for sape for all of the combining methods are smaller than the corresponding values for each of the individual methods. The notion of trimming or Winsorizing when combining forecasts seems more compelling when some of the individual forecasts are relatively extreme. We divided the 3003 forecast situations into three categories, based on the size of the coefficient of variation of the individual forecasts, using the tertiles to determine the categories. The rankings of the smapes of the combining methods for these three categories are given in Table 4. When the coefficient of variation is low, the mean outperforms all of the trimmed and Winsorized means, and the best trimmed and Winsorized means are those with i=1. For coefficients of variation in the middle range, the rankings are more mixed, with the mean dropping somewhat and the Winsorized mean with i=4 doing best. For situations with a high coefficient of variation, the mean has the worst performance of all the 20 combining methods, and the best combining methods are W(i) for i=4, 3, and 5, followed by T(i) for i=3, 2, and 4. One caveat is that the differences in smape within each category are not large, but it does appear that the robust methods are most useful when there is higher variability among the individual forecasts. Also, the smapes increase dramatically from category to category, averaging about 3.3, 10.2, and 24.6, respectively, for the low, medium, and high coefficients of variation. This suggests that higher variability among the individual forecasters should lead us to expect higher forecast errors from the combined forecasts. However, we do not always have the luxury of having 22 different forecasts to combine. To see how T(i) and W(i) perform for smaller values of n,werandomlychose 1000 subsets, each of sizes 5 9, from the 22 methods, and evaluated the combined methods based on each of those subsets. The results for n =5, 7, and 9 are summarized in Table 5, and they are consistent with the results for n = 22. The trimmed and Winsorized means are roughly comparable, and they outperform both the mean and the median. For n=5, the only option is i = 1 (40% trimming or Winsorizing); i = 1 does slightly better than i=2 for n=7 (29% trimming or Winsorizing, compared with 57% for i=2) and n=9 (22% trimming or Winsorizing). Note that increasing the value of n improves the accuracy of the combined forecasts, but at a decreasing rate, as would be expected. The improvements in smape in going from n=5 to n = 9 are roughly comparable to the gains in going from n=9ton=22. Moreover, starting with n=5 and no trimming or Winsorizing, the improvement due to trimming or Winsorizing with i=1 is roughly the same as the improvement due to increasing the number of forecasts to n = 9 without trimming or Winsorizing. For the NGDP data set, the number of forecasters participating in a quarter varied between 9 and 87. For Table 5 Accuracy measures for combinations of 5, 7, and 9 individual methods from the M3 Competition smape smedape % Better Mean T(1) T(2) T(3) Median W(1) W(2) W(3)

6 168 V.R.R. Jose, R.L. Winkler / International Journal of Forecasting 24 (2008) Table 6 Accuracy measures for combinations of 5, 7, and 9 individual forecasts from the NGDP data set smape smedape % Better Mean T(1) T(2) T(3) Median W(1) W(2) W(3) Individ each quarter, we randomly chose 10,000 subsets, each of sizes 5 9, from the forecasts available, and evaluated the combined methods based on each of those subsets. The results for n=5, 7, and 9 are summarized in Table 6.The smapes and the smedapes are much smaller than in Table 5, reflecting the fact that the forecasters are able to forecast NGDP more accurately than the forecasting methods can forecast the mélange of series in the M3 Competition. Of course, the forecasters are experts at forecasting economic variables, and they no doubt consider much more information than just the historical series of NGDP. What is particularly notable is that the relative performances of the different methods replicate the results of Table 5 quite closely, providing further support for the conclusions from our analysis of the M3 Competition data. 4. Summary and discussion The trimmed and Winsorized means seem to be viable alternatives for use when combining forecasts. The empirical results support Armstrong's (2001) recommendation to use trimmed means, and also support the use of Winsorized means. Trimmed and Winsorized means are simple to use and are quite robust. For the data sets analyzed here, they provide slightly more accurate forecasts than the simple average of the forecasts, and also beat all of the individual forecasts being combined. They do well with accuracy measures such as smape, and perhaps even more important, they do well on measures of the risk of high errors. How much trimming or Winsorizing is recommended? The results here suggest that for T(i), moderate trimming of 10 30% (i.e., 5 15% on each side) yields good results. For W(i), slightly greater Winsorizing of 15 45% ( % on each side) works well. Compared to the trimmed mean, the performance of the Winsorized mean seems to be a little less sensitive to the choice of i than the trimmed mean, which might make it preferable on balance. Winsorizing is also more appealing in the sense that the modifications to the mean provided by Winsorizing are less extreme in spirit than those from trimming. Winsorizing adjusts some forecasts to make them less extreme, whereas trimming eliminates those forecasts entirely. Nonetheless, our empirical results indicate that the overall improvements in the combined forecasts are similar for trimming and Winsorizing. Therefore, the nature of the situation and the forecasts should be considered when deciding whether to use one of these robust methods, and if so, which one to use (i.e., trimming or Winsorizing, and which value of i). For example, the rankings of combining methods for the low, medium, and high coefficients of variation of the forecasts indicate that the robust methods are more advantageous, and that greater amounts of trimming or Winsorizing might work better when there is more variability among the individual forecasts. More variability among the individual forecasts also leads to higher smapes, in which case getting additional forecasts might be desirable. Also, insofar as getting additional forecasts is concerned, accuracy improves at a decreasing rate as more forecasts are added. For example, improvements in smape in going from n=5 to n=9 with the forecasts from the M3 Competition are comparable to gains in going from n=9 to n=22.

7 V.R.R. Jose, R.L. Winkler / International Journal of Forecasting 24 (2008) Our conclusions echo the spirit of some of the results of the M Competitions and of various other studies of combined forecasts: (1) Combined methods do better than their component parts, the individual methods. (2) Specific details (e.g., how much trimming or Winsorizing works best) differ depending on the accuracy measure used, but the general pattern is quite consistent and robust. (3) These simple robust combining methods do better than the equally simple but perhaps less robust simple average, which has been shown in previous studies to tend to do better than more complex combining methods. We have only considered symmetric trimming or Winsorizing, where equal numbers of forecasts are deleted or adjusted at the low and high ends. Asymmetric trimming could also be used, but would be closer to outlier detection. More complex and robust measures could also be considered. A whole strand of literature on such measures can be found in Huber (2004).However, the beauty of aggregation schemes such as trimmed and Winsorized means (and indeed, the simple average as well) is that they are simple to apply, understand, and explain to the end-users of the forecast, regardless of their technical backgrounds. Trimming can be explained as the removal of extreme values, while Winsorizing could be thought of as a reduction in the extremity of the outermost values. Recent behavioral studies such as Yaniv (1997) and Harries, Yaniv, and Harvey (2004) have even shown that trimming is commonly used as a decision heuristic when individuals aggregate information subjectively under conditions of uncertainty. As noted above, the trimmed and Winsorized means are appealing because of their simplicity and robust performance. On the question of which is better, trimming or Winsorizing, we have a slight preference for Winsorizing, but believe that the answer in a given situation depends on the circumstances and perhaps also on taste. In any event, the evidence suggests that in averaging forecasts, we could benefit from incorporating some degree of robustness into our aggregation schemes. Acknowledgements We are grateful to Michèle Hibon for providing the forecasts submitted by the forecasters in the M3 Competition, and to Bob Clemen, Luca Rigotti, and two reviewers for helpful comments. Appendix A. Supplementary data Supplementary data associated with this article can be found at References Agnew, C. (1985). Bayesian consensus forecasts of macroeconomic variables. Journal of Forecasting, 4, Armstrong, J. S. (2001). Combining forecasts. In J. S. Armstrong (Ed.), Principles of forecasting: A handbook for researchers and practitioners (pp ). Norwell, MA: Kluwer Academic Publishers. Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, Clemen, R. T., & Winkler, R. L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, Croushore, D. (1993, November/December). Introducing the survey of professional forecasters. Business review (pp. 3 13). Federal Reserve Bank of Philadelphia. Graham, J. R. (1996). Is a group of economists better than one? Than none? Journal of Business, 69(2), Harries, C., Yaniv, I., & Harvey, N. (2004). Combining advice: The weight of a dissenting opinion in the consensus. Journal of Behavioral Decision Making, 17, Huber, P. J. (2004). Robust statistics. New York: Wiley-Interscience. Makridakis, S. (1993). Accuracy measures: Theoretical and practical concerns. International Journal of Forecasting, 9, Makridakis, S., & Hibon, M. (2000). The M-3 competition: Results, conclusions and implications. International Journal of Forecasting, 16, McNees, S. K. (1992). The uses and abuses of consensus forecasts. Journal of Forecasting, 11, Stock, J. H., & Watson, M. W. (2004). Combination forecasts of output growth in a seven-country data set. Journal of Forecasting, 23, Winkler, R. L., & Clemen, R. T. (1992). Sensitivity of weights in combining forecasts. Operations Research, 40(3), Yaniv, I. (1997). Weighting and trimming: Heuristics for aggregating judgments under uncertainty. Organizational Behavior and Human Decision Processes, 69(3), Zarnowitz, V. (1984). The accuracy of individual and group forecasts from business outlook surveys. JournalofForecasting, 3,