Forecasting Using Consistent Experts Dr. Bernard Menezes Professor Dept. Of Comp Sc & Engg IIT Bombay, Powai, Mumbai

Transcription

1 M.Vijayalakshmi PhD Student Dept. Of Comp Sc & Engg IIT Bombay, Powai, Mumbai Forecasting Using Consistent Experts Dr. Bernard Menezes Professor Dept. Of Comp Sc & Engg IIT Bombay, Powai, Mumbai Venu Gopal MTech Dept. Of Comp Sc & Engg IIT Bombay, Powai, Mumbai Abstract Combining forecasts from different models has shown to perform better than single forecasts in most times series. In this paper new techniques for combining a large number of forecasting models in order to achieve better forecasting performance are introduced. This class of new techniques for combining is based on using consistent experts for forecasting. Keywords: Times series forecasting, combining methods, Rank based Combining, Experts, TopK, Dynamic TopK (DTopK), Consistent Experts 1. Introductions Sales forecasting is an important part of supply chain management - both at the retailer end and at the end of the distributors, manufacturers and suppliers. Timely and accurate sales forecasts are crucial in bridging the gap between supply and demand, thereby decreasing the inventory holding costs. Thus, sales forecasting finds applications in managerial decision making, helping to reduce the risk involved when inventory decisions are made. One class of sales forecasting techniques use the concept of times series analysis. A time series is a set of observations (i.e., sales) measured at regular intervals (i.e., daily, weekly, monthly) over a period of time (i.e., three months, one year, five years). Time series modeling methods assume that history repeats itself, so that by studying the past, you can make better decisions, or forecasts, for the future. The sales forecasting system described in this paper is based on performing a set of preprocessing and post processing tasks on a time series of interest. We decompose the series into multiple components such as trend, seasonality and irregular component. We then International Conference on Management of Data COMAD 2008, Mumbai, India, December , 2008 Computer Society of India, 2008 produce individual forecasts for these components. These forecasts are then combined to generate the actual forecast. Since earlier results have shown that multiplicative combining of the components works better for sales forecasting,, we use the Cartesian product for combining the components[1] [2]. In our paper, post processing of a time series, refers to the use of multiple forecasters for generating a single forecast. One way to use multiple forecasting methods is model selection, where we determine the best model based on certain criteria like error measures, and use that model to forecast. Model selection is associated with instability problem when we have less amount of data available or when two or more models are performing well [3]. The other option is to combine the results of all the methods in some way and generate a forecast. Our post processing techniques aim at improving the accuracy of the forecast by dynamically choosing a subset of forecasts, from a large available pool of forecasting methods to combine at each point [5]. The challenge lies in choosing that subset of forecasts, which will give a better forecast compared to using all the available forecasts or model selection. It is well known that combining reduces the risk in forecasting [5]. We base our work on a type of combining called Rank Based Combining. Rank Based Combining was introduced in [9], where the class of TopK and Dynamic TopK (DTopK) methods for combining was suggested. In this paper we present an extension of the above class of methods, which uses consistent experts to combine and forecast a future point. Section 2 gives the relevant literature survey and the background study carried out in this work. It also the dataset used, how the forecasting models used in the work were selected, and the combining work carried out. Section 3 introduces our set of new techniques developed based on using consistent experts for forecasting. Section 4 explains some of the interesting observations and subsequent analysis work carried out which tries to justify the use of combining techniques. In Section 6 we present our conclusions, highlighting the future work that could be done in this area. 1

2 2. Literature Survey In this section we discuss the setting for our work. We also describe existing research and results in this area. Prior research in forecasting has indicated that a combination forecast works better than single forecasts, in sales domain. [3]. Combining can reduce errors arising from faulty assumptions, bias, or mistakes in data. Combining is based on the Delphi method [3], which takes the opinions of various experts and then combines the opinions of all depending on some criteria. Each of the models, whose results would be combined, has to produce individual forecasts at each point. Then to produce the combined forecasts at any point, a simple average of the forecasts produced from all these models for that point can be taken.[14] Alternatively, a weighted average can be taken, where higher weight is given to the model that fits the data more appropriately [12]. Yang proposed an Aggregated Forecast Through Exponential Re-weighting (AFTER) Algorithm for combining forecasts [13]. In the AFTER method of combining results from different forecasting methods, looking at the past performance of the different methods they are assigned weights and a weighted average of the forecasts are taken as the final forecast. Rank based combining was introduced in [6], where a new class of combining methods called DTOPK was suggested. In Rank based combining, the experts (models) at each point of time in the past are ranked and some good subset of experts is chosen. The mean/median of this subset is then chosen as the final forecast. Experiments have shown that DTOPK method works as well as or even better than the standard Holt Winter approach for sales forecasting. The number of base experts chosen was about a million. At every point the forecasts of a million experts have to be compared and ranked based on some criteria. The choices for criteria were any one of the several error measures available, like Mean Square Error (MSE), Absolute Percentage Error (APE), and Mean Absolute Percentage Error (MAPE) etc. In our work the MAPE measure is used to rank the experts. Based on this ranking, Top K experts are chosen for the combination. 2.1 Decomposition Techniques The existing forecasting system is based on preprocessing and post processing of a time series. The kind of preprocessing is decomposition. The series is decomposed into multiple components such as trend, seasonality and irregular component. Each of these components is individually forecasted and their forecasts are joined back (using their Cartesian product) to get the overall forecasts. Five different decomposition techniques were used [6]. 2.2 Rank Based Combining Techniques Rank based combining was introduced in [5], where a new class of combining (post processing) methods called TopK and DTopK (Dynamic Top K) were suggested. In the TopK method, at each point of time t, we calculate the MAPE of each expert, which is obtained by looking at the full past performance (Up to time t 1). A set of top K experts are then chosen at this point t and their forecasts are combined by taking simple mean of them, to give us a forecast for time (t+1). In this way, we obtain forecasts for all time instants. In this approach, we need to determine the parameter K to decide how many top experts we need to combine. In the Dynamic TopK method, all K values are evaluated with the performance of Top K experts at each point of time. The top K experts are determined at each previous time by looking at the respective past performances. The K which has given optimal average performance in the past is selected. This class of methods is called DTOPK because the K parameter is determined dynamically at each time t. 2.3 The Expert Pool A method used in forecasting a component is referred to as an atomic forecaster. Since the original series is decomposed into three components, a forecaster for the original series is a triplet made up of the atomic forecasters for each component. The set of such triplets is the Cartesian product of the sets of forecasters for the Trend, Seasonality and Irregular components. As already mentioned, literature indicates that a multiplicative model is good for sales forecasting. Each such triplet of atomic forecasters (T, S, I) is referred to as an Expert. Appendix B includes a list of atomic forecasters used in this work. The following is the description of the group of experts used in the experiments: Expert Set D-1 (96492 Experts): In this experts group, the various experts are obtained from the Cartesian product of 86 Trend Experts, 33 Seasonal Experts and 34 Irregular Experts (96492 = ). All the individual experts belong to the ARIMA family. Expert Set D ( experts) five different decomposition methods [6]. Hence the total number of forecasts when all the 5 seasonal-variation based decomposition schemes are considered, are 5*86*33*34 = (close to half a million forecasts). 3. Using Consistent Experts Based on the ranking of the experts (using their MAPEs) at each point of time, we identify experts which are relatively more consistent across all the previous points in the series. The idea is to then use these identified consistent experts (experts which were consistent enough in the past) while forecasting the next point.. 2

3 3.1 Expert Consistency Qualifying an expert as being consistent intuitively implies that the expert has always featured in the top X% of the rankings. However, it was observed that of all the experts used (about half a million), there is not a single expert in the top 60% of the expert rankings at all points in any of the time series. We thus redefine the notion of expert consistency to be a pair (X, Y) and an expert is qualified as being consistent, if the expert features in the top X% of the past expert rankings, at least Y % of the time. We can observe this by looking at a Consistent Experts Number Matrix (CEN matrix). performance of all our algorithms has been compared with the performance of the standard Holt Winter (HW) forecasting technique. There are two reasons for this; one being that almost all Demand Sales times series have components of both seasonality and trend and they are also stationery at all points [3]. The second reason is that for such type of series the default best forecaster has been the Holt Winter method. It was observed that the (50, 50) pair gives the maximum improvement over the HW method, as amongst the different (X, Y) pairs tried out. Figure 1: Consistent Experts Number Matrix- Abraham Series A Consistent Experts Number Matrix can be obtained by counting the number of experts which feature in the top X% of the expert rankings, at least Y% of the time, over all points in the series. The matrix has X as columns and Y as rows. The matrices obtained for Abraham series is as shown in the figure 1. It can be observed that only a small number of experts feature in the top X% of the rankings all the time (i.e. Y=100). While the numbers of experts present in the lower left triangle of the number matrix are few, the numbers of experts in the upper triangle are large. 3.2 Fixed (X, Y) Method One simple method of forecasting using consistent experts, is to identify the consistent experts with some prior fixed values for X and Y. These consistent Experts can then be used to get final forecasts. For example, we can fix X=10%, Y=80% (that is, experts in the top 10% of the rankings, at least 80% of the time). By observing the consistent experts number matrix, it seems logical to use those values for X and Y which fall along the diagonal. Hence experiments were carried out using different (X, Y) pairs most of which fall on the diagonal line. The Figure 2: Fixed(X, Y) For Different (X, Y) Pairs This indicates that using consistent experts which are found in the top 50% of all the past rankings, at least 50% of the time, show a 9.85% improvement over standard Holt-Winter method. This is better, when compared with the performance of the algorithm that the mean value of all the experts as forecasted value. (8.31%). Figure 2 3

4 shows the MAPE values obtained for different values of (X, Y) pairs used in the experiments. The table indicates the overall improvement also. 3.3 Fixed-X, varying-y Method In this heuristic, X is fixed at a particular value (say 40%) and Y is reduced from 100 %( in steps of 10%) until there are a minimum number (K) of consistent experts. These experts are then used for forecasting. Here the variable K is the minimum number of experts used for forecasting. forecasting using X = 40% and varying Y for different K values. 3.4 Fixed-Y, Varying-X Method This method is a mirror image of Fixed-X, varying-y method. Here, Y is fixed at a particular value, and X is increased from 10% in steps of 10% until minimum number (K) of consistent experts are available. Experiments with different fixed values of Y showed that Y = 60% gives the best forecasting performance for the same value of K. Hence fixing Y at 60% experiments were conducted and the forecasting performance for different k values was obtained. The results obtained by using this heuristic are shown in the figure 4. Figure 3: Fixed-X varying Y, X=40% Experiments using different values of X indicate that X = 40% gives the best forecasting performance for the same value of K. Hence, X = 40% was chosen for further experiments and the forecasting performance for different K values were obtained. Figure 3 indicates the results of Figure 4: Fixed-Y Varying X, Y=60% 3.5 Optimal (X, Y) Values To be able to estimate the maximum improvement obtainable, we need to identify the best X value and best Y value at each point a prior. At each point, by looking at actual values we identify the best X and Y value and use 4

5 that for forecasting. The best X value is chosen from 10% to 60% and Y value is chosen from 10% to 100%. Figures 5 and 6 show the plots of best X and Y values at various points for Sweet and Rose. The experiments were conducted for the other series as well. From the results it appears having X value greater than 50% and Y value greater 60% is optimal. Method 1: Choosing Best (X, Y) Pair One method for choosing X, Y values dynamically is to choose the (X, Y) pair which has performed the best in the past (has the minimum MAPE). This means, for forecasting at point t, different (X, Y) pairs are used for forecasting up to point (t 1), and the pair whose MAPE value up to point (t 1) is the least, is chosen. X is varied from 10% to 60% and Y is chosen from 10% to 100%. We could also choose the best few (X, Y) pairs instead of just the best. Figure 5: Optimal X Y for Sweet Series Figure 6: Optimal X Y For Sweet Series 3.6 Dynamic-XY Methods As can be observed from the results obtained by using the optimal (X, Y) detailed in the previous section, forecasting performance can be improved if we can estimate best values for X and Y. Hence, the idea of choosing both X and Y dynamically seems promising. Figure 7: Best X, Varying Y Method 2: Best- X, Varying- Y Method The X value to be used at point (t) is chosen as the X (amongst X values of 10%, 20% and so on) which leads to minimum MAPE upto point (t 1). Note that for a given X value, Y is chosen by reducing it from 100% (in steps of 10%) until at least K experts are available. Hence the value of K (the minimum number of consistent experts to 5

6 be used at each point) is fixed in this heuristic. The results obtained by this method are given in Figure 7. Method 3: Enhanced Best- X, Varying- Y Method It is observed that for some series, using the mean of the consistent experts gives greater forecasting improvement than their median. But in some cases, for e.g. the beer, dry and hsales series, the median works better. Hence the Best-X, Varying-Y method is improved to dynamically select the mean or the median. This is done by choosing the best X value and the corresponding mean/ median which did best in the past. This heuristic gives a slight improvement (10.48%) over the previous heuristics (10.40%). The heuristic was further enhanced by using a new error measure, EWMAPE (Exponentially Weighted MAPE), which is defined as: T t) APE(X, t) * α(t t =1 EWMAPE = ; α<= 1 T (T t) t =1 α The X which is associated with the minimum EWMAPE is then chosen by the heuristic. Figure 8 shows the results for the two extensions of Best-X, Varying-Y method. Method 4 Best-Y, Varying-X method The heuristic described in the previous section (Best-X, varying-y) can be modified so that the Y which did best in the past is chosen (instead of X) and X is increased from 10% (in steps of 10%) until sufficient number of experts are available for forecasting. The results obtained by forecasting using the Best-Y, varying-x method are given in Figure 9. Figure 9: Best- Y, Varying- X Figure 8: Enhanced Best- X best- Y 3.7 Averaging different forecasting models We can have a combination forecast by taking the average (mean) of two (or more) forecasted values from (preferably unidentical) different forecasting systems. While combining could be done on similar forecasting experts, averaging here refers to taking the average of point forecasts of dissimilar forecasting systems. The idea is that forecasted values from different systems tend to be on the either sides of the actual value, and hence their average could be better than their individual forecasts. This follows the basic principle of combining forecasts. 6

7 The results obtained by averaging the consistent experts based heuristics explained in the previous sections and DTopK methods are shown in Figure Fixed- X Varying- Y method As explained earlier in section 3.3, the results are as shown in figure 3. All these experiments use the same set of experts. An improvement of 10.07% over Holt- Winter was achieved using this method. The optimal value of the heuristic turned out to be X = 40%, and K=20,000 (which is about 4% of the total number of experts). 4.3 Fixed- Y Varying- X method This method is similar to the earlier one, with X and Y reversed. Again the results, Figure 4, indicate that a K value of 20,000 is a good number of experts to choose. Here optimal value of Y is 60%. This bears out our earlier analysis indicating that (X, Y) value above (40, 60) will perform better. The improvement obtained is about 9.98%. Figure 10: Averaging Method 4. Results And Observations In this section the results obtained from the various methods described in the earlier section are analysed. 4.1 Fixed (X, Y) method The results obtained by forecasting using Fixed (X, Y) method were depicted in Figure 1. Even though it is evident that combining consistent experts is better than the standard Holt Winter method, there is no fixed (X, Y) pair which is best for most of the series. Examples of good (X, Y) pairs are (40%, 60%), (50%, 50%) and (60%, 80%). The (50%, 50%) pair has given the best performance amongst the many (X, Y) pairs tried out (giving 9.85% improvement over Holt-Winter). It seems that X Y values above 50% will perform well. 4.4 Dynamic X Y Methods This class of methods were developed in an attempt to vary both X and Y dynamically. The Best-X, Varying-Y method is an improvement over Fixed-X, Varying-Y method as seen from Figure 7. The experiment was tried out for different values of K. The best forecasting improvement of 10.40% was obtained for K = 10,000 (which is about 2% of the total number of experts used). This indicates that the dynamic methods need lesser number of experts to produce a good forecast. With more number of experts there is no significant improvement. The Best-Y, Varying-X method is an improvement over Fixed-Y, Varying-X method (Figure 9). The best forecasting improvement of 10.27% was obtained again for K = 10,000. A slightly better forecasting performance was obtained by using exponential weighting with α = It is further seen that, in individual series, X, and Y values are initially high and then slowly decrease as we move further in the series. 4.5 Comparison With DTOPK and AFTER Methods Amongst the various methods based on using consistent expert for forecasting, the method which has performed best is Best-X, varying-y (E). Figure 11 shows the comparison amongst the various combining methods existing in literature and the best method based on using consistent experts. This method has given a 10.53% improvement which is better than both, the rank based combining (DTopK) method, and weight based combining (AFTER*) method. However there still exist series where, each of the above methods has done better than all the other methods. 4.6 Averaging Different Forecasting Models Figure 10 showed the results of using the average of two different rank based combining methods. The best method based on using consistent experts is averaged with 7

8 DTopK, another combining method. A good forecasting improvement of 10.77% over the Holt-Winter method was achieved by using the mean of two rank methods whose individual improvements were 10.42% and 10.53%. In general, it is observed that when two methods having comparable forecasting performances are averaged, the results tend to better than the individual methods being averaged participating forecasting systems have similar forecasting performances individually. Future work includes Using data mining techniques to identify best values for X, Y[10] Clustering the series based on similar optimal X Y values and using some cluster specific good algorithm[13] All this leading to the design of a rule based forecasting system. Such a system can analyze the characteristics of a given times series, identify the best set of participating experts and method to forecast.[11] Appendix Figure 11 Comparison of Different methods 5 Conclusions and Future Work The following conclusions can be drawn from this work: Forecasting using consistent experts is effective in combining, and is better than the existing rank based forecasting (DTopK) and weight based combining (AFTER) methods. Averaging of different forecasting systems can be useful and can substantially reduce the risk involved. It is especially seen to be useful when the A Some Of The Times Series Used[14][15] 1. ABRAHAM4.DAT Monthly car sales in Quebec Source: Abraham & Ledolter (1983). 2. ABRAHAM14.DAT Monthly sales of U.S. houses (thousands) Source: Abraham & Ledolter (1983). 3. ADV_SALES.DAT More advertising and sales data: 36 consecutive monthly sales and advertising expenditures of a dietary weight control product. Source: Abraham and Ledolter. 4. ADVERT.DAT Advertising and sales data Source: Makridakis, Wheelwright and Hyndman (1998). 5. ANDERSON14.DAT Monthly sales of company X Jan '65 - May '71 C. Cahtfield. Source: O.D. Anderson and O'Donovan (1983). 6. BLUME.DAT Monthly unit sales, Winnebago Industries, Nov Feb Source: Hipel and Mcleod (1994). 7. BOOKS.DAT Daily sales of paperback and hardcover books. Source: Makridakis, Wheelwright and Hyndman (1998). 8. COLA.DAT Monthly sales of Tasty Cola. Source: Bowerman and O'Connell (1993), 9. DRYWHITE.DAT Monthly Australia sales of dry white wine: thousands of litres. Jan Jul Source: ABS. 10. EKNIVES.DAT Sales of electric knives for the period Jan 1991 through April Source: Makridakis, Wheelwright and Hyndman (1998). 11. FANCY.DAT Monthly sales for a souvenir shop on the wharf at a beach resort town in Queensland, Australia. Jan 1987-Dec Source: Makridakis, Wheelwright and Hyndman (1998). 8

9 12. FORTIF.DAT Monthly Australian sales of fortified wine: thousands of litres. Jan Jul Source: ABS. 13. HSALES.DAT Monthly sales of new one-family houses sold in the e USA since Source: Makridakis, Wheelwright and Hyndman (1998). B Some Of The Experts Used Trend Experts: 1. ARIMA(1,1,0)(0,0,1)s 2. ARIMA(2,1,0)(1,0,0)s 3. ARIMA(2,1,0) 4. Log ARIMA(2,1,2) 5. Log ARIMA(1,1,0)(0,0,1)s Seasonality Experts: 1. ARIMA(0,1,2)(0,1,1)s 2. Log ARIMA(1,0,0)(0,1,1)s 3. ARIMA(1,0,0)(0,1,1)s 4. Log ARIMA(1,0,1)(0,1,1)s 5. ARIMA(1,0,1)(0,1,1)s Irregular Experts: 1. ARIMA(1,0,0) 2. Log ARIMA(1,0,0)s 3. ARIMA(1,0,0)s 4. Log ARIMA(1,0,1)s 5. ARIMA(1,0,1)s The rest of the experts are available in [8] [8] B. Menezes, A. Seth, and R. Singh, Can a million experts improve your sales forecasts? European Symposium on Time Series Prediction, [9] A. Seth, Using a multitude of experts to improve forecasts, Master s thesis, Kanwal Rekhi School of Information Technology, IIT Bombay, Powai, Bombay, , INDIA, [10] R. Agrawal, T. Imielinski, A. Swami. "Mining association rules between sets of items in very large databases." Proceedings of the ACM SIGMOD Conference on Management of data, pages , [11] Adya J. Scott Armstrong and Collopy. Automatic identification of time series features. International Journal of Forecasting, pages , October [12] J. Scott Armstrong. Combining forecasts: The end of the beginning or the beginning of the end? International Journal of Forecasting, 5(4): , October [13] Jeffrey Xu Yu, Michael K. Ng, Joshua Zhexue Huang. Patterns Discovery Based on Time-Series Decomposition, Advances in Knowledge Discovery and Data Mining: 5th Pacific-Asia Conference, PAKDD 2001 Hong Kong, China, April 2001, Proceedings. [14] Jeremy Smith and Kenneth F. Wallis. Combining Point Forecasts: The Simple Average Rules, OK? February 2005, University of Warwick. References [1] Brockwell, P.J. and Davis, R.A. (1991) Time series: Theory andmethods, Second edn. Springer International Edition. [2] Makridakis, S., Wheelwright, S., and Hyndman, R. (1998). Forecasting methods and Applications. Third Edition. Wiley:NY. [3] Robert T. Clemens. Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, [4] The Bayesian Approach to Forecasting An Oracle White Paper Updated September 2006 [5] P. Gulhane, B. Menezes, &others, Forecasting using decomposition and combination of experts, Kanwal Rekhi School of Information Technology, IIT Bombay, Powai, Bombay, , INDIA, T.R [6] R. Singh, On using various decomposition methods in time series forecasting, Master s thesis, Kanwal Rekhi School of Information Technology, IIT Bombay, Powai, Bombay, , INDIA, [7] H. Zou and Y. Yang, Combining time series models for forecasting, Dept. of Statistics, Snedecor Hall, Iowa State University, Ames, IA , USA, Tech. Rep.,