A MACRO-DRIVEN FORECASTING SYSTEM FOR EVALUATING FORECAST MODEL PERFORMANCE

Size: px

Start display at page:

Download "A MACRO-DRIVEN FORECASTING SYSTEM FOR EVALUATING FORECAST MODEL PERFORMANCE"

Wesley Thompson
5 years ago
Views:

1 A MACRO-DRIVEN ING SYSTEM FOR EVALUATING MODEL PERFORMANCE Bryan Sellers Ross Laboratories INTRODUCTION A major problem of forecasting aside from obtaining accurate forecasts is choosing among a wide range of forecasting models. This problem places a major burden upon the forecaster in two respects. First the forecaster spends a substantial amount of time developing and maintaining a set of programs covering the potential forecast models. And second the forecaster must somehow aggregate the results of numerous test runs in order to confidently evaluate and choose among these models. I have developed a forecasting system which aids the forecaster in both of these respects. This system consists of a set of programs which can be grouped into 3 categories: 1) an input module 2) a processing module and 3) an output module. Rather than developing and maintaining a set of programs for each model the user only has to maintain a parameter list for each one. In the input module these parameters are converted into SAS@ macros which generate the selected forecasting model along with controlling input output and other processing. For each model tested several iterations are run. Each period in a user-selected time range serves as the ending period for an iteration and a starting point for a forecast. The results from all of these iterations are aggregated according to user-selected specifications. Thus in a single run the user obtains a set of aggregated statistics on a model covering several iterations eliminating the necessity of running a model multiple times over a time range and then manually aggregating the results. The system offers flexibility. From the SAS@ System and SAS/ETS@ software the user can select the regression autoregression or Box-Jenkins procedure combined with any type of seasonality from the XII procedure. Any time range for the model analysis and forecast can be selected through a set of time parameters. In addition by following the macro- and variable-naming conventions of the system the user is able to add any number of procedures such as exponential smoothing or data steps tailored to particular processing or reporting needs. The system was developed on a VM/CMS system but could be used in the batch mode with only minor modifications. PHILOSOPHY OF A PARAMETER-DRIVEN SYSTEM A couple philosophies were incorporated into this forecasting system to help alleviate some of the difficulties of building a forecasting model. One philosophy is development of a parameter-driven system whereby parameter lists can be saved and modified at run time. The user can also combine parameter lists at run time with one list for the input-output and time specifications and the other for the forecast equation. This speeds up model development and improves efficiency of computer resource usage from several standpoints. The user does not have to spend time writing and debugging programs. All models start with a common base and framework. The user only has to be concerned with the parameter list for each model. Multiple versions of programs with much duplicated source code do not have to be developed and maintained for each forecasting model. By simply changing a couple parameters the user has an entirely new forecasting model. All output results are easily standardized. By only changing the forecast model parameters output from every model will be in identical format. The user can maintain separate input-output parameter lists and combine them with separate forecast model parameter lists to generate the same set of output reports or files for each model. To include a new time period in the model and output the user simply changes a parameter. Within the forecasting system these parameters are converted to macros. These macros in turn set the values of several variables and generate the source code of the different forecasting models. Because of several situations where the SAS system does not allow the use of variable names macros are the only way of creating a parameter-driven system like this one. PHILOSOPHY OF MODEL EVALUATION A second philosophy incorporated into the system is an emphasis on forecast performance rather than statistical measures of model fit. A practical evaluation of a forecast model must be based on how well it can forecast. To evaluate a model's forecasting ability the model must end at a time period before the present and its forecast must be compared to actual data available up to the present. By using this method the user can evaluate the model on the basis of actual performance rather than statistical measures of theoretical performance. The problem with many statistical measures is they only describe how well the formulated model fits the actual data which may give no indication of how well the model can forecast. Moreover confidence intervals extending beyond the end of the model are not valid measures because the time series data often are not stationary. The error is biased because it is minimized by fitting the model to the actual data used in generating it. Whereas a forecast does not have the advantage of using the unknown future data in minimimizing the error which is therefore likely to be much greater. 58

2 A related serious problem is that significant bias and lack of fit in the model may not be revealed by the standard output statistics. These statistics may show a good overall fit of the model to the data even though the fit is poor during the critical stretch of the last few time periods in the model and its forecasts of future periods. Nevertheless my system has not abandoned the statistics on model fit. The user can select these statistics for output over the time range of interest. My system also enhances evaluation of forecast model performance in another way. By running the same model multiple times with ending periods covering a considerable time range the user can view forecast performance under dynamic conditions. With each run the model coefficients are re-determined and the errors aggregated. This offers a distinct improvement over a single model run which only shows static performance with fixed parameters at a single point in time. By averaging several runs together the system shows the overall average performance of the model. The user can then better judge the range of forecasting errors to be expected with the model at hand and how well it compares to other models. Since the output from all the models is in terms of the difference between the model estimates or forecasts and the actual data for each period users can develop their own routines to calculate the mean square error and other familiar statistical measures. MEASURING ERRORS There are two general measures important in selecting a model or judging whether it is adequate. Most important is bias or lack of fit in the model especially in recent periods and in the forecast. The for~asting system measures this as average error. The average error even in a highly volatile time series may be small because the positive and negative errors offset each other. However if the model has a tendency to either over- or underestimate the actual data the average error will be large and little confidence can be placed in forecasts of an unknown future. On the other hand if the errors tend to offset each other with the average accumulated error approaching some minimal percent the user would have some confidence that forecasts over an extended number 6f future time periods would exhibit comparable overall accuracy. The average error of the forecast should be the basis for including or excluding models from consideration for selection. The user can focus on the time periods or time ranges of interest by appropriate parameter selection. A second important measure is absolute error which indicates the degree of volatility in the data series. This error is caused by fluctuations in the data which the model as formulated cannot explain or forecast. Adding other variables or changing the model may reduce some of the absolute error but the remainder is caused by random fluctuations or other factors that cannot be predicted. The error reaches its maximum percentage in short time intervals such as a month and converges to some minimum level in longer time intervals depending on the nature of the data's volatility. Unlike av'erage errors absolute errors do not offset each other over time. Absolute errors indicate the amount of error to be expected in model forecasts of individual time periods. They are especially important if accurate short-term forecasts are needed. The input data along with the model results and errors are accumulated in terms of months quarters half-years and years. Users can incorporate any of the data and results into their own routines for special analyses. The absolute and average errors could even be derived for exponential smoothing and other procedures lacking statistical error measures. These errors form a valid basis for comparing the forecast performance of exponential smoothing to that of regression and other statistical techniques. Another part of the system designed to aid the user is the relative and absolute time p~riod output. Relative time periods are labeled according to there relation to the last period or ending in the model which is labeled "On. One month and one quarter and one year ago are labeled "-1" two months ago "-2" and so on. One month and one year ahead are labeled "+1" two months ahead n+21t and so on. The output statistics show a model's performance in predicting the first month ahead the first year ahead the second month ahead or any other time period relative to the ending of the model. Absolute time periods are in terms of specific s - e.g January 1984 or the 2nd Quarter of The output statistics show a model's performance in predicting any specific time period in the past or future. If the period occurs before the ending of the model the errors represent a measure of the model's fit. If it occurs after the ending and actual data is available for comparison the errors reflect a measure of forecasting accuracy. If the period occurs after the available data only the forecast is output since errors cannot be calculated. The diagram on the next page shows these above relationships. This diagram assumes data is available through August Three iterations of the model are run with ending s of February March and April Increasing the number of iterations increases the number of observations for calculating model fit and forecast errors increasing confidence in the results. In the diagram there are three observations for calculating relative time period model fit errors in monthly periods 0 to -no There are also three observations for calculating relative time period forecast errors in monthly periods +1 to +4 and quarterly period +1 (consisting of monthly periods +1 through +3). There is only one observation for calculating forecast errors in monthly period +6 and quarterly period +2 (consisting of monthly periods +4 through +6). Forecast errors for monthly periods after +6 59

3 (Feb.8S) MODEL FIT ERRORS end ERRORS model - 3rd iteration I -n relative time periods NO ERRORS CALCULATED +S model - 2nd iteration -n S n I 0 end (Apr.8S) model - 1st iteration ACTUAL DATA past history DIAGRAM: Relationships between errors S current (Aug.8S) data not available future available data and time periods cannot be calculated since actual data is not available for comparison. In the diagram absolute time period model fit errors can be calculated for s of April 1985 and earlier. Absolute time period forecast errors can be calculated for s from May to August No errors can be calculated for s after August TIlE SYSTEM The forecasting system consists of several programs and options which can be selected through input parameters. A control program processes these parameters to determine which procedures processing routines and input-output routines will be run. Parameter lists can be placed in SAS data steps and stored under SAS program names. The user can input one or more of these parameter lists simply by including their SAS names when running the forecast system. The parameter list is then displayed through the SAS/FSP@ full-screen procedure and the user can add or change any parameters before they are processed. Among the many parameters the user can select to control processing input and output are the following: a) The dependent Y-variable-or-equation obtained or computed from the input data set. b) A transform variable-or-equation (e.g. units/hour) for converting the final output (e.g. to hours). c) A forecast series of one main independent Variable. (Actual values will be used in the model up to its ending with these forecast values used thereafter.) d) Up to 7 independent X-variables any of which can be time lags of the depenedent variable. e) A seasonal factor - the dependent variable is deseasonalized before being input into the forecasting procedure and re-seasonalized before being output. f) The time range for the ending of each iteration of the model. g) The number of past time periods before the model's ending over which model fit statistics will be gathered and summarized. (This has no effect on the model or its forecast.) h) The number of periods after the ending over which forecast results will be gathered and summarized. (Again there is no effect on model or forecast.) i) The number of past and forecast months quarters and years for which you want statistics printed. j) The SAS forecasting procedure used along with its parameters and options. (Routines for regression autoreggression and Box-Jenkins have been developed.) k) Whether output should be in terms of absolute or relative time periods. 60

4 1) OlITPUT Whether the results are to be output to a disk file for later processing or only printed. Results of the processing module can be captured at various steps and used in a customized output module to meet specific needs. Initial output from this module consists of one record for each observation or synonymously for each time period. Each record contains: (1) the time period (either relative to the end of the model or the actual ) (2) the model estimate or forecast for that period (3) the actual observed value for comparison and (4) the residual error from their difference. Other information contained in each record consists of the values of the independent variables (including time lags of the dependent variable) the seasonal factor and parameter estimates where they are available from the forecasting procedure. Results can also be captured at a later step after further processing and aggregation. At this point the data_ is summarized in monthly quarterly and annual time period groupings. For each time period grouping a record is generated containing: (1) the time period label (2) the model estimates and forecasts (3) the actual values (4) the residual error in value and percent and (5) the maximum and minimum observed errors among the detailed observations composing the grouping. The output sample listings on the next page show these aggregated results for two different time series models. For the relative time period model the past and future periods are the time period groupings relative to the last period used in each model iteration represented by "On. Twelve model iterations were run one ending each month from June 1983 through May Only four yearly observations with forecast errors could be calculated because data was only available through August Overall for all the models combined there were 432 past monthly observations and 108 future observations for calculating errors based on the time parameters selected. For the absolute time period model a single iteration was run with an ending of June In this case the observations reflect the number of months in the time period grouping rather than the number of iterations for the groupings used in calculating errors. This is because the model ends in the middle of the year and a year ahead forecast for example consists of six months to the end of 1983 plus the first six months of CONCLUSION In my own personal experience use of this forecasting system has brought major improvements to forecasting analyses. I now can concentrate solely on the forecasting itself without expending efforts on programming and other computer tasks. Before developing the system I was inund by all the programs and models that had to be maintained and modified in testing forecasts for a set of time series. I had stacks of computer listings and my programs cluttered computer disk storage. I had difficulty organizing all of the programs and listings and validating new and modified forecasting models. From each listing I would manually gather some of the important statistics in a chart. Using this system I eliminated most of these inefficiencies. All of the forecasting analyses are performed with a small set of programs which have been thoroughly tested and debugged. The parameter lists for new forecasting models are easy to develop and maintain. The reports are standardized with output limited to only what is important for model evaluation. In testing several forecast models and data series I have found the standard theoretical statistics output by the procedures to be insufficient for a proper evaluation of forecast performance. Models theoretically superior according to their output statistics may not forecast as well as other models. These theoretical statistics covering the overall time frame may not reveal serious problems of bias and nonstationarity in the models in recent and future time periods. The errors and confidence intervals often significantly underestimate forecasting errors. I have found the forecast errors and other statistics output by my system to be more practical and meaningful than the theoretical statistics. These forecast errors have allowed the testing of forecast performance of such unsophisticated techniques as exponential smoothing and trend fitting. Tracking model fit and forecasts during the latest time ranges reveals underlying changes in trends and cycles in the data after only a few periods prompting the user to adjust future forecasts or alter the forecast model. I no longer spend an excessive amount of time interpreting the theoretical statistics and comparing them between models. The forecast errors from the system are output in terms of time periods and time frames relevant to the forecasting needs allowing quick judgment of whether a model is poor adequate or superior in forecasting a data series. Bryan Sellers Ross Laboratories Dept Cleveland Ave. Columbus Ohio phone: (614)

5 OUTPUT SAMPLE LISTINGS RELATIVE TIME PERIOD MODEL ARIMA UNITS = PRDTS + autoregrssive ) MAY 1984 PAST OBSER- ACTUAL ESTIMATED RELATIVE ABSOLUTE MINIMUM MAXIMUM PERIODS VATIONS VALUES VALUES ERROR ERROR ERROR ERROR Overall % 20% month % 14% o month % 12% quarter Oll 1% 6% o quarter % 4% o year l % 1% PERIODS VALUES Overall ll% 26% month % 15% month % 25% month % 24% quarter % 8% quarter % 12% quarter % 15% quarter % 25% year % 7% ~ r ABSOLUTE TIME PERIOD MODEL ARIMA UNITS = autoregrssive ( ) SEASONAL JUNE 1983 PAST OBSER- ACTUAL ESTIMATED RELATIVE ABSOLUTE MINIMUM MAXIMUM PERIODS VATIONS VALUES VALUES ERROR ERROR ERROR ERROR May 83 I % 2% June 83 I % 94% Qtr-I % 28% Qtr % 33% Yr % 11% Yr % 29% PERIODS VALUES July 1983 I % 94% August 1983 I % 3% Sept I % 41% Qtr % 32% Qtr over 100% l Qtr-I % 31% f Qtr % 56% t Yr % 77% " Yr % 44% I t " E i * l SAS SAS/ETS and SAS/FSP are registered trademarks of SAS Institute Inc. Cary Net USA. 62

Forecasting. Copyright 2015 Pearson Education, Inc.

Forecasting. Copyright 2015 Pearson Education, Inc. 5 Forecasting To accompany Quantitative Analysis for Management, Twelfth Edition, by Render, Stair, Hanna and Hale Power Point slides created by Jeff Heyl Copyright 2015 Pearson Education, Inc. LEARNING