LUMS Department of Management Science

Size: px

Start display at page:

Download "LUMS Department of Management Science"

Clemence Preston
6 years ago
Views:

1 Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Forecasting High- and Low-Frequency Data with Neural Networks Challenges in Modelling the Input Vector Nikolaos Kourentzes Sven F. Crone LUMS Department of Management Science

2 Agenda I. Motivation DMIN 2008 i. Why Neural Networks in forecasting? II. III. IV. ii. iii. Why focus on the input vector? Why focus on high frequency data? Experimental Design I. The experiment II. The dataset Results I. Forecasting errors II. What are the challenges for HF-data? Conclusions & Further Research

3 Motivation Neural Networks in forecasting Artificial Neural Networks in Forecasting ANN are universal approximators [Hornik et al., 98, Hornik, 99] able to model linear & nonlinear functions and generalise well [Zhang, 01, Zhang et al., 01] Promising forecasting performance 77% (out of 84) articles found ANN outperforming benchmarks, 73% according to Adya & Collopy, 98 Large scale studies (100+ time series) ANN at least as good as benchmarks [Hill et al., 96, Liao & Fildes, 05] Criticism in literature?

4 Motivation Neural Networks in forecasting Criticism of ANN in literature Universal approximation challenged in practice, due to processing requirements [Levelt, 90] need to evaluate more ANN setups [Curry et al., 02] M3 forecasting competition ANN fail to show their strength [Makridakis & Hibbon, 00] ANN not produce consistent solutions criticised as not being a good forecasting tool [Armstrong, 06] How does research addresses this criticism?

5 Motivation Neural Networks in forecasting Modelling complexity gives rise to the problems Problem caused by inconsistent trial and error modelling approaches [Zhang et al., 98] Input vector selection the most important issue in ANN modelling for forecasting [Zhang, 01, Zhang et al., 01, Zhang et al., 98, Darbellay and Slama, 00 ] No widely accepted methodology how to set the input vector & hidden layers [Anders and Korn, 99, Zhang et al., 98] Focus on the input vector! Any difference for low or high-frequency data?

6 High frequency business time series High frequency high sampling frequency Motivation High Frequency Data No formal definition! Relies on the available techniques, computational resources and what is the common practice [Engle, 00] 1.7 x 1010 Low frequency monthly data A year of TV viewership data low and high frequency x 10 7 High frequency daily data Is standard forecasting practice adequate for high frequency data?

7 Motivation High Frequency Data x 10 7 High frequency daily data Weekly Annual 1.7 x 1010 Low frequency monthly data Several modelling problems: a. vast amounts of data daily data: 30 times longer than monthly 1.2 b. multiple seasonalities intraday, intraweek c. increased detail increased noise level d. high computational requirements Conventional methods difficult to interpret/use information [Granger, 98] Demand a different approach to forecasting [Taylor et al., 06] What about ANN?

8 ANN and high frequency business time series Motivation High Frequency Data ANN have been used outperforming other established methods in fields like electricity [e.g. Hippert et al., 05], traffic prediction [e.g. Dougherty and Cobbett, 97], etc They should be good! high frequency data large samples ANN perform better [Markham & Rakes, 98, Hu et al., 99] BUT inconsistent results (again) Electricity: Only 1 paper provides some information on how to model the ANN No standardised modelling paradigm! No published research on how to model ANN on high frequency data Modelling problems unexplored

9 Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar

10 Experimental Design Objectives & Setup Use a consistent modelling approach across three different frequency domains Daily, Weekly and Monthly Reveal the additional complexity as the frequency increases Time series and elements of setup originate from the NN5 competition ATM money withdrawal data Setup 1. SMAPE rolling origin evaluation 2. Forecast 56 days / 8 weeks / 2 months 3. Holdout sample: 84 days/ 12 weeks / 3 months (equal to validation set) 4. Bottom-up comparison Benchmarks 1. (Seasonal) Naive model 2. Exponential smoothing family Proven to be robust [Makridakis & Hibbon, 00]

11 NN5-035 time series Experimental Design Time Series 60 NN5-035 daily May Aug Nov Jan May Jul Oct Jan-98 Time span: 18/03/96 22/03/98 trimmed for bottom-up comparisons 14 missing values imputed using the average of the neighbouring observations Processed time series form the low frequency time series

12 60 Daily time series Experimental Design Time Series High frequency Summed into calendar weeks Summed into calendar months May Aug Nov Jan May Jul Oct Jan-98 Weekly time series Monthly time series days Medium frequency 100 weeks Low frequency 23 months Outliers - No trend (Phillips-Perron Perron test) - Seasonality?

13 Seasonality Experimental Design Time Series Seasonal Diagram Periodogram ACF/PACF 60 Day of the week seasonal diagram 8 Amplitude Mon Tue Wed Thu Fri Sat Sun Daily: 7, 365? frequency (Hz) From the ACF/PACF analysis the yearly cycle is more evident Daily: 7 Daily: 7, 365 Weekly & Monthly Yearly seasonality but not enough data to model the yearly seasonality!

14 Artificial Neural Networks Experimental Design ANN Models All ANN use similar scalling and learning parameters 40 initialisations error distribution / statistical tests Single hidden layer (universal approximator [Hornik, 99]). Number of hidden nodes specified through simulations. Input vector?

Experimental Design ANN Models Input vector Neural networks are autoregressive models PACF Yule-Walker algorithm a. Common in literature [Ghiassi et al., 05] b.

15 Experimental Design ANN Models Input vector Neural networks are autoregressive models PACF Yule-Walker algorithm a. Common in literature [Ghiassi et al., 05] b. Minimises forward error in the LS sense PACF Burg algorithm a. Better estimation of true PACF [McCullough, 98], but not used in ANN literature b. Minimises forward and backward error ACF & PACF PACF: Yule-Walker (Daily) Lag Input Vector: t -1, t -3, t -5, t Stepwise Linear Regression (Daily) a. ARIMA analogue for ANN [Lachtermarcher & Fuller, 95] Stepwise Linear Regression a. Popular approach [Swanson & White, 97, Qi & Maddala, 99, Dahl & Hylleberg, 04] How many lags to evaluate? Literature?

16 Experimental Design ANN Models Input vector How many lags to evaluate for the input vector? Only 1 paper partially discusses this problem! A rule of thumb (with no statistical analysis backup) is given for low frequency data [Balkin and Ord, 00] Common practice arbitrary select a number of lags some cases is not even a full season [Zhang and Qi, 05, Curry, 07] Low frequency data trial and error approaches common in literature [Zhang, 98] High frequency data the amount of data makes a detailed trial and error modelling very challenging A methodology that will use the information of the time series is necessary

17 Experimental Design ANN Models Input vector Assume no seasonality information split time series in different possible seasons find euclidean distance Euclidean Distance 400 Season: Euclidean distance: Seasonal length plot Season: 7 Euclidean distance: Season Minimise distance minimum deviation (in RSE terms) of the seasons in a seasonal plot seasonality identification Seasonality additional modelling information include in input vector [Crone, 2005, Curry, 07] Multiples of seasonality that minimises deviation guideline how many seasons to include in the input vector identification

18 Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar

19 Two sets of inputs Number of inputs and number of hidden units a. Yule/Burg/ACF-Yule/Regression b. -S suffix inputs. The same with the addition of the maximum lag identified by the Euclidean distance c. In many cases these are the same As the frequency becomes higher: a. Longer input vectors b. More information extracted c. Number of hidden units uncorrelated? Monthly time series barely any autoregressive information Results Input / Hidden Layer Daily Number of Inputs S Hidden Units Yule Burg ACF-Yule Regression YuleS BurgS ACF-YuleS RegressionS Weekly # Hidden Units Yule Burg ACF-Yule Regression YuleS BurgS ACF-YuleS RegressionS Monthly # Hidden Units Yule Burg ACF-Yule Regression YuleS BurgS ACF-YuleS RegressionS - 7 -

20 Neural network error based on model selection (best on validation set) ANN better than benchmarks on daily & weekly Model selection criterion not optimum look at ANN error distributions SMAPE results Models ANN Benchmarks Results SMAPE errors Frequency Model Daily Weekly Monthly Yule Burg ACF-Yule Regression YuleS BurgS ACF-YuleS RegressionS Selected Model Regression(S) Yule/ACF- Yule/Burg/ Yule ACF-Yule Naive Naive Naive EXSM EXSM EXSM Use Friedman and Nemenyi nonparametric statistical tests for comparison [Demsar Demsar, 06]

21 Daily T-test (Wilcoxon test) p-values Mean Median Benchmark Best Model 0.077(0.000) 0.000(0.000) 0.023(0.000) 0.014(0.413) 0.283(0.219) 0.280(0.004) Errors Yule Burg ACF-Yule(S) Regression(S) YuleS BurgS Models: Yule Burg ACF-Yule(S) Regression(S) YuleS BurgS 45 6 Results Friedman p-value: Different CritDist: Average Rank Weekly p-values T-test (Wilcoxon test) 0.000(0.000) 0.000(0.000) 0.000(0.000) 0.000(0.000) Errors Yule/ACF-Yule Burg Yule/ACF-Yule BurgS Friedman p-value: Different CritDist: 4.7 Models: Yule/ACF-Yule Burg Yule/ACF-YuleS BurgS Average Rank Monthly T-test (Wilcoxon test) p-values 0.000(0.000) 0.002(0.000) Errors All (-regres.) All (-regres.)s Friedman p-value: All same CritDist: 2.0 Models: 2 1 All (-regres.) All (-regres.)s Average Rank

22 Results Bottom-up Results Bottom-up comparisons across frequencies - SMAPE Model used to create forecasts Time Series Daily Weekly Monthly Daily time series Weekly time series Monthly time series Increased frequency forecasts provide increase in accuracy! Differences in accuracy between weekly and daily effect of outlier coding

23 Outlier coding binary and integer coding insufficient Challenges ANN can forecast high frequency data What are the challenges? Input vector identification methods problems with confidence intervals Effect of sample size on confidence interval observations observations Confidence Interval Sample size Sample Autocorrelation Lag Non significant Significant CI Too many significant lags ANN training problems

24 Computational resources implications for wrapper methods Challenges ANN can forecast high frequency data What are the challenges? Time series Computational Time (secs) % more from monthly Daily % Weekly % Monthly Computer: Core-Duo 2.2Ghz 3mb Ram 32 bit OS Filters and smart heuristics gain more importance economy of computational resources

25 Challenges ANN can forecast high frequency data What are the challenges? How many lags to evaluate to model the input vector? Connected to computational needs sample size, overlaying seasonalities no guideline in literature! Calendar problems leap years, number of weeks in months/years, etc... problems for statistical methods

26 Conclusions ANN gain advantages in forecasting as frequency increases A set of new modelling problems appear A more sophisticated way to code the outliers (inputs) is necessary A non-simulation based approach to deciding how many lags to evaluate for the input vector - Euclidean distance approach More computational resources needed. Parallel computing and distributing computing the way forward? Applicable in practical problems?

27 Nikolaos Kourentzes Lancaster University Management School Centre for Forecasting Lancaster, LA1 4YX, UK Tel. +44 (0)

28 References 1. K. Hornik, M. Stinchcombe, and H. White, Multilayer feedforward networks are universal approximators Neural Networks, vol. 2, no. 5, 1989, pp K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural Networks, vol. 4, no. 2, 1991, pp W.J.M. Levelt, Are multilayer feedforward networks effectively Turing Machines?, Psychological Research, vol. 52, no. 2-3, 1990, pp G.P. Zhang, B.E. Patuwo, and M.Y. Hu, A simulation study of artificial neural networks for nonlinear time-series forecasting, Computers & Operations Research, vol. 28, no. 4, 2001, pp G.P. Zhang, An investigation of neural networks for linear time-series forecasting, Computers & Operations Research, vol. 28, no. 12, 2001, pp G.Q. Zhang, B.E. Patuwo, and M.Y. Hu, Forecasting with artificial neural networks: The state of the art, International Journal of Forecasting, vol. 14, no. 1, 1998, pp M. Adya, and F. Collopy, How effective are neural networks at forecasting and prediction? A review and evaluation, Journal of Forecasting, vol. 17, no. 5-6, 1998, pp T. Hill, M. O'Connor, and W. Remus, Neural network models for time series forecasts, Management Science, vol. 42, no. 7, 1996, pp K.P. Liao, and R. Fildes, The accuracy of a procedural approach to specifying feedforward neural networks for forecasting, Computers & Operations Research, vol. 32, no. 8, 2005, pp S. Makridakis, and M. Hibon, The M3-Competition: results, conclusions and implications, International Journal of Forecasting, vol. 16, no. 4, 2000, pp J.S. Armstrong, Findings from evidence-based forecasting: Methods for reducing forecast error, International Journal of Forecasting, vol. 22, no. 3, 2006, pp U. Anders, O. Korn, and C. Schmitt, Improving the pricing of options: A neural network approach, Journal of Forecasting, vol. 17, no. 5-6, 1998, pp R.F. Engle, The econometrics of ultra-high-frequency data, Econometrica, vol. 68, no. 1, 2000, pp C.W.J. Granger, Extracting information from mega-panels and high-frequency data, Statistica Neerlandica, vol. 52, no. 3, 1998, pp

29 References 15. J.W. Taylor, L.M. de Menezes, and P.E. McSharry, A comparison of univariate methods for forecasting electricity demand up to a day ahead, International Journal of Forecasting, vol. 22, no. 1, 2006, pp H.S. Hippert, D.W. Bunn, and R.C. Souza, Large neural networks for electricity load forecasting: Are they overfitted?, International Journal of Forecasting, vol. 21, no. 3, 2005, pp H.R. Kirby, S.M. Watson, and M.S. Dougherty, Should we use neural networks or statistical models for short-term motorway traffic forecasting?, International Journal of Forecasting, vol. 13, no. 1, 1997, pp M.Y. Hu, G.Q. Zhang, C.Z. Jiang, and B.E. Patuwo, A cross-validation analysis of neural network out-of-sample performance in exchange rate forecasting, Decision Sciences, vol. 30, no. 1, 1999, pp I.S. Markham, and T.R. Rakes, The effect of sample size and variability of data on the comparative performance of artificial neural networks and regression, Computers & Operations Research, vol. 25, no. 4, 1998, pp E.J. Gardner, Exponential smoothing: The state of the art--part II, International Journal of Forecasting, vol. 22, no. 4, 2006, pp M. Ghiassi, H. Saidane, and D.K. Zimbra, A dynamic artificial neural network model for forecasting time series events, International Journal of Forecasting, vol. 21, no. 2, 2005, pp B.D. McCullough, Algorithm choice for (partial) autocorrelation functions, Journal of Economic and Social Measurement, vol. 24, 1998, pp G. Lachtermacher, and J.D. Fuller, Backpropagation in Time-Series Forecasting, Journal of Forecasting, vol. 14, no. 4, 1995, pp M. Qi, and G.S. Maddala, Economic factors and the stock market: A new perspective, Journal of Forecasting, vol. 18, no. 3, 1999, pp N.R. Swanson, and H. White, Forecasting economic time series using flexible versus fixed specification and linear versusnonlinear econometric models, International Journal of Forecasting, vol. 13, no. 4, 1997, pp C.M. Dahl, and S. Hylleberg, Flexible regression models and relative forecast performance, International Journal of Forecasting, vol. 20, no. 2, 2004, pp

30 Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar

31 Artificial Neural Networks Experimental Design ANN Models All ANN use similar setup, apart from the input vector and number of hidden nodes Scalling [-0.6, 0.6] no pre-processing Learning: gradient backpropagation. Learning rate 0.5 Cool down 0.01, Momentum 0.04, Early stopping (MSE), 1000 epochs Single hidden layer (universal approximator [Hornik, 99]). Number of hidden nodes specified through simulations. Four different input vector selection methodologies Outliers integer dummy variable 40 initialisations error distribution / statistical tests Multiple initialisations: Model selection Best ANN in validation set

32 Euclidean distance Experimental Design ANN Models Input Vector The square root of the sum of the distance of points between two (or n) time series ( ) 2 = Σ

Input-variable Specification for Neural Networks An Analysis of Forecasting Low and High Time Series Frequency

Universität Hamburg Institut für Wirtschaftsinformatik Prof. Dr. D.B. Preßmar Input-variable Specification for Neural Networks An Analysis of Forecasting Low and High Time Series Frequency Dr. Sven F.