Predicting Time Series Using Integration of Moving Average and Support Vector Regression

Transcription

1 Inernaional Journal of Machine Learning and Compuing, Vol. 4, No. 6, December 204 Predicing Time Series Using Inegraion of Moving Average and Suppor Vecor Regression Chihli Hung, Chih-Neng Hung, and Szu-Yin Lin daase. For he predicion of hese ime series, we use wo differen ime-based moving average approaches, i.e. weekly moving average and daily moving average, in order o reduce he amoun of variaion in he daa. We use he median of differen weekly and daily moving averages insead of he radiional moving average. As i is no easy o predic he arge variable, Y, direcly, we rea he difference beween he acual value and he median of moving average as he predicion arge for SVR. The remainder of his paper is organized as follows. The relaed work, focusing on compuaional inelligence approaches, is described in Secion II. Secion III inroduces our mehodologies, which include suppor vecor regression and our proposed predicion approach. The experimenal design, including he daase and he evaluaion, is presened in Secion IV. The experimenal resuls are given in Secion V and finally, he conclusion is provided in Secion VI. Absrac Time series predicion is one of he major asks in he field of daa mining. The approaches of ime series predicion can be divided ino saisical echniques and compuaional inelligence echniques. Mos researchers use one specific approach and compare he performance wih oher approaches. This paper proposes a novel hybrid approach, which inegraes radiional moving average models wih suppor vecor regression for he predicion of ATM wihdrawals in England. The use of moving average modeling is no only for he purpose of smoohing bu also for ime series predicion. We rea a weekly median moving average as he benchmark. Based on experimenal resuls, our proposed approach consisenly ouperforms he benchmark. Index Terms Suppor vecor regression, cash wihdrawal analysis, ime series predicion, daa mining. I. INTRODUCTION This paper proposes a novel approach, which inegraes radiional moving average models wih suppor vecor regression [] for he predicion of ATM wihdrawals in England. Time series predicion is one of he major asks in he field of daa mining [2]-[4]. Mining useful knowledge from large amouns of daa has araced a grea deal of ineres. A large porion of daa in he real world is presened as ime series, in which daa iems are ime dependen and obained over repeaed ime inervals [5]. Time series predicion is beneficial for many applicaions, such as predicions for financial reurns, sock marke prices, reail ransacions, river floods, elecrical consumpion, and so on. There are several ime series predicion compeiions, which no only group researchers from differen fields ogeher, bu also provide valid and reliable evaluaion plaforms for he empirical performance of differen approaches [6], [7]. Generally speaking, he approaches o ime series predicion include saisical echniques and compuaional inelligence echniques. Mos researchers use one specific approach and compare he performance wih oher approaches. This paper combines he compuaional inelligence echnique, i.e. he suppor vecor regression (SVR) [8], wih saisical moving average models, for eleven ime series from he NN5 ime series predicion compeiion II. RELATED WORK A vas amoun of research has been devoed o ime series predicion. The approaches o ime series predicion can be broadly divided ino wo groups. The firs group uses saisical echniques and he second group uses compuaional inelligence echniques. The major mehods in he saisical group conain moving average (MA), linear regression, muliple regression, logisic regression, cluser analysis, muliple group discriminan analysis, auoregressive moving average (ARMA), auoregressive inegraed moving average (ARIMA), and so on [9]. Even hough such approaches have performed well for several ime series asks, he fas developmen of compuer hardware and sofware echniques provides an efficien simulaion environmen recenly. Thus, researchers in he compuaional inelligence field use many echniques including he mulilayer percepron neural nework (MLP) [0], recurren neural nework (RNN) [], self-organizing map (SOM) [2], suppor vecor machine (SVM) [3], geneic algorihm (GA) [4], fuzzy logic [5], ec. for ime series predicion. For example, Hill e al. [6] compared six saisical mehods wih he arificial neural nework (ANN) echnique on ime series. Based on he experimenal resuls, he ANN ouperforms he saisical mehods. Xie and Hu [7] prediced Shanghai housing prince index using ARIMA, ANN and SVM, and found ha ANN and SVM are superior o ARIMA. Many ANN models have been proven useful for Manuscrip received April 0, 204; revised June 8, 204. This work was suppored in par by he Minisry of Science and Technology, Taiwan under Gran of NSC H MY2. Chihli Hung and Szu-Yin Lin are wih he Deparmen of Informaion Managemen, Chung Yuan Chrisian Universiy, Chungli 32023, Taiwan, ROC ( chihli@cycu.edu.w, san@cycu.edu.w). Chih-Neng Hung is wih he Deparmen of Informaion and Technology Educaion, Minisry of Educaion, Taipei, Taiwan, ROC ( hcn069@gmail.com). DOI: /IJMLC.204.V hp://

2 ime series, e.g. [8], [9]. However, some researchers have reached he opposie conclusion. Afer exensive analysis of he forecasing accuracy of ANN and saisical forecasers, Foser e al. [20] found ha linear regression and he simple average of he exponenial smoohing mehod are superior o heir ANN counerpar. Some work was based on a hybrid approach, which combines he ANN and saisical mehods, such as ARIMA [2], or several echniques in compuaional inelligence [22], [23] in order o achieve beer performance. In his paper, we focus on using compuaional inelligence echniques and inegrae moving average mehods wih suppor vecor regression for eleven ime series in he NN5 ime series predicion ask. III. METHODOLOGIES A. Suppor Vecor Regression The suppor vecor machine (SVM), based on he compuaional approach ermed he principle of srucural risk minimizaion, proposed by Vapnik [], was originally mainly used for binary classificaion asks. This principle is relaed o he saisical learning heory in order o develop a novel ype of neural nework. The main idea of SVM is o find he hyper plane wih he bigges margin o divide daa ino wo groups. This hyper plane is called he opimal separaing hyper plane. For linearly indivisible daa, SVM should map hem ono a higher dimensional feaure space by he ransformaion of is kernel funcion. In oher words, he opimal separaing hyper plane can be found by he ransformaion from linearly indivisible daa in a lower dimensional feaure space o linearly divisible daa in a higher dimensional feaure space. Based on he same concep of SVM, Vapnik e al. [8] proposed anoher version of SVM for regression, which is called suppor vecor regression (SVR). SVR has been proven valuable for modeling and predicing ime series in several pracical applicaions [24]. However, he quadraic programming solver is he only known raining algorihm for SVM for a long ime. This solver causes SVM o suffer from consuming long processing ime while dealing wih a large daase. Several raining algorihms have recenly been proposed for SVM. In his paper, we use a sequenial minimal opimizaion (SMO) algorihm [25]. More heoreical analysis can be found in he echnical repor of Smola and Schölkopf [26]. B. The Predicion Approach I is necessary o model ime series by analyzing he mechanism ha produces he ime series before predicion. The daa used in he NN5 ime series predicion ask conains abou wo years of daily cash wihdrawals a various auomaic eller machines (ATMs) which are locaed hroughou England. As such wihdrawals from ATMs are mainly relaed o differen day of he week, and he weekly flucuaion rae, i is necessary o deal wih rends and seasonal variaions. The wihdrawals are also influenced by special evens or srucural sysem breaks, which may cause he ime series o include ouliners, zeros and missing values. Tha is, hese ime series also conain rends, cyclic variaions, seasonal variaions and random movemens. The ime series predicion arge variable, Y, can be illusraed as Y T C S I or heir sum, where T, C, S and I indicaes rend, cyclic seasonal and random movemens, respecively. Even hough here are several approaches which are able o model and predic ime series usefully, some of hem may no be easy o build. For example, Rojas e al. [2] argued ha obaining he srucure of he ARMA model is a problem iself, and i seems difficul o develop an auomaic sysem wihou he direc inervenion of a human exper in he Box-Jenkins mehodology. Conversely, a simple approach such as moving average ends o reduce he amoun of variaion presen in ime series and may have a beer performance han oher very complex algorihms [27]. Moving average (MA) is an approach for predicing he arge variable, Y, by is k following sequence of arihmeic mean, MA k as in (). Thus moving average can be reaed as a smoohing approach for ime series as i is able o eliminae unwaned flucuaions of ime series. However, his smoohing feaure also makes moving average reac o an even wih some ime lag. For example, in Fig., he solid line illusraes he acual value and he doed line is moving average of hree previous acual observaions. The difference beween acual value and is moving average a he even A is made due o his smoohing feaure of MA. Therefore, he predicion arge made by moving average is a lagging indicaor, which responds o he informaion behind he arge. Y Acual value Moving average Difference beween acual value and is MA even A Time Fig.. The difference beween acual value and moving average a he even A. Y MA ( Y Y... Y ) k 2 K K The NN5 daase generaed from ATM cash wihdrawals, which are mainly relaed o differen day of he week. In oher words, he relaionship of he same day of he week in differen weeks is significan. Thus we consider moving average on a weekly base. Moreover, he median of several weekly moving average values is more sable han a single moving average for he predicion of he difference beween he acual value and he MA median. According o our experimens, he median of wo week MA, hree week MA and four week MA is quie useful. () 492

3 Fig. 2. The concepual diagram of our proposed approach. D indicaes he difference beween median of MAs and he acual value a ime. The equaions of hese MAs are shown as (2)-(4), respecively. In his paper we use he suppor vecor regression (SVR), based on he sequenial minimal opimizaion (SMO) algorihm [25] wih a polynomial kernel funcion and an epsilon of 0.00, o model and predic ime series by evaluaing he difference beween acual value and is median weekly moving average. This difference is named as D. We can hen ge he predicion arge, Y, as in (5) and he concepual diagram is shown in Fig. 2. Week ( Y 2 Y _ MA2 4 ) (2) (3) Week _ MA3 ( Y 4 2) 3 (4) Week _ MA4 ( Y ) 4 Y D median ( Week _ MA2, Week _ MA3, Week _ MA4 ) (5) The arge oupu variable used by SVR is he difference beween acual value and is median weekly moving average, i.e. D. By analyzing he feaures of he NN5 daase, cash wihdrawals are influenced by hree groups of facors. The firs group conains previous acual wihdrawals, median of weekly MA and median of daily MA. Inpu facors in he second group are flucuaion raes. The purpose of facors in he hird group is o rack deviaion of hisoric observaions. We name his approach as SVR-median. On he oher hand, he radiional MA approach is no only for he purpose of smoohing bu also for ime series predicion. We direcly use weekly median MA o predic ime series wihou using any compuaional inelligence algorihm. This approach is our benchmark model and is named as MA-median. A. Daase IV. EXPERIMENTAL DESIGN The 2008 NN5 ime series compeiion for arificial neural neworks and compuaional inelligence exends he earlier 2007 NN3 ime series compeiion. Boh compeiions provide wo daases: a complee ime series daase and is reduced subse, which conains ime series. In his paper, we evaluae our approach based on he reduced subse as hese wo daases conain similar complexiy. The NN5 reduced daase includes abou wo years of daily cash wihdrawals a various auomaic eller machines (ATMs) locaed in differen pars of England. Each ime series is formed by cash wihdrawals from an ATM. More specifically, he daily cash wihdrawals of each ime series sar from 8 March 996 o 22 March 998, and conain abou 735 known observaions as he daase. However, since he cash wihdrawals are influenced by special evens and some srucural sysem breaks, each individual ime series conains he differen numbers of missing values, zeros and ouliners. We use he las 97 observaions from he daase as he es se and he res as he raining se so as o evaluae he performance of our proposed approach. The missing values are replaced by heir associaed median weekly MA. When heir associaed median canno be found, hen heir associaed moving average of he previous wo days is used insead. B. Evaluaion As he ime series predicion arge variable, Y, is numeric, i is more difficul o evaluae is performance han oher nominal predicion or classificaion arge variables used in a classificaion ask. Tha is, he performance of ime series predicion canno be evaluaed solely on he crierion of accuracy. We should evaluae he similariy beween acual value and is predicion arge value. On he oher hand, as differen individual ime series may conain a differen number of observaions, i is necessary o consider he differen scale beween individual ime series. We follow he evaluaion crierion defined in he NN5 ime series compeiion, which uses he symmeric mean absolue percen error (SMAPE) [28] as in (6). n X Y SMAPE 00 (6) n ( X Y ) 2 V. EXPERIMENTAL RESULTS The NN5 reduced daase consiss of eleven ime series. Each series consiss of 735 known observaions. The las 97 observaions from he original raining se are used as our es se. We use he weekly moving average approach, i.e. MA-median as he benchmark for our proposed approach, i.e. SVR-median. Table I shows he performance of hese wo approaches evaluaed by SMAPE. Our proposed approach performs consisenly beer han he benchmark on boh raining se and es se for all ime series. A -es is used o es wheher or no a difference beween he proposed SVR-median and he MA-median predicion models achieves saisical significance. According o he -es resul, his proposed SVR-median predicion model is significanly beer han he MA-median one due o of he p-value for boh raining and es ses. 493

4 TABLE I: COMPARISON BETWEEN MA-MEDIAN AND SVR-MEDIAN FOR TIME SERIES Time Series Training Se Tes Se MA-median SVR-median MA-median SVR-median (0) 22.9% 6.4% 28.5% 22.0% (02) 23.2% 8.0% 33.4% 25.% (03) 23.7% 7.7% 26.4% 22.% (04) 26.3% 2.9% 25.2% 20.5% (05) 23.2% 8.8% 3.2% 26.% (06) 24.4% 9.4% 27.2% 24.% (07) 22.0% 6.7% 34.3% 27.8% (08) 2.8% 6.5% 3.7% 26.8% (09) 6.6% 3.5% 26.% 20.7% (0) 27.6% 9.% 30.0% 24.2% () 24.0% 8.8% 26.4% 2.7% Mean SMAPE 23.3% 7.9% 29.% 23.7% VI. CONCLUSION This paper has proposed a novel hybrid approach, by he inegraion of saisical moving average models and suppor vecor regression in he compuaional inelligence field. These ime series are generaed from daily cash wihdrawals of differen auomaic eller machines, which are influenced by differen day of he week, special evens and occasional sysem srucural breaks. This paper uses he sequenial minimal opimizaion algorihm for SVR o predic he difference beween he acual wihdrawal and is median of various weekly moving averages. We consider a weekly median moving average as he benchmark for his research. Experimenal resuls show ha our proposed approach performs beer han he benchmark for all ime series evaluaed by he crierion of he symmeric mean absolue percen error. REFERENCES [] V. Vapnik, "Principles of risk minimizaion for learning heory," Proceedings of Neural Informaion Processing Sysems, pp , 99. [2] A. K. Pali and D. Popovic, Compuaional Inelligence in Time Series Forecasing: Theory and Engineering Applicaions, London: Springer-Verlag London Limied, [3] T.-C. Fu, "A review on ime series daa mining," Engineering Applicaions of Arificial Inelligence, vol. 24, pp. 64-8, 20. [4] P. Esling, and C. Agon, "Time-series daa mining," Compuing Surveys, vol. 45, no., paper 2, 202. [5] E. Keogh, "A decade of progress in indexing and mining large ime series daabases," in Proc. he 32nd Inernaional Conference on Very Large Daa Bases, 2006, pp [6] J. McNames, J. Suykens, and J. Vandewalle, "Winning enry of he K.U. Leuven ime-series predicion compeiion," Inernaional Journal of Bifurcaion Chaos, vol. 9, no. 8, pp , 999. [7] A. Lendasse, E. Oja, O. Simula, and M. Verleysen, "Time series predicion compeiion: The CATS benchmark," Neurocompuing, vol. 70, no. 3-5, pp , [8] V. Vapnik, S. E. Golowich, and A. Smola, "Suppor vecor mehod for funcion approximaion, regression esimaion, and signal processing," in Advances in Neural Informaion Processing Sysems, vol. 9, M. C. Mozer e al., Eds. Cambridge: MIT Press, 997, pp [9] B. L. Bowerman, R. T. O Connell, and A. B. Koehler, Forecasing, Time Series, and Regression: An Applied Approach, Belmon, CA: Thomson Learning, [0] R. Yadav, P. Kalra, and J. John, "Time series predicion wih single muliplicaive neuron model," Applied Sof Compuing, vol. 7, no. 4, pp , [] R. C. Saudemeyer and C. W. Omlin, "Evaluaing performance of long shor-erm memory recurren neural neworks on inrusion deecion daa," in Proc. he Souh African Insiue for Compuer Scieniss and Informaion Technologiss Conference (SAICSIT), pp , 203. [2] G. Barreo, "Time series predicion wih he self-organizing map: A review," Sudies in Compuaional Inelligence, vol. 77, pp.35-58, [3] A. Sorjamaa, J. Hao, N. Reyhani, Y. Ji, and A. Lendasse, "Mehodology for long-erm predicion of ime series, Neurocompuing, vol. 70, no. 6-8, pp , [4] M. M. M. Fuad, "Differenial evoluion versus geneic algorihms: Towards symbolic aggregae approximaion of non-normalized ime series," in Proc. he 6h Inernaional Daabase Engineering & Applicaions Symposium, pp , 202. [5] H. G. Seedig, R. Grohmann, and T. A. Runkler, "Forecasing of clusered ime series wih recurren neural neworks and a fuzzy clusering scheme," in Proc. he 2009 Inernaional Join Conference on Neural Neworks (IJCNN), [6] T. Hill, M. O Connor, and W. Remus, "Neural nework models for ime series models forecass," Managemen Sciences, vol. 42, no. 7, pp , 996. [7] X. Xie and G. Hu, "A comparison of Shanghai housing price index forecasing," in Proc. he Third Inernaional Conference on Naural Compuaion, pp , [8] T. H.-K. Yu and K.-H. Huarng, "A bivariae fuzzy ime series model o forecas he TAIEX," Exper Sysems wih Applicaions, vol. 34, no. 4, pp , [9] G. P. Zhang, "A neural nework ensemble mehod wih jiered raining daa for ime series forecasing," Informaion Sciences, vol. 77, pp , [20] W. R. Forser, F. Collopy, and L. H. Ungar, "Neural nework forecasing of shor, noisy ime series," Compuers and Chemical Engineering, vol. 6, no. 2, pp , 992. [2] I. Rojas, O. Valenzuela, F. Rojas, A. Guillen, L. J. Herrera, H. Pomares, L. Marquez, and M. Pasadas, "Sof-compuing echniques and ARMA model for ime series predicion," Neurocompuing, vol. 7, pp , [22] H.-J. Kim and K.-S. Shin, "A hybrid approach based on neural neworks and geneic algorihms for deecing emporal paerns in sock markes," Applied Sof Compuing, vol. 7, pp , [23] K. W. Lau and Q. H. Wu, "Local predicion of non-linear ime series using suppor vecor regression," Paern Recogniion, vol. 4, pp , [24] L. Cao, "Suppor vecor machines expers for ime series forecasing," Neurocompuing, vol. 5, pp , [25] J. Pla, "Fas raining of suppor vecor machines using sequenial minimal opimizaion," in Advances in Kernel Mehods-Suppor Vecor Learning, B. Schoelkopf e al. Eds., Cambridge: MIT Press, pp , 999. [26] A. Smola, and B. Schölkopf, "A uorial on suppor vecor regression," Technical Repor, Royal Holloway College, NeuroCOLT Technical Repor (NC-TR ), Universiy of London, UK, 998. [27] R. T. Clemen, "Combining forecass: A review and annoaed bibliography," Inernaional Journal of Forecasing, vol. 5, pp , 989. [28] J. S. Armsrong and F. Collopy, "Error measures for generalizing abou forecasing mehods: Empirical comparisons," Inernaional Journal of Forecasing, vol. 8, pp , 992. Chihli Hung received a PhD degree a he School of Compuing and Technology from he Universiy of Sunderland, UK in He is an associae professor a he Deparmen of Informaion Managemen, Chung Yuan Chrisian Universiy, Taiwan, R.O.C. His curren research ineress include ex mining, inelligence sysems, compuaional inelligence, and daa mining. 494

5 Chih-Neng Hung received a PhD degree a he Insiue of Managemen Science from Tamkang Universiy, Taiwan in 203. His curren research ineress include daa mining and informaion securiy governance. Szu-Yin Lin received his Ph.D. degree in informaion managemen from Naional Chiao Tung Universiy, Taiwan, 202. He is an assisan professor in he Deparmen of Informaion Managemen a Chung Yuan Chrisian Universiy, Taiwan. His experise is on service-oriened archiecure and compuing, inelligen dynamic daa analysis, muli-agen sysems, and inerne echnology. 495