SEASONAL TIME SERIES PREDICTION WITH ARTIFICIAL NEURAL NETWORKS AND LOCAL MEASURES. R. Pinto, S. Cavalieri

Transcription

1 SEASONAL TIME SERIES PREDICTION WITH ARTIFICIAL NEURAL NETWORKS AND LOCAL MEASURES R. Pnto, S. Cavaler Unversty of Bergamo, Department of Industral Engneerng, Italy vale Marcon, 5. I Dalmne BG Ph.: E-mal: Abstract: Forecastng s one of the most challengng felds n the ndustral research, due to ts mportance n practce and to the varablty and number of elements that should be consdered. In ths context, the usage of Artfcal Neural Networks has proved to be partcularly advsable, thanks to ther ablty to approxmate any knd of functon wthn a desrable range. Whle most of the lterature concerns wth the defnton of the characterstcs of an ANN, there s only a few number of contrbutons that address the pre-analyss of the data, and seems there s no recent work about the pre-processng of the patterns to submt as nput to the ANN. The am of ths paper s to propose a novel approach to the tme seres forecastng actvtes through the dentfcaton and explotaton of nformaton hdden or latent nto the values and the structure of a seasonal tme seres. In order to do ths, the tme seres s decomposed nto parts, for each of whch some local measures are evaluated: such measures are ntended to mprove the forecastng ablty of the ANN. Moreover, to explot the regularty of a seasonal tme seres, the concept of Seasonal Perodc Index (SPI) has been ntroduced. Results obtaned from the tests confrm the effectveness of the usage of the local measures and of the SPI. Copyrght 2005 IFAC Keywords: Artfcal Neural Networks, Forecastng, Local measures, Seasonal tme seres, Levenberg-Marquardt learnng algorthm,. INTRODUCTION Forecastng s one of the most challengng felds n the ndustral research, manly due to ts mportance n any demand plannng process and to the varablty and number of elements that should be consdered n the mplementaton of an effectve predcton process, supported by sutable methods and technques. Forecastng technques range from quanttatve to subjectve evaluaton, dependng on the objectve of the forecast actvty and the type of predcton beng obtaned. Focusng on quanttatve methods, two man classes can be sngled out: Extrapolatve methods: these methods am to fnd some characterstcs of a set of sequental observatons (referred to as tme seres) that show a repettve behavour. Exponental smoothng technques are an example of these methods. Explcatve methods: these methods correlate one or more ndependent varables to one observed varable (.e. dependent from the other varables), whch has to be predcted. Examples are smple and multple regresson models. In ths context, Artfcal Neural Networks (ANN), whose applcaton n forecastng s dscussed n ths paper, are quanttatve technques whch can be used ether as unversal regressors thanks to ther ablty to approxmate almost any knd of functon (manly, feedforward (Thesng, and Vornberger, 997; Crone, 2002) and Radal Bass Functon networks (Cocou,

2 998; Zemour, et al., 2003) or extrapolatve methods, due to ther feature to recognze pattern s characterstcs through tme (especally wth tme delay (Conway, et al., 998) and recurrent neural networks. Whle most of the lterature concern wth the characterstcs of an ANN (Hegazy and Salama, 995).e. topology, number of layers, number of neurons per layer, actvaton functons an so on there are only a few contrbutons that address the pre-analyss of the data. For example, the analyss of the nput data could be performed through the Prncpal Component Analyss, n order to reduce the sze of the nput space (Cloarec and Rngwood, 998) and avod the curse of dmensonalty (the exponental growng calculaton effort needed as the nput space grows). On the other hand, t seems that there s no recent work about the formaton process of the patterns to submt as nput to the ANN. The dea at the bass of ths work s that nput patterns should be carefully organzed n order to contan not only the values of the tme seres as done generally n most applcatons but also other nformaton gathered from the tme seres tself. Ths nformaton could be obtaned through an opportune pre-processng actvty performed on the data. In the followng of ths paper, an nput pattern defnton process s proposed, n order to enrch the nformaton that the ANN could explot for the forecastng. More precsely, the paper s organzed as follows: secton 2 descrbes the dfferences between the tradtonal approach to forecastng wth ANN and the proposed new approach; secton 3 presents the assumpton at the bass of the proposed forecastng model and the structure of the nput pattern, characterzed by the concept of local measures. In secton 4 the defnton process of the nput pattern s presented: ths process s the bass of the forecastng model, ntroducng the concept of Seasonal Perod Index. Secton 5 brefly exposes the structure of the ANN, whle secton 6 reports about the results obtaned durng the applcaton of the model. Fnally, n secton 7 some concluson remarks and further extensons are proposed. 2. FORECASTING TIME SERIES USING NEURAL NETWORK The forecastng of a tme seres through an ANN could be performed n several ways, as stated n the prevous secton. The general approach (wth the excepton of the usage of tme delay ANNs) conssts n the parttonng of the tme seres n a set of equally szed patterns.e. the same number of observatons per pattern that are presented to the ANN durng the learnng phase, together wth the expected response. Ths approach s based on the assumpton that each observaton A depends on the prevous t values of the tme seres A -, A -2,,A -t. Startng from ths approach, the dea at the bass of the model proposed n ths work s to mprove the performance of the ANN usng nformaton that s not explctly represented by the tme seres values, but s hdden both n the values and n the structure of the tme seres. In order to accomplsh ths, the assumpton becomes more extended: each observaton A of the tme seres could be seen as a combnaton of the nformaton somewhat connected or represented by prevous t observatons. It s mportant to note that we used the term nformaton, whch mples only an ndrect reference to the numercal values of the observaton, whle referrng strongly to other type of knowledge that s consdered embedded but hdden n the tme seres. Stated ths assumpton, the key of the forecast task becomes the dentfcaton and the explotaton of such nformaton. For example, the value of the next observaton could be related to the mean of prevous t values (as n the movng average methods): the mean of the t values s a pece of nformaton whch s dfferent from the numercal value of the observatons. The reference to the movng average methods s sgnfcant also to explan another extenson to the startng assumpton, that seems to be reasonable: snce the movng average consders only the last t values of the tme seres, we could assume that the nformaton s local,.e. s related only to a pece of the tme seres, and could be dfferent dependng on the part consdered (see for example Nottngham and Cook, 200) For all these reasons, the proposed model, nstead of focusng the attenton on the structure and the characterstcs that the ANN should have n order to perform adequately, focuses ts attenton on the data (.e. the tme seres n nput) and on the nformaton that could be extrapolated from them, n order to support the forecastng actvty. In partcular, the attenton s concentrated on seasonal tme seres (wth and wthout trend), because of the ntrnsc dffculty nvolved n the predcton of such seres. 3. THE ASSUMPTION OF THE FORECASTING MODEL Each forecastng technque s based on some assumptons and hypothess: our assumptons (or hypothess) could be stated as follows: a) Any behavour of the tme seres observed n the past wll repeat tself n the future. b) Any observaton of the tme seres could be seen as the result of the combnaton of the nformaton embedded mplctly or explctly n prevous observatons. c) Such nformaton s local to a partcular subset of the avalable tme seres,.e. such nformaton changes dependng on the tme seres under analyss.

3 d) There s a totally random fluctuaton that could not be reasonably elmnated, but the entre seres s not completely random (Not Random Walk Hypothess) The frst assumpton s the well-known Hypothess of Contnuty, whch s the base for almost all the forecastng methods based on tme seres analyss: we could only predct the future studyng the past and assumng that the past wll repeat tself n the future, hopefully almost unvared. The second assumpton s derved from the so called Weak Form Effcent Market Theory (Fama, 970; Stte and Stte, 2002) whch the Random Walk model (mentoned n the fourth assumpton) s an extenson of. As explaned n secton 2, each observaton s seen not only as a combnaton of prevous values (such as n the movng average methods or n the exponental smoothng technque), but t s also nfluenced by the nformaton (.e. not explct, but n some case easly obtanable) embedded nto prevous values. Such nformaton s quanttatve, due to the mathematcal nature of the ANN. The thrd assumpton states that, f we consder only a subset of the tme seres, we could fnd n such a set enough nformaton to be used to forecast the next value of the seres. In other words, each observaton has n ts neghborhood a hdden, latent nformaton that could help n the predcton. Fnally, the fourth assumpton says that t s almost mpossble to predct exactly the values n a real tme seres, because of the presence of random and unknown hdden phenomena. It s dffcult to model or quantfy such elements, so the predcton should not be represented by a sngle value, but by a range of possble values. Gven these assumptons, n our work we attempted to mprove the forecastng results provded by a feedforward ANN by explotng nformaton embedded nto the tme seres. More precsely, we would say local nformaton, meanng a set of measures (as sad, the nformaton consdered s quanttatve) evaluated consderng only a few observatons at a tme. In practce, tme seres are parttoned nto subsets, each charactersed by the followng measures: () mean value; () varance; () slope of the regresson lne that fts the subset (referred to as regresson slope n the followng). Each subset, wth ts related local measures, s the bass for the constructon of the nput patterns for the network learnng process. 4. DEFINITION OF THE INPUT PATTERN The man task of the proposed model s the defnton of the nput patterns of the ANN, startng from the sequence of values of the tme seres. Before descrbng the structure of the nput patterns, t s useful to ntroduce some defntons: LS: the length of the tme seres (.e. the number of the observatons n the seres); W: the wndow extenson, meanng the number of tme seres observatons that consttute an nput pattern; L: the number of the observatons wthn a sngle seasonal cycle; for example, f the season s over one year and the observatons are monthly regstered, L wll be equal to 2; f, nstead, the observatons are gathered every 3 months, L wll be equal to 4. Startng from a seasonal tme seres of length LS, each nput pattern wll be formed by W sequental observatons, whle the target value s represented by the W+ value. As defned prevously, we make use of local nformaton for each pattern, represented by the mean and the varance of the W observatons of the patterns, plus the slope of the regresson lne that fts such W observatons. Moreover, n order to explot the regularty of seasonal tme seres, we ntroduce another element to the dscusson: f the tme seres has a seasonalty perod of L observatons (.e. the seres repeats tself every L observatons) we could provde the ANN wth such nformaton, thus reducng the tranng perod and mprovng the qualty of results. For ths reason, beyond the above llustrated measures, we have added a further set of nputs, namely the Seasonal Perod Index (SPI). Ths s an array of L bnary values, whch s subjected to the followng constrant: L = SPI[ ] = where SPI[] represents the seasonal array. Thus, for each pattern only one element of the SPI could be set to. The poston p of the array whch stores the non-zero value n a pattern s defned accordng to the followng expresson: ( ) p = h mod L + where h s the poston of the target value n the tme seres for the formng pattern (so h [ ; LS]). Obvously, p [ ; L]. As a result, we have a set of patterns composed by W+3+L elements. Fgure may help n understandng the process llustrated above. As shown, the frst pattern s composed by three sequental values of the tme seres (W = 3). The mean, the varance and the regresson slope of these three values are computed and compose the other three elements of the pattern. Most nterestng s the last part of the pattern: supposng that the season s over four perods, we add four bnary values as defned before.

4 LS MSW = n n 2 w = MEAN, VARIANCE AND REGRESSION SLOPE EVALUATION ,3 37, INPUT PATTERN MEAN W OBSERVATIONS VARIANCE REGRESSION SLOPE SEASONAL PERIOD INDEX 4 TARGET VALUE Fg.. The nput pattern defnton process In order to defne whch s the only non-zero value, we have to look at the target value: snce the target value s n the fourth poston (h = 4) and L = 4, the non-zero poston n the SPI s: ( 4) + P = 4 mod = So, the should be postoned n the frst cell of the SPI array. The second pattern wll have the n the second poston of the SPI array and so on. 5. THE ANN AND THE LEARNING ALGORITHM The ANN used n ths work s a feedforward backpropagaton network wth one hdden layer (wth logstcs actvaton functon) traned wth the Levenberg-Marquardt (LM) supervsed learnng algorthm. Ths algorthm, as the quas-newton methods, has been purposely desgned for provdng an approxmaton of the Hessan matrx of the performance functon, mprovng the speed of learnng, especally for moderate-szed networks. Ths algorthm revealed to be the best one n the test we conducted, both n the epochs needed for reachng the target error and n the qualty of the result, measured n terms of Mean Average Percentage Errore (MAPE): MAPE = o t t 00 where o s the output of the network and t s the target value, that s the desred value. Moreover, the LM algorthm s partcularly sutable whenever a relatvely small number of observatons s avalable. Two dfferent objectve functons have been tested: the smple Mean Squared Error (MSE) and the MSE wth regularzaton (MSEREG), as defned n the Matlab package: MSEREG = λ MSE + λ ( ) MSW where λ s called performance rato (λ [0 ; ]; f λ =, then MSEREG MSE) and w s the -th weght of the ANN. Ths functon, used as the objectve functon of the learnng phase, ams to reduce the weght of the network, forcng the network response to be smoother and avodng the rsk of data overfttng, that occurs f the ANN has memorzed well the patterns used for the tranng, but shows really poor performances on new data sets. 6. RESULTS The proposed approach has been tested over a number of seasonal sales tme seres, obtaned from varous sources. For the sake of clarty, only the test conducted over a seasonal tme seres wth trend (Fgure 2) s reported. Tests have been conducted over an ANN wth a hdden sgmodal layer of a varable number of neurons (rangng from 5 to 0), traned wth the same learnng algorthm (the LM) but wth dfferent nputs: Fg. 2. Sample of the data used for the test A) wth the set of pattern consttuted from the rollng W observaton of the tme seres; B) wth the set of pattern ncludng the W rollng observaton, the local measures and the SPI. The results of a seres of test are reported n Table, where NN_LM ndcates an ANN traned wth the patterns usng the local measures and SPI, constructed as llustrated n secton 4. Each network and pattern confguraton has been tested 7 tmes, and Table reports the mean, the mnmum and the maxmum MAPE reached both on tranng and test set. As could be seen, local regressons measures mprove the response of ANNs n almost all the tests conducted. Moreover, the usage of the MSEREG functon results n an mprovement of the generalzaton ablty of the network.

5 Table Results of the test Tranng Set Test Set NN_LM NN NN_LM NN Wndow Sze Objectve Mean Error (%) 9,42 7,63 3,9 25,65 Mn Error (%) 6,79 8,90 0,5 6,97 5 Max Error (%),00 30,68 6,44 49,5 Mean Error(%) 0,83 7,46 6,69 24,34 Mn Error (%) 9,37 3,65 4,42 8,62 Max Error (%) 5,34 2,5 24,28 3,42 Mean Error (%) 9,77 6,93 5,39 24,34 Mn Error (%) 6,25,95 9,79 5,4 Max Error (%) 4,8 23,80 22,89 36, MSEREG Mean Error (%) 0,06 32,02 5,90 54,35 Mn Error (%) 7,88 20,87,76 34,73 Max Error (%),89 38,98 9,5 66,73 Mean Error (%),5 27,76 8,34 46,82 Mn Error (%) 7,80 9,26,55 32,5 Max Error (%) 20,26 42,46 33,74 72,74 Mean Error (%) 3,48 2,7 22,3 36,5 Mn Error (%) 8,35, 3,3 7,49 Max Error (%) 2,38 34,27 36,27 58, MSE Fgure 3 reports more clearly the dfferences between the two approaches: wth the usage of local measures and SPI the forecast on the tranng set s more accurate. As stated before, ANN could approxmate arbtrarly well almost any knd of functon, gven a sutable set of data. avods the overfttng problem, but nothng could be sad precsely about the performance of the network when tested on a set of nputs that have not been presented to the ANN durng the tranng phase. Ths s the reason why a test set has also been used: the test set s made up by observatons that have not been presented to the ANN durng the tranng process. Fg. 3. Tme seres (dotted lne) versus forecast on the tranng set Most of the problems arse when the ANN has to generalze ts knowledge,.e. when the ANN has to predct the next value of a tme seres on the bass of nputs never seen before. The usage of the MSEREG Fg. 4. Tme seres (dotted lne) versus forecast on test set

6 As could be seen n Fgure 4, that depcts the forecast on the test set, the proposed model outperforms the normal pattern network also on the test set. Analysng the results of the test, the followng evdences emerge: The assumptons of the model are valdated: the assumptons stated n secton 3 has revealed to be reasonable; that s, n seasonal tme seres could be found a latent nformaton that could mprove the forecastng process. The local measures are approprate: n the proposed model, the mean, varance and regresson slope are adopted. Although these measures seem sutable to mprove the forecast, there could be other local measures that could be exploted. The usage of Seasonal Perod Indcator mproves the performances: provdng the nformaton about the perod to whch the forecast wll belong contrbutes to mprove the performances of the ANN. A smple network archtecture works fne: although t s possble to use dfferent typologes of ANN, the tests confrm that also a relatvely smple feedforward neural network can show good performances. The generalzaton ablty of the network s mproved: perhaps the most mportant result, the added nformaton provded by local measures and SPI mproves the forecastng ablty on new data. The model s robust aganst trend: ths result emerges from the comparson of the result swth the applcaton of de-seasonalzng methods: such methods, n fact, could perform well only f the tme seres s stable, whle the proposed method works fne also n presence of trend (as n the example of the tme seres depcted n Fgure 2) 7. CONCLUSION: REMARKS AND FURTHER EXTENSIONS In ths work, a novel approach to the usage of ANN for forecastng has been proposed. Instead of focusng the attenton on the characterstcs of the ANN, we have proposed a model whch ams to better explot the nformaton embedded nto the hstorcal data. The results prove that the usage of so called local measures (lke mean, varance and slope regresson) could mprove sgnfcantly the performance of an ANN. Moreover, the ntroducton of the Seasonal Perod Index (SPI) contrbutes to the effectveness of the model. In ths work, only a part of the possble measures and pattern confguraton has been examned. In order to generalze the results, some of the most challengng extensons of the present work could be the followng: the dentfcaton of the sutable structure of the ANN (n partcular, the number of the neurons of the hdden layer) by usng some correlaton measures of the characterstcs of the tme seres tself, n order to reduce the sze of the network and the tranng epochs; although the feedforward network wth LM algorthm performs well, could be nterestng to extend the analyss to other network archtectures, such as the Generalzed Regresson Neural Networks (GRNN) often used for functon approxmaton; another nterestng topc s the defnton of the extenson of the wndow W to consder n the transformaton of the tme seres nto nput pattern. Ths parameter could be lnked, for example, to the seasonalty of the tme seres or to other characterstcs; fnally, the proposed method could be extended to other non-seasonal tme seres, that could represent one of the most attractve felds of future research. REFERENCES Cocou, J.B., (998). Tme seres analyss usng RBF networks wth FIR/IIR synapses. Neurocomputng, 20, Cloarec, G.M., Rngwood, J., (998). Incorporaton of Statstcal Methods n Mult-step Neural Network Predcton Models /9, IEEE Conway, A.J., Macpherson, K.P., Brown, J.C., (998). Delayed tme seres predctons wth neural networks. Neurocomputng, 8, 8 89 Crone, S.F., (2002). Tranng artfcal neural networks for tme seres predcton usng asymmetrc cost functons. Techncal Workng Paper IWI-020, accepted at: ICONIP 2002 Proceedngs Fama, E.F., (970). Effcent captal market: A revew of theory and emprcal work. Journal of Fnance, 25, Hegazy, Y.G., Salama, M.M.A., (995). An Effcent Algorthm for Identfyng the Structure of Artfcal Neural Networks for Forecastng Problems /95, IEEE Nottngham, Q.J., Cook, D.F., (200). Local lnear regresson for estmatng tme seres data. Computatonal Statstc & Data Analyss, 37, Stte, J., Stte, R., (2002). Neural Networks Approach to the Random Walk Dlemma of Fnancal Tme Seres. Appled Intellgence, 6, 63 7 Thesng, F.M., Vornberger, O., (997). Sales forecastng Usng Neural Networks. Proceedngs ICNN 97, Huston, Texas, 9-2 June 997, vol. 4, pp , IEEE Zemour, R., Racoceanu, D., Zerhoun, N., (2003). Recurrent radal bass functon network for tmeseres predcton. Engneerng Applcaton of Artfcal Intellgence, 6, au/~hyndman/tsdl/