Forecasting Time Series with Multiple Seasonal Cycles using Neural Networks with Local Learning

Transcription

1 Forecasng Tme Seres wh Mulple Seasonal Cycles usng Neural Neworks wh Local Learnng Grzegorz Dudek Deparmen of Elecrcal Engneerng, Czesochowa Unversy of Technology, Al. Arm Krajowej 7, Czesochowa, Poland Absrac. In he arcle a smple neural model wh local learnng for forecasng me seres wh mulple seasonal cycles s presened. Ths model uses paerns of he me seres seasonal cycles: npu ones represenng cycles precedng he forecas momen and forecas ones represenng he forecased cycles. Paerns smplfy he forecasng problem especally when a me seres exhbs nonsaonary, heeroscedascy, rend and many seasonal cycles. The arfcal neural nework learns usng he ranng sample seleced from he neghborhood of he query paern. As a resul he arge funcon s approxmaed locally whch leads o a reducon n problem complexy and enables he use of smpler models. The effecveness of he proposed approach s llusraed hrough applcaons o elecrcal load forecasng and compared wh ARIMA and exponenal smoohng approaches. In a day ahead load forecasng smulaons ndcae he bes resuls for he one-neuron nework. Keywords: seasonal me seres forecasng, shor-erm load forecasng, local learnng, neural neworks. Inroducon Tme seres may conan four dfferen componens: rend, seasonal varaons, cyclcal varaons, and rregular componen. Seasonaly s defned o be he endency of me seres daa o exhb some paern ha repeas perodcally wh varaon. Somemes a me seres conans mulple seasonal cycles of dfferen lenghs. Fg. shows such a me seres, where we can observe annual, weekly and daly varaons. Ths seres represens hourly elecrcal load of he Polsh power sysem. From hs fgure can be seen ha he daly and weekly profles change durng he year. In summer hey are more fla han n wner. The daly profle depends on he day of he week as well. The profles of he weekdays are smlar o each oher n he same perod of he year. To he characersc feaures of hs me seres s nonsaonary and heeroscedascy should be ncluded as well. These all feaures have o be capured by he flexble forecasng model. The mos commonly employed mehods o modelng seasonal me seres nclude []: seasonal auoregressve negraed movng average model (ARIMA), exponenal adfa, p., 20. Sprnger-Verlag Berln Hedelberg 20

2 smoohng (ES), arfcal neural neworks (ANNs), dynamc harmonc regresson, vecor auoregresson, random effec models, and many ohers. (a) 20 L, GW Year (b) wner sprng summer auumn L, GW Mon Tue Wed Thu Fr Sa Sun Hour Fg.. The load me seres of he Polsh power sysem n hree-year (a) and one-week (b) nervals. The base ARIMA model wh jus one seasonal paern can be exended for he case of mulple seasonales. An example of such an exenson was presened n [2]. A combnaoral problem of selecng approprae model orders s an nconvenence n he me seres modelng usng mulple seasonal ARIMA. Anoher dsadvanage s he lnear characer of he ARIMA model. Anoher popular model he Hol-Wners exponenal smoohng was adaped by Taylor so ha can accommodae wo and more seasonales [2]. An advanage of he ES models s ha hey can be nonlnear. On he oher hand can be vewed as beng of hgh dmenson, as nvolves nalzaon and updang of a large number of erms (level, perods of he nraday and nraweek cycles). In [] more parsmonous formulaon of ES s proposed. New exponenally weghed mehods for forecasng me seres ha conss of boh nraweek and nraday seasonal cycles can be found n [3]. Gould e al. [4] nroduced he nnovaon sae space models ha underle ES mehods for boh addve and mulplcave seasonaly. Ths procedure provdes a heorecal foundaon for ES mehods and mproves on he curren approaches by provdng a common sense srucure o he models, flexbly n modelng seasonal

3 paerns, a poenal reducon n he number of parameers o be esmaed, and model based predcon nervals. ANNs beng nonlnear and daa-drven n naure, may be well sued o he seasonal me seres modelng. They can exrac unknown and general nformaon from mul-dmensonal daa usng her self-learnng ably. Ths feaure releases a desgner from a dffcul ask of a pror model selecon. Bu new problems appear: he selecon of nework archecure as well as he learnng algorhm. From many ypes of ANN mos ofen n forecasng asks he mullayer percepron s used, whch has a propery of unversal approxmaon. ANNs are able o deal wh he seasonal me seres whou he pror seasonal adjusmen bu deseasonalzaon and also derendng s recommended [5]. The me seres decomposon s used no only n ANNs, bu also n oher models, e.g. ARIMA and ES. The componens showng less complexy han he orgnal me seres can be modeled ndependenly and more accurae. Usually he me seres s decomposed on seasonal, rend and sochasc componens. Oher mehods of decomposons apply he Fourer or wavele ransform. The smple way o remove seasonaly s o defne he separae me seres for each observaon n a cycle,.e. n he case of cycle of lengh n, n me seres s defned ncludng observaons n he same poson n successve cycles. Ths paper consders smple neural forecasng model ha approxmaes he arge funcon usng paerns of seasonal cycles. Defnng paerns we do no need o decompose a me seres. A rend and many seasonal cycles as well as he nonsaonary and heeroscedascy s no a problem here when usng proper paern defnons. The proposed neural model learns n a local learnng procedure whch allows o model he arge funcon n he neghborhood of he query paern. As a resul we ge a local model whch s beer fed n hs neghborhood. 2 Paerns of he Tme Seres Seasonal Cycles Our goal s o forecas he me seres elemens n a perod of one seasonal cycle of he shores lengh. In he case of he me seres shown n Fg. hs s a daly cycle conanng n = 24 elemens (hourly loads). The me seres s dvded no sequences conanng one seasonal cycle of lengh n. In order o elmnae rend and seasonal varaons of perods longer han n (weekly and annual varaons n our example), he sequence elemens are preprocessed o oban her paerns. The paern s a vecor wh componens ha are funcons of acual me seres elemens. The npu and oupu (forecas) paerns are defned: x = [x x 2 x n ] T and y = [y y 2 y n ] T, respecvely. The paerns are pared (x, y ), where y s a paern of he me seres sequence succeedng he sequence represened by x. The nerval beween hese sequences s equal o he forecas horzon τ. The way of how he x and y paerns are defned depends on he me seres naure (seasonal varaons, rend), he forecas perod and he forecas horzon. Funcons ransformng seres elemens no paerns should be defned so ha paerns carry mos nformaon abou he process. Moreover, funcons ransformng forecas se-

4 quences no paerns y should ensure he oppose ransformaon: from he forecased paern y o he forecased me seres sequence. The forecas paern y = [y, y,2 y,n ] encodes he successve acual me seres elemens z n he forecas perod +τ: z +τ = [z +τ, z +τ,2 z +τ,n ], and he correspondng npu paern x = [x, x,2 x,n ] maps he me seres elemens n he perod precedng he forecas perod: z = [z, z,2 z,n ]. Vecors y are encoded usng curren process parameers from he neares pas, whch allows o ake no consderaon curren varably of he process and ensures possbly of decodng. Some defnons of he funcons mappng he orgnal space Z no he paern spaces X and Y,.e. f x : Z X and f y : Z Y are presened n [6]. The mos popular defnons are of he form: f ( z x, ) = n z l=, ( z z, l z ) 2, f ( z y, ) = n z l= + τ, ( z z, l z ) 2, () where: =, 2,, N he perod number, =, 2,, n he me seres elemen number n he perod, τ he forecas horzon, z, he h me seres elemen n he perod, z he mean value of elemens n perod. The funcon f x defned usng () expresses normalzaon of he vecors z. Afer normalzaon hese vecors have he uny lengh, zero mean and he same varance. When we use he sandard devaon of he vecor z componens n he denomnaor of equaon (), we receve vecor x wh he uny varance and zero mean. Noe ha he nonsaonary and heeroscedasc me seres s represened by paerns havng he same mean and varance. Forecas paern y s defned usng analogous funcons o npu paern funcon f x, bu s encoded usng he me seres characersc ( z ) deermned from he process hsory, wha enables decodng of he forecased vecor z +τ afer he forecas of paern y s deermned. To calculae he forecased me seres elemen values on he bass of her paerns we use he nverse funcon f ). y ( y, 3 Local learnng The ranng daa can have dfferen properes n dfferen regons of he npu and oupu spaces hus s reasonable o model hs daa locally. The local learnng [7] concerns he opmzaon of he learnng sysem on a subse of he ranng sample, whch conans pons from he neghborhood around he curren query pon x*. By he neghborhood of x* n he smples case we mean he se of s k neares neghbors. A resul of he local learnng s ha he model accuraely adjuss o he arge funcon n he neghborhood of x* bu shows weaker fng ousde hs neghborhood. Thus we ge model whch s locally compeen bu s global generalzaon propery s weak. Modelng he arge funcon n dfferen regons of he space requres relearnng of he model or even o consruc dfferen model, e.g. we can use a lnear

5 model for lnear fragmens of he arge funcon whle for he nonlnear fragmens we can use a nonlnear model. The generalzaon can be acheved by usng a se of local models ha are compeen for dfferen regons of he npu space. Usually hese models are learned when a new query pon s presened. The error creron mnmzed n local learnng algorhm can be defned as follows: N E( x*) = K ( d ( x, x*), h) δ ( y, f ( x )), (2) = where: N number of ranng paerns, K(d(x,x*),h) kernel funcon wh bandwdh h, d(x,x*) dsance beween he query paern x* and ranng paern x, δ(y,f(x )) error beween he model response f(x ) and he arge response y when npu paern x s presened (hs response can be a scalar value). Varous kernel funcons mgh be used, ncludng unform kernels and Gaussan kernels whch are ones of he mos popular. The kernel s cenered on he query pon x* and he bandwdh h deermnes he wegh of he h ranng paern error n (2). When we use unform kernel he ranng paerns for whch d(x,x*) h = d(x k,x*), where x k s he kh neares neghbor of x*, have uny weghs. More dsan paerns have zero weghs, and herefore here s no need o use hese pons n he learnng process. For Gaussan kernels all ranng pons have nonzero weghs calculaed from he formula exp( d 2 (x,x*)/(2h 2 )), whch means ha her weghs decrease monooncally wh he dsance from x* and wh he speed dependen on h. In order o reduce he compuaonal cos of deermnaon of errors and weghs for all ranng pons we can combne boh kernels and calculae weghs accordng o he Gaussan kernel for only k neares neghbors of x*. The compuaonal cos s now ndependen of he oal number of ranng paerns, bu only on he number of consdered neghbors k. In he expermenal par of hs paper we use local learnng procedure wh unform kernel. 4 Expermenal Resuls As an llusrave example of forecasng me seres wh mulple seasonal cycles usng neural neworks wh local learnng we sudy he shor-erm elecrcal load forecasng problem. Shor-erm load forecasng plays a key role n conrol and schedulng of power sysems and s exremely mporan for energy supplers, sysem operaors, fnancal nsuons, and oher parcpans n elecrc energy generaon, ransmsson, dsrbuon, and markes. In he frs expermens we use he me seres of he hourly elecrcal load of he Polsh power sysem from he perod Ths seres s shown n Fg.. The me seres were dvded no ranng and es pars. The es se conaned 3 pars of paerns from July The ranng par Ψ conaned paerns from he perod from January 2002 o he day precedng he day of forecas.

6 We defne he forecasng asks as forecasng he power sysem load a hour =, 2,, 24 of he day j =, 2,, 3, where j s he day number n he es se. So we ge 744 forecasng asks. In local learnng approach for each ask he separae ANNs were creaed and learned. The ranng se for each forecasng ask s prepared as follows: frs we prepare he se Ω = {(x, y, )}, where ndcaes pars of paerns from Ψ represenng days of he same ype (Monday,, Sunday) as days represened by a query par (x*, y *), hen based on he Eucldean dsances d(x, x*) we selec from Ω k neares neghbors of he query par geng he ranng se Φ = {(x, y, )} Ω Ψ. For example when he forecasng ask s o forecas he sysem load a hour on Sunday, model learns on k neares neghbors of he query paern whch are seleced from x-paerns represenng he Saurday paerns and h componens of y-paerns represenng he Sunday paerns. ANN (he mullayer percepron) learns he mappng of he npu paerns o he componens of oupu paerns: f : X Y. Number of ANN npus s equal o he x- paern componens. To preven overfng ANN s learned usng Levenberg- Marquard algorhm wh Bayesan regularzaon [7], whch mnmzes a combnaon of squared errors and ne weghs. The resulng nework has good generalzaon quales. In he frs expermen we assume k = 2. Snce he arge funcon f s modeled locally, usng a small number of learnng pons, raher a smple form of hs funcon should be expeced, whch mples small number of neurons. We esed he neworks: composed of only one neuron wh lnear or bpolar sgmodal acvaon funcon, wh one hdden layer conssng of m = 2,..., 8 neurons wh sgmodal acvaon funcons and one oupu neuron wh lnear acvaon funcon. Such a nework archecure can be seen as a unversal approxmaor. APE and MAPE (absolue percenage error and mean APE) s adoped here o assess he performance of he forecasng models. The resuls (MAPE for he ranng and es samples and he nerquarle range (IQR) of MAPE s ) of he 9 varans of ANNs are presened n ab.. Tes errors for hese varans are sascally ndsngushable (o check hs we use he Wlcoxon rank sum es for equaly of APE medans; α = 0,05). I s observed ha for he wo-layered neworks n many cases mos weghs ends o zero (weghs decay s a resul of regularzaon), hus some neurons can be elmnaed. As an opmal ANN archecure ha one wh one neuron wh sgmodal acvaon funcon s chosen. Ths one-neuron ANN s used n he nex expermens. In he second expermen we examne he nework performance dependng on he number of he neares neghbors k,.e. he sze of he ranng se Φ. We change k from 2 o 50. The resuls are shown n Fg. 2, where MAPE for he cases when he ANN s raned usng all ranng pons represenng days of he same ype as days represened by query par,.e. pons from he se Ω, s also shown. As we can see

7 from hs fgure he es error remans around when k [6, 50]. For hese cases MAPE s are sascally ndsngushable when usng Wlcoxon es. When we ran ANN usng paerns from he se Ω MAPE s s sascally dsngushable greaer han for k [6, 50]. Table. Resuls of 9 varans of ANNs. Number of neurons ln sg MAPE rn MAPE s IQR s MAPE for rn Ω.25 MAPE s for Ω MAPE Number of neares neghbors k Fg. 2. MAPE for he ranng ses (rngs) and es se (crosses) dependng on k. In he local learnng approach he horny ssue s he rao of he ranng pons number o he number of free parameers of he nework. Ths rao for our example even for one-neuron ANN s oo small (2/25), whch means ha he model s overszed ( has oo many degrees of freedom n relaon o he problem complexy expressed by only a few ranng pons). The regularzaon whch has a form of a penaly for complexy s a good dea o solve hs problem. Anoher dea s he feaure selecon or feaure exracon as a form of dmensonaly reducon. The mos popular mehod of feaure exracon s he prncpal componen analyss (PCA). Ths procedure uses an orhogonal ransformaon o conver a se of muldmensonal vecors of possbly correlaed componens no a se of vecors of lnearly uncorrelaed componens called prncpal componens. The number of prncpal componens s less han or equal o he dmenson of orgnal vecors. In he nex expermen we ransform he 24-dmensonal x-paerns no paerns wh a smaller number of uncorrelaed componens usng PCA. Fg. 3 shows relaonshp beween MAPE and he number of prncpal componens. From hs fgure can be seen ha he levels of errors are very smlar. MAPE s are sascally ndsngushable for dfferen number of prncpal componens. Usng only frs prncpal componen we can bul good neural forecasng model for our daa. Such a model has only wo parameers. The percen var-

8 ance explaned by he correspondng prncpal componens are shown n Fg. 4. The frs prncpal componen explans 30% of he oal varance MAPE Number of prncpal componens Fg. 3. MAPE for he ranng ses (rngs) and es se (crosses) dependng on he number of prncpal componens. 00 Varance explaned (%) Prncpal componen Fg. 4. The percen varance explaned by he correspondng prncpal componens. Now we compare he proposed one-neuron ANN wh oher popular models of he seasonal me seres forecasng: ARIMA and ES. These models were esed n he nex day elecrcal load curve forecasng problem on hree me seres of elecrcal load: PL: me seres of he hourly load of he Polsh power sysem from he perod (hs me seres was used n he expermens descrbed above). The es sample ncludes daa from 2004 wh he excepon of 3 unypcal days (e.g. holdays), FR: me seres of he half-hourly load of he French power sysem from he perod The es sample ncludes daa from 2009 excep for 2 unypcal days, GB: me seres of he half-hourly load of he Brsh power sysem from he perod The es sample ncludes daa from 2009 excep for 8 unypcal days.

9 In ARIMA he me seres were decomposed no n seres,.e. for each a separae seres was creaed. In hs way a daly seasonaly was removed. For he ndependen modelng of hese seres ARIMA(p, d, q) (P, D, Q) m model was used: m m D d m ) φ ( B)( B ) ( B) z = c + ( B ) θ ( B) ξ Φ ( B Θ, (3) where {z } s he me seres, {ξ } s a whe nose process wh mean zero and varance σ 2, B s he backshf operaor, Φ(.), φ(.), Θ(.), and θ(.) are polynomals of order P, p, Q and q, respecvely, m s he seasonal perod (for our daa m = 7), d and D are orders of nonseasonal and seasonal dfferencng, respecvelly, and c s a consan. To fnd he bes ARIMA model for each me seres we use a sep-wse procedure for raversng he model space whch s mplemened n he forecas package for he R sysem for sascal compung [8]. Ths auomac procedure reurns he model wh he lowes Akake's Informaon Creron (AIC) value. ARIMA model parameers, as well as he parameers of he ES model descrbed below, were esmaed usng 2-week me seres fragmens mmedaely precedng he forecased daly perod. Unypcal days n hese fragmens were replaced wh he days from he prevous weeks. The ES sae space models [9] are classfed no 30 ypes dependng on how he seasonal, rend and error componens are aken no accoun. These componens can be expressed addvely or mulplcavely, and he rend can be damped or no. For example, he ES model wh a dumped addve rend, mulplcave seasonaly and mulplcave errors s of he form: Level : Growh : Seasonal : Forecas : l = ( l s = s b = φb µ = ( l m + φb + β ( l ( + γξ ), + φb )( + αξ ), ) s + φb m, ) ξ, (4) where l represens he level of he seres a me, b denoes he growh (or slope) a me, s s he seasonal componen of he seres a me, µ s he expeced value of he forecas a me, α, β, γ (0, ) are he smoohng parameers, and φ (0, ) denoes a dampng parameer. In model (4) here s only one seasonal componen. For hs reason, as n he case of he ARIMA model, me seres s decomposed no n seres, each of whch represens he load a he same me of a day. These seres were modeled ndependenly usng an auomaed procedure mplemened n he forecas package for he R sysem [8]. In hs procedure he nal saes of he level, growh and seasonal componens are esmaed as well as he smoohng and dampng parameers. AIC was used for selecng he bes model for a gven me seres. In Table 2 resuls of PL, FR and GB me seres forecass are presened. In hs able he resuls of forecas deermned by he naïve mehod are also shown. The forecas rule n hs case s as follows: he forecased daly cycle s he same as seven days ago. The Wlcoxon es ndcaes sascally sgnfcan dfferences beween MAPE s

10 for each par of models and each me seres, so we can ndcae he one-neuron ANN as he bes model for hs daa and ES as he second bes model. Table 2. Resuls of forecasng. Model PL FR GB MAPE s IQR MAPE s IQR MAPE s IQR ANN ARIMA ES Naïve The las expermen concerns me seres forecasng up o seven daly perods ahead. In such asks he y-paerns are defned usng τ =, 2,, 7. For each horzon τ he one-neuron ANN s raned usng he same local learnng scheme as for τ = descrbed above. The forecas errors for PL, FR and GB me seres n Fg. 5 are presened. For FR and GB daa ANN gave he lowes errors. For PL daa and τ > 2 ES model s beer, and for τ > 3 also ARIMA model s beer. The acual and forecased fragmens of he me seres are shown n Fg. 6. MAPE s PL ARIMA ES ANN Forecas horzon MAPE s FR Forecas horzon MAPE s GB Forecas horzon Fg. 5. The forecas errors for dfferen horzons. Noe ha n he case of ARIMA and ES he model parameers are esmaed on he bass of he me seres fragmen (2 weeks n our example) drecly precedng he forecased fragmen. ANN learns on he ranng se composed of paerns represened daly perods from longer hsory. In local learnng case he ranng paerns are seleced usng creron based on he smlary o he curren npu paern. 5 Conclusons In hs arcle we examne a smple neural model wh local learnng for forecasng seasonal me seres. A he nal sage of he forecasng procedure daa are preprocessed o ge paerns of he me seres seasonal perods. An approach based on he paerns of he seasonal cycles smplfy he problem of forecasng of he nonsaonary and heeroscedasc me seres wh rend and many seasonal vara-

11 ons. Afer smplfcaon he problem can be modeled usng smpler ools. The exsence of many seasonal cycles s no a problem when we use forecasng model based on paerns. We resgn from he global modelng, whch does no necessarly brngs good resuls for he curren query pon. Insead, we approxmae he arge funcon locally n he neghborhood of he query pon. The dsadvanage of he local learnng s he need o learn he model for each query pon. Snce he local complexy s lower han he global one, we can use a smple model ha s quckly learned PL, ARIMA ES ANN L a L, GW 4 3 L, GW 2 τ= τ=2 τ=3 τ=4 τ =5 τ=6 τ= , h FR, τ = τ =2 τ =3 τ =4 τ =5 τ =6 τ = , h GB, L, GW τ= τ=2 τ=3 τ=4 τ=5 τ=6 τ = , h Fg. 6. The fragmens of load me seres and her forecass for dfferen horzons.

12 Ths approach s accepable when we have enough me (some seconds) o learn model and prepare forecas. The learnng speed s penalzed by he selecon of he neares neghbors. As shown by smulaon sudes o model he local relaonshp beween npu and oupu paerns he one-neuron model s suffcen. Ths model urned ou o be beer han he convenonal models (ARIMA and exponenal smoohng) n one-day ahead forecasng of he elecrcal load me seres and compeve n forecasng over longer me horzons. Acknowledgmens. The auhor would lke o hank Professor James W. Taylor from he Saïd Busness School, Unversy of Oxford for provdng French and Brsh load daa. The sudy was suppored by he Research Projec N N fnanced by he Polsh Mnsry of Scence and Hgher Educaon. References. Taylor, J.W., Snyder, R.D.: Forecasng Inraday Tme Seres wh Mulple Seasonal Cycles Usng Parsmonous Seasonal Exponenal Smoohng. Deparmen of Economercs and Busness Sascs Workng Paper 9/09, Monash Unversy (2009) 2. Taylor, J.W.: Shor-Term Elecrcy Demand Forecasng Usng Double Seasonal Exponenal Smoohng. Journal of he Operaonal Research Socey 54, (2003) 3. Taylor, J.W.: Exponenally Weghed Mehods for Forecasng Inraday Tme Seres wh Mulple Seasonal Cycles. Inernaonal Journal of Forecasng 26(4), (200) 4. Gould, P.G., Koehler, A.B., Ord, J.K., Snyder, R.D., Hyndman, R.J., Vahd-Aragh, F.: Forecasng Tme-Seres wh Mulple Seasonal Paerns. European Journal of Operaonal Research 9, (2008) 5. Zhang, G.P., Q, M.: Neural Nework Forecasng for Seasonal and Trend Tme Seres. European Journal of Operaonal Research 60, (2005) 6. Dudek, G.: Smlary-based Approaches o Shor-Term Load Forecasng. In: Zhu, J.J., Fung, G.P.C. (eds.): Forecasng Models: Mehods and Applcaons, pp Concep Press (200) hp:// paperid= Foresee F.D., Hagan M.T.: Gauss-Newon Approxmaon o Bayesan Regularzaon. Proc. 997 Inernaonal Jon Conference on Neural Neworks, (997) 8. Hyndman, R.J., Khandakar, Y.: Auomac Tme Seres Forecasng: The Forecas Package for R. Journal of Sascal Sofware 27(3), 22 (2008) 9. Hyndman, R.J., Koehler, A.B., Ord, J.K., Snyder, R.D.: Forecasng wh Exponenal Smoohng: The Sae Space Approach. Sprnger (2008)