Neural Networks-Based Time Series Prediction Using Long and Short Term Dependence in the Learning Process

Transcription

1 Neural Neworks-Based Tme Seres Predcon Usng Long and Shor Term Dependence n he Learnng Process J. Puchea, D. Paño and B. Kuchen, Absrac In hs work a feedforward neural neworksbased nonlnear auoregresson (NAR) model for forecasng me seres s presened. The learnng rule used o adjus he neural ne weghs s based on he Levenberg-Marquard mehod. In funcon of he long and shor sochasc dependence of he me seres, we propose an on-lne heursc law o se he ranng process and o modfy he neural ne opology. The npu paerns for he neural nework-based model are he values of he me seres afer applyng a medelay operaor. ence, he neural-ne oupu wll end o approxmae he curren value avalable from he seres. The coeffcens of he nonlnear fler are adjused on-lne n he learnng process, by consderng a creron ha modfes a each me-sage he number of paerns, he number of eraons, and he lengh of he apped-delay lne, n funcon of he urs s value () calculaed for he me seres. Accordng o he sochasc behavor of each seres, can be greaer or smaller han.5, whch means ha each seres ends o presen long or shor erm dependence, respecvely. The algorhm s appled o he me seres o forecas he nex 8 values gven n he NN3 Forecasng Compeon for Neural Neworks and Compuaonal Inellgence. I. INTRODUCTION A. Overvew of he NN Approach Ths work presens a soluon o he NN3 Forecasng Compeon for he Neural Neworks & Compuaonal Inellgence, whch s organzed as specal sessons of he Inernaonal Symposum of Forecasng, ISF 7, Inernaonal Jon Conference on Neural Neworks, IJCNN 7, and Inernaonal Conference on Daa Mnng, DMIN 7. The proposed soluon s based on he classcal nonlnear auoregresson fler usng me lagged feedforward neworks. The nnovaon s made on he learnng process, whch employs he Levenberg-Marquard rule and consders he long and shor erm sochasc dependence of passed values of he me seres o adjus a each me-sage he number of paerns, he number of eraons, and he lengh of he apped-delay lne, n funcon of he urs s value () of he sgnal. Accordng o he sochasc characerscs of each seres, can be greaer Manuscrp receved March, 7. Ths work was suppored n par by he Naonal Agency for Scenfc and Technologcal Promoon (ANPCyT) under gran PAV-TIC-76, PICT/ N 53. J. A. Puchea s wh he Deparmen of Elecronc Engneerng, Faculy of Exac, Physcal and Naural Scences, a Naonal Unversy of Córdoba. X56GCA, Córdoba, Argenna. (julan.puchea@gmal.com). The auhors.d. Paño and B. Kuchen are wh he Insue of Auomacs (INAUT), Faculy of Engneerng a Naonal Unversy of San Juan, 5 San Juan, Argenna. (dpano@nau.unsj.edu.ar). or smaller han.5, whch means ha each seres ends o presen long or shor erm dependence, respecvely. In order o adjus he desgn parameers and see he performance of he proposed predcon model, snusodal and square sgnals are used. Then, he neural nework-based nonlnear fler s appled o he me seres o forecas he nex 8 values gven n he NN3 Forecasng Compeon. B. Fraconal Brownan Moon In hs work he urs s value s used n he learnng process o modfy on-lne he number of paerns and number of eraons presened. The parameer s useful for he defnon of he Fraconal Brownan Moon (fbm). The fbm s defned n he poneerng work by Mandelbro and van Ness [6], hrough s sochasc represenaon B Γ Η + () = ( ) ( ) () + ( ) () s s db s s db s (.) where, Γ( ) represens he Gamma funcon Γ ( α ) = (.) and << s called he urs parameer. The negraor B s a sochasc process, ordnary Brownan moon. Noe, ha B s recovered by akng =/ n (.). ere, s assumed ha B s defned on some probably space (Ω, F, P), where Ω, F and P are he sample space, he sgma algebra (even space) and he probably measure, respecvely. So, a fbm s a connuous-me Gaussan process dependng on he socalled urs parameer <<. I generalzes he ordnary Brownan moon correspondng o =.5, and whose dervave s he whe nose. The fbm s self-smlar n dsrbuon and he varance of he ncremens s gven by Var B (.3) B s = ν s where, v s a posve consan. Ths specal form of he varance of he ncremens suggess varous ways o esmae he parameer. In fac, here are dfferen mehods for compung he parameer assocaed o Brownan Moon [] [3] [5]. In hs work, he algorhm uses a wavele-based mehod for esmang from a race of he fbm wh parameer [] [3] []. The race pah from he fbm are shown n Fg., where can be noed he dfference n he velocy and he amoun of s ncremens. α x x e dx ( ( ) ( )),

2 =. = Inpu sgnal Z - I Error-correcon sgnal Esmaon of predcon error NN-Based Nonlnear Fler One-sep predcon =.8 - Fg.. Block dagram of he nonlnear predcon Fg.. Three sample pah from fraconal Brownan moon for hree values of. C. Problem Saemen The classcal predcon problem may be formulaed as follow. Gven pas values of a process ha are unformly spaced n me, as shown by x(n-t), x(n-t),..., x(n-mt), where T s he samplng perod and m s he predcon order, s desred o predc he presen value x(n) of such process. Therefore, we lke o oban he bes predcon (n some sense) of he presen values correspondng o a random or pseudo-random sgnal. The predcor sysem may be mplemened usng eher an auoregresson model-based lnear or a nonlnear adapve fler, dependng on wheher he process s lnear or nonlnear. In he second case, neural neworks are used as a nonlnear model buldng, n he sense ha smaller he predcon error s (n a sascal sense), he beer he ne serves as model of he underlyng physcal process responsble for generang he daa. In hs work, me lagged feedforward neworks are used. Thus, he presen value of he sgnal s used as he desred response for he adapve fler, and he pas values of he sgnal supply as npu of he adapve fler. Then, he adapve fler oupu wll be he one-sep predcon sgnal. In Fg. s shown he block dagram of he nonlnear predcon scheme based on a neural nework fler. In hs work, a predcon devce s desgned such ha sarng from a gven sequence {x n } a me n correspondng o a me seres, can be obaned he bes predcon {x e } for he followng 8 values sequence. ence, s proposed a predcor fler wh an npu vecor l x, whch s obaned by applyng he delay operaor, Z -, o he sequence {x n }. Then, he fler oupu wll generae x e as he nex value, ha wll be equal o he presen value x n. So, he predcon error a me k can be evaluaed as ek ( ) = xn( k) xe( k), whch s used for he learnng rule o adjus he neural nework weghs. II. DESCRIPTION OF TE PREDICTION MODEL A. NN-Based Nonlnear Auoregresson Model We propose a neural nework-based nonlnear fler based on a nonlnear auoregresson model [7] [8] [9]. The neural nework used s a me lagged feedforward neworks ype. The neural ne opology consss of l x npus, one hdden layer of o neurons, and one oupu neuron. The learnng rule used n he learnng process s based on he Levenberg- Marquard mehod. The learnng rule modfes he number of paerns and he number of eraons a each me-sage accordng o he parameer, whch gves shor and long erm dependence of he sequence {x n }, or from a praccal pon of vew gves he ruggedness of he me seres. x n Z - Z - Z - x n- x n- x n-lx Feedforward Neural Nework x e - Learnng Law Fg. 3. Neural Nework-based nonlnear predcor fler. In order o predc he sequence {x e } one-sep ahead, he frs delay aken off from he apped-lne x n s used as npu. Therefore, he oupu predcon can be denoed by x e ( n + ) = Fp ( Z I( { x n} )) (.) where, F p s he nonlnear predcor fler operaor, and x e (n+) he oupu predcon a n+. B. The Proposed Learnng Process The wegh of he ne are adjused based on he Levenberg-Marquard rule, whch consders he long and

3 shor erm sochasc dependence of he me seres measured by he urs s parameer. The proposed learnng process consss on changng boh he number of paerns and he number of eraons n funcon of he parameer for each correspondng me seres. The learnng process s performed usng a bach model. In hs case he wegh updang s performed afer he presenaon of all ranng examples, consung an epoch. The pars of he used npu-oupu paerns are ( x (.5), y ) =,,..., N p where, x and y are he correspondng npu and oupu paern respecvely, and N p s he number of npu-oupu paerns presened a each epoch. ere, he npu vecor s defne as X (.6) = Z I( { x} ), and s correspondng oupu vecor as (.7) Y = x. Furhermore, he ndex s whn he range of N p gven by o Np 3 lx where, o s he number of he hdden neurons and l x s he dmenson of he npu vecor. In addon, hrough each epoch he number of eraons performed s gven by ( o ). The proposed creron o modfy he par (, N p ) s gven by he sascal dependence of he me seres {x n }, supposng ha s a fbm. The dependence s evaluaed by he urs s parameer, whch s compued usng a wavelebased mehod [] []. Then, a heursc adjusmen for he par (,N p ) n funcon of accordng o he membershp funcons shown n Fg. s proposed. Fnally, he number of npus of he nonlnear fler s uned ha s he lengh of apped-delay lne, accordng o he followng heursc creron: when he ranng process s compleed, boh sequences, {x n } and {{x n },{x e }}, should have he same parameer. If he error beween ({x n }) and ({{x n },{x e }}) s graer han a hreshold parameer θ he value of l x s ncreased (or decreased), accordng o l x ±. Explcly, l = l + sgn θ ( ) x x. ere, he hreshold θ was se abou 5%. III. MAIN RESULTS A. Se-up of Model and Learnng Process The nal condons for he fler and learnng algorhm are shown n Table. The nal number of hdden neurons and eraon are se n funcon of he npu number. Table shows he nal condons of he learnng algorhm used for forecasng he me seres, whch szes have a varable lengh, beween and 7 values. (, N p ) N p = o = ( o -) Fg.. eursc adjusmen of (, N p ) n erms of. Varable Inal Condon l x 6 o l x /3. o -.5 Table. Inal condon of he learnng algorhm. B. Prelmnary Resuls Usng Oher Tme Seres In order o es he proposed desgn procedure of he neural nework-based nonlnear predcor, an expermen wh snusodal and square sgnals was performed. The performance of he fler s evaluaed usng he mean Symmerc Mean Absolue Percen Error (SMAPE) proposed n he NN3 evaluaon: SMAPE S (.8) where, s he me observaon, n s he es se sze, s each me seres, X and F are he acual and he forecas me seres values a me respecvely. The SMAPE of each seres s calculaes he symmerc absolue error n percen beween he acual X and s correspondng forecas F value, across all observaons of he es se of sze n for each me seres s. Fg. 5 shows he predcor nonlnear-fler response, gvng he 6 fuure values for a snusodal me seres. The used sne me seres has a perod T=.8 s, and s sampled a T =.5 s. The nal lengh of he apped-delay lne was se-up a 6 aps, and a he end of he learnng process go be equal o 6. For a square me seres, Fg. 6 presens he forecased 8 values. ere he value of, across for he complee me seres {x n } and {x e }, dffers a a 5%. To mprove he forecasng performance of he neural nework fler, s used as nal condon of l x = 7, n order o ncrease of he {x e }. The new resuls are shown n Fg. 7, where he percenage s declned n he order of %. The fler srucure and learnng parameers are adjused a each me- sample n funcon of he value. The evoluon of hose parameers s shown n Fg. 8, where can be noe n X F = n X F = ( + ) = o N p = l x

4 he varaon of he number of used paern n he learnng process. Gven ha he number of eraons a each epoch s small a he begnnng, here are no changes a he parameers. Then, for more learnng me he compuaon of can be evaluaed more accuraely, whch can go o se he values of (N p, ), as s shown n Fg Ieraons Number of paerns urs parameer Tme Fg. 8. Evoluon of he learnng parameers SMAPE = =.87. =.666. l = 6. e x Fg. 5. Predcon of a snusodal me seres. C. Man Predcon Resuls for he NN3 Tme Seres In he followng fgures are shown he forecas for he me seres number and of he ones proposed n he NN3 compeon p SMAPE = = e = l x = 6. Fg. 6. Predcon of he nex 8 values of a square me seres SMAPE = = e = l x = 7. Fg. 7. Fnal predcon afer adjusng he lengh of he apped-delay lne of neural nework n funcon of he error Temporal sere No 5 5 SMAPE = =.399. e =.59. Fg. 9. Tme forecas for he me seres No Ieraons 5 5 Number of paerns 5 5 urs parameer Tme [monh] Fg.. Evoluon of he learnng algorhm s parameers.

5 Temporal sere No he urs parameer, an on-lne heursc adapve law s proposed o updae he neural ne opology, number of npu aps, and he number of paerns and eraons a each mesage. The man resuls shows a good performance of he predcor sysem appled o he me seres, proposed n he NN3 compeon, due o we obaned smlar roughness for boh he orgnal and he forecas me seres, evaluaed by and e respecvely. These resuls encourage one o go on workng wh hs new learnng algorhm, applyng o oher neural nework models, duo o he me seres generaed by humans neracon presens shor and long erm sochasc dependence. 6 8 SMAPE =.999. =.7. =.53. e Fg.. Tme seres number wh nearly. SMAPE s ndex Tradonal. Mean(SMAPE s ) =8.7 Modfed. Mean(SMAPE s ) = Ieraons 6 8 Number of paerns urs parameer 6 8 Tme [monh] Fg.. Evoluon of he learnng algorhm s parameers. D. Man Resuls The performance of he neural nework-based predcor fler s evaluaed hrough he SMAPE ndex, Eq. (.8), across he me seres gven n he NN3 compeon. Fg. 3 shows he evoluon of he SMAPE ndex for a radonal neural nework fler, whch uses a learnng algorhm wh fxed parameers. And anoher named modfed neural nework fler, whch s proposed n hs work and use he parameer o adjus heurscally eher srucure of he ne or parameers of he learnng rule. IV. CONCUSSION In hs work a feedforward neural neworks-based nonlnear auoregresson (NAR) fler for forecasng me seres has been presened. The learnng rule proposed o adjus he neural ne weghs s based on he Levenberg- Marquard mehod. And n funcon of he long and shor erm sochasc dependence of he me seres, evaluaed by No of each Temporal sere Fg. 3. The SMAPE ndex appled over he me seres. REFERENCES [] Abry, P.; P. Flandrn, M.S. Taqqu, D. Vech (3), "Self-smlary and long-range dependence hrough he wavele lens," Theory and applcaons of long-range dependence, Brkhäuser, pp [] Barde, J.-M.; G. Lang, G. Oppenhem, A. Phlppe, S. Soev, M.S. Taqqu (3), "Sem-paramerc esmaon of he long-range dependence parameer: a survey," Theory and applcaons of longrange dependence, Brkhäuser, pp [3] Deker, T.,. Smulaon of fraconal Brownan moon. MSc heses, Unversy of Twene, Amserdam, The Neherlands. [] Flandrn, P. (99), "Wavele analyss and synhess of fraconal Brownan moon," IEEE Trans. on Informaon Theory, 38, pp [5] Isas, J.; G. Lang (99), "Quadrac varaons and esmaon of he local ölder ndex of a Gaussan process," Ann. Ins. Poncaré, 33, pp [6] Mandelbro, B. B., (983). The Fracal Geomery of Naure, Freeman, San Francsco, CA. [7] aykn, S (999), Neural Neworks: A comprehensve Foudaon, nd Edon, Prence all. [8] Mozer, M. C., (99). Neural Ne Archecures for Temporal Sequence Processng. A. S. Wegend and N. A. Gershenfeld, eds., Tme Seres Predcons: Forecasng he Fuure and Undersandng he Pas, pp Readng, M.A.: Addson-Wesley. [9] Zhang, G.; B.E. Pauwo, and M. Y. u, (998). Forecasng wh arfcal neural neworks: The sae of ar, J. In. Forecasng, vol., pp